Page 27 of 123

Re: Señor Casaroja's Noesis

Posted: Wed Sep 14, 2011 5:23 am
by Trishty
I want to ask this. How to extract summon model of FF8? everyone told me it in .exe

Re: Señor Casaroja's Noesis

Posted: Wed Sep 14, 2011 9:10 am
by LUBDAR
Trishty wrote:I want to ask this. How to extract summon model of FF8? everyone told me it in .exe
In the "Battle" folder you can find a couple of the summoned monsters which you had to battle against: Brothers, Bahamut, Ifrit, Diabolos, Cerberus.
Satoh wrote:
I see a lot of people asking questions like this, and honestly I don't know how they expect to get a simple answer. MrAdults programmed Noesis, with the exception of a few of the plugins, but he doesn't know what format a file is simply by hearing the name of the game or the extension on the file.

Just as the man who creates a wrench doesn't know which size you need for your washing machine simply because you told him where you bought it...

Noesis is a tool, and a good one, one that takes away most of the user's need to be involved, but there is still a small amount of common sense that says a user must be there to start it up and test it out on things.

I thought this was a great analogy!! Whatever happened to peoples sense of adventure to look at all available resources and just try things out?

Re: Señor Casaroja's Noesis

Posted: Wed Sep 14, 2011 10:17 am
by Trishty
LUBDAR wrote: In the "Battle" folder you can find a couple of the summoned monsters which you had to battle against: Brothers, Bahamut, Ifrit, Diabolos, Cerberus.
I talk about Phoenix, Leviathan, Alexander,... -.-!

Re: Señor Casaroja's Noesis

Posted: Wed Sep 14, 2011 5:35 pm
by LUBDAR
Those I'm not sure about, the textures are in the same folder though

Re: Señor Casaroja's Noesis

Posted: Wed Sep 14, 2011 7:41 pm
by Satoh
Well I'm not sure they're in the exe, it seems like a possibility, but the PSX game uses the same file formats for most of the things the PC does, so that means they'd have to be in the PSX's program as well, but that seems a bit strange.

I think most likely the summons are in the magic folder.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 18, 2011 9:11 pm
by invisghost
I've had this idea for a general 3d file exploration tool for some time now but I haven't found the right person to talk to about it. Hopefully this will at least put me on the right track. What the feature would do is pretty simple but would speed things up tremendously. You would select a file and it would treat the file as a array of xyz vectors. Then it would draw a bunch of points to a viewport where you could orbit around the points, zoom in and out and just see if there is anything that looks like a model. You could set the beginning and end offset where the program would get the vectors from and see real time updates in the viewport. I know there's a tool that can dump a file to 3ds max but it wouldn't be as easy to find where the model's points start compared to this. I've previously made my own simple program that does this to see if it would work and it does. There could be more features to this but I'll explain those later. If you need me to elaborate on anything feel free to ask, tell me what you think!

Re: Señor Casaroja's Noesis

Posted: Sun Sep 18, 2011 9:37 pm
by chrrox
you can already do that with noesis he supports python just code up a script to look at 3d points.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 18, 2011 10:39 pm
by Satoh
@chrrox, I believe you once wrote a script that viewed 3D points as dots did you not? it was specifically for the Xenosaga III models that I recall it... or am I confusing you with some other very code-capable person?

Anyway
@invisighost, what chrrox said is correct, in fact I'm considering it myself, but I'm not fluent in python or in fact -viewing things code-.

If you(or me or even MrAdults) decide to undertake such a module for Noesis, I suggest having an option to start at a specific offset (as perhaps the relevant bytes are not aligned perfected with 0 or by 4's), an option to dictate a byte-stride or pattern (again, the bytes may be one byte XYZ, or could be 16bit X Y and Z, or could even be plain text, as in OBJ, though unlikely)

and a separation option, for instance it could be XYZ[ ][ ]XYZ or XYZ[ ][ ][ ][ ][ ][ ][ ][ ][ ][ ]XYZ.

It's not necessarily for finding specific formatting, but if there is extraneous data between points, such as bone weight data, vertex color information, dummy info, polygon strip ID's, etc, the points may not line up as expected, and it will likely cause data that should be XYZ to be interpreted as 'YZU ZUV UVX VXY' for example.

Also, a way to track the position of a point on the screen would be good, as it could lead to finding not only a file that has model data, but -where- in the file the vertex pool is.

Aside from tracking each vertex to an offset, just drawing the points at a certain stride and separation based on user input, should theoretically be easy... once you learn the Noesis API.

Re: Señor Casaroja's Noesis

Posted: Mon Sep 19, 2011 10:45 pm
by invisghost
chrrox wrote:you can already do that with noesis he supports python just code up a script to look at 3d points.
Having some control over it other than just dumping the entire file is the main purpose of it. Satoh gets the idea, being able to change things like stride or offset easily is the key feature.

I'm willing to try and make the module but as Satoh said, I need to know the Noesis API which I don't. I would also like to chat with someone who works on Noesis and be able to ask questions while working on it but I haven't been able to get in contact with anyone.

Re: Señor Casaroja's Noesis

Posted: Mon Sep 19, 2011 11:18 pm
by Satoh
Noesis can take user input, however, as it is mostly customized to batch processing, I think the user input properties available in the api are probably lacking.

That said, I think it's still possible with the functions supported. If you need a crash course in the API, it's a bit of a steep learning curve this way, but you can check out the __NPExample.txt and a list of some functions is in __NPReadme.txt. You should also check out inc_noesis.py for a more comprehensive list of python functions and what they do... (Literally what they do, as it contains the method definitions for them)

If you want to use something other than python for it... I couldn't even begin to give advice. I can't even get a DLL that's already fully written and set up in a solution to compile properly...

@MrAdults, I've written a plugin for some Xenosaga3 XTX images, (it's only partial, as it doesn't read the swizzeled versions. It does have a check for them however.)

A problem I'm having, however, is that since the textures it is designed to work on are -not- power of two dimensions, the textures in the Noesis preview window appear white. They export properly, however they cannot be appropriately previewed.
This is an issue in 3.51 and earlier, you may have fixed it with the newest updates, but I'm only just now updating.

Still it's something I thought you should be aware of.

EDIT: Ahm... also, is there any simple way to deal with 128 (0x80) length Alpha values? These files have 0x80 as their max alpha value, rather than 0xFF.

Re: Señor Casaroja's Noesis

Posted: Tue Sep 20, 2011 2:36 am
by MrAdults
You could do whatever you want if you wrote it as a native plugin DLL. Being a native binary would allow you to embed your own dialog resources and use them to prompt/display whatever you want.

In a Python script, there are a few options. You could just hardcode the parameters as variable defines at the top of the .py file so you can easily modify them, seeing as it's just a text file after all. You could also use the addOption stuff to get the options in the form of commandline parameters, but that wouldn't be very convenient. You may also be able to try using something like Tkinter with the Noesis Python implementation, but I've never tried that and won't really know what to tell you if/when you run into trouble with Tkinter.

Satoh: You should be able to use the PS2 untwiddle function on those files pretty easily. For expanding the alpha channel, you could do something like r8g8b8a7p1 instead of r8g8b8a8. That way it will only use the fist 7 bits of the alpha and expand from 127 to 255 automatically. Kind of a hack but it's probably the easiest solution.

Non-power of 2 textures being white is caused by your graphics drivers/hardware not supporting the non-power texture extension. I could resize before uploading in that case, I suppose. What kind of GPU do you have? You may find a driver upgrade automagically adds non-PoT texture support for you. [Edit: Went ahead and made a new build to do the resize before GL texture uploads if GL_ARB_texture_non_power_of_two isn't supported.]

Re: Señor Casaroja's Noesis

Posted: Tue Sep 20, 2011 4:03 am
by Satoh
My GPU is probably the most unfortunate thing I've ever seen on a modern computer... An Intel something-or-other chipset that has minimal support for even standard windows OGL stuff... (which it complains to me about in more than one program I've had to give up on lately) Though things with software renderers, seem to be mostly unaffected, like Maya and 3DS Max, which run fine (though with possible errors I've not noticed)

Unfortunately there is no other drivers for this particular computer than the ones made in like '99... it is an office laptop that is basically a bare minimum replacement for my old one (R.I.P.) since I have about -4 USD to my name.

However, I had the same white texture problem on the computer I was borrowing until recently. Oddly the issue only appears in Noesis' preview window... but that may just be due to all my other 3D applications using software rendering methods or something...

I hadn't considered it was my computer that was a problem, but it is a definite possibility I suppose. If you add a quickfix for it that would be nice, but I suppose it is probably more my responsibility in that case, so I won't hold my breath or test my luck. After all, you're one of the last people I want to be on the bad side of.

I tried using the variations of a7p1, p1a7, and such but that produced a few strange, but logical results on export. The problems with that method are that it actually uses 8bits for the alpha channel, but it never goes higher than 128 (0b10000000 0x80) whereas 7 bits is capped at 127 (0x7F, 0b1111111) leading 0x80 to be treated as 0 since the only bit that's on is out of range, or leading some values like 1 to being treated like 0...

I did manage to cobble together a remedy by loading the bytes into an array and manually changing every fourth one to be roughly 2*byte & 0xFF... Which works and is probably about as close to accurate as I can figure out how to make it.

I also tested out the various unswizzle and untwiddle functions on some of the files that are obviously twiddled or swizzled (as per the xenosaga 3 thread in 3d/2d board for more detail)
but they don't seem to be the right format, which I expected anyway. (Though it could also be that I used them wrong, since I have absolutely no understanding of swizzles or twiddles.)

Then loading it into the raw decoder.

This is the script, it doesn't work on all XTX's as some are swizzled, like I said, so whether or not to upload it is your prerogative. If I figure out how the swizzle in question works and understand it well enough I'll probably attempt to add it to here. However, since it DOES do what I intended it to do, and there may be other who want to process these images, I'm posting it here (mainly because it's noesis related, I'll be quoting it into the Xenosaga thread as well)

If anyone sees something I did stupidly, give me a suggestion and I can certainly fix it... or you can...

Code: Select all

from inc_noesis import *
import noesis
import rapi

def registerNoesisTypes():
	handle = noesis.register("Xenosaga XTX Texture", ".xtx")
	noesis.setHandlerTypeCheck(handle, xtxCheckType)
	noesis.setHandlerLoadRGBA(handle, xtxGetRGBA)
	return 1
	
XTX_TAG = 0x00585458 # XTX.
	
def xtxCheckType(data):
	if len(data) < 0x50:#If the file is smaller than the header size, quit.
		return 0
	bs = NoeBitStream(data)

	if bs.readInt() != XTX_TAG:#if the header tag is wrong, quit.
		return 0
		
	return 1

def xtxGetRGBA(data, texList):
	bs = NoeBitStream(data)
	
	#tag
	bs.readInt()
	#start of pixels
	imageOffset = int(bs.readByte())
	
	#unknown
	bs.readBytes(11)
	
	#width
	w = int(bs.readUShort())
	#swizzel indicator...?
	swiz = int(bs.readUShort())
	#height
	h = int(bs.readUShort())
	
	imgPix = data[imageOffset:]
	
	if (swiz != 0x04):#if this is not 0x04, the only value I've found that is raw RGBA32, quit.
		return 0
	
	#This next loop takes care of the half size alpha channel. The files max alpha at 0x80 instead of 0xFF
	#this is here to spread the values so they max at 0xFF for fully opaque
	#if there is a better way to do this, let me know.
	
	for i in range(0,w*h):#For each pixel
		if imgPix[((i+1)*4)-1] > 0:#if the alpha is not 0 (this is needed due to overflow of 0 - 1 = FF)
			imgPix[((i+1)*4)-1] = ((imgPix[((i+1)*4)-1] * 2)-1)&0xFF #Every fourth byte, perform operations to roughly double the value, then cap it to 8 bits
	
	pix32 = rapi.imageDecodeRaw(imgPix, w, h, "r8g8b8a8")
	
	texList.append(NoeTexture("xtx", w, h, pix32, noesis.NOESISTEX_RGBA32))
	return 1
Sorry for the size, Xentax could really use a spoiler bbcode...

EDIT: Also thanks for getting back to me on the aforementioned things, and I'll test out the new Noesis in a sec, thanks again.

EDIT_DOS: Thanks for the internal power of two fix MrAdults, it works beautifully! Much appreciated!

Re: Señor Casaroja's Noesis

Posted: Tue Sep 20, 2011 6:58 am
by MrAdults
Oh, right. Yeah, I've come across a lot of textures with a 0-128 alpha range on PS2. I went ahead and added some stuff and put another build up:

-3.58 - Added base orientation change option to anim-only preview mode.
-3.58 - Added an optional new flags parameter to imageDecodeRaw/imageDecodeRawPal in the Python API. See the DECODEFLAG_* values listed in __NPReadMe.txt.
-3.58 - Added imageScaleRGBA32 to the Python API.

noesis.DECODEFLAG_PS2SHIFT is usable with imageDecodeRawPal now. If you use it as the last param it will shift the palette index to compensate for the native hardware palette ordering. (a good number of games do that as well)

In your case the new scale function would be used like pix = rapi.imageScaleRGBA32(pix, (1.0, 1.0, 1.0, 2.0), pixWidth, pixHeight), although it wouldn't really benefit you much to use other than being faster since it's in native code.

I can't test your script since I don't have Xenosaga 3 (or its data), but I think that last pix32 = rapi.imageDecodeRaw(imgPix, w, h, "r8g8b8a8") you have in there is unnecessary, since imgPix already appears to be in rgba32 format. You could just pass imgPix right into the texture constructor, instead.

Oh, and did you actually find r8g8b8a8 textures in Xenosaga 3 that are twiddled? If so, I could use some for testing, as I've never come across any before.

Re: Señor Casaroja's Noesis

Posted: Fri Sep 23, 2011 6:07 am
by MrAdults
I finally got around to looking at the TXY when I couldn't get back to sleep before work this morning. Kind of a weird one, but par for the course on PS2:

Code: Select all

from inc_noesis import *
def registerNoesisTypes():
	handle = noesis.register("Xenosaga TXY Image", ".txy")
	noesis.setHandlerTypeCheck(handle, txyCheckType)
	noesis.setHandlerLoadRGBA(handle, txyLoadRGBA)
	return 1

def txyGetImageCut(texPage, pageWidth, pageHeight, ofsX, ofsY, width, height):
	r = bytes()
	for i in range(0, pageHeight):
		imgOfs = ofsY*pageWidth + i*pageWidth + ofsX
		r += texPage[imgOfs:imgOfs+width]
	return r

def txyRetile8(pix, pixW, pixH):
	dst = bytearray(pixW*pixH)
	for y in range(0, pixH):
		for x in range(0, pixW):
			blockOfs = (y & ~15)*pixW + (x & ~15)*2
			sw = (((y+2)>>2) & 1) * 4
			ofsY = (((y & ~3)>>1) + (y & 1)) & 7
			colOfs = ofsY*pixW*2 + ((x+sw) & 7) * 4
			ofsX = ((y>>1) & 1) + ((x>>2) & 2)
			dst[blockOfs+colOfs+ofsX] = pix[y*pixW + x];
	return dst

class TXYFile:
	def __init__(self, reader):
		self.reader = reader

	def loadImageInfo(self):
		reader = self.reader
		if reader.dataSize <= 20:
			return 0
		id = reader.readInt()
		ver = reader.readInt()
		fsize = reader.readInt()
		if id != 0x797874 or ver != 1 or fsize > reader.dataSize:
			return 0
		reader.seek(16, NOESEEK_ABS)
		self.infoOfs = reader.readInt()
		if self.infoOfs <= 0 or self.infoOfs >= reader.dataSize:
			return 0
		reader.seek(self.infoOfs, NOESEEK_ABS)
		self.hx = reader.readInt()
		self.bd = reader.readInt()
		self.pageW = reader.readInt()
		self.pageH = reader.readInt()
		reader.seek(4, NOESEEK_REL)
		self.numSlices = reader.readInt()
		self.numBlocks = reader.readInt()
		if self.numSlices <= 0 or self.numBlocks <= 0:
			return 0
		reader.seek(4, NOESEEK_REL)
		self.slicesOfs = reader.tell()
		return 1

	def loadPage(self):
		reader = self.reader
		slices = bytes()
		slicesH = 0
		self.bitsPP = 8
		for i in range(0, self.numSlices):
			reader.seek(self.slicesOfs + i*32, NOESEEK_ABS)
			pixOfs = reader.readInt()
			rofs = reader.readInt()
			sliceW = reader.readInt() * 2
			sliceH = reader.readInt() * 2
			reader.seek(pixOfs+32, NOESEEK_ABS)
			imgSize = sliceW*sliceH if self.bitsPP >= 8 else (sliceW*sliceH)//2
			pix = reader.readBytes(imgSize)
			pix = rapi.imageUntwiddlePS2(pix, sliceW, sliceH, self.bitsPP)
			for j in range(0, sliceH):
				row = pix[j*sliceW:j*sliceW + sliceW]
				row += bytes([0])*(self.pageW-sliceW)
				slices += row
			slicesH += sliceH
		self.pageH = slicesH
		self.texPage = slices

	def getPaletteBlock(self, palX, palY):
		pal = bytearray(1024)
		idxCut = txyGetImageCut(self.texPage, self.pageW, self.pageH, palX, palY, 32, 32)
		return rapi.imageScaleRGBA32(txyRetile8(idxCut, 32, 32), (1.0, 1.0, 1.0, 2.0), 16, 16)

	def loadTextures(self, texList):
		reader = self.reader
		reader.seek(self.infoOfs+288, NOESEEK_ABS)
		texInfos = []
		for i in range(0, self.numBlocks):
			texInfo = {}
			texInfo["ofsa"] = reader.readInt()
			texInfo["na"] = reader.readInt()
			texInfo["width"] = reader.readInt()
			texInfo["height"] = reader.readInt()
			texInfo["ofsx"] = reader.readInt()
			texInfo["ofsy"] = reader.readInt()
			texInfo["ofsb"] = reader.readInt()
			reader.seek(4, NOESEEK_REL)
			texInfo["palx"] = reader.readInt()*2
			texInfo["paly"] = reader.readInt()*2
			reader.seek(24, NOESEEK_REL)
			texInfo["name"] = noeStrFromBytes(reader.readBytes(32))
			texInfos.append(texInfo)
		for texInfo in texInfos:
			pal = self.getPaletteBlock(texInfo["palx"], texInfo["paly"])
			idxCut = txyGetImageCut(self.texPage, self.pageW, self.pageH, texInfo["ofsx"], texInfo["ofsy"], texInfo["width"], texInfo["height"])
			pix = rapi.imageDecodeRawPal(idxCut, pal, texInfo["width"], texInfo["height"], self.bitsPP, "r8g8b8a8", noesis.DECODEFLAG_PS2SHIFT)
			texList.append(NoeTexture(texInfo["name"], texInfo["width"], texInfo["height"], pix, noesis.NOESISTEX_RGBA32))
			

def txyCheckType(data):
	txy = TXYFile(NoeBitStream(data))
	if txy.loadImageInfo() == 0:
		return 0
	return 1

def txyLoadRGBA(data, texList):
	txy = TXYFile(NoeBitStream(data))
	if txy.loadImageInfo() == 0:
		return 0

	txy.loadPage()
	txy.loadTextures(texList)
	return 1
Hard to tell if some of those values are right since you only uploaded a couple of files. The page width value may not really be page width, but worst case you could calculate that from the max ofs+width of the textures from the texture list. If there are 4bpp textures you'd also have to change some numbers around for that. But it should be enough for you to go on, anyway.

Basically they're broken up into what I called "slices", and if you untile each slice into a single page you end up with this:
Image
The garbled-looking data is actually the palette data. Each texture has an actual page-space x/y offset for its palette. So you grab the rect for each texture and apply the appropriate palette using the texture entry and get the final image.
Image

Re: Señor Casaroja's Noesis

Posted: Fri Sep 23, 2011 6:11 am
by LUBDAR
MrAdults wrote:I finally got around to looking at the TXY when I couldn't get back to sleep before work this morning.
YESS!!!! I love that, sometimes I get "Active Brain" and just can't sleep cuz I'm messing around with stuff.