Spotlight: Señor Casaroja's Noesis

[MrAdults]

I just put a new build up as well, cause I figured it would be useful to have the color-format-string functionality for paletted images as well as for raw image decoding. If you upgrade to Noesis 3.51, you can chop that script down to this:

Code: Select all

from inc_noesis import *
def registerNoesisTypes():
	handle = noesis.register("Star Ocean image", ".unslz")
	noesis.setHandlerTypeCheck(handle, noeCheckGeneric)
	noesis.setHandlerLoadRGBA(handle, soiLoadRGBA)
	return 1
def soiLoadRGBA(data, texList):
	imgWidth = imgHeight = 256
	pix32 = rapi.imageDecodeRawPal(data[1024:], data[:1024], imgWidth, imgHeight, 8, "r8g8b8a8")
	texList.append(NoeTexture("soetex", imgWidth, imgHeight, pix32, noesis.NOESISTEX_RGBA32))
	return 1

[Satoh]

Will do!

On the header exploration, they seem to be pretty straight forward, however there aren't any identifiers as to what type of file is what, data-wise. (I can obviously tell by opening them in a raw image viewer like GGD and messing around some.)

Some files contain a group header that routes to multiple subfile headers, while some files simply jump straight into the subfile header... Also the presence of the group header changes the file header format slightly, causing an extra line of data to be a part of it...

I'm not certain how to go about writing the logic to differentiate automatically between whether it is a multifile type or a simple 1 file type. I'll post a few examples of files along with the data I've gathered.


Files tagged Fight or Run are player battle files, Ene are enemy battle files, and NPC are field map sprites.

They appear slightly different graphically but seem to have the same header setup.

Any suggestions as to how I might make the computer recognize one header type form another?

To better explain:
a grouped file which contains an extra set of headers, could be followed to the end of the offsets and find the graphical data, but to do the same on a non grouped, single image file, it would be fewer header jumps and in a different order, I need to know how to differentiate whether to jump 4 times... or say, 1 time, by having the computer identify what type of file header it is currently reading... I'm not sure how to do that.


EDIT: For people who don't want to download and will trust my observations, here is the spec

Code: Select all

SO1 Sprite headers

Dealing with group headers and by proxy their file header part 1 counterparts may be a bit of trouble


Group header (Battle sprites seem to lack this and jump straight to the file header part 2)
Offset	Size		Description
0x00	1dword		Number of files
0x04	1dword		absolute offset to a sub header
0x08	1dword		absolute offset to a sub header
...	1dword		absolute offset to a sub header
0x3C	1dword		absolute offset to a sub header


File header part 1
0x00	1dword		always 0x01??
0x04	1dword		unknown
0x08	1dword		unknown
0x0C	1dword		unknown
		Leads directly to a file header part 2 after this

File header part 2
0x00	1dword		relative offset to some part of this header from 0x00 (0x00 of the part 1 if there is one)
0x04	1dword		relative offset to some part of this header from 0x00 (0x00 of the part 1 if there is one)
0x08	1dword		relative offset to animation header starting from 0x00 of this header
0x0C	1dword		relative offset to sprite header from 0x00

Animation header
		To come later... not terribly important for image exporting...

sprite header
0x00	1dword		relative offset to palette header start, counting from 0x00
0x04	1dword		relative offset to image data, counting from 0x00
0x08	1word		0x02 for field sprites 0x04 for battle?
0x0A	1word		Number of image frames in the sprite data

		0x02 files have this format
0x0C	5bytes		Nothing here...?
...
0x11	1byte		width in pixels of first image frame
0x12	1byte		height in pixels of first image frame
0x13	pad?		0x00
0x14	1byte?		???
0x15	pad?		0x00
0x16	1byte?		???
0x17	pad?		0x00
0x18	5bytes		more padding...?
...
0x1D	1byte		second frame's width
0x1E	1byte		second frame's height...
		And so on like this for offset [0x0A]'s number of times...
		
		0x04's have this one
0x0C	6bytes		Nothing here...?
0x10	2bytes		nothing...?
0x12	1word		width in pixels of first image frame
0x14	1word		height in pixels of first image frame
0x16	1word		???
0x18	1dword		???
0x1C	1dword		???
0x20	2bytes		Nothing...?
0x22	1word		second frame's width
0x24	1word		second frame's height...
		And so on like this for offset [0x0A]'s number of times...

		until reaching.....

Palette header
0x00	1dword?		Seems to be a relative offset to the last palette for this file
0x04	24bytes		Nothing apparently... always seems to be 0x00
...
0x10	16words		palette, 16bit ABGR 5bit color 1bit alpha

		Directly after the last palette, is the beginning of image data.

EDIT2: with NoeBitStream, do the read functions have explicit bitsize types? for instance a readInt16() readInt32() readInt64() etc? I notice the example uses readBytes(#) a lot to cover space (which I tend to do likewise) and for arbitrary lengths of data, is there a single readByte or would I need to use readBytes(1)?

would I need to define special byte length readInt()'s?

I don't see (obviously) all of the datatypes explained in the Noepy documents. [Edit:] Which is understandable; the man of red houses is a programmer, not a technical documenteer.

[Satoh]

Sorry for the double post, but this is more pertaining to the code I'm (slightly blindly) writing to read the files.

Code: Select all

from inc_noesis import *
def registerNoesisTypes():
	handle = noesis.register("Star Ocean sprite", ".unslz")
	noesis.setHandlerTypeCheck(handle, noeCheckGeneric)
	noesis.setHandlerLoadRGBA(handle, sosiLoadBGRA)
	return 1
def sosiLoadBGRA(data, texList):
	bs = NoeBitStream(data)
	
	//groupedFile = some test I haven't figured out yet that returns boolean 
	
	
	
	if groupedFile:
		numFiles = bs.readInt()
		for i in range(0, numFiles):
			fileHeaderOffset[i] = bs.readInt32()
		
	spriteMode = bs.readInt32()
	
	for i in range(0, numSprites):
		if spriteMode == 2:
			imgWidth[i] = bs.readByte()
			imgHeight[i] = bs.readByte()
		else if spriteMode == 4:
			imgWidth[i] = bs.readInt16()
			imgHeight[i] = bs.readInt16()
		imgPix[i] = rapi.imageDecodeRaw(imgPix[i], imgWidth[i], imgHeight[i], "a1b5g5r5")
		texList.append(NoeTexture("soetex"+i, imgWidth[i], imgHeight[i], pix32, noesis.NOESISTEX_RGBA32))

This is obviously unfinished, but it occurred to me that I might be able t use a try...except to determine whether the files are grouped or singular...

or rather than "determine" it, just execute the code for one, and when it fails, execute the code for the other...

However, this strikes me as a bit of a hack rather than the proper method of handling the issue... and with the possibility of horrendous errors in the event that a single file manages to open as a grouped one, and ends up presenting a pile of graphical garbage...

But I figured since I'm asking so many questions, I should at least show that I am, in fact, working on it.

Then again I also have no idea whether what I'm doing is even viable, as I don't use python much and I've never written a plugin for anything, or in fact, any piece of code that wasn't entirely from scratch for a single purpose, and by that token, everything I've ever written is somewhat inflexible and not extensible.

So if anyone has any experience, could you at least tell me if I'm going in the right direction?

[Dinoguy1000]

Keeping in mind I've never done any work with file format identification, the only conclusion I've ever been able to come to is that the only surefire, 100% guaranteed way to correctly identify a file's format anywhere near 100% of the time is to just try opening it as that format (there is a very slim chance that a file of one format will open as another format, but that should be mind-bogglingly rare, and even if it did happen, any output would almost certainly be garbage) - extensions and magic numbers can both be changed or faked, and MIME types rely on one or both of those. If trying to actually read the file fails, though, it means either the format is unsupported, or it's based on a supported format, but has been modified in some way - e.g. a new or changed field, or encryption used in the file header/metadata, etc.

On the other hand, if someone else who knows better says otherwise, you should of course ignore what I've just said and listen to them instead. =)

[finale00]

This is obviously unfinished, but it occurred to me that I might be able t use a try...except to determine whether the files are grouped or singular...

or rather than "determine" it, just execute the code for one, and when it fails, execute the code for the other...

That's usually how one would go about trying different approaches if one or another fails.

Of course, as you've pointed out, you might end up with garbage cause it just happens to not throw an exception in a previous try clause, but if you already had a way to check which one to use you wouldn't need a try/except for that purpose.

If that's what you're referring to as whether "it is viable" or not.

[Satoh]

[EDIT: Sorry to get ahead of myself, let me thank you all for the help you've volunteered so far first. Thanks, it is appreciated. Also if there's a better forum for this, despite the code being specific to Noesis, let me know and I'll move it there.]

Ok so obviously I've done a bit more on this...
1: I figured out a more reliable check for file type... grouped files almost never have more than 6 individual files, which is the first byte, and single files are almost always referencing 0x40 in the first byte, so I used a check for that byte to differentiate... potential for error... low to moderate I think.

2: I've tested this and it certainly does...something... but not what I need, so, I can't say what's going wrong, but I have a guess.
My guess is that rapi.imageDecodeRaw() works the way I'm trying to use it.
I solved my issues with explicit data types by reading the inc_noesis.py file, in case anyone needs more info on the Noepy functions, you can find a few of them there.
Problem is, I didn't see the rapi functions, likely due to them not being part of noepy (duh)
Also, I have no idea how the image functions work, so I'm sort of bashing my head at them and hoping. (which so far hasn't worked) I'm thinking, since my image is obviously not "RGBA32" the other image functions won't work either, not that I understand how they work...

Anyway, this is the current state of my code... I could really use some help at this point...

Code: Select all

from inc_noesis import *
def registerNoesisTypes():
	handle = noesis.register("Star Ocean sprite", ".unslz")
	noesis.setHandlerTypeCheck(handle, noeCheckGeneric)
	noesis.setHandlerLoadRGBA(handle, sosiLoadBGRA)
	return 1
def sosiLoadBGRA(data, texList):
	bs = NoeBitStream(data)
	check=bs.readInt()
	if (check >= 0x40):
		groupedFile = False
	else:
		groupedFile = True
	bs.seek(0,0)
	if groupedFile:
		numFiles = bs.readInt()
		for h in range(0, numFiles):
			fileHeaderOffset = []
			spriteHeaderOffset = []
			paletteOffset = []
			imgDataOffset = []
			spriteMode = []
			numSprites = []
			fileHeaderOffset.append(bs.readInt())
			bs.seek(fileHeaderOffset[h]+0x1C,0)
			spriteHeaderOffset.append(bs.readInt())
			bs.seek(spriteHeaderOffset[h]-0x20,1)
			paletteOffset.append(bs.readInt())
			imgDataOffset.append(bs.readInt())
			spriteMode.append(int(bs.readUShort()))
			numSprites.append(int(bs.readUShort()))
			imgPal = []
			imgWidth = []
			imgHeight = []
			imgPix = []
			for i in range(0, numSprites[
				if spriteMode[h] == 2:
					bs.readBytes(
					imgPal.append(int(bs.readByte()))
					imgWidth.append(int(bs.readByte()))
					imgHeight.append(int(bs.readByte()))
					bs.readBytes(5)
				elif spriteMode[h] == 4:
					bs.readBytes(4)
					imgPal.append(int(bs.readUShort()))
					imgWidth.append(int(bs.readUShort()))
					imgHeight.append(int(bs.readUShort()))
					bs.readBytes(
			#for i in palettes do: palette[i] read palettes...? dunno how to do 
			#imgPal = rapi.imageGetPalette(imgPix, imgWidth, imgHeight, 256, 0, 1)
			for i in range(0, numSprites[h]-1):
				imgPix = rapi.imageDecodeRaw(imgPix, imgWidth[i], imgHeight[i], "a1b5g5r5") #I think this is wrong for this application
				texList.append(NoeTexture("soetex"+"."+h+"."+i, imgWidth[i], imgHeight[i], pix32, noesis.NOESISTEX_RGBA32)) #No idea how this works

[KnightDragon]

The contents of this post was deleted because of possible forum rules violation.

[Satoh]

The contents of this post was deleted because of possible forum rules violation.

[KnightDragon]

The contents of this post was deleted because of possible forum rules violation.

[Satoh]

The contents of this post was deleted because of possible forum rules violation.

[nitekrawler1285]

Howdy there MrAdults. I was hoping you could answer a question for me. I was trying to extract the models from MVC3 using Noesis 3.49 and recently v 3.51. When I try to export the PS3 models Noesis crashes as in wont respond to any input and must be closed via a task manager. I tried two different 360 versions and it says that it doesn't recognize the compression and wont export the arcs for me to get at the models.

[Orohu]

I'd eventually like to see support for the XPPS files used in inFamous 2.

p.s. About the earlier post I deleted, I fixed the problem by restarting my computer. XD

[Satoh]

Well, after a series of very industry standard bash-your-head-into-the-computer-until-it-works techniques, I succeeded... eh-mostly.

I can properly read and output NPC images... I can read enemy and battle images as well... but the palettes are a bit off...

the only -problem- is the little ghost that keeps asking for it's long lost integer... (I seriously can't find any place that is missing an integer)

[MrAdults]

That message is caused by not returning a 1 or 0 from your handler to indicate success or failure. I'd comment on more of this, but I'm on my phone and typing on this thing is like water torture.

[viperzerofsx]

this new version works for me I don't know what was done differently but it solved my problem of it not starting