Page 78 of 123

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 8:23 am
by finale00
The textures are stored in the same file as the model.

Code: Select all

for i in range(numTex):
  offset, size, unk, unk = self.inFile.read('4L')
  self.inFile.seek(offset)
  texData = self.inFile.readBytes(size)
  texName = "texture[%d]" %i
  tex = rapi.loadTexByHandler(texData, ".pvr")
  if tex:
      tex.name = texName
      self.texList.append(tex)
I wrote the contents of texData out to verify that it's actually a pvr image, and when I loaded it using noesis it loaded fine.

However noesis is telling me "warning could not load external texture"
I did the same sort of thing for another game that stored textures inside the model file so it shouldn't be an issue that the texture is not an "external" file

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 8:59 am
by Vindis
Thank you for your feedback. Now one more thing [roll]

If i have to assign different material to the same mesh what is the correct method? I mean the mesh let's say build from 12k vertice and the first 2000 face use texture1, the second 2000 faces use texture2. Do I have to assign two separate vertex buffer and face buffer, or is it have to be enough to use one vertbuff and separate face buffs for different materials.
I crash noesis every time when i try to assign different face than the total face count (i mean assing 2000 face tex1 2000face tex2 thats 4 multiple with 3 i get 12k but still crash). Bit confusing.

The header at 0x70 says number of vertices 3999 at 0x84 the face number 4083. Tex1 use 361 face, tex2 1000face, if I multiple it with 3 i get 4083, but then what is 3999. If i use 4083 for buffer crash again, if i use 3999 noesis can draw the mesh for me, but cant assign different face mats. Is it understandeable (hard to describe it)? :?:

faceBuff=bs.readBytes(xxx*2) <--- and what 2 stands for, why/when 2 or 3 or 4?

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 11:12 am
by Demonsangel
The faceBuff=bs.readBytes(xxx*2) is most likely because faces/triangles/whatever name they use are usually stored in unsigned shorts ( = 2 bytes) therefore you read double the amount of bytes read to get your buffer

The reason you crash is because you're telling noesis to read the wrong amount of bytes in your rpgCommitTriangles
You have to tell it to parse the amount of indices and not the amount of faces. Most formats I've seen use triangles (3 indices for a face), I'm guessing it's because you're using noesis.RPGEO_TRIANGLE but the number you give isn't dividable by 3.

For the textures, I can't say because I don't know how your facebuffers are built.
Are they stored as 1 large buffer or are they seperate buffers where the numbers start allover again

Meshes with the same material might get merged into 1 mesh, I haven't really looked into Noesis' behavior for that, but you basically need to redefine

Code: Select all

rapi.rpgSetMaterial() // set the material for the mesh
rapi.rpgBindPositionBufferOfs   () //You can keep using the large vertexbuffer or you can cut it up in pieces, it doesn't really matter just make sure the numbers match
rapi.rpgBindUV1BufferOfs        () 
rapi.rpgCommitTriangles         () //everything must be put before this

for each mesh
There are a bunch of formats you can use with the rpgCommitTriangles:

Code: Select all

	PYNOECONSTN(RPGEO_NONE),
	PYNOECONSTN(RPGEO_POINTS),
	PYNOECONSTN(RPGEO_TRIANGLE),
	PYNOECONSTN(RPGEO_TRIANGLE_STRIP),
	PYNOECONSTN(RPGEO_QUAD), //ABC_DCB
	PYNOECONSTN(RPGEO_POLYGON),
	PYNOECONSTN(RPGEO_TRIANGLE_FAN),
	PYNOECONSTN(RPGEO_QUAD_STRIP),
	PYNOECONSTN(RPGEO_TRIANGLE_STRIP_FLIPPED),
	PYNOECONSTN(NUM_RPGEO_TYPES),
	PYNOECONSTN(RPGEO_QUAD_ABC_BCD),
	PYNOECONSTN(RPGEO_QUAD_ABC_ACD),
	PYNOECONSTN(RPGEO_QUAD_ABC_DCA),
if you just want to do a quick test
try this
rapi.rpgCommitTriangles(None, noesis.RPGEODATA_USHORT, numVertices, noesis.RPGEO_POINTS,1) to only have points instead of faces that way you can see if your vertexbuffer is set up correctly without having to feed facedata.

As for the "self" I'm not sure how to explain you can look at it like a custom-made container/dictionary/wrapper (i'm not sure what to use here) for which you can create functions, they belong to classes

The NoeBitStream is a class, which has serveral variables and functions.
You usually do bitstream = NoeBitStream(data)
the 'internal' noesis python scripts take the "data" and begin performing functions on it and putting it in containers/variables
for instance bitstream.data now holds all the bytes.

When defining the class you can define functions for it

for example the readByte would sort of look like this:

Code: Select all

def readByte(self): //self tells this function the first arguement is the container itself
It can then access the data variable ( which would be the bitstream.data) as self.data inside the class itself.

[url]http://cpython_sandbox.readthedocs.org/en/latest/tutorial/classes.html[/url]

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 5:13 pm
by ShinRa Inc
MrAdults wrote:
finale00 wrote:Regarding this other stuff with people not seeing textures after exporting to FBX/DAE/etc., it's likely a case of the texture output files not being on a path the importing app can find, or that app needs file extensions on texture names because it isn't smart enough to scan for its own supported image formats. In the latter case you'd have to append the appropriate extension (like .png or whatever the extension is for the texture format you used) to the texture names yourself. That is easy to do to COLLADA files in a text editor, or with FBX if you use the option to export the FBX in ASCII mode. (see the FBX section of the advanced options list from the export dialog) I think I even made a FBX-specific option to auto-append extension strings, but I can't recall for sure. As mentioned, I'm posting from an iPad and can't run Noesis to see right now.
Checking the exported .daes in a text editor shows this;

Code: Select all

<image id="Image3">
      <init_from>../../../noesisv252/ffccdif_3</init_from>
    </image>
Now, I'm not exporting my files to a noesisv252 folder. Might that have something to do with it?

I'll also reiterate, the end result needs to be a .dae file, since that's the only format the destination program will accept. If needed, I can convert things around with Blender (or other option), but that's the format the end result needs to be in.

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 6:38 pm
by Vindis
Thanks Demonsangel your input was really helpful.
This is the draw part, dunno if it'll be any help, but... I always get correct point cloud in viewport and correct uv with it.

Code: Select all

for i in range(0,numMeshes[0]): #loop for number of meshes
		bs.seek(meshHeader[i][5],NOESEEK_ABS) #set pointer to vertex offset
		if (meshHeader[i][2])==12: #vertexbyte check
			VertBuff = bs.readBytes(meshHeader[i][3] * 0xC)
			rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 12, 0)
		elif (meshHeader[i][2])==24:
			VertBuff = bs.readBytes(meshHeader[i][3] * 0x18)
			rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 24, 0)
			rapi.rpgBindUV1BufferOfs(VertBuff,noesis.RPGEODATA_HALFFLOAT,24,20)
		elif (meshHeader[i][2])==32:
			VertBuff = bs.readBytes(meshHeader[i][3] * 0x20)
			rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 32, 0)
			rapi.rpgBindUV1BufferOfs(VertBuff,noesis.RPGEODATA_HALFFLOAT,32,28)
		
		for j in range(0,meshHeader[0][1]): #loop for each texture used by meshX
			bs.seek(meshHeader[i][7],NOESEEK_ABS)
			FaceBuff=bs.readBytes(meshHeader[i][4]*2)
			rapi.rpgSetMaterial(matName[0])
			rapi.rpgCommitTriangles(FaceBuff, noesis.RPGEODATA_USHORT, meshHeader[i][4],noesis.RPGEO_TRIANGLE, 1)
I have one big vertex buffer that's in the "for i" loop, and if it's possible i would like to separate the mesh in the "for j" loop. the for j loop now is just a mess, that i'm messing around with, to figure out how to slice. But as i said Noesis always crash.
(this shitty file format is tricky. lot of file made up from X number of faces and x/2 of them use material1, but no info about the other x/2 :mrgreen: )
Maybe i need another loop to read the mesh1-->mat1,mat2-->indice1,indice2 group...

Is it possible to comit face triangles and then add different materials to different face "range",some dedicated "slice" method or should I/we have to code it?

Senor Casaroja do you have any input for me about this? :D

Edit: Okay, it seems i successfully figured out the reading method. I assigned the correct material to the first group of face, but no matter where i put construct, it only builds the first one. The second,third, fourth always left out.

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 7:22 pm
by Demonsangel
You need to put a rapi.rpgBindPositionBuffersOfs() for each rapi.rpgCommitTriangles(), also what's in matName? You seem to keep setting it to the same material, could you also show me the code that declares matName.

I'm assuming meshHeader[4] holds the number of indices and not the triangles?

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 8:51 pm
by ShinRa Inc
Checking another export with the newer version of Noesis (Appearently the other file i looked at in the previous post is from an old export, from and older version;

Code: Select all

    <effect id="Effect" name="C__Documents_and_Settings_HP_Administrator_My_Documents_Installers__PSP_Final_Fantasy_VII_Crisis_Core__USA__FULL__FF7CC_File_Viewer_0_25_CCDATA_ffccdif_0">

Code: Select all

 <user_properties>C:%5CDocuments and Settings%5CHP_Administrator%5CMy Documents%5CInstallers%5C[PSP]Final Fantasy VII Crisis Core [USA][FULL]%5CFF7CC_File_Viewer_0.25%5CCCDATA%5Cffccdif_0</user_properties>
Could these two lines be throwing things off, especially in the end case where I'm uploading everything into a server that presumably is storing things it's own way? Both because the file locations are different, and the format for the file paths are wonky?

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 10:11 pm
by Vindis
Sorry no spoiler, so i have to paste it...

Code: Select all

#Noesis Python model import+export test module, imports/exports some data from/to a made-up format

#bs.seek(0xEC,NOESEEK_REL) #seek to header info
#red=bs.read("hh")
#testU=noesis.getFloat16(red[0])
#testV=noesis.getFloat16(red[1])
#print(testU,testV)
from inc_noesis import *

import noesis

#rapi methods should only be used during handler callbacks
import rapi

#registerNoesisTypes is called by Noesis to allow the script to register formats.
#Do not implement this function in script files unless you want them to be dedicated format modules!
def registerNoesisTypes():
   handle = noesis.register("!Star Wars: The Old Republic", ".gr2")
   noesis.setHandlerTypeCheck(handle, noepyCheckType)
   noesis.setHandlerLoadModel(handle, noepyLoadModel) #see also noepyLoadModelRPG
       #noesis.setHandlerWriteModel(handle, noepyWriteModel)
       #noesis.setHandlerWriteAnim(handle, noepyWriteAnim)
   noesis.logPopup()
       #print("The log can be useful for catching debug prints from preview loads.\nBut don't leave it on when you release your script, or it will probably annoy people.")
   return 1

NOEPY_HEADER = "GAWB"

#check if it's this type based on the data
def noepyCheckType(data):
   bs = NoeBitStream(data)
   if len(data) < 4:
      return 0
   if bs.readBytes(4).decode("ASCII").rstrip("\0") != NOEPY_HEADER:
      return 0
   return 1       


#load the model
def noepyLoadModel(data, mdlList):
	ctx = rapi.rpgCreateContext()
	bs = NoeBitStream(data)
	bs.seek(0x18,NOESEEK_ABS) #seek 0x18h/24
	numMeshes=bs.read("h") #number of meshes
	numTextures=bs.read("h") #number of textures
	bs.seek(0x50,NOESEEK_ABS) #seek 0x50h/80
	offset50offset=bs.read("i")
	offsetMeshHeader=bs.read("i")
	offsetMaterialNameOffsets=bs.read("i")
	bs.seek(0x70,NOESEEK_ABS) #seek to Mesh Header
	
	meshHeader=[] #mesh header data
	meshName=[] #mesh names
	matName=[] #materialname used by meshes
	texName=[] #texture names will be used
	matList=[] #Noesis built-in must have material list
	texList=[] #Noesis built-in must have texture list
	offsetMaterialUsageMesh=[]
	offsetMaterialUsageTexture=[]
	
	#for loop to get meshHeader data
	for i in range(0,numMeshes[0]): 
		offsetMeshName=bs.read("if") #unkFloat=bs.read("f") nem kell offsetMeshName olvassa
		numUsedTextures=bs.read("h")
		numBones=bs.read("hh") #unKnown=bs.read("h") nem kell numBones olvassa
		numVertexBytes=bs.read("h")
		numVertices=bs.read("i")
		numFaces=bs.read("i")
		offsetVertices=bs.read("i")
		offsetMaterialUsage=bs.read("i")
		offsetFace=bs.read("ii") #unkOffset=bs.read("i") nem kell offsetFace olvassa
		meshHeader.append([offsetMeshName[0],numUsedTextures[0],numVertexBytes[0],numVertices[0],numFaces[0],offsetVertices[0],offsetMaterialUsage[0],offsetFace[0]])
	#-------------------------------------------------------------
	
	#for loop to get meshName data
	for i in range(0,numMeshes[0]):
		bs.seek((meshHeader[i][0]),NOESEEK_ABS)
		nameLength=0
		boolP=True
		while (boolP==True):
			wak=bs.read("b")
			if (wak[0]!=0):
				nameLength=nameLength+1
			else:
				boolP=False
		bs.seek((meshHeader[i][0]),NOESEEK_ABS)
		meshName.append(bs.readBytes(nameLength).decode("ASCII").rstrip("\0"))
	#-------------------------------------------------------------
	
	#for loop to get matName data
	matNameOffsetList=[]
	bs.seek(offsetMaterialNameOffsets[0],NOESEEK_ABS) #seek to 0x50 offsetMaterialNameOffset
	for i in range(0,numTextures[0]): #fill matNameOffsetList
		matNameOffsetList.append(bs.read("i"))
	
	for i in range(0,numTextures[0]):
		bs.seek((matNameOffsetList[i][0]),NOESEEK_ABS)
		nameLength=0
		boolP=True
		while (boolP==True):
			wak=bs.read("b")
			if (wak[0]!=0):
				nameLength=nameLength+1
			else:
				boolP=False
		bs.seek(matNameOffsetList[i][0],NOESEEK_ABS)
		matName.append(bs.readBytes(nameLength).decode("ASCII").rstrip("\0"))
		texName=[s + "_d" for s in matName]
	#-------------------------------------------------------------
	
	#for loop to get materialUsage data
	for i in range(0,numMeshes[0]): # meshszamszor vegrehajt pl 3 mesh
		bs.seek((meshHeader[i][6]),NOESEEK_ABS)
		for j in range(0,(meshHeader[i][1])): #hasznalt textura szamszor vegrehajt pl 4 texture
			materialFacesIndex=bs.read("i")
			materialNumFaces=bs.read("i") #szorozni harommal a vertex szamhoz
			textureID=bs.read("i")
			bs.seek(0x24,NOESEEK_REL)
			offsetMaterialUsageMesh.append([materialFacesIndex[0],materialNumFaces[0],textureID[0]])
		offsetMaterialUsageTexture.append(offsetMaterialUsageMesh)
		offsetMaterialUsageMesh=[]
	#-------------------------------------------------------------
	
	#test to display materialUsage
	#for i in range(0,numMeshes[0]):
	#	for j in range(0,meshHeader[i][1]):
	#		print("Face number for mesh",i," texture",i,offsetMaterialUsageTexture[i][j][1])
	#-------------------------------------------------------------
	
	#material naming
	for i in range(0,numTextures[0]):
		material=NoeMaterial(matName[i],"")
		material.setTexture(texName[i]+".dds")
		matList.append(material)
		#print(matName)
	#-------------------------------------------------------------
	
	#the draw
This is the code yet. I deleted the draw section, because i have to rewrite the whole part. First I thought it's working, but tried with several mesh, and it wasn't perfect every time.
The material assign that you mentioned was just a dummy, for testing purposes. I work on one thing at a time. The material assignment system is the last on my list.
This was the part that i deleted

Code: Select all

for i in range(0,numMeshes[0]):
		bs.seek(meshHeader[i][5],NOESEEK_ABS) #set pointer to vertex offset
		for j in range(0,meshHeader[i][1]):
			#bs.seek(meshHeader[i][5],NOESEEK_ABS) #set pointer to vertex offset
			if (meshHeader[i][2])==12:
				VertBuff = bs.readBytes(meshHeader[i][3] * 0xC)
				rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 12, 0)
			elif (meshHeader[i][2])==24:
				VertBuff = bs.readBytes(meshHeader[i][3] * 0x18)
				rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 24, 0)
				rapi.rpgBindUV1BufferOfs(VertBuff,noesis.RPGEODATA_HALFFLOAT,24,20)
			elif (meshHeader[i][2])==32:
				VertBuff = bs.readBytes(meshHeader[i][3] * 0x20)
				rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 32, 0)
				rapi.rpgBindUV1BufferOfs(VertBuff,noesis.RPGEODATA_HALFFLOAT,32,28)
			rapi.rpgCommitTriangles(None, noesis.RPGEODATA_USHORT, meshHeader[i][3], noesis.RPGEO_POINTS, 1)
			#bs.seek(meshHeader[i][7],NOESEEK_ABS)
			#bs.seek(offsetMaterialUsageTexture[i][j][0],NOESEEK_REL)
			#FaceBuff=bs.readBytes((offsetMaterialUsageTexture[i][j][1]*3)*2)
			#rapi.rpgSetMaterial(matName[offsetMaterialUsageTexture[i][j][2]])
			#rapi.rpgCommitTriangles(FaceBuff, noesis.RPGEODATA_USHORT, (offsetMaterialUsageTexture[i][j][1]*3), noesis.RPGEO_TRIANGLE, 1)
			#print(offsetMaterialUsageTexture[i][j][1])
			#print(meshHeader[i][1])
	mdl=rapi.rpgConstructModel()
	mdl.setModelMaterials(NoeModelMaterials(texList, matList))
	mdlList.append(mdl)
	rapi.rpgClearBufferBinds()
	return 1
morning update: i'd rewritten the code. i get correct UV layouts and faces for each texture. It seems that i forgot to multiple the indice offset with 3, and that's why i didn't get correct start position. Now just have to change paste it into a loop.

update2: i use rapi.rpgClearBufferBinds(), but the second UV set still display UV1 underneath

Re: Señor Casaroja's Noesis

Posted: Thu Aug 30, 2012 11:33 pm
by Satoh
MrAdults wrote:
Satoh wrote:I've run into an issue with a BND archive from a PSP game
Lots of games use BND containers in seemingly slightly or very different formats. If you just want to get the GMO's then you might as well just use GitMO. It's on my personal web site. I would walk you through writing an archive script, but doing that much typing from the iPad is too terrible to endure. If you read the Duke3D GRP script carefully, it should tell you everything you need to know. Also, justChecking is set when Noesis just wants to validate the type, without performing any kind of extraction. It differs from how other data handlers have separate validation methods, and is inherited from legacy Noesis archive handling from years ago.
I see, thank you. I did manage to write a script that dumps the BND archive, and though it took a while to figure out what parts were actually relevant to the archive. With all the secondary ART file stuff, it took a while to realize only one of the functions had anything to do with the archive. After that it didn't take long at all to gather what it was doing.

And also, no problem on the iPad typing thing. I can only imagine the torture of writing a script on one. In any case, I managed to get it done, and am quite satisfied with the results.

Re: Señor Casaroja's Noesis

Posted: Sat Sep 01, 2012 8:50 pm
by Vindis
Thanks for All of You guys! For your help, for your patience. :up:
The main part is ready, now i just have to tidy up, and optimize a bit :lol:

Image

Re: Señor Casaroja's Noesis

Posted: Sun Sep 02, 2012 12:45 am
by MrAdults
It's nice to come back after a few days and see so many of the things I would have had to address myself have already been addressed. Thanks for helping, Demonsangel. Very much appreciated.
finale00 wrote:I wrote the contents of texData out to verify that it's actually a pvr image, and when I loaded it using noesis it loaded fine.

However noesis is telling me "warning could not load external texture"
I did the same sort of thing for another game that stored textures inside the model file so it shouldn't be an issue that the texture is not an "external" file
That looks like it should work. And loadTexByHandler asks each handler if the data is valid for that format, so multiple .pvr handlers should not be an issue. Can you send me the full script and one of the files in question? I'll debug it engine-side and see what's happening.

Satoh: Glad you got it working.

ShinRa Inc: FCollada does weird shit sometimes as well with automatically shoving the current working directory into texture paths. I don't know why. You should really be going to FBX instead. You may also need to use -fbxtexext .png (or whatever format your textures are output to) if you want to append that in the FBX scene file to all the texture names, as I referenced in an earlier post. Then you can export from there to whatever form of COLLADA you're trying to end up in using whichever COLLADA exporter is recommended for whatever you're trying to get the model into. Standards between COLLADA implementations are always a mess.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 02, 2012 5:21 am
by Darko
MrAdults wrote:It's nice to come back after a few days and see so many of the things I would have had to address myself have already been addressed. Thanks for helping, Demonsangel. Very much appreciated.
finale00 wrote:I wrote the contents of texData out to verify that it's actually a pvr image, and when I loaded it using noesis it loaded fine.

However noesis is telling me "warning could not load external texture"
I did the same sort of thing for another game that stored textures inside the model file so it shouldn't be an issue that the texture is not an "external" file
That looks like it should work. And loadTexByHandler asks each handler if the data is valid for that format, so multiple .pvr handlers should not be an issue. Can you send me the full script and one of the files in question? I'll debug it engine-side and see what's happening.

Satoh: Glad you got it working.

ShinRa Inc: FCollada does weird shit sometimes as well with automatically shoving the current working directory into texture paths. I don't know why. You should really be going to FBX instead. You may also need to use -fbxtexext .png (or whatever format your textures are output to) if you want to append that in the FBX scene file to all the texture names, as I referenced in an earlier post. Then you can export from there to whatever form of COLLADA you're trying to end up in using whichever COLLADA exporter is recommended for whatever you're trying to get the model into. Standards between COLLADA implementations are always a mess.
Any news regarding noesis or you're still busy Mr. Adults??

See ya.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 02, 2012 6:03 am
by ShinRa Inc
Ultimately, I'm trying to upload the models into an offshoot of SecondLife, known as Aurora; The only format it will accept is Collada .dae. The only other program I have at the moment is Blender, which doesn't seem to work with autodesk that I can tell. Any attempt I've made to import anything from Noesis into blender has lost the texturing assignments (I've tried .dae and .obj). The mapping seems intact; if I assign the textures manually in blender, they go in the right places. Even these are having some display problems in the final destination, however;
Image

Again, apologies in advance if I'm missing anything 'simple', or mixing up any terminology, I'm very new to all this.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 02, 2012 11:43 am
by Demonsangel
http://usa.autodesk.com/adsk/servlet/pc ... d=10775855
FBX 2013.2 Converter for Windows (exe - 26327Kb)

Utility that converts between different versions of fbx and dae iirc. See if the -fbxtexext works for you and then just convert everything with the tool to dae.

If it still doesn't work at all, pm me the script( if it isn't builtin), one of the models and it's textures.

Re: Señor Casaroja's Noesis

Posted: Sun Sep 02, 2012 6:07 pm
by MrAdults
Darko wrote:Any news regarding noesis or you're still busy Mr. Adults??

See ya.
I'm still working on the next release, it's just taking a while, since I don't tend to get an average of an hour or so of work done on it every couple days. Maybe sometime in the next few days. I have a lot of features/fixes/additions being worked on concurrently.