Dragon Age 2 (PC)

[figuresculptor]

Check this thread: viewtopic.php?f=10&t=6117&hilit=dragon+age+2

It contains a QuickBMS for extracting files from the Dragon Age 2 .erf files

It will expand a bunch of numbered files, and then a subfolder called output that will have the named .msh files.

The texture files are .dds images. Using any of the scripts above, you can import the .msh file into a 3D program. It will not have its armature, and you'll have to manually find the textures if you want those.

There is currently no way to get the changes back into the game, though I would imagine there will be at some point.

[figuresculptor]

Revised Script.

I found why certain models (mostly hair) wouldn't load - they had a vertSize of 40, which wasn't handled by any of the code. I've replaced the if statement with new code that calculates the number of unknown bytes between the vertex position and the texture coordinates, and it should now load every data model from Dragon Age 2.

Code: Select all

bl_addon_info = {
	"name": "Import Dragon Age 2 Mesh files (.msh)",
	"author": "",
	"version": (2, 0),
	"blender": (2, 5, 6),
	"api": 31847,
	"location": "File > Import ",
	"description": "Import Dragon Age 2 Mes (.msh)",
	"warning": "",
	"wiki_url": "",
	"tracker_url": "",
	"category": "Import-Export"}

"""
Version': '1.0' 

This is a straight port of the 3DS Max Script from this forum
posting:

http://forum.xentax.com/viewtopic.php?f=16&t=6119&hilit=dragon+age+2

This version is based on the script posted with this timestamp

Posted: Mon Mar 14, 2011 3:22 am 

All credit to chrrox for figuring the format out.

"""

import bpy
import mathutils
import os
import sys
import string
import math
import re
from string import *
from struct import *
from math import *
import mathutils
from bpy.props import *


DEBUGLOG = False
LONGSIZE = 4
FLOATSIZE = 4
HALFFLOATSIZE = 2
SHORTSIZE = 2

def unpack_list(list_of_tuples):
	l = []
	for t in list_of_tuples:
		l.extend(t)
	return l

def halfToFloatPrivate(h):
	s = int((h >> 15) & 0x00000001)		# sign
	e = int((h >> 10) & 0x0000001f)		# exponent
	f = int(h &			0x000003ff)		# fraction

	if e == 0:
		if f == 0:
			return int(s << 31)
		else:
			while not (f & 0x00000400):
				f <<= 1
				e -= 1
			e += 1
			f &= ~0x00000400
	elif e == 31:
		if f == 0:
			return int((s << 31) | 0x7f800000)
		else:
			return int((s << 31) | 0x7f800000 | (f << 13))

	e = e + (127 -15)
	f = f << 13

	return int((s << 31) | (e << 23) | f)

def halfToFloat(h):
	result = halfToFloatPrivate(h)
	str = pack('I',result)
	f = unpack('f', str)
	return f[0]
	
def mshImport(infile):
	global DEBUGLOG
	print ("--------------------------------------------------")
	print ("---------SCRIPT EXECUTING PYTHON IMPORTER---------")
	print ("--------------------------------------------------")
	print ("Importing file: ", infile)
	
	mshfile = open(infile,'rb')
	if (DEBUGLOG):
		logpath = infile.replace(".msh", ".txt")
		print("logpath:",logpath)
		logf = open(logpath,'w')
		
	def printlog(strdata):
		if (DEBUGLOG):
			logf.write(strdata + "\n")

	basename = os.path.basename(infile)
	# basename = os.path.splitext(infile)[0]		
	
	
	printlog("basename:" + basename)
	
	mshfile.seek(0x1A0)
	baseoff = 0x1A0
	

	namebase, garbage, unkbase, vertbase, facebase, garbage1, garbage2, meshcount = unpack('<8l', mshfile.read(8*LONGSIZE))

	printlog("baseoff: " + str(baseoff))
	printlog("unkbase: " + str(unkbase))
	printlog("vertbase: " + str(vertbase))
	printlog("facebase: " + str(facebase))
	printlog("meshcount: " + str(meshcount))
	
	vertSizeArray = []
	vertCountArray = []
	faceCountArray = []
	vertPosArray = []
	facePosArray = []
	
	for i in range(meshcount):
		printlog("assessing mesh " + str(i))
		float01, float02, float03, float04, float11, float12, float13, float14, float21, float22, float23, float34 = unpack('<12f', mshfile.read(12*FLOATSIZE))
		unk01, vertSize, vertCount, faceCount, vertPos, facePos, unk2, long01, long02, long03, long04, vertCount2 = unpack('<12l', mshfile.read(12*LONGSIZE))
		printlog("\tvertsize: " + str(vertSize))
		printlog("\tvertCount: " + str(vertCount))
		printlog("\tfaceCount: " + str(faceCount))
		printlog("\tvertPos: " + str(vertPos))
		printlog("\tfacePos: " + str(facePos))
		vertSizeArray.append(vertSize)
		vertCountArray.append(vertCount)
		faceCountArray.append(faceCount)
		vertPosArray.append(vertPos)
		facePosArray.append(facePos)
	
	faceStart = baseoff + facebase + 4
	vertStart = baseoff + vertbase + 4
	nameStart = baseoff + namebase
	unkStart = baseoff + unkbase
	
	if (DEBUGLOG):
		printlog("faceStart: " + str(faceStart))
		printlog("vertStart: " + str(vertStart))
		printlog("nameStart: " + str(nameStart))
		printlog("unkStart: " + str(unkStart))
	
	for a in range(meshcount):
		me_ob = bpy.data.meshes.new(basename + str(a))
		printlog(("New Mesh = " + me_ob.name))
		vertArray = []
		uvArray = []
		normalArray = []
		faceArray = []
		printlog("===Offset = " + str(vertStart + vertPosArray[a]))
		printlog("===vertStrt = " + str(vertStart))
		printlog("===vertPos[a] = " + str(vertPosArray[a]))
		mshfile.seek(vertStart + vertPosArray[a])
		for b in range(vertCountArray[a]):
			data = mshfile.read(3*FLOATSIZE)
			vx, vy, vz = unpack('<3f', data)
			bytesOfUnknown = (vertSizeArray[a] - 16)
			unknown = mshfile.read(bytesOfUnknown)
			tu_h = unpack('H', mshfile.read(HALFFLOATSIZE))
			tv_h = unpack('H', mshfile.read(HALFFLOATSIZE))
			tu = halfToFloat(tu_h[0])
			tv = halfToFloat(tv_h[0])
			vertArray.extend([(vx, vy, vz)])
			uvArray.extend([(tu, tv)])
	
		printlog("facePosArray: " + str(facePosArray[a]))
		mshfile.seek(faceStart + facePosArray[a] * 2)
		faceLen = faceCountArray[a]
		
		printlog("faceCountArray length: " + str(faceLen))
		
		if (faceLen > 0):
			faceLen = faceLen / 3
		else:
			faceLen = 0
		printlog("facelen: " + str(faceLen))
		
		faceUVArray = []
		
		for b in range(int(faceLen)):
			f1, f2, f3 = unpack('<3h', mshfile.read(3*SHORTSIZE))
			if (f1 < len(vertArray) and f2 < len(vertArray) and f3 < len(vertArray)):
				faceArray.extend([f1, f2, f3, 0])
				faceUVArray.extend([(uvArray[f1], uvArray[f2], uvArray[f3])])
			#printlog("Adding array:" + str(f1) + ", " + str(f2) + ", " + str(f3))
			
		printlog("vertArray length: " + str(len(vertArray)))
		printlog("faceArray length: " + str(len(faceArray)))
		me_ob.vertices.add(len(vertArray))
		me_ob.faces.add(len(faceArray)//4)
		me_ob.vertices.foreach_set("co", unpack_list(vertArray))
		me_ob.faces.foreach_set("vertices_raw", faceArray)
		me_ob.faces.foreach_set("use_smooth", [False] * len(me_ob.faces))
		
		texture = []
		texturename = basename + "_tex_" + str(a)
		if (len(faceUVArray) > 0):
			uvtex = me_ob.uv_textures.new() #add one uv texture
			uvtex.name = "Imported UV"
			for i, face in enumerate(me_ob.faces):
				blender_tface= uvtex.data[i] #face
				blender_tface.uv1 = faceUVArray[i][0] #uv = (0,0)
				blender_tface.uv2 = faceUVArray[i][1] #uv = (0,0)
				blender_tface.uv3 = faceUVArray[i][2] #uv = (0,0)
			texture.append(uvtex)
		
		materialname = "mat"
		materials = []

		matdata = bpy.data.materials.new(materialname)
		matdata.diffuse_color=(float(0.04),float(0.08),float(0.44))#blue color
		
		texdata = None
		texIndex = len(bpy.data.textures) - 1
		if (texIndex >= 0):
			texdata = bpy.data.textures[len(bpy.data.textures)-1]
			if (texdata != None):
				#print(texdata.name)
				#print(dir(texdata))
				texdata.name = "texturelist1"
				matdata.active_texture = texdata
		materials.append(matdata)
		for material in materials:
			#add material to the mesh list of materials
			me_ob.materials.append(material)
			
		# me_ob.uv_textures.new(basename + "_uv_" + str(i))
		# uvtex = me_ob.uv_textures[0]
		# for n,tf in enumerate(uvArray):
		# 	datum = uvtex.data[n]
		# 	datum.uv1 = tf[0]
		# 	datum.uv2 = tf[1]
		# 	datum.uv3 = tf[2]
		# 
		# uvtex.foreach_set("co", unpack_list(uvArray))
		# me_ob.uv_textures.foreach_set("co", unpack_list(uvArray))
		me_ob.update()
		
		obmesh = bpy.data.objects.new(basename,me_ob)
		bpy.context.scene.objects.link(obmesh)
		bpy.context.scene.update()
	
def getInputFilename(filename):
	checktype = filename.split('\\')[-1].split('.')[1]
	print ("------------",filename)
	if checktype.upper() != 'MSH':
		print ("  Selected file = ",filename)
		raise (IOError, "The selected input file is not a *.msh file")
	mshImport(filename)

class IMPORT_OT_msh(bpy.types.Operator):
	'''Load a Dragon Age 2 Mesh File'''
	bl_idname = "import_scene.msh"
	bl_label = "Import MSH"

	# List of operator properties, the attributes will be assigned
	# to the class instance from the operator settings before calling.
	filepath = StringProperty(name="File Path", description="Filepath used for importing the OBJ file", maxlen= 1024, default= "")

	def execute(self, context):
		getInputFilename(self.filepath)
		return {'FINISHED'}

	def invoke(self, context, event):
		wm = context.window_manager
		wm.fileselect_add(self)
		return {'RUNNING_MODAL'}


def menu_func(self, context):
	self.layout.operator(IMPORT_OT_msh.bl_idname, text="Dragon Age 2 Mesh (.msh)")


def register():
	bpy.types.INFO_MT_file_import.append(menu_func)
	
def unregister():
	bpy.types.INFO_MT_file_import.remove(menu_func)

if __name__ == "__main__":
    register()

Here's the desire demon imported - I had to manually load the image and body texture (face diffuse = 16511.dds, body diffuse = 9902.dds, eyeball diffuse = 16510.dds - there are normal maps also, but they don't appear to be tangent space normal maps or object space normal maps, so not sure how to use them yet).

[chrrox]

Find anything that looks like bones in any of the files that's why i got bored with this game i cant finish the importer.

[figuresculptor]

Find anything that looks like bones in any of the files that's why i got bored with this game i cant finish the importer.

I'm guessing they might be in the .mmh file, or one of the other ones, but there's no easy way to match up the named mesh files with the numbered mmx files. But no, I haven't seen anything that looked like bones yet, though most of my time porting the stuff you figured out. I don't have a lot of experience reverse engineering 3D data formats..

[sirew]

Hi guys, have you ever thought doing converter, with UI?, with just a click of a button and not using any command to convert, just asking

[Modman69]

This Script I found on the Dragon Age Forums

Although, it is for Dragon Age: Origins maybe it will contain some useful insight into the .mmh format for DA 2.


http://hotfile.com/dl/110281794/5ccda19/Readme.pdf.html (Here's the .pdf also for explanation of format).

Also, Fantastic work by both Chrrox and figuresculptor

Edit:

This is the Latest Dragon Age: Origins Importer/Exporter by Author Eshme and found on the Dragon Age Nexus

http://hotfile.com/dl/110283196/8ff46a6 ... 42.7z.html

[figuresculptor]

Hi guys, have you ever thought doing converter, with UI?, with just a click of a button and not using any command to convert, just asking

Well, 3DS Max and Blender both provide a better UI than I'm likely to come up with

Honestly, I haven't given it any thought myself. For one or a few files, loading them into a 3D program is the easiest way for me to work because that's where I will edit or convert them. If I'm going to be doing multiple files, I prefer the command line. I can do something like:

find. | grep 'msh$' | xargs -I@ python da2obj.py @

and it'll convert a whole directory for me., Much faster than booting a GUI converter.

The other reason you don't want me writing a GUI is because (deep dark secret here) I work primarily on a Mac. I boot into Windows for some games, but for day-to-day stuff, I'm in OS X.

There is a pretty good Windows GUI program out there called something like 3D Object Converter - and the author is pretty good about following these boards and adding new game formats, might be worth checking it out, though I doubt the author's added DA2 yet since Chrrox just posted the file format recently.

[greywaste]

When I was taking a peek at some of the models, I cleared the folder of all files apart from .msh and .dds to make it easier to browse the textures, I noticed the .anb files were only about (+/-) 150, Maybe these have skeleton info since so many of the objects were statics? Anyway though, figuring which one goes with which mesh : /

[figuresculptor]

Normal Maps

It turns out that the orange normal maps used by Dragon Age 2 are just run-of-the-mill tangent space normal maps, but they've been byte swapped. Instead of being in RGBA byte order. They appear to be in BGXR, where X is an unused value of 0 (tangent space normal map uses 3 values, so alpha is unneeded). Converting an orange normal map to a standard purple tangent space normal map is just a matter of swapping the bytes around. Here's the desire demon's body rendered with the converted alpha map. Notice the hint of abdominal muscles - those aren't in the model, they're coming from the normal map.



I'm attaching a converter as a binary for Mac OS X. For Windows, I'm attaching the source code - somebody else will have to compile it, as I don't have a compiler installed under Windows. It requires libpng, but otherwise, this is straight, platform agnostic C that should compile fine as a Windows command line program.

Code: Select all

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <stdarg.h>

#define PNG_DEBUG 3

// Depending on platform, you may need to change the include
// #include <png.h>
#include "/usr/X11/include/png.h"

void abort_(const char * s, ...)
{
    va_list args;
    va_start(args, s);
    vfprintf(stderr, s, args);
    fprintf(stderr, "\n");
    va_end(args);
    abort();
}

int x, y;

int width, height;
png_byte color_type;
png_byte bit_depth;

png_structp png_ptr;
png_infop info_ptr;
int number_of_passes;
png_bytep * row_pointers;

void read_png(char* file_name)
{
    char header[8]; 
    FILE *fp = fopen(file_name, "rb");
    if (!fp)
        abort_("[read_png_file] File %s could not be opened for reading", file_name);
    fread(header, 1, 8, fp);
    if (png_sig_cmp((png_bytep)header, 0, 8))
        abort_("[read_png_file] File %s is not recognized as a PNG file", file_name);
    
    png_ptr = png_create_read_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL);
    
    if (!png_ptr)
        abort_("[read_png_file] png_create_read_struct failed");
    
    info_ptr = png_create_info_struct(png_ptr);
    if (!info_ptr)
        abort_("[read_png_file] png_create_info_struct failed");
    
    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[read_png_file] Error during init_io");
    
    png_init_io(png_ptr, fp);
    png_set_sig_bytes(png_ptr, 8);
    
    png_read_info(png_ptr, info_ptr);
    
    width = (int)png_get_image_width(png_ptr, info_ptr);
    height = (int)png_get_image_height(png_ptr, info_ptr);
    color_type = png_get_color_type(png_ptr, info_ptr);
    bit_depth = png_get_bit_depth(png_ptr, info_ptr);
    
    number_of_passes = png_set_interlace_handling(png_ptr);
    png_read_update_info(png_ptr, info_ptr);
    
    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[read_png_file] Error during read_image");
    
    row_pointers = (png_bytep*) malloc(sizeof(png_bytep) * height);
    for (y=0; y<height; y++)
        row_pointers[y] = (png_byte*) malloc(png_get_rowbytes(png_ptr,info_ptr));
    
    png_read_image(png_ptr, row_pointers);
    
    fclose(fp);
}


void write_png(char* file_name)
{
    FILE *fp = fopen(file_name, "wb");
    if (!fp)
        abort_("[write_png_file] File %s could not be opened for writing", file_name);
    
    png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, NULL, NULL, NULL);
    
    if (!png_ptr)
        abort_("[write_png_file] png_create_write_struct failed");
    
    info_ptr = png_create_info_struct(png_ptr);
    if (!info_ptr)
        abort_("[write_png_file] png_create_info_struct failed");
    
    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[write_png_file] Error during init_io");
    
    png_init_io(png_ptr, fp);

    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[write_png_file] Error during writing header");
    
    png_set_IHDR(png_ptr, info_ptr, width, height,
                 bit_depth, color_type, PNG_INTERLACE_NONE,
                 PNG_COMPRESSION_TYPE_BASE, PNG_FILTER_TYPE_BASE);
    
    png_write_info(png_ptr, info_ptr);
    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[write_png_file] Error during writing bytes");
    
    png_write_image(png_ptr, row_pointers);
    if (setjmp(png_jmpbuf(png_ptr)))
        abort_("[write_png_file] Error during end of write");
    
    png_write_end(png_ptr, NULL);
    
    for (y=0; y<height; y++)
        free(row_pointers[y]);
    free(row_pointers);
    
    fclose(fp);
}


void process_dragon_age_normal_map(void)
{
    
    for (y=0; y<height; y++) 
    {
        png_byte* row = row_pointers[y];
        for (x=0; x<width; x++) 
        {
            png_byte* ptr = &(row[x*4]);
//            printf("Pixel at position [ %d - %d ] has RGBA values: %d - %d - %d - %d\n",
//                   x, y, ptr[0], ptr[1], ptr[2], ptr[3]);
            
            png_byte tmp = ptr[2];
            ptr[2] = ptr[0];
            ptr[0] = ptr[3];
            ptr[3] = 255 - tmp;
        }
    }
}


int main(int argc, char **argv)
{
    if (argc != 3)
        abort_("Usage: danorm <file_in> <file_out>");
    
    read_png(argv[1]);
    process_dragon_age_normal_map();
    write_png(argv[2]);
    
    return 0;
}

[codo85]

Hi guys, can somebody please explain how can I use figuresculptor's port to blender of chrrox's script?
I mean, I save the text in a .py file... and then?
I want to import DA2 models in blender, but Dragon Age Tools works only with DA:O models, and I don't know how to replace its scripts with the one posted in this thread.

Thanks in advance!

[figuresculptor]

Hi guys, can somebody please explain how can I use figuresculptor's port to blender of chrrox's script?
I mean, I save the text in a .py file... and then?

1) Save script to a text file. You may have to convert spaces to tabs (four spaces = 1 tab) depending on your version of Python. Make sure it has a .py extension. (if you have problems, I can upload the script in a zip file)
2) Launch Blender 2.56b
3) Go to the File menu and select "User Preferences"
4) Navigate to the Add-Ons tab
5) Click the "Install Add-On…" button at the bottom
6) Select the file where you saved the script
7) On the left side, select "Import-Export" to filter the sripts
Look for the one that says "Import-Export: Import Dragon Age 2 Mesh files (.msh) and click the check-box on the right side so it's checked
9) Close the user preferences window
10) Type ctrl-u to save the preference changes
11) Select File menu, Import submenu, Dragon Age 2 Mesh (.msh)
12) Find the file you want to import
13) Profit!

[codo85]

Awesome man, awesome!
I could import several DA2 models and edit them with no problems.
Now we have to wait for a script to export the models in .msh format, so that we can pack them back in .erf files and see them in game, am I right?

[figuresculptor]

Now we have to wait for a script to export the models in .msh format, so that we can pack them back in .erf files and see them in game, am I right?

Well, somebody needs to figure out how to extract the bones and which vertices are attached to which bones, and THEN we need to be able to get the data back into .erf files (which I think is now possible - the v3.0 erf file seems to be laid out here: http://social.bioware.com/wiki/datoolset/index.php/ERF.

Over at Dragon Age Nexus, tazpn appears to have figured out the file format changes for DA2: http://social.bioware.com/project/4253/#details However, I'm unable to get the erf extraction to work - it doesn't seem to do anything, and when I try to convert a mesh, I get an error (possibly from not having the .mmh available). His source code is available, so if I had time, I'd look through is code and see where the bones are, but it will probably be a few more weeks before I have that kind of time to spend on this stuff again.

It looks like it's possible to create modified meshes now, but since I can't get tazpn's tool to work, I can't confirm it, and I haven't seen any body creating mesh mods yet, so not sure where that leaves us.

[chrrox]

if someone sends me a bone file ill look at the format no problem

[codo85]

Umh... I fear I'm missing something...
This is my naive approach to the matter.
My plan was to extract the different files (.mmh, .msh and .phy), edit them and pack them back in .erf.
Extracting the files from .erf is easy with GFF editor.
Packing the files back is even easier with Erf packer.
To edit the models I think I have to import the .msh in blender with the script you posted and, when the editing is finished, to get another .msh file as output. In order to do this I think I need an export script, which is the only thing we're lacking... or not?