[TDUF Research] Tutorials : Understand TDU files with FileTool

[Djey]

Closed thread: you'll find updated documentation within TDUCP project:

 

https://github.com/djey47/tdu-cp/wiki/TDUF-Reverse-Engineering-Files

 

[TDUF] Understand TDU files with FileTool

Hello,

this thread will explain how to use a TDUF feature, that is FileTool.

Also, it's a place where people may collaborate, to help with understanding of TDU Files.

 

PRE-REQUISITES

 

• Installed TDUF Toolset (0.2.1+): http://forum.turboduck.net/threads/32570-TDUF-Djey-Tools-and-Discussion-about-new-modding-possibilities
  
• Web browser (easy!)
  
• Hex Editor software your are comfortable with (optional, but recommended)
  
• Sense of logic (a little)
  
• Free time (a bunch)
  
• Fierceness!
  
• That's all for now :)
  

 

TUTORIALS

If you'd need to remember only one thing: you don't have to know the complete structure of a file to begin playing with it. Favor baby steps ;)

 

1. OPENING EXISTING FILE

 

To achieve this, you will need latest structure (xxxxxxx-map.json file) for the file you wanna open. Structures are available in each TDUF release; they are provided as separate download in this thread, as well.

Please note that structures are not always complete - thus providing limited support to what can be read or written in the file.

 

1. Make sure TDUF CLI (command line interface) is started properly
   
2. Get the file you want to get opened, make it ready (extract it from BNK, then decrypt it with TDUDEC tool if necessary)
   
3. Locate needed structure file , it is generally named <file extension>-map.json. If there's no such file, that means that the structure will have to be found and written. This topic will be addressed in 4th tuorial below.
   
4. Run following command in TDUF CLI :
   

   FileTool jsonify -i "C:\Users\Bill\Desktop\Sector-0-2-0-6-A.pmi" -s "C:\Users\Bill\Desktop\PMI-map.json" -o "C:\Users\Bill\Desktop\Sector-0-2-0-6-A.json"

   
   Explanation: FileTool jsonify -i "pathToFileToOpen" -s "pathToStructureFile" -o "pathToNewFileToCreate"
   

5. Make sure the new file has been created (Sector-0-2-0-6-A.json in current example), else report error written in console in this thread.
   
6. To open this newly created file, use any JSON editor (currently recommending http://jsoneditoronline.org/ )
   
7. Here are displayed contents: [ATTACH=CONFIG]22455[/ATTACH]
   
8. That's all for this first tutorial ;)
   

 

 

 

2. EDITING EXISTING FILE

 

As you're about to start reading this tutorial, I assume you already get generated files xxxxxxx.json from TDU files (as explained in previous tutorial).

1. Start your favorite JSON Editor, if you don't have one just t head to http://jsoneditoronline.org/
   
2. From editor, open file generated file (or paste contents into area at left-hand side: [ATTACH=CONFIG]22455[/ATTACH]
   
3. Synchronize contents with editor: click on => (left-to-right arrow) to get a more comfortable view on JSON contents
   
4. From the area at right-hand side, following changes are available:
   
   • Changing field value, that is right-hand part of entry (name : value)
     
   • Adding one or more fields - use with caution as it may cause issues when saving back to TDU file !
     
   • Deleting one or more fields - same remark!
     
   • Reordering fields, though new order will be lost upon next JSON import
     

Important notes:

- I don't recommend changing field names (left part of entry) as it will desynchronize current file with structure; if you really need this, make sure you have changed field name in structure as well!

- Don't even try to change values of fields typed UNKNOWN in structure (bytes written as Base64-encoding)!

[*]Now time to save to TDU file ! See next tutorial for this.

 

 

 

3. SAVING EXISTING FILE

 

This part will deal with saving modified JSON file with latest changes then exporting it back to a TDU file.

1. From JSON editor, save current file to disk, or use <= (right-to-left) arrow to transfer data to left-hand area and copy it to a new file.
   
2. Back to TDUF CLI (finally) run following command:
   

   FileTool applyjson -i "C:\Users\Bill\Desktop\Sector-0-2-0-6-A.json" -s "C:\Users\Bill\Desktop\PMI-map.json" -o "C:\Users\Bill\Desktop\Sector-0-2-0-6-A.pmi"

   
   Explanation: FileTool applyjson -i "pathToModifiedFile" -s "pathToStructureFile" -o "pathToTDUFile"
   

3. Make sure the new file has been created (Sector-0-2-0-6-A.pmi in current example), else report error written in console in this thread.
   
4. Et voila!
   

 

 

 

4. ADVANCED: ADD SUPPORT FOR NEW FILE

TODO

 

FILE STRUCTURE UPDATE

Here are all latest known file structures to play with. Feel free to test, modify, understand and enhance them ;)

DOWNLOAD

 

REFERENCE

FileTool instructions

• applyjson : writes a json back to TDU file.
  
• decrypt : make XTEA encrypted TDU file readable again.
  
• encrypt : transform file contents with XTEA algorithm - used by some (db, btrq, playersave...).
  
• jsonify : converts a supported TDU file to JSON file, allowing it to be edited.
  

 

File Structure reference

Example:

{
 "name": "TDU BTRQ (BinaryTorque?)",
 "littleEndian": true,
 "cryptoMode": 1,
 "fields": [
   {
     "name": "entryCount",
     "type": "INTEGER",
     "signed": true,
     "size": 4
   },
   {
     "name": "gap1",
     "type": "GAP",
     "size": 4
   },
   {
     "name" : "graph",
     "type" : "REPEATER",
     "size" : "?entryCount?-1",
     "subFields" : [
       {
         "name": "torqueNm",
         "type": "FPOINT",
         "size": 4
       },
       {
         "name": "engineSpeed",
         "type": "FPOINT",
         "size": 4
       }
     ]
   },
   {
     "name": "gap2",
     "type": "GAP",
     "size": 4
   },
   {
     "name": "unknown",
     "type": "UNKNOWN"
   }
 ]
}

First level information reference:

• name : to identify file you want to decode
  
• littleEndian : true/false, indicates byte order - most of TDU files are little endian
  
• cryptoMode : in case of encrypted file, indicates the encryption mode used - 0=Savegames, 1=Others (db, btrq...)
  
• fields : list of fields contained in this TDU file (see Field reference below)
  

Note that cryptoMode set to 0 or 1 will make file to be automatically encrypted/decrypted by FileTool, so you won't have to.

 

Field/Subfield reference:

• name : give clearest meaning as possible - must be unique !
  
• type: one of predefined values indicating data kind (see Field types below)
  
• signed: (from version 0.6.0) true if value should be signed, false otherwise. When absent, value is considered as unsigned.
  
• size: number of bytes used by this field (see Size tricks below)
  
• subFields: used when field type is a REPEATER, list of repeated fields
  

Field types:

• TEXT: clear text data
  
• INTEGER: an integer, numeric value (e.g.: 53)
  
• FPOINT: a floating-point, numeric value (e.g : 235,6547)
  
• GAP: an area filled with 0's
  
• UNKNOWN: to be discovered still
  
• REPEATER: embeds a list of complex values (with their own layout)
  

Size tricks:

- Size attribute is basically the number of bytes used by corresponding field, but with some exceptions:

• When UNKNOWN field type, it is not mandatory - if omitted, the field is expected to occupy all remaining bytes in file.
  
• When REPEATER, its meaning is the count of repeated items.
  
• When REPEATER, it's not mandatory either - if omitted, the sub-items are expected to be repeated till the end of the file.
  

Important: when INTEGER field type, only supported values are 1 or 2 or 4.

When FPOINT field type, 2 or 4 are supported.

 

- Value of this attribute may also be a formula

e.g : size: "=(2+3)*6" will interpret it as 30.

 

- A formula may contain a reference to a previously read field

e.g size: "=5*?fieldName?" will interpret it as 30, as long a field named fieldName has been read before and has a value of 6.

Please note:

• Field reference writes itself as field name declared earlier in structure, surrounded by question marks characters (?)
  
• Concerning repeated fields, they get unique name each, following: repeaterFieldName[itemRank].repeatedFieldName
  e.g ?itemList[5].speed?, will reference speed field, repeated under itemList field, at rank 6 (as first item is always at rank 0).
  
• Also, it is possible to reference a repeated field with the same rank without giving the full name:
  e.g ?speed?, will reference speed field, repeated under the same field as current one and as same rank.
  

 

CREDITS

• 2CVSUPERGT &TDUCK community => BIN (cameras) research
  
• mcv100 => BTRQ research
  
• Speeder => PMI/SHK research
  

[Djey]

-Reserved-

 

File structure description is complete, I think. Feel free to ask if you shall need some things to be clarified! ;)

[Tob-Racer]

It doesn't really talk to me all this stuff but I'm sure it's gonna be beneficial for many modders :) It's great to see you developing all this Djey. Can't believe TDUMT was released I think in 2009 and 6 years later you're still doing tools for TDU. Thank you so much for still improving the game today ;)

[Djey]

Yeah, that's been a while - but I made long breaks as well (and many others in the future, probably). :)

 

I totally understand and assume that these contents are for advanced modders and developpers overall.

But since TDUMT toolset lacked of documentation, I don't want TDUF prevents motivated people to have a look at it, as that did ages ago.

[mcv100]

Is the provided btrq json reference file just for reference or should it actually return something meaningful? I was trying out the jsonify option on a btrq file from a random car, and I just cannot get anything meaningful out of it. The floating point values are all over the place it seems, and the entrycount is in the billions... I'm new to the forums btw, but I might be interested in helping out a bit, if I have time that is :)

[Djey]

In fact, the .btrq files have been encrypted by EDEN devs :)

You should use tdudec to encrypt/decrypt it (with non-savegame mode I believe)

 

I will add a simple btrq read/write example as tutorial.

 

 

Also, next TDUF version (0.3.0) will handle encrypted files automatically.

[Djey]

First tutorial just added: opening existing file :D

 

Please let me know if some things need to be precised.

[Dux]

Agree with Tob.

I hope some modders will have use of this and make something nice.

This is alien language for me but someone will surely make something. :D

Good job Djey.

[Djey]

Thanks for the messages anyhow, I keep my hopes up that one day regular players can enjoy what's now preparing under the hood.

[Dux]

Well it might be ok if someday we can add new slots and new cars on them ourselfs, but ok, its good like this too, modders can make megapacks and stuff.

And i dont know what else might be possible.

Maybe importing crew map, hmmm . . . :hmmm:

Lol, just kidding.

:)

[Djey]

Tutorials 2 & 3 just added :)

 

See updated first post.

 

---

 

I've updated structure package with PMI and cameras.bin support, as well!

Dropbox - TDUF-structures.zip

[Djey]

Updated structure for cameras.bin files !

http://bit.ly/1GARTAJ