[C#] Read DBC-Files

[Mythli]

Today i want to figure out how to read DBC files with c# so i started programming and i wonder how i can read Strings properly (assign them to "Rows") and how to detect and remove empty columns (columns that just have "0"-Values) like many DBC-Editors do

Here is my Source:

(Pastebin)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Windows.Forms;

/* --DBC FILE STRUCTURE--
[Header]
Column      Field         Type         Notes
1          Signature     String         (4-bytes) string, always 'WDBC'
2          Records     Integer     (4-bytes) number of records in the file
3          Fields     Integer     (4-bytes) number of fields per record
4          Record     Size         Integer (4-bytes) Fields*FieldSize (FieldSize is usually 4, but not always)
5          String     Block Size     Integer Size of the string block
*/

namespace VisualAI.Classes.World.Data
{
   enum DBCFileDefines
   {
       SIGNATURE_LENGTH = 4,
       FIELD_SIZE = 4,
       HEADER_SIZE = 20,
   }

   class DBCFileInfo
   {
       public Int32 pRecordCount;
       public Int32 pFieldCount;


       public Int32 pRecordSize;      //unused
       public Int32 pStringBlockSize; //unused

       public Int32 pStringBlockStart;

       public DBCFileInfo() { }
       public DBCFileInfo(Int32 pRowCount, Int32 pColumnCount)
       {
           this.pRecordCount = pRowCount;
           this.pFieldCount = pColumnCount;

           this.pStringBlockStart = (pRowCount * pColumnCount * (Int32)DBCFileDefines.FIELD_SIZE) + (Int32)DBCFileDefines.HEADER_SIZE +1;
       }
   }

   class DBCParser
   {
       private BinaryReader lBinaryReader;
       public String pFileName;
       public DBCFileInfo pDBCFileInfo = new DBCFileInfo();

       private Int32 lCurrentRecordId = -1;

       private void lLoadDBCFile()
       {
           this.lBinaryReader = new BinaryReader(File.Open(this.pFileName, FileMode.Open));

           UTF8Encoding lUTF8Encoder = new UTF8Encoding();
           //should read string WDBC if file is a valid DBC file
           String lSignature= lUTF8Encoder.GetString(this.lBinaryReader.ReadBytes((Int32)DBCFileDefines.FIELD_SIZE));
           if (lSignature != "WDBC")
               throw new Exception("DBCParser only read WDBC-Files!");
           //after this we are @byte12
           this.pDBCFileInfo = new DBCFileInfo(this.lBinaryReader.ReadInt32(), this.lBinaryReader.ReadInt32());

           //unused
           this.pDBCFileInfo.pRecordSize = this.lBinaryReader.ReadInt32();
           this.pDBCFileInfo.pStringBlockSize = this.lBinaryReader.ReadInt32();
       }

       private Boolean lCheckForStringTerminator(Byte pByte)
       {
           if ((Int32)pByte != 0)
               return false;
           else
               return true;
       }

       public DBCParser(String pFileName)
       {
           this.pFileName = pFileName;
           this.lLoadDBCFile();
       }

       #region misc functions
       public List<String> pGetColumnList(String pPrefix)
       {
           List<String> lTmpColumnList = new List<String>();
           for (int i = 0; i < this.pDBCFileInfo.pFieldCount; i++)
           {
               lTmpColumnList.Add(pPrefix + i);
           }
           return lTmpColumnList;
       }
       #endregion

       public Boolean pNextRecord()
       {
          while (this.lCurrentRecordId < this.pDBCFileInfo.pRecordCount-1)
          {
              this.lCurrentRecordId++; 
              return true;
           }
           return false;
       }

       #region Read-Based functions
       public Object pGetFieldValue(Int32 pFieldId)
       {
           if (pFieldId <= this.pDBCFileInfo.pFieldCount)
           {
               Int64 lCurrentOffset = (Int32)DBCFileDefines.HEADER_SIZE + (this.lCurrentRecordId * (Int32)DBCFileDefines.FIELD_SIZE * this.pDBCFileInfo.pFieldCount + pFieldId);
               this.lBinaryReader.BaseStream.Seek(lCurrentOffset, SeekOrigin.Begin);

               return this.lBinaryReader.Read();
           }
           else
               throw new Exception("Unknown Column!");
       }

       public Int32 pGetInt32(Int32 pFieldId)
       {
           return Convert.ToInt32(this.pGetFieldValue(pFieldId).ToString());
       }

       public String pGetStringValue(Int32 pFieldId)
       {
           // this.pStringBlockStart = (pRowCount * pColumnCount * (Int32)DBCFileDefines.FIELD_SIZE) + (Int32)DBCFileDefines.HEADER_SIZE + 1;
           Int64 lCurrentOffset = this.pDBCFileInfo.pStringBlockStart;

           this.lBinaryReader.BaseStream.Seek(lCurrentOffset, SeekOrigin.Begin);

           Int32 lStringDataLength = (Int32)(this.lBinaryReader.BaseStream.Length - (Int64)lCurrentOffset);
           Byte[] lStringData = this.lBinaryReader.ReadBytes(lStringDataLength);
           List<Byte> lBytes = new List<Byte>();
           Int32 lCurrentStringIT = 0;

           for (int i = 0; i < lStringData.Length; i++)
           {
               Byte lCurrentByte = lStringData[i];
               if (this.lCheckForStringTerminator(lCurrentByte))
               {
                   Byte[] lStrByteArr = lBytes.ToArray();
                   UnicodeEncoding lUTF8Encoder = new UnicodeEncoding();

                   if (lCurrentStringIT >= pFieldId)
                       return lUTF8Encoder.GetString(lStrByteArr);
                   else
                   {
                       lCurrentStringIT++;
                       lBytes.Clear();
                   }
               }
               else
               {
                   lBytes.Add(lCurrentByte);
               }
           }
           return "";
       }
       #endregion
       public void pClose()
       {
           this.lBinaryReader.Close();
       }
   }
}

Example reading data from Map.dbc:

(Pastebin)

public static List<MapData> pGetMapList()
       {
           List<MapData> lTmpMapDataList = new List<MapData>();
           DBCParser lDBCParser = new DBCParser(Settings.Instance.pDBCPath + "/Map.dbc");
           Int32 i = 0;

           while (lDBCParser.pNextRecord())
           {
               MapData lTmpMapData = new MapData();
               lTmpMapData.pID = lDBCParser.pGetInt32(0);
               lTmpMapData.pName = lDBCParser.pGetStringValue(i);

               lTmpMapDataList.Add(lTmpMapData);
               i++;
           }

           return lTmpMapDataList;

       }

Reading Strings work just "fine" but i can't assign them to rows because i just have many strings one after one without any logical "order" (in my eyes).

My code in "GetStringValue" is working but 100% no solution because it exponential take longer if you want to select a entry in a column/row (also a problem i can't assign string values to rows) in reason i always have to loop through all bytes, count existing strings and return the right one because i cant (?) set a pointer to specific string value.

Any suggestions?

[darkstalker]

you could just copy mangos code

[Mythli]

I have done some research on mangos code (DBCLoader) and figured out that you have string-reference fields which contain a Int32 value that point to a string with the following offset:

StrReadOffset = StringStartOffset + FieldValue

Problem now is that i have to detect if column is a string reference or not.

[Warlord123]

Look here http://github.com/LordJZ/spellwork_cs .

[Mythli]

If you look at DBCReader.cs you will see that there is no detection:

for (int r = 0; r < header.RecordsCount; ++r)
               {
                   uint key = reader.ReadUInt32();
                   reader.BaseStream.Position -= 4;

                   T T_entry = reader.ReadStruct<T>();

                   dict.Add(key, T_entry);
               }

He try to read a struct from DBCFile with predefined types, structure of Spell.dbc is defined in "Structure.cs" -

this is exactly that what i don't want.

[Khira]

Hi, you need to check the existence of the string with your int32 and if it is true for 100 lines for exemple, you can say that it is really a string.

[Konctantin]

Tip: Read first, all data in memory, and then work with the collections, or you will lose performance

[faramir118]

http://gist.github.com/463304 should be what you want.

[TOM_RUS]

You can't be 100% sure about field types. That's why predefined types is a best approach. The DBC file format designed that way and you can't do anything about that...

[faramir118]

I figured it would be a problem best handled heuristically.

If someone does use the code I posted, please let me know if you find cases where it is getting false positives or negatives. I can try to improve it.

Read this and this.