[Dev] remove DBC files and make life easily for everyone

[antz]

Hi All,

I have been looking at this for a while now.

Basically Mangos only uses the DBC (and DB2) files as a lookup for certain information (spells etc)

My Idea is during the extraction process there is a step that reads these files and dumps the information into a database (DBC maybe).

This potentially has several advantages:

- Remove the DBC handling code to reduce the memory footprint of Mangos slightly

- Remove the need to have the DBC files on the server

- Allows 3rd Party Applications access to the lookup fields, such as races / classes, displayid's etc.

- Allows the Database team to be able to do full id checking/cross referencing (rather than having to start Mangos and check the error log)

To this end I currently looking at the the DBC loading routines with a view to lifting the required code out of Mangos and put it into its own tool and including a SQL script creation step onto it.

Any thoughts ?

[lillecarl]

I just want to say: HEY YOU STOLE MY IDEA! Haha no but i wanted to do this as well, but stuff seems to be a bit too much for me at the moment. But i really would appreciate someone doing this. Then dbc fields could be understood by "averanges" easier then now! Hope someone csn help you, good luck!

[michalpolko]

Small digression: I have an idea in a "TODO" state to store in mysql DB all the data that is now stored in dbc. Primary idea for me is to easily modify the data (and even add custom columns to make things better/easier) to make things like spells or vehicles work as intended instead of thinking "it looks like this flag is not what we all thought it was" and this way remove a lot of hardcoded stuff from code.

On topic: why is the data stored in dbc anyway (and not in mysql)?

[cyanidel337]

just thinking doesnt this idea then put all the stress/strain on the mysql server instead? whats the point in that?

[Mark_]

The mysql server should be better optimized to do data manipulation than the Mangos core code. And even if it isn't, having the possibility of mysql running on it's own core or even a separate machine should be a net gain.

[antz]

On topic: why is the data stored in dbc anyway (and not in mysql)?

The dbc files are files extracted from the client.

[antz]

just thinking doesnt this idea then put all the stress/strain on the mysql server instead? whats the point in that?

It should in theory make no difference.

The DBC file information is loaded into memory during the server startup. All we would be doing is changing this to read some tables into memory instead of opening files and extracting the information.

[shlainn]

Uhm, guys, actually the quest tracker does a fair amount of dbc extraction. It is quite easy, really. Check https://github.com/shlainn/tracker/blob/develop/extract.php for details. So, third party tools / db verification is already happening.

MaNGOS memory footprint will not be reduced by this, as AFAIK core loads everything from SQL into memory anyway.

As far as having dbc files on the server vs. having dbc data in your sql tables is concerned, I also do not see any advantage nor disadvantage from either.

(as an aside, I am not sure what can be _learned_ from this - MaNGOS University wise... )

[michalpolko]

On topic: why is the data stored in dbc anyway (and not in mysql)?

The dbc files are files extracted from the client.

DBC files are files created after extracting data from MPQ files - those are client files.

[antz]

Exactly..

What I was suggesting was doing away with these files and putting the infomation into the database and getting the server to read the database rather than the files.

[shlainn]

Current procedure

1) extract dbcs from MPQ

2) copy dbcs into dbc folder

3) core reads data from DBC files on startup

Suggested procedure

1) extract dbcs from MPQ to SQL format

2) insert SQL into database

3) core reads data from SQL database on startup

Difference for the core?

[antz]

Difference for the core?

For the core, some changes to the load routines, but should be minimal

one of the advantages of the core reading the infomation from the database is that the impact of the client dbc field changes will be minimised

- instead the changes can be made to the extractor and the DB guys can check the changes and ensure everything links up nicely

anyone who knows their way around the DB but not a coder can easily generate queries to check the state of the update changes.

as a bonus, any 3rd Party tools can also access the DB for the required information rather than them all having to modify their code each time the DBC fields change (unless blizz decide the remove key fields)

[shlainn]

As I said - the extractor is in the repo I linked. It even directly inserts chosen bits of data into the database. Expanding it is trivial. I also have an DBC viewer written in php - I can share that as well if you want.

[amaru]

Hi All,

I have been looking at this for a while now.

Basically Mangos only uses the DBC (and DB2) files as a lookup for certain information (spells etc)

My Idea is during the extraction process there is a step that reads these files and dumps the information into a database (DBC maybe).

This potentially has several advantages:

- Remove the DBC handling code to reduce the memory footprint of Mangos slightly

- Remove the need to have the DBC files on the server

- Allows 3rd Party Applications access to the lookup fields, such as races / classes, displayid's etc.

- Allows the Database team to be able to do full id checking/cross referencing (rather than having to start Mangos and check the error log)

To this end I currently looking at the the DBC loading routines with a view to lifting the required code out of Mangos and put it into its own tool and including a SQL script creation step onto it.

Any thoughts ?

Database team can always read dbc directly to get what they need.

The only advantage here might be reducing the direct interaction between mangos emulator and client data, but as long as there is public test server and client patcher, it is no longer important

[lillecarl]

Dont be so conservative! This could help many!

[antz]

As I said - the extractor is in the repo I linked. It even directly inserts chosen bits of data into the database. Expanding it is trivial. I also have an DBC viewer written in php - I can share that as well if you want.

Thanks for that, Using that info I am currently looking into modifying the standard extraction tool 'ad.exe' to produce the sql files

[lillecarl]

As I said - the extractor is in the repo I linked. It even directly inserts chosen bits of data into the database. Expanding it is trivial. I also have an DBC viewer written in php - I can share that as well if you want.

Thanks for that, Using that info I am currently looking into modifying the standard extraction tool 'ad.exe' to produce the sql files

Awesome! How much modification will have to be done to mangos?

[antz]

should be fairly trivial.

DBC files are loaded by the DBCLoader functions into a memory Datastore, Mangos currently has functions to Load Database tables into Datastores as well.

It should be a case of extending the Database Loader routine to read the additional tables into memory and removing the DBCLoad stuff

[antz]

(as an aside, I am not sure what can be _learned_ from this - MaNGOS University wise... )

I'm hopefully going to learn (as well as anyone following the project) some basic principles in doing this:-

- Updating legacy code (i'm not that experienced in C++ programming)

- Implementing new functionality to existing code

- Designing a way to allow flexible updating without using hardcoded values

If you don't mind shlainn, I may call on your assistance to ensure that the end results are the same - plus possibly racking your brains if I get stuck with any of the DBC stuff.

[shlainn]

Antz wrote:

If you don't mind shlainn, I may call on your assistance to ensure that the end results are the same - plus possibly racking your brains if I get stuck with any of the DBC stuff.

I do not mind. This is a learning community after all But DBC is REALLY easy to understand. May I recommend the excellent writeup at http://www.pxr.dk/wowdev/wiki/index.php?title=DBC ?

[Unkle Nuke]

Apologies for any confusion due to the small edit to the title.

Although it is a minor issue, the MaNGOS University label is intended for topics that offer instruction and education for those who are new to developing and wanting to learn. Such posts can consist of questions by MaNGOS students or guides and tutorials from more experienced developers.

This thread is obviously discussing a development project, rather than introducing novices to the concepts presented in a manner that is more like a tutorial.

Thank you for your patience. You may now resume your coding awesomosity! :cool:

[antz]

Thought i'd pass on some good news...

The author of WowParser (a DBC to CSV converter) has given me access to the sourcecode.

This has made the coding challenge that bit easier

[desteny_]

nice idea, it will make DBC modding much easier

but it would be nice to make a documantation witch columns could be modified without modifying the client DBC data.

a other interesting idea would be multi-lingual dbc data, that means we extract the main data from EN client, and could add language dependet database columns or tables for data extracted from alternativ clients (only a idea, but don't even know if this is needed, or if every dbc data will be translated by client...)

[antz]

I intend to document the files/tables as i go along. A lot of the DBC column have already been documented so i'm intending on build on that work to build up the DB reference data/information.

I'm not sure how much the modding will work, since the original dbc files will still be read by the client.

[antz]

if your interested the latest version of my AD.exe modifications is located at :-

git://github.com/billy1arm/server.git