[Map/Grid] New Map/Terrain Management System

[XTZGZoReX]

Hi,

This is the result of 3 days of non-stop work, consisting of countless hours of pulling my hair trying to understand the grid system . It implements a new terrain management system in MaNGOS which will have several benefits, particularly for future multithreading.

Somewhat full list of changes:

• * Added new TerrainMgr with reference-counted TerrainData objects (as well as reference counting for each GridMap allocated within it). It will serve as the proxy for obtaining terrain information, and can also do terrain calculations when you don't have a Map object available.
  * Added support for pre-loading all maps. Yes, this means that you people with loads of RAM will benefit, as you can avoid map file I/O. The current memory usage of both maps and VMaps fully loaded in memory is approximately 2.5 GB. Loading on-demand is still supported. Loading all map data takes around 5 minutes on my i7 960.
  * Rewrote MapManager almost completely. It now only has the functions it *really* needs to have, though some more cleanup will also happen in the future.
  * Removed the MapInstanced class and all "base map" functionality completely. Now that TerrainMgr can give us terrain information without loading a Map object, that's how it should be done. This also gives us the advantage that multiple instances of the same map ID can be updated concurrently in the future.
  * Made several functions related to area/zone ID retrieval from map files static. These functions never needed a Map object to begin with, and thus unnecessarily created Map/MapInstanced objects in memory - just to use DBC files.
  * Lots of random cleanups, some typo fixes, and nicer code because of the above .
  

What I need most of all right now is some live testing on servers with players. I've done lots of local tests in different scenarios, and everything looked to be good, but of course, one single player isn't enough to determine the stability of this patch.

Link to full patch: http://paste2.org/p/743853

Comments, suggestions, bug reports, etc are all welcome. Be advised, this patch should NOT be run on a server that uses some multithreaded maps patch, as it was not written with this in mind - yet.

Also, thanks to everyone who's helped me in some way in the process (I won't give specifics, in case I forget someone.....)!

[cyrex]

On Ubuntu:

./patches/Terrain.patch:3558: trailing whitespace.

After removing the trailing it started compiling fine but then:

CXX AccountMgr.o

In file included from ../../../src/game/Map.h:36,

from ../../../src/game/BattleGround.h:24,

from ../../../src/game/Group.h:24,

from ../../../src/game/Player.h:30,

from ../../../src/game/ObjectAccessor.h:32,

from ../../../src/game/AccountMgr.cpp:21:

../../../src/game/TerrainMgr.h:39: error: use of enum ‘ZLiquidStatus’ without previous declaration

../../../src/game/TerrainMgr.h:79: error: ‘ZLiquidStatus’ does not name a type

In file included from ../../../src/game/Map.h:36,

from ../../../src/game/BattleGround.h:24,

from ../../../src/game/Group.h:24,

from ../../../src/game/Player.h:30,

from ../../../src/game/ObjectAccessor.h:32,

from ../../../src/game/AccountMgr.cpp:21:

../../../src/game/TerrainMgr.h:143: error: ‘ZLiquidStatus’ does not name a type

In file included from ../../../src/game/BattleGround.h:24,

from ../../../src/game/Group.h:24,

from ../../../src/game/Player.h:30,

from ../../../src/game/ObjectAccessor.h:32,

from ../../../src/game/AccountMgr.cpp:21:

../../../src/game/Map.h: In member function ‘ZLiquidStatus Map::GetLiquidStatus(float, float, float, uint8, LiquidData*)’:

../../../src/game/Map.h:418: error: ‘struct TerrainData’ has no member named ‘GetLiquidStatus’

make[2]: *** [AccountMgr.o] Error 1

using Core 9637 + SD2

[XTZGZoReX]

Right, forgot to mention... I didn't test the Linux build. If anyone could check that out for me, that'd be great. If not, I'll get around to it... soon.

[tehavatar]

Hi XTZGZoReX, this looks quite impressive. I dont have time to test this patch, so do you mind answering some questions?:

Does this patch have any performance advantage over the current MapMgr ?

How much of advantage will it be to be able to completely cache map data ?

Does your system compliment the MMAPS project?

Thanks again, I will make time to test your developments soon!

[XTZGZoReX]

Does this patch have any performance advantage over the current MapMgr ?

In MaNGOS as it is now, there's no significant gain in this - the actual gain will be clear when MaNGOS gets map multithreading.

How much of advantage will it be to be able to completely cache map data ?

Depending on player count, a huge advantage! Whenever a grid is unloaded, map data is freed, and thus needs to be reloaded the next time that grid is loaded. In fact, map file I/O happens very often on high population servers. You would gain a lot of performance from not having to do this at runtime.

Does your system compliment the MMAPS project?

While it has nothing to do with movement pathing as such, it certainly should make the job easier for whoever wants to implement it.

[The_Game_Master]

Thank you very mutch for your work!

One question tough, as i understand the maps/Vmaps will be loaded at start-up every time (optional feature) and take about 5 minutes. Wouldn't that be a very big disavantage if the process will crash every 5 minutes? Or in the future this can be avoided by creating independent threads?

[XTZGZoReX]

One question tough, as i understand the maps/Vmaps will be loaded at start-up every time (optional feature) and take about 5 minutes. Wouldn't that be a very big disavantage if the process will crash every 5 minutes? Or in the future this can be avoided by creating independent threads?

Why would the process crash every 5 minutes in the first place? Then something is clearly wrong elsewhere... But yes, if that really would be the case, it would be a disadvantage. But really, it's up to you. On-demand loading is still there, if you really want to use it. Anyway, we can't always take possible crashes into account everywhere in the core.

[cyrex]

One last question here. How much LESS memory will this help mangos use up. I have a 1GB Server and i have to literally kill mangos every 24 hours so it does not crash later.

[kero99]

Really impresive work XTZGZoReX, i would test your patch in 3K server but without mtmaps support my CPU will die =(. Some posibility to do compatible actual ACE Mtmaps with your patch?

Thank and best regards

[FragFrog]

Whenever a grid is unloaded, map data is freed, and thus needs to be reloaded the next time that grid is loaded.

Doesn't GridUnload = 0 have the same effect though? And if that is the case - does applying this patch mean the same problems for long uptimes? Specifically, with GridUnload=0 I keep getting reports about mines not being minable, creatures not respawning, etc. after a day or two, three of uptime. Setting GridUnload=1 effectively seems to fix that, but if this patch keeps the same data in memory we might get the same trouble?

[XTZGZoReX]

Doesn't GridUnload = 0 have the same effect though? And if that is the case - does applying this patch mean the same problems for long uptimes? Specifically, with GridUnload=0 I keep getting reports about mines not being minable, creatures not respawning, etc. after a day or two, three of uptime. Setting GridUnload=1 effectively seems to fix that, but if this patch keeps the same data in memory we might get the same trouble?

Grids consist of two things - objects and map data (or rather, they did, before this patch); either of which can keep a grid loaded. GridUnload in config simply sets an explicit lock on the grid (which is never unlocked, thus, keeping the grid constantly active).

I'm not sure what would cause the problems you're facing; those are spawn-related and not really related to this patch as such, so...

Anyway, with my patch, map data is handled separately. It is not directly tied to the grid system at all - in fact, you could just go ahead and do something like this anywhere you wanted (almost; of course not inside the terrain code itself):

...
TerrainData* data = sTerrainMgr.GetData(571);
float height = data->GetHeight(100.0f, 200.0f, 300.0f);
sTerrainMgr.DisposeData(data);
...

Basically meaning that grid unloading is not directly related to map data anymore - memory management of map data is handled solely by the TerrainMgr, nothing else. Of course, bugs can happen if the code that uses TerrainMgr doesn't properly dispose of map data when done with it (i.e. reduce the reference count), but I haven't encountered such issues so far.

One last question here. How much LESS memory will this help mangos use up. I have a 1GB Server and i have to literally kill mangos every 24 hours so it does not crash later.

Strictly speaking, there's no difference from before. From a developer's point of view, though, the memory management is much nicer now.

[XTZGZoReX]

New patch for current master (untested, but has necessary changes, so should apply): http://paste2.org/p/745173

[XTZGZoReX]

New patch: http://paste2.org/p/745250

Some random cleanups + re-organization of GridMap and related classes. Also should fix Linux build issues.

[cyrex]

Confirmed. Patch works perfect on linux and compile finished correctly. Now what should i test or could test with a population between 20 and 30 players?

[nanounico]

For me right now, in small server, ~400-600 no crash!

[XTZGZoReX]

Confirmed. Patch works perfect on linux and compile finished correctly. Now what should i test or could test with a population between 20 and 30 players?

Sure thing. In fact, anything above 1 player will do .

[KAPATEJIb]

does it fixes a crash http://paste2.org/p/745979 (happens without patch) when using pre-loaded maps? Because my server is often fails at map file reading. So if server load maps only once at server start this crash can disappear?

[Forgotten]

does it fixes a crash http://paste2.org/p/745979 (happens without patch) when using pre-loaded maps? Because my server is often fails at map file reading. So if server load maps only once at server start this crash can disappear?

not sure. This is terrain data management system, but it does nothing with vmaps

[KAPATEJIb]

not sure. This is terrain data management system, but it does nothing with vmaps

I mean this

Added support for pre-loading all maps. Yes, this means that you people with loads of RAM will benefit, as you can avoid map file I/O. The current memory usage of both maps and VMaps fully loaded in memory is approximately 2.5 GB.

can help with my crash problem from map/vmap file reading at runtime?

[XTZGZoReX]

can help with my crash problem from map/vmap file reading at runtime?

Nope. It only uses the functions that the VMap API provides. But, I'll have Lynx3d look at that crash...

[Lynx3d]

@KAPATEJIb: That looks a lot like "out of memory" exception throw to me...you are running 32bit linux?

Would help to know the memory usage at time of crash...

If that is the case, mangos probably won't even start if you try to preload everything.

But there's hope for oniyou, i'm just finishing up a rewrite of Vmaps. The new structure references spawned objects instead of clng it each spawn. Also the geometry storage was horrible...

I'm currently down to ~410MB total (from 1.8GB), and that's without sacrificing precision by storing coordinates with only 16bit precision.

Memory consumption (with everything loaded) should be roughly equal to disk space in both cases btw...

[xeross155]

These kinds of patches really make the future of MaNGOS look a lot brighter, I will add it to my server on the next patching run.

[XTZGZoReX]

New patch: http://paste2.org/p/746088

Mostly just cleanups (suggestions from vladimir) and random small fixes.

[XTZGZoReX]

And yet again, new patch against latest master: http://paste2.org/p/746655

This one has some bug fixes for grid reference counting. It *should* work now...

[XTZGZoReX]

And again, a new one: http://paste2.org/p/748603

Made for latest master, and fixes a crash at BG updating.