MMaps Redux

[faramir118]

@ KAPATEJIb

Can you apply this patch, so that we can see why it's not loading those files?

[qsa]

faramir118 : tell me, are you sure assigning those bits to tiles, don't hurt polygons?

I'm asking, since .. well, just look at those crash dumps.

pathPolys = {4213792, 4214256, 4214257, 4214227, 1967148500, 4213273,

4213280, 4212993, 4205097, 4205099, 4205095, 39, 0, 20, 4204946, 2485034720,

2, 2485033640, 1, 0, 141375826, 173306232, 4, 2485037879, 2485037879,

2941076064, 83541108, 2485033544, 83150438, 2485212808, 2485037879,

2485037879, 4, 484, 0, 83, 15601689, 2485034828, 0, 2485037879, 1, 0, 15, ....

All those polygon references are overflows. It even dies at SIGFPE(arithmetic exception).

Looks like something is affected, and not in a good way.

Giving 12bit to the tiles making us have only 10 bits for the poly. Limiting the number of polys per time to 1024, which obviously, we have more, much more.

[KiriX]

Crash like in 617 post every 10-20 min make me tune off mmaps

[KAPATEJIb]

Crash like in 617 post every 10-20 min make me tune off mmaps

same for me, i was turned mmaps off too for fixing it

[faramir118]

We need minimum 10 bits for tile, but in theory need 12

We need at least 14 for the polys, maybe more. Of the tiles I currently have built, the largest poly count is over 10000 (5325235.mmtile)

Salt is 10 bits (checked in dtNavMesh::init, implicitly set in MapBuilder::buildNavMesh, line 635).

10 + 14 + 10 > 32

We need to reduce the size of salt. Short term fix.

Then we can work on updating to a more recent version of recast, so we can take advantage of the wider-typed dtPolyRef.

edit: this math isn't 100% sound, since 532 only has a few tiles and thus only needs 4 bits for tile id. but we still have cases where large maps, like 0 or 1, have poly counts over 4096 on single tiles.

[Iowahc]

Well still have the problems, trying the extractor now

Thank so much

[faramir118]

Finally found a spot with an overflowed polygon id:

.go -10982.000000 -1965.010010 39.733799 0 2.290960

Doesn't crash, just overflows and travels to the wrong polygon. Tested by summoning a vanity pet.

I fixed it, going to push changes.

You will need to re-run generator to fix the bits (with this patch again.)

[faramir118]

Also pushed some code cleanup, and changes to the mmap GM commands.

Read commit notes for details.

[Nick Taylor]

Ok, I am getting a segmentation fault after I redid my mmaps (did them with the high resolution option, and generated the continent maps), and I get the following error a couple seconds after logging in:

getLiquidStatus(): vmap liquid level: 612.213562 ground: 613.243164 type: 1
getLiquidStatus(): vmap liquid level: 612.213562 ground: 613.243164 type: 1
getLiquidStatus(): vmap liquid level: 612.213562 ground: 613.243164 type: 1
getLiquidStatus(): vmap liquid level: 612.213562 ground: 613.243164 type: 1
getLiquidStatus(): vmap liquid level: 612.213562 ground: 613.243164 type: 1
Segmentation fault

I am not certain that the getLiquidStatus() thing is part of the error, but it seems to show up each time it crashes.

[faramir118]

Do you have a crash dump, or at least the location of your player?

What commit version are you running?

[Nick Taylor]

I am in Dalaran when it happens.

I am running 925318e60faddecccf0e revision for mangos (10717) and revision c6232a6bca3647fbaf02 of mmaps.

I have to get the crashdump, so I will post it when I have it.

[KAPATEJIb]

if someone still can't extract mmaps from compiled application, try this http://filekeeper.org/download/insider/Mangos335/mmap_mod/add/extractor_10717%2B.7z (full set of extractors: for dbc,maps,vmaps,mmaps)

You will need to re-run generator to fix the bits (with this patch again.)

ok, i will try

EDIT:

seems it works. Server doesn't crashes now.

[qsa]

...

We need to reduce the size of salt. Short term fix.

Then we can work on updating to a more recent version of recast, so we can take advantage of the wider-typed dtPolyRef.

...

I think the real solution should be doing something like:

Define : map, cell, tile, polygon.

While :

• map - ingame map ( just like we have right now )
  
• cell - subregion of the map ( what is called tile now )
  
• tile - subregion of cell
  
• polygon - same as now.
  

When gridXY is loaded, we load cell, with all its tiles. Instead of making gridXY=single tile.

This will solve both overflows due too too many polygons per tile and too many tiles per region and the problem of long triangular polygons that screw some of our paths.

Problems:

• extra overhead - since tiles have their own data, pretty much neglectable, since the real number of polygons will not change, but still ...
  
• pathfinding between neighboring cells. detour do not support this ( as far as I know). We will have to change findPath/findStraghtpath algorithms to support this.
  Atleast we'll have to make them return the status of last poly in path. We have to know if location is out of cell. Then we can calculate two (or more) subpuths and paste them together.
  This (*) is the real problem.
  

Those are just some thoughts on the subject.

[faramir118]

Are you saying that each mangos tile should be a separate navmesh? That is a very large hurdle... I vote no! (navmesh-per-tile is how the project started out, so returning to that would be going backwards )

Fortunately, we can increase the number of tiles without making things more difficult for ourselves.

We just have to make sure dtPolyRef can address all possible polygons. e33d285 solved this problem for now.

With more tiles, we may need dtPolyRef to be a wider type - more recent versions of recastnavigation make this change easier.

What I was thinking about...

mangos (and wow in general):

map -> 1..64x64 tiles -> 8x8 cells

detour:

navmesh -> 1..64x64 mmtile -> 8x8 dtTile (-> 1..1024 polygon)

Our mmtile would become a logical grouping of 8x8 dtTiles. This means a theoretical maximum of 262144 dtTiles, which would require 18 bits. After giving 4 bits to salt, that leaves 10 bits for polygons. It should work without widening dtPolyRef, but the only way to know for certain is to make the changes and see if any dtTile's polycount is over 1024.

(again, these numbers aren't fixed - they depend on the actual number of tiles in the map)

Of course, we can choose any arbitrary number of dtTiles per mangos tile. The system is rather flexible. It might even work around some outstanding bugs, like the spiral stairways.

[qsa]

Are you saying that each mangos tile should be a separate navmesh? That is a very large hurdle... I vote no! (navmesh-per-tile is how the project started out, so returning to that would be going backwards )

no, not exactly.

But reading your post made me understand we will not avoid widening dtPolyRef.

The thing is that every polygon in same navmesh must have unique ID. Since the rough number of polygons per map do not change, it is around ~2^30. So, splitting those to different tiles wont help us. It will only solve the long poly issue.

We also need pretty big salt to minimize the chance for wrongly selecting polys.

EDIT: actually, we don't use the feature salt was created for at all, for our loading/unloading. Maybe we can cut it out giving us 4 more bits.

Fortunately, we can increase the number of tiles without making things more difficult for ourselves.

We just have to make sure dtPolyRef can address all possible polygons. e33d285 solved this problem for now.

With more tiles, we may need dtPolyRef to be a wider type - more recent versions of recastnavigation make this change easier.

We can always make dtPolyRef 64bit. I can do the nessesery changes, if you provide me with diff with all our changes from clean recast.

It might even work around some outstanding bugs, like the spiral stairways.

http://code.google.com/p/recastnavigation/issues/detail?id=102

EDIT: we need to think first, if we need more than 32bits for dtPolyRef. Cutting salt to 0 or 1bit, will hopefully give some room for polygon numbers we need.

[faramir118]

We also need pretty big salt to minimize the chance for wrongly selecting polys.

EDIT: actually, we don't use the feature salt was created for at all, for our loading/unloading. Maybe we can cut it out giving us 4 more bits.

Whatever the case with salt, we could cheat and just never unload tiles until the map dtor is called :rolleyes:

We can always make dtPolyRef 64bit. I can do the nessesery changes, if you provide me with diff with all our changes from clean recast.

https://github.com/faramir118/recastnavigation

There are two branches

• master is a clone of SVN
  
• mangos is the version we are on, with our changes (missing some things, like makefiles)
  

I had planned to do a complete merge to the newest version. A simple fastforward will break a lot generator and mangos code - better to do small merges and update our code as we go.

Or you can merge just r235, r240, and r242 to start. These will not fix it completely though - for example, 1) dtNavMesh::init still assumes 32bit and 2) we are advised to use a 64bit hash

http://code.google.com/p/recastnavigation/issues/detail?id=102

I know

http://groups.google.com/group/recastnavigation/browse_thread/thread/c52d951a4a00f90a

Mikko's suggestion, monotone regions, aren't possible with our huge tiles. But since we're going to use smaller tiles soon anyway...

[Wojta]

Thanks for that crash fix, so far, so good for me

[KiriX]

***

I fixed it, going to push changes.

You will need to re-run generator to fix the bits (with this patch again.)

Thx for fix! =)

Maybe will be usefull add new generator's key for generate only .mmaps instead of this patch

[KAPATEJIb]

will be usefull add new generator's key for generate only .mmaps instead of this patch

and the key to skip existed tiles at extraction

leaks still isn't fixed... http://paste2.org/p/1084058 (3015 mbyte dump)

but now this crash happens more less (1 crash per day)

and another crash http://paste2.org/p/1084105 but crash dump has only 2700 mbytes

[qsa]

leaks still isn't fixed... http://paste2.org/p/1084058 (3015 mbyte dump)

but now this crash happens more less (1 crash per day)

and another crash http://paste2.org/p/1084105 but crash dump has only 2700 mbytes

Some of them fixed ... you cant get it all at once

Can you post your partial server log?

What I'm really looking for, is errors that might have been generated by mmap code.

PS: doubt 2nd crash has anything to do with mmap code.

[KAPATEJIb]

What I'm really looking for, is errors that might have been generated by mmap code.

after last mmap code update i don't see any errors with mmaps in logs. But i know some places where mmaps doesn't works properly and mobs in these zones tries to jump into the air at aggro (even if you trying to attack them from land) and then evades. http://filebeam.com/9c40d57a6c794e296ce5abc3c565a155.jpg

[qsa]

Merged and pushed both new Mangos and new Recast (r252).

Somewhat tested, but, as always, there may be more than enough well hidden problems.

Next stage is converting for full dtPolyRef-64bit support.

[KAPATEJIb]

dump http://paste2.org/p/1084668 using this https://github.com/faramir118/mangos/commit/38af7beeae9109521508cfc5f6b1c9c539a82e82

[qsa]

dump http://paste2.org/p/1084668 using this https://github.com/faramir118/mangos/commit/38af7beeae9109521508cfc5f6b1c9c539a82e82

Another memory leak...

I pushed something interesting. Killing one more piggy.

Hehe, now you'll have to test all the crashes introduced with 252 switch

[pyza]

After Update Recast to r252. and Changes to fit new Recast API. mobs won't aggro and evade immediately.

Mmaps extracted with hi res paremeter.

Console log:

2010-11-12 12:39:51 ERROR:5712530.mmtile was built with Detour v4, expected v6

2010-11-12 12:39:52 ERROR:5712532.mmtile was built with Detour v4, expected v6

2010-11-12 12:39:52 ERROR:5712431.mmtile was built with Detour v4, expected v6

2010-11-12 12:42:31 ERROR:0005032.mmtile was built with Detour v4, expected v6

2010-11-12 12:42:32 ERROR:0004932.mmtile was built with Detour v4, expected v6