MMaps Redux

[Edder]

Whats with the liquid data?

Is it necessary too (underwater pathfinding)?

[faramir118]

Liquid data is needed for preserving inhabitType

If creature can't swim, we shouldn't build a path that takes it through water.

If creature can't walk, we shouldn't build a path that takes it on land.

@qsa

With underwater pathfinding code you pushed, how can we prevent swimming-only creatures from leaving the water?

Flying sharks are cool, but it can be overdone

edit: I should clarify... how do we prevent swimming-only creatures from going from pool1 to pool2:

|-------| g |-------|
|       | r |       |
| pool1 | o | pool2 |
|       | u |       |
|-------| n |-------|
         d

[faramir118]

http://valhalla-team.com/WoWScrnShot_110610_205934.jpg

http://valhalla-team.com/WoWScrnShot_110610_214725.jpg

http://valhalla-team.com/WoWScrnShot_110610_215311.jpg

These are caused by long, thin triangles in the navmesh.

I will work on a solution. I have a feeling it will be a major change to the .mmtile file format...

[qsa]

@qsa

With underwater pathfinding code you pushed, how can we prevent swimming-only creatures from leaving the water?

While I haven't test this particular case, I'm pretty sure it will be fine. Since there's no "water only" path between the pools. So it will work just like "land only" trying to pass a river -> evading.

You asking since you there's a bug somewhere, or?

EDIT: actually, when I think about it little bit more, there can be some problematic corner cases. Like when both are in deep water, too far from any polygons. But i think in general it should be pretty much reasonable for now.

By the way, same applies to flying creatures. There are cases when flying creatures will fly directly into the walls. You really have to work hard for this (almost impossible to make while normally playing), but its still doable.

caused by long, thin triangles in the navmesh.

I will work on a solution. I have a feeling it will be a major change to the .mmtile file format...

There is maximum triangle edge parameter. Wont it work playing with this value a little bit?

I mean, it does make a difference while you generate those under recastDemo.

There's also things like the max. number of edges per poly which can be useful.

KAPATEJIb / Wojta : thank you very much guys for testing.

PS: it seems that result vary a little bit from one system to another. Maybe we have x64 compatibility issue?

For example I cannot confirm a fair portion of those errors on my test system 32b.

[faramir118]

While I haven't test this particular case, I'm pretty sure it will be fine. Since there's no "water only" path between the pools. So it will work just like "land only" trying to pass a river -> evading.

You asking since you there's a bug somewhere, or?

I didn't spend time looking for a location to test this

I agree that it's a rare corner case.

There is maximum triangle edge parameter. Wont it work playing with this value a little bit?

I mean, it does make a difference while you generate those under recastDemo.

There's also things like the max. number of edges per poly which can be useful.

Even when I edit the recast source so that maxEdgeLen affects all contours (not just tile and area boundaries), the navmesh still has a lot of long, thin polygons.

Changing maxVertsPerPoly doesn't seem to solve this either, just creates different number of tessellated shapes in the same general patern.

rcBuildRegions seems to have the bigest impact - it places a large region which must later be broken into smaller subregions in order to create convex polygons. Unfortunately, there isn't a way to limit region size directly.

This isn't working well for us, so I'm going to reduce the size of tiles - then recast will always create reasonably size regions.

It may affect performance (more polygons), but the visual result that we have now is unacceptable.

The file format for mmtile will be incompatible with the current implementation, also.

PS: it seems that result vary a little bit from one system to another. Maybe we have x64 compatibility issue?

I have a x64 win box I can compare against, but all of my dev so far is x86 win.

Mikko recently posted some very interesting things on his blog. I think we should see if anything from this post or this post are applicable to our environment. They both deal with path-corridor patching, similar to what we want to achieve with the suffix optimization.

[Iowahc]

hy there,

Today I merged my mangos with ahbot and playerbot with mmaps.

But now, my mangos tells me that the maps are out of date. worked yesterday

Do I have to recompile ad.exe from mmaps branch and extract all again (maps, dbc, vmaps, mmaps) ?

Thx in advance

[faramir118]

Do I have to recompile ad.exe from mmaps branch and extract all again (maps, dbc, vmaps, mmaps) ?

Just maps, dbc, and mmaps.

[KAPATEJIb]

dbc

why?

maybe my problem appears because i'm using "ad.exe -f 0" to extracting maps ?

[Iowahc]

But with the recompiled ad.exe from mmaps, am I right?

[qsa]

Mikko recently posted some very interesting things on his blog. I think we should see if anything from this post or this post are applicable to our environment. They both deal with path-corridor patching, similar to what we want to achieve with the suffix optimization.

Beside "Visibility Optimizations" idea, the other things were already implemented in my first patch.

He said that he implements suffix optimization in crowd tool. I'll check his way when I get back home, but it can't be too different.

About "Visibility Optimizations" idea : he doesn't implement it at all. While it sounds good, the cost of checking if we can cut the corners may cost more than we are ready to pay for paths with few less corners. It is something we have to implement, then test - only then we can see the effects. But in general I don't think we should mind paths with couple extra corners.

I have a x64 win box I can compare against, but all of my dev so far is x86 win.

Pretty much the same. But identifying those problems ( if any ) can be tricky.

Main question is : do we really have amy related problems?

What I did notice while checking the svn, is that there were alot of x64 related changes. Need to check the exact nature of those changes. We may need to update ...

Which brings me to next question : do we have changes made into recast package beside changes in recastDemo? I sure hope not.

[KAPATEJIb]

Fresh crash dumps with https://github.com/faramir118/mangos/commit/38faa3dae7328159086d5d5a3f9bff1ab87c94b1 and my _not clean_ sources ^^

http://paste2.org/p/1077185

http://paste2.org/p/1077186

i'm using x86 Fedora

[KiriX]

dbc

why?

maybe my problem appears because i'm using "ad.exe -f 0" to extracting maps ?

No... I don't use key -f 0 but have error with immediately evade...

[qsa]

http://paste2.org/p/1077185

Any chance you know how much memory used while it crashed? since it appears that malloc() failed to allocate additional memory.

http://paste2.org/p/1077186

Again, it is memory allocation error.

I have no idea why this can happen. Either there are memory leaks somewhere, or your system just running out of memory.

[Iowahc]

Hy, I have succesfully extracted map files and vmap files from my 335a client (build 12340), but when creating mmaps, or starting the server, the map files seem to be outdated, any chance to see which version the map files are?

[qsa]

Hy, I have succesfully extracted map files and vmap files from my 335a client (build 12340), but when creating mmaps, or starting the server, the map files seem to be outdated, any chance to see which version the map files are?

You have to build and use AD extractor from our source, then extracting maps/mmaps.

[Iowahc]

Well I built it from this source, doesnt worked

[faramir118]

What I did notice while checking the svn, is that there were alot of x64 related changes. Need to check the exact nature of those changes. We may need to update ...

Which brings me to next question : do we have changes made into recast package beside changes in recastDemo? I sure hope not.

From what I could tell, they were all simply widening the dtPolyRef type.

That is something I am interested in though, regardless if it fixes a problem or not. It would alleviate some concerns about too many tiles...

I only changed one part of recast, the rcSpan height members' widths.

[KAPATEJIb]

ok, all problems with mobs aggro seems solved by using hiRes mmaps. using this revision https://github.com/faramir118/mangos/commit/793bc30ad10f8c5c4842c13ec779b43ac1945510

[KAPATEJIb]

http://paste2.org/p/1077185

Any chance you know how much memory used while it crashed? since it appears that malloc() failed to allocate additional memory.

as i understand crash dump file size shows memory using at program termination. All crash dumps (i have many crashes at "dep/recastnavigation/Detour/Source/DetourNode.cpp:39") has 3000-3100 mb file size

as i already said i'm using Fedora x86, so i have memory limit

Here the crash dumps:

http://paste2.org/p/1079018 - as is see it crashed at same game zone as in old crash dump http://paste2.org/p/1077185

http://paste2.org/p/1079031

http://paste2.org/p/1079032

as i see i have another crash happened after 10 minutes (!) with different dump (at grid load) with malloc() and dump has 3000 mb file size too

i feel i have a memory leak somewhere ... but it start to happen after updating from 10683 to 10690 (with updating from mmap sources ofc). I'm also thinking about this problem as problem of mmaps, because some peoples said they have problem with leaks at their own core too (they have totally different sources... more clean than mine and etc).

[blueboy]

Well I built it from this source, doesnt worked

I had exactly the same problem. Do as qsa suggests

qsa wrote:

You have to build and use AD extractor from our source, then extracting maps/mmaps.

Using the precompiled AD.exe provided doesn't work. If possible compile both vmapExtractor3.exe & vmap_assembler.exe too. It might not be necessary to compile these but it only takes a few minutes, and you then know that all exec files are compatible. They all compile without issue, and cerstainly solved the issue for me. If you use gcc on *nix like me to develop, you can download a trial version of "Microsoft Visual C++ Express Edition" if you do not have a full version.

If you still don't have any joy, contact me

Hope this helps

[KAPATEJIb]

Well I built it from this source, doesnt worked

I had exactly the same problem. Do as qsa suggests

qsa wrote:

You have to build and use AD extractor from our source, then extracting maps/mmaps.

Using the precompiled AD.exe provided doesn't work. If possible compile both vmapExtractor3.exe & vmap_assembler.exe too. They all compile without issue. You can download a trial version of "Microsoft Visual C++ Express Edition" if you do not have a full version.

Hope this helps

i can upload working extractors for you if need

http://filekeeper.org/download/insider/Mangos335/mmap_mod/add/extractor_10698%2B.7z

[blueboy]

Hi KAPATEJIb

KAPATEJIb wrote:i can upload working extractors for you if need

http://filekeeper.org/download/insider/Mangos335/mmap_mod/add/extractor_10698%2B.7z

I was going to do the same, if Iowahc still had problems

Cheers

[qsa]

i feel i have a memory leak somewhere ... but it start to happen after updating from 10683 to 10690 (with updating from mmap sources ofc). I'm also thinking about this problem as problem of mmaps, because some peoples said they have problem with leaks at their own core too (they have totally different sources... more clean than mine and etc).

We haven't made any changes that can affect memory usage between 10683 and 10690.

Actually the svn is at 10677, but I don't think mangos did anything too extreme too.

So if there's actual leak in our code, its much older.

I tried to run our latest under profiler (purify), and there were no definite memory leaks found (or any at all related to our code).

If anyone can run under valgrind with TBB disabled! (important), and actually get leak trace, I'll be able to fix it. But prior to that, I cannot even confirm its there

PS: you have grid unloading disabled, right? try enabling it. Maybe your memory actually just fills up.

PSS: anyone have similar issues?

[KAPATEJIb]

PS: you have grid unloading disabled, right?

no.

Maybe i can test without TBB under valgrind if you teach me how to do it and this will work without additional lags at live server

[qsa]

PS: you have grid unloading disabled, right?

no.

Maybe i can test without TBB under valgrind if you teach me how to do it and this will work without additional lags at live server

Well, dropping TBB under FC wont affect the performance too much. But running under valgrind MOST defiantly will

But if you run something under 50 testers, it will be reasonable.

I'll make a little guide here, maybe someone will find it useful.

1) install valgrind package.

2) disable TBB. You can do it by adding "#define USE_STANDARD_MALLOC" at top of src\\framework\\Policies\\MemoryManagement.cpp or just commenting the entire file.

The reason for this, is that TBB takes over memory management so the valgrind's sandbox can't hook properly.

EDIT: obviously you have to recompile.

3) start mangosd under valgrind.

"screen -A -m -d -L -S worldserver valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes $srv_dir/bin/mangos-worldd"

This assumes you are using screen utility. The output will be written into screen.X file (usually located in your home dir, or mangos dir)

You can also run it directly by "valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes $srv_dir/bin/mangos-worldd > output.txt"

4) Start the server -> do staff - > shutdown properly -> see the results.

The logs are more or less self explainable. You can always look for detailed explanation at google.

5) Enjoy

EDIT: you can also use it as profiling tool - can be pretty nice to get those logs too. To see the frequency of path calls and their relative exec times over long time periods.

"valgrind --tool=callgrind $srv_dir/bin/mangos-worldd"