MMaps Redux

[Undergarun]

That's pretty good news.

How about we make a deal?

I will merge with leatest mangos and all its changes, and you in return, will make a detailed report about mmap memory/cpu consumption as function of number of players. With mmaps and without.

How does that sound?

I can provide you screenshots of htop w/out mmaps but with mtmaps at 4 threads load is continuously jumping from one thread to another (CPU with 24 threads) so is quite difficult to get accurate information. But if is enought for you, deal.

[qsa]

I can provide you screenshots of htop w/out mmaps but with mtmaps at 4 threads load is continuously jumping from one thread to another (CPU with 24 threads) so is quite difficult to get accurate information. But if is enought for you, deal.

The common case will do for us, thanks.

Due to recent changes (spline movement) it will take me bit longer to fulfill my side of the agreement.

All movement logic will have to be made from scratch. The good news are that the result, as I see it, will be more centralized, clear code.

[SilverIce]

Due to recent changes (spline movement) it will take me bit longer to fulfill my side of the agreement.

All movement logic will have to be made from scratch. The good news are that the result, as I see it, will be more centralized, clear code.

I think now movement part of mmap code can be simplified alot or maybe removed at all

[Undergarun]

Due to recent changes (spline movement) it will take me bit longer to fulfill my side of the agreement.

All movement logic will have to be made from scratch. The good news are that the result, as I see it, will be more centralized, clear code.

Perfect, i'll do tests when you finish to provide performance data from latest mmaps version.

[qsa]

Merged with master.

Added mmaps to RandomMovementGenerator.

All movement code remade fresh, pile of changes all over the place.

Keep your eyes open for new issues.

@Undergarun: Waiting for your side of the deal

I think now movement part of mmap code can be simplified alot or maybe removed at all

I didn't manage to remove mmap integration from movement generators completely, but it sure became simpler in some places, trivial in others.

I also had to make couple tinny modification to your spline code.

Feel free to comment on general design.

Cheers.

[Undergarun]

Tests will be done as soon as i can fix stability issues on my realms. Thanks for all work made!

[Undergarun]

With low chance: http://pastebin.com/6JBf0JxK

[qsa]

With low chance: http://pastebin.com/6JBf0JxK

Well, if that keeps happening, we'll have to add some asserts. Otherwise, without catching it under debugger there's no chance I can solve it.

You have no idea how to reproduce, do you? Had to ask ...

If you have more of those, please post them all, even if the trace looks the same.

On, not so related note:

Below is little modification turning our A* into weighted A*.

Two simple modification:

1) Weight up the heuristic function: in our case most of the time distance plays the role of perfect heuristics, thus expending 'only' nodes directly on the path to target - we can't go wrong here.

2) Do not return nodes from closed set back to open.

Both of the changes will do the following two things:

1) Expend less nodes. In other words, make path generator take less time and memory - should be about 30% less - no idea to now much overall it translates.

2) Provide non optimal results. While A* ensures optimal result (optimal by our metric - distance), this variant, while still ensuring completeness, will provide suboptimal solutions.

I'd love to see the performance difference under load. (thanks in advance)

diff --git a/dep/recastnavigation/Detour/Source/DetourNavMeshQuery.cpp b/dep/recastnavigation/Detour/Source/DetourNavMeshQuery.cpp
index 44aafe2..be66667 100644
--- a/dep/recastnavigation/Detour/Source/DetourNavMeshQuery.cpp
+++ b/dep/recastnavigation/Detour/Source/DetourNavMeshQuery.cpp
@@ -68,7 +68,7 @@ inline float dtQueryFilter::getCost(const float* pa, const float* pb,
}
#endif    

-static const float H_SCALE = 0.999f; // Search heuristic scale.
+static const float H_SCALE = 2.0f; // Search heuristic scale.


dtNavMeshQuery* dtAllocNavMeshQuery()
@@ -695,7 +695,7 @@ dtStatus dtNavMeshQuery::findPath(dtPolyRef startRef, dtPolyRef endRef,
            if ((neighbourNode->flags & DT_NODE_OPEN) && total >= neighbourNode->total)
                continue;
            // The node is already visited and process, and the new result is worse, skip.
-            if ((neighbourNode->flags & DT_NODE_CLOSED) && total >= neighbourNode->total)
+            if ((neighbourNode->flags & DT_NODE_CLOSED)/* && total >= neighbourNode->total*/)
                continue;

            // Add or update the node.

Best regards

[Detrol]

This might be a weird question, but makes little sense to me.

Since the whole project is merged into a mangos branch instead of a patch file.

How are we supposed to use it with the current mangos branch? ( my case TBC )

Thanks.

[faramir118]

Learn to use git... here's a very simple (but not very proper) way to add mmaps to your current branch in your local repository:

(sorry about the code block, need to get the css for this fixed...)

git remote add faramir118 git://github.com/faramir118/mangos
git pull faramir118 mmaps_rewrite

If you use mangos zero (client 1.12), replace mmaps_rewrite with mmaps_zero

If you use mangos one (client 2.4), replace mmaps_rewrite with mmaps_one

A little disclaimer... these branches are a bit behind. Again, we welcome any mangos zero/one users to step forward and maintain these branches!

Unfortunately, further development of these two branches is frozen until the new spline code is backported. I had to do a bit of asm debugging to fix our linear spline for older clients when I backported mmaps, so I assume that doing the same to SilverIce's code may need extra research.

[KuraiNeko]

I can upload to github MZero z1765 version vith all changes untill [11720] Implement spline movement subsystem - was applied.

Also regulary update MZ + MMAP to actual revision changes.

PS.

But as i says previously in MZ exist bug with tile unload in more places

[Wojta]

I tried to read most of this thread and I didnt see this one noticed: http://valhalla-team.com/snap4.png http://valhalla-team.com/WoWScrnShot_072911_162911.jpg - the whole wmo is misplaced in mmap mesh, but vmaps are all right. Same thing happened for bridge on the other side of EoS.

But I have slightly modified and most likely little outdated version of mmap patch, can someone test this please?

EDIT: It seems like it is just rotated by 180 or the rotation wrong, because bridge is too low on one side and too high on the other: http://valhalla-team.com/snap5.png http://valhalla-team.com/snap6.png

EDIT2: Now when I look more closely, I think the left and right bridge is just swapped, lol Oo

EDIT3: No, they are not, more like rotated by 180 degrees.

EDIT4: By the way, the bridges are on two tiles at once: for example the one between mage tower and fel reaver ruins is on both tile 566 28 28 and 566 27 28

[qsa]

I tried to read most of this thread and I didnt see this one noticed: http://valhalla-team.com/snap4.png http://valhalla-team.com/WoWScrnShot_072911_162911.jpg - the whole wmo is misplaced in mmap mesh, but vmaps are all right. Same thing happened for bridge on the other side of EoS.

But I have slightly modified and most likely little outdated version of mmap patch, can someone test this please?

EDIT: It seems like it is just rotated by 180 or the rotation wrong, because bridge is too low on one side and too high on the other: http://valhalla-team.com/snap5.png http://valhalla-team.com/snap6.png

EDIT2: Now when I look more closely, I think the left and right bridge is just swapped, lol Oo

EDIT3: No, they are not, more like rotated by 180 degrees.

EDIT4: By the way, the bridges are on two tiles at once: for example the one between mage tower and fel reaver ruins is on both tile 566 28 28 and 566 27 28

The mmaps generated are accurate, according to input mesh - look at it, the bridges are little bit misplaced.

There's error in our TerrainBuilder::loadVMap() logic, or maybe model data (or entire map angle)? It looks like pitch angle is off by ~3-5deg.

[Wojta]

The mmaps generated are accurate, according to input mesh - look at it, the bridges are little bit misplaced.

There's error in our TerrainBuilder::loadVMap() logic, or maybe model data (or entire map angle)? It looks like pitch angle is off by ~3-5deg.

If so, then not only in one axis - the bridge near mage tower in-game and from RecastDemo: http://valhalla-team.com/WoWScrnShot_072911_200016.jpg http://valhalla-team.com/snap7.png

[qsa]

If so, then not only in one axis - the bridge near mage tower in-game and from RecastDemo: http://valhalla-team.com/WoWScrnShot_072911_200016.jpg http://valhalla-team.com/snap7.png

Should be fixed in latest commit.

All maps must be re-extracted fresh (delete mmap dir's content at the extractor, otherwise it will just skip those).

Thanks for report. This should fix fair pile of issues.

@Undergarun: how's that report is going? We're getting old

[Wojta]

Thank you. Kinda surprised nobody noticed this Oo

[qsa]

Thank you. Kinda surprised nobody noticed this Oo

Most models do not have pitch/roll values at all, or they are set to low values.

PS: some testing needed.

[Wojta]

WMOs seems to be positioned correctly, but walkable areas are generated little bit off the map (or map is bit misplaced), not sure if its because i only took "+ Change in WMO rotation matrix. Should fix couple issues. " and not "+ MMAP Extractor: ..." commit. Will do some more testing tomorrow, need to sleep, good night.

[Undergarun]

Hi QSA, thanks for

https://github.com/faramir118/mangos/commit/30cd0ee2bd1d23ba0952d528e720dfe9808e4a83

https://github.com/faramir118/mangos/commit/2e70c84b625b6c956d912bc4cf9ffc34bd246bc6

I haven't forgotten our deal, i'm just waiting the work of rsa on multithreading locking for make fully multithreaded performance tests. I still use mmaps and notice no new crashdumps related to, so consider mmaps as stable (not fully free bugs but stable at least).

Cheers.

[lillecarl]

Hi QSA, thanks for

https://github.com/faramir118/mangos/commit/30cd0ee2bd1d23ba0952d528e720dfe9808e4a83

https://github.com/faramir118/mangos/commit/2e70c84b625b6c956d912bc4cf9ffc34bd246bc6

I haven't forgotten our deal, i'm just waiting the work of rsa on multithreading locking for make fully multithreaded performance tests. I still use mmaps and notice no new crashdumps related to, so consider mmaps as stable (not fully free bugs but stable at least).

Cheers.

This might not fit at all, but even the trinity mmaps is stable (working on a trinity server:() So mangos mmaps should really be stable aswell, but yeah bug free is awhile to go Im not sure if it does matter at all but i mean for how long hasnt vmaps been developed and still not "bug free"?

- LilleCarl

[Wojta]

http://valhalla-team.com/snap8.png wait..whaat?

I generated all from scratch using extractors from your mmap branch Oo

EDIT: never mind, i must have done something wrong, going to re-extract all mmaps.

EDIT2: So far, so good.

[qsa]

...

I haven't forgotten our deal, i'm just waiting the work of rsa on multithreading locking for make fully multithreaded performance tests. I still use mmaps and notice no new crashdumps related to, so consider mmaps as stable (not fully free bugs but stable at least).

Cheers.

The reason I'm so impatient is since I think its about the time this thread move to "under review" section.

Without proper test results, I cannot do it with clear conscience.

The reason I want it done, is not that I think its bug-free or perfect. The reason is to speed up development.

Proper threading model can be nice, since mmaps do scale well - but it is not "a must".

Unrelated:

There's tinny issue with terrain I noticed couple of days ago:

In some places the liquid ends just before the shore line- as seen in the picture:

http://i51.tinypic.com/dpiu4x.jpg

This causes really bad mesh generation in that area, since that shoreline is unpassable.

Was looking around both in map extractor and in our code - wasn't able to find logical error. It might be problem with the actual provided terrain. But then we'll have to complicate for it somehow.

Anyone noticed this king of issues before? faramir118?

Best regards.

[faramir118]

I noticed that, and know that even after this commit there are lots of places with lingering issues.

I thrashed a bit with the liquid vertex placement and tessellation, and obviously got it wrong...

• A picture from in-game: http://img829.imageshack.us/img829/4180/gamey.png
  Taken from below, the terrain is culled.
  
• Now one from recast, the same location: http://img812.imageshack.us/img812/2462/mesh.png
  Taken from above, with terrain polygons excluded from the build process.
  

I circled the area of interest - in recast, notice that all of the edge-quads' bottom-left triangle is missing.

This, combined with the above issue, probably means someone needs to redo the liquid part

Thanks all, for continued efforts!

[aming51]

I found another bug on Undergarun's server : if hunter stands on edge of floor/stage（e.g. the towers in north Western Plaguelands and stage of Lethtendris in Dire Maul East) which connects to ground with stairs/slope, the aggo mobs will not go up stairs/slope but go directly to the position on the ground under hunter and just watch hunter without any motion(except range spell). In this case, hunter can easily kill the boss or mobs and even hunter moved back to floor/stage center from edge, the mobs still can not go up stairs/slope but follow postion on the ground under the floor where hunter is and evade after few seconds.

Regards,

AMING51

Same problem as I said above. Hunter/Warlock/range spells can easily kill the boss when I stand on edge of stage/floor. Any news about this?

[faramir118]

There are some problems in the extractor (ad.exe) that might be contributing to the issues qsa and I posted.

In the mmaps_rewrite version of contrib/extractor/System.cpp, the loop @ line 775 is wrong.

It iterates over quads (128x128), but acts on vertices

• sets the top-left vertex of a quad to invalid height, which affects all quads on that corner.
  Conceptually, I can see this causing both the problems qsa and I reported.
  
• ignores the right- and bottom-most edge of vertices
  This most likely isn't causing any problems, but it could in theory.
  

Offhand, I think the easiest solution is to throw this loop away, write both liquid_height and liquid_show to the .map file, and go from there. Unfortunately that is a bit of a rewrite for the core as well.

Thoughts?