qsa

Lookup "screen" utility, should do the trick for you.

qsa, sorry, I am semi-newb

it only takes good will, never too late to learn something.

It's sizeof(dtMeshTile) * 2^21, which on x86 works out to about 120mB (without padding)

sizeof(dtMeshTile) = 8 byte, so sizeof(dtMeshTile) * 2^21 = 2^24byte, = 2^14kilobyte = 2^4megabyte

It actually allocated twice, half its size for m_tileLutSize. But I don't see my math mistake

Thank you, faramir118. Now Deadmines takes only 54 MBytes.

Thank you again.

qsa, problem solved, as I see, and it was NOT a Trinity problem

If that indeed solves the problem, my compliments.

But you can't blame me for being sceptical after seeing mile long memory leak log from mentioned project.

See if this helps (you need to re-extract maps, but it will skip all tiles so it won't take long)

diff --git a/contrib/mmap/src/MapBuilder.cpp b/contrib/mmap/src/MapBuilder.cpp
index 27d8eab..d53635c 100644
--- a/contrib/mmap/src/MapBuilder.cpp
+++ b/contrib/mmap/src/MapBuilder.cpp
@@ -646,7 +646,7 @@ namespace MMAP
        int tileBits = STATIC_TILE_BITS;
        int polyBits = STATIC_POLY_BITS;

-        int maxTiles = 1 << tileBits;
+        int maxTiles = tiles->size();
        int maxPolysPerTile = 1 << polyBits;

        /***          calculate bounds of map         ***/

While it will save some memory, It won't help you much.

STATIC_TILE_BITS = 21. so ..tiles*tile_ref_size 2^21 *8 = 2^24bytes = 16mega -- that is what reverser per map. See dtNavMesh::init()

If it will, well, I'll be pleasantly surprised

and my test is:

extract mmaps, start server, look at memory usage

shutdown server, delete "mmaps" folder

start server, look at memory usage

Better try the test I suggested.

Start the server login with character on map without mmaps. Measure memory used.

Teleport to another map without mmaps - measure memory used.

Teleport to map with mmaps - measure again.

Cheers

a new issue with navmesh in booty bay (almost same place)

X: -14435.139648 Y: 420.237518 Z: 8.844869 Map: 0

--bigBaseUnit true

http://filebeam.com/e2cb821599cfe493dd34cd67abb881de.jpg

--bigBaseUnit false

http://filebeam.com/89c92f181cf826d54df8816887b3caee.jpg

We are trying to apply discrete logic and methods to continuous world. Therefore, corner cases will always be possible.

The best we can hope for, is to minimize their occurrence. There's no "silver bullet" for this problem.

I don't think we can dynamically solve this particular issue.

I'd love to see someone proving me wrong.

Alternatively, you can test our parameters to find the best values which will give you best results for that particular tile.

1545 MB memory used on just-started server with new mmaps

after loading map Deadmines 1770 MB memory used

628 MB memory used on just-started server without new mmaps

after loading map Deadmines 661 MB memory used

deadmines with new mmaps = 225 MBytes

deadmines w\\o mmaps = 33 MBytes

sorry for my english, I'm almost sleeping

You see, I'll say it again ( just like I told you in PM) : memory log, or it didn't happen.

This report provides us with no valid information.

I can tell you as a fact, that on my test system, with one player loading 5~ mmap grids take at _very most_ 20mega.

Pretty easy to test. Start server, log into character on map with no mmap files, teleport to map with mmap - see the change.

I bet someone can provide 2nd opinion on this one.

Besides, knowing this report is for mmap port to Trinity, doen't make it too reliable, sorry.

As I said, report problems in code in our repository. We cannot maintain changes/bugs made by others, sorry.

By no means, I'm not saying there are no memory issues in mmap code, its just people cannot show there are.

Either show me valgrind log, OR test I mentioned on CLEAN mangos+empty database.

Best regards

how to detect memory leaks under *nix\\win64?

my developer machine has 5GB of memory and win7 64, and centos 5.5 on virtual machine

Post #592 in this topic can give you basic idea.

Activate fast-math on g3dlite, Detour, Recast and MoveMapGen (FloatingPointModel="2" in vcproj files or in project properties C/C++ | Code Generation | Floating Point Model).

Ambal beat me to it.

It may not be very safe. Some proper testing needed. Simply since, only god knows what assumptions Recast/g3dlite code takes.

Making 0.0f == 0.0f error, is well too easy to make. Even tho, none in right mind will write this code regardless of optimizations used.

If anyone can properly test it for mmaps and report, we'll be grateful.

navigation in Booty Bay:

Looks like there are consequences to such small voxel sizes... small, insignificant gaps become problems.

I'm really surprised to see this...

I think we can allow some particular problematic tiles to use different world metrics.

I'll push it, if I'll have luck making them coexist properly, link together.

Pushed something to git.

It should "solve" this little issue in bootybay and similar.

The good news, is that you can extract only affected tile using bigger metrics, and due to chosen sizes, we maintain same ratio, thus neighboring tiles ( which use normal metric) will link with this one.

The bad news are, it is not really generic solution. Having some tiles extracted one way, others another.

Guess we'll have to make list of those or something and script to extract them afterwards.

qsa:

before commit 4c154a764cba03560d0d176528d4ea3c06f6328a memory usage was 6.5Gb after 2d uptime.

That commit is from August 23, 2010, since then we had like... I don't want to count, how many changes. You better update.

navigation in Booty Bay:

Looks like there are consequences to such small voxel sizes... small, insignificant gaps become problems.

I'm really surprised to see this...

I think we can allow some particular problematic tiles to use different world metrics.

I'll push it, if I'll have luck making them coexist properly, link together.

...

Have fun.

PS: on Windows, properly setup your compilation flags and you get a -33% time to extract compared to what is on GIT currently (btw this is true for core too, release mode can have 20% speed increase by putting other compilation flags, but that's another story).

Thanks, I'll push it once I have time to test it on *nix machine, or maybe someone can do it.

This sure can be useful to some people.

About compiler flags, you had anything particular in mind? Since all I can think about are cpu dependent.

...

Online: 200

GridUnload: off

and crazy lags after .mmap stats command

mmaps directory size is 1.4 GB

9 hours ain't too bad with mmaps and mtmaps (which you obviously use). Cores aren't too busy either.

About memory usage : since you don't unload grids, it actually may be expected sizes. Don't really know, have nothing to compare it with. Its not like we have formula for "proper" memory usage for mangos. online_count*X*grid_loaded*Y*...

Still, 7g +9g virtual looks just too much.

Again, if you imply memory leaks, please do provide us a trace. You can find example few pages back in this topic.

"crazy lag" is nothing unexpected from "mmap stats" command, the more grids you have loaded, the slower it goes. keep in mind, that it is debug command, thus should not be used on anything but test machine.

Thanks for testing and feedback. The better quality of reports we get, the faster problems will be solved.

Cheers people

This not bugs but not implemented features. I think you not understand difference bugs from missing features.

I'm a developer by trade mate, I know what the difference is. My point is that they're not implemented. Regardless of what you want to call them, they've been around since about 3.0 and they're not working. When I replied to Kich0 in my original post, I was saying that functionality in this case wasn't in place. The features are missing. I never mentioned anything about bugs, so please refrain from judging my ability to understand simple differences when it's actually in my day-to-day work.

You can always use your skills for something good, instea ...

This topic makes kitty sad.

Noticed couple problems with VMAPs which affect mmaps.

Nothing new, they just became visible due to mmap related movement.

Case 1:

http://i55.tinypic.com/29atcus.jpg

Just check the diff between the actual ground level and level claimed by VMAP.

Case 2:

".go 3624.958 2034.717 75.228 571" and soot the bear next to you. Now observe the Z coordinate GetClosePoint() gives you as your target position. Needless to say it is far from actual ground level -> no path can be generated.

While not directly related to mmap code, we may want to do something about it. Or at least properly test and fill bug report Those things sure cause some nasty problems to mmap generated movement.

Another issue is the following: http://i53.tinypic.com/1087f43.jpg

Only partial water data and partial terrain is there, something to do with how we generate the terrain.

Not very problematic in most cases, but, those things below water line shouldn't be there.

We really need to think how underwater terrain can be handled.

forbiddenn : you have problem with merging. It compiles and links just fine on vs9. PS: some actual info can be nice. But hey! I must be a prophet, guessing those.

scargo: That 2nd report directly related to first case. Thanks.

PS: pushed some things that may need some real testing.

He starts in a place where there isn't a path to players.

Technically, he turns into a bat so he should fly somehow...

Brings up a point (one mentioned before) that some mobs probably should cheat, like bosses and guards.

We most defiantly should not allow cheating by default.

Just like Schmoozerd said, script should handle this case, since obviously it is scripted event.

Script can "cheat" by using something as simple as :

target->GetMotionMaster()->MovePoint(0, x,y, z, false);

Last optional parameter allows skipping pathfinding in movement.

I was thinking to add in our "canFly" check not only check for static inhabit type, but also auras, but then, I figured, that there are no real fly auras. Only things like mounts and levitate, but they are not really related to actual pathfinding.

A question, for example, how about Jeklik on the wall at map 309, x -12289, y -1382, z 146 (ZG) ?

What about him?

I was thinking more in direction of bash script that does something as simple as :

MoveMapGen 0&

MoveMapGen 1&

MoveMapGen 530&

MoveMapGen 571&

MoveMapGen --skipContinents true&

You can get reporting, distribution based on number of CPU's, etc. But this basic thing will do just fine for a start.

Pushed new tile format, some additional changes, cleanup.

You will have to extract all maps fresh. This time it will take longer.

So, it can be good idea, if you have multi-core systems, to parallelize things by simply extracting different maps in different instances on the extractor simultaneously.

If I remember right, someone was about to make bash+batch script for us. Something nice, which detects number of CPUs and balances load of extraction accordingly. Guess I was imagining ... ehh, I'll get there one day.

I also added check, to prevent generating maps without any actual geometry, those flat tiles.

Saves some disc space. Their chances to be loaded on runtime are slim, but still.

I didn't see this part in diff

Just a little, very basic check. Around line 1021 (MapBuilder.cpp)

We may want some real logic there. Since most, if not all instance maps are filled with junk tiles.

For example, places like map 33 - two actual tiles out from ~22.

General formula can be really hard to come by, but it can be good idea. Saving us alot of time.

If we manage to check it before extracting anything.

I think you should use uint32, uint16, int32... everywhere like in the core.

That is problematic, because Recast and Detour would need a lot of modifications

Unfortunately, both of you are right. Both Faramir and Neo.

We do have to think about some solution.

Easiest, will be just converting Recast+Detour from native sizes. But, it will make future syncs, updates really painful.

But, currently I don't see any other option. Beside doing nothing, that is ...

Maybe we can suggest it for commit for Mikko's repo.

One thing I noticed today: we have like 3 different sets of definitions for things like MMAP_VERSION, MmapTileHeader, etc. Those should be in some separate file included by all involved.

So, I guess, this makes it 2 TODO items. Beside the usual testing.

Then, its time to think, what do we want next.

Not too many people run netBSD nowdays. Don't be surprised why none takes part in this WiP with you.

Someone have to be the first, lead the way, making the adjustments needed.

Best regards.

Did some more testings and tweaking, and this is what came out.

http://pastie.org/1347195

BASE_UNIT_DIM lowered to 0.266666f. This gives us much better results. All those corners, passes and doorways are much nicer now. Came with a price, longer extraction times, but meh ... you don't switch client version too often.

Some other parameters were tweaked a little bit too.

I also added check, to prevent generating maps without any actual geometry, those flat tiles.

Saves some disc space. Their chances to be loaded on runtime are slim, but still.

Some little changes here and there, etc.

I was thinking about making BASE_UNIT_DIM parameter, something like hiResHeight we have, but then, I came to conclusion it will only create mayhem. People having different version of maps, making different reports, etc.

So, my next though, following same logic, was removing hiResHeight option altogether.

Any objections?

If no problems will be found, I'll push it soon enough.

One more though: Noticed that in some portions of the extractor code we use "int" and in others+core "uint32". Can't it cause problems? Lets say, on 64bit system int is 8 bytes, while uint32 is still 4. Can this sort of thning happen with ACE defines? it sure looks so. Do we really care?

Best regards.

Some of the doors on map 34 look like they are only 1 voxel: http://img689.imageshack.us/i/tight.png/

There's no problem with those passes. Just like I explained in previous post.

We don't care how thin they are. Just try that place in game, you'll see that the movement there is just fine.

It's the detail mesh - it won't affect pathing, but it will affect movement with findSmoothPath.

You can fix it be decreasing maxSimplificationError. Experiment with that number a bit... 1.9 seems ok

maxSimplificationError is too good to lower it.

I wasn't able to understand why they are generated by Recast.

But, I did 2nd best thing : hack-fixed them.

Add anywhere before running dtCreateNavMeshData and after meshes are merged.

   // little ugly hack to prevent unexpected polygon angles
   // TODO: find why they are generated in the first place.
       for (int k = 1; k < iv.polyMeshDetail->nverts; ++k)
       {
           float* v1 = &iv.polyMeshDetail->verts[k*3];
               float* v0 = &iv.polyMeshDetail->verts[(k-1)*3];
           if(fabsf(v1[1]- v0[1]) > 300.0f)
               v1[1] = v0[1];
       }

Will have to do for now

https://gist.github.com/726329

edit: that cellSize comes really close to making doors impassable on the maps I've tried. Wouldn't be surprised if some are blocked.

I guess you meant BASE_UNIT_DIM(0.53333333f).

In order passage not to be walkable, it has to be less than 1.6y and some bad luck to be optimized out.. hm... we might actually have such places, that should be walkable. I'm 99% sure there's minimal walkable size defined in wow. Can be really nice finding it out.

We can always change BASE_UNIT_DIM to 0.333333f - but.. it will double extraction time and other parameters will have to be recalculated accordingly. Namely maxSimplificationError and walkableRadius. Since we still want to keep about the same ration of 0.5y borders, so 2*BASE_UNIT_DIM will have to be used, making actual borders even bigger. Maybe we can use 0.266666f , or something ... needs testing - but in general above applies to it too.

Not sure if it is worth it. Unless we can find some real places where passages are cut due to this metrics.

Barely walkable = good for us, since what really moves on our mesh is point, with zero radius. As long as there's connectivity - we're good, no matter how thin the passage is.

PS: thanks for merging those changes.

EDIT: I did noticed something strange, something I never seen before.

http://i51.tinypic.com/dpyclc.jpg

That screwed polygon is totally fine on polymap but something went wrong on converting to mesh.

Has no affect on pathfinding, but still annoying.

Any ideas?

As continuation to my previous post.

Below attached patch that changes once again the way we store our data.

Basically, we're back having 1:1 relationship between ingame tiles and mmap tiles we extract.

The reason we moved from this design, was problem with the way some polygons were shaped. Long and thin.

Splitting tiles to subtiles solved this problem but introduced couple very unpleasant corner cases, as explained in my previous post.

Below patch tries different approach to exactly same problem. However, it uses 1:1 (ingame:mmap tiles).

Basically, it generates polygons in such a way, that they are shaped just as we want them to be, without using subtiles.

Most modifications are done to the extractor. The core is basically back to old code we used to have loading tiles.

All metrics moved to native unit sizes. This solves all truncation problems. It also makes things much clearer.

See code comments for more info.

Some other generation parameters were tweaked too. I had time to play with those, to truly understand their effect, so some of them modified to give better results.

maxSimplificationError : 2.5-3.0 gives by far the best result in preserving our geometry ( assuming ~0.5 base unit : vertex size) while removing almost all jagged edges. This greatly reduces the number of overall polygons with no side effects. Those small polies are not walkable in 99% of the cases anyway, they just slow us down.

walkableClimb: this one must be less than our agent height. We really, really don't want to allow units climbing over things their height, it just looks wrong.

tileSize: about 20yads is more or less optimal here, but we can push it up for smaller mmap sizes.

EDIT: The bigger number is, the less polygons we have. Up to some limit. But we get more and more "bad" polygons.

This can be nice thing to have as parameter. We can have different sizes for different maps. They can co-exist just fine.

Overall compared to current solution (multiple tiles):

The good:

- Solve mentioned above problematic cases.

- Allow safely using any size as cell size.

- Smaller mmap sizes -> less memory usage.

- Spiral stairs are finally solved, this time for good.

- Much simpler core code. We don't need any extra logic. Basically what we had before.

- Solves visuals at debugging tools.

- Smoother maps, almost no jagged corners.

The bad:

- We won't have compactHeightfield/ContourSet/HeightField in debug info (or atlest I haven't found a way adding those .. yet ... not a big loss)

The ugly:

- It is based on revision 4da57201204, so some useful parts that was added with multi-tile will have to be re-added. Like mmap version, etc.

- You have to extract all mmaps again

- Lots of testing needed.

They say one picture worth 100 words , so... http://i53.tinypic.com/2rot5cw.jpg http://i55.tinypic.com/1zn3j1t.jpg

This should give basic idea what it is all about.

Patch against latest (beb30c28604f11ef27cb) EDIT: http://pastie.org/1342639

Updated a patch a little bit to include some changes we need from multi-tile patch. Shouldn't have real functionality change.

Feedback is always welcome.

Go and test!

Found this while looking into the new detours: http://img140.imageshack.us/gal.php?g=demob.png

How did our code generate a different poly path?

You can see on the picture, that the first waypoint not exactly where you stand.

I can only guess its due to some problem with our new format.

One other thing I noticed, is path's gets "trapped" in polygons while trying to pass between neighboring tiles.

You can observe it nicely on map 33. try making path from -209.369, 2242.350, 79.850 to -205.268, 2213.742, 79.764. or from -177.160, 2212.520, 80.096 to -223.471, 2196.717, 79.764.

You get maximal length path which trapped inside connection between tiles.

After spending half day, trying to understand why we cannot make custom sized tiles and how do the actual linking works. I realized, we took wrong approach.

Auto-linking between neighboring tiles is really, really primitive and will totally fail (path wise) when there's "fragmented" borders.

You can see this phenomenon on some tile connections : generate path that should have been straight between two neighboring tiles : you sometimes (rather often

) will get zig-zag at tile connection.

Started working on different way to generate polygons. Something that doesn't involve having small tiles.

I'll post it soon enough. So, if you plan making some huge changes, better wait a little bit.

Best regards.

Personally I still like qsa's idea to write one(or three) classes for loot, but afaik a multi-heritage OOP design throws a few problems (C-casts everywhere) - and I didn't understand why he didn't implement these classes as member variables of the creatures, corpses, or whatever

As long as you do not cast to lootable, you are safe. As I see it, there's no principle problem using multiple inheritance here.

There reasons I don't use it as member variable is that it fits much better to OOP idea. Object (GO/creature/whatever) "is a" lootable. It differs from Object "contains/ has a" lootable.

Besides, it saves you access : obj->loot->something to obj-> something

You can always say "object has a loot" but, meh...

You are right in general, that its exactly the same thing. The only difference is approach. The way you look at things. Bottom line, C++ is level 1 language, therefore, classes and their relationships exist only in source code.

You can call this patch "proof of concept". It offers no functionality over other idea, beside arranging things differently.

I totally forgot about recent changes in item loot. Guess patch has to be revised a little bit. Don't have the time right now, but I'll get to it eventually.

Best regards.

Looks lovely, thanks - I'll check it out as soon as I have time.

And we can probably compensate with some of the optimizations from Mikko (visibility stuff from CrowdManager.cpp).

Can you point to something we haven't discussed before? I couldn't find anything we can use, which we haven't yet.

unrelated: I made few tests playing with agent size to see the real ingame impact, and I ahve to say, that having it around 0.5 makes wonders on creatures scratching corners.

We defiantly should use it as default.

About bigger models, we we already spoke about having multiple meshes. But for now, 0.5 will work just lovely.

This is how I imagined this is done.

Thanks for the explanation.

You assume there's no R/W shared data between the threads, therefore it is safe.

My concern, as explained in previous post, is that there are.

What I'm really saying here, is that determining not safe data relations between the threads is 99% of things that have to be done to make it multi-threaded. Things may appear working, but, once in a while you get data corruption - those things are next to impossible to debug.

oh, well, if it works, it works.

Thanks once again.

First of all, I have to admit, that I'm not fully understand the threading mechanism used by different thread patches. Never had the opportunity to check it.

If I'm asking something silly, please do explain. This will sure help me (and others) understand the model.

This said, I have a very basic question about this, and similar patches:

How can it be thread safe without actually locking shared objects?

For example, movement packets (per thread per map like in this patch) manipulate player objects. What assures us that same object cannot be manipulated from other thread simultaneously? For example, by chat handler.

I really fail to see how being in world ("belonging" to some map) makes it thread safe.

I know this is bad example, due to granularity, but lets assume our granularity is player object and not its fields. Since, if you count on fact, that different threads change different parts, you have to be 100% sure you checked ALL the fields involved. This is, far, far from trivial. If they don't, we don't need this patch at all.

So, the way I see it, we can get undefined values due to R/W, W/R, W/W, basically everything but R/R.

Assuming store/loads are atomic don't help either (they are not).

Thanks in advance.