mtmaps (OpenMP)

[Auntie Mangos]

Using two threading libs at the same time is always bound to cause some sort of trouble.

Mnn i never used together :huh: , I assume that it isn't any doubt about you nerver have to use it together... sorry again if my english is a lillte crapy.

To GriffonHeart , in my experience, right now, there isn't should be any noticiable difference about one or the other (I tested openmp mtmaps but long time ago when raczman post it)...but altnernative you can try this patch http://getmangos.eu/community/showpost.php?p=90117&postcount=49 posted by balrok or infinty patch...

Balrok's, should not crash in ... our crash.. even with multithreaded player updates (this should bring us ,a lot of less load in main thread), but when I tested map threads start in 6X% of cpu ussage by thread making mangos consume 500% of cpu (8 cores) with 1 character ON :confused:...But for balrok it works fine...so meaby it's a problem with my OS, so if you test it please let me know how it works for you.

Infinty's patch, i will test it a soon i got free time to do it.

[Auntie Mangos]

Could you please add some hints about tweaking DB performance?

I've been playing around a bit, but I have the feeling, that there is still some tweak possible ; )

thanks,

Termitt

http://getmangos.eu/community/showthread.php?12411-High-Performance-Tips

=)

[Auntie Mangos]

On a windows system, how do I update ACE?

http://getmangos.eu/community/showthread.php?12077-ACE-Netcode-5-7-5-rev-88569&p=102744&viewfull=1#post102744

[Auntie Mangos]

Perhaps that can help some peeps to compile properly

http://getmangos.eu/community/showthread.php?12771-How-to-use-custom-ACE-and-TBB-libs-without-screwing-the-one-already-installed-on-your

[Auntie Mangos]

Hey i get error

How to fix , i have installed tbb correclty i think

Exactly the same problem..

Install TBB libs ...

Here is a howto: http://getmangos.eu/community/showthread.php?12771-How-to-use-custom-ACE-and-TBB-libs-without-screwing-the-one-already-installed-on-your

[Auntie Mangos]

Seems to be more and more incompatible with each new revision, anyone know a working Revision where the Patch can be at least manually still with some modifications applied?

I tested it here with one of the later revisions but got a few issues http://getmangos.eu/community/showthread.php?8896-mtmaps-%28OpenMP%29&p=106912&viewfull=1#post106912

[Auntie Mangos]

kero patch - is only improvement communication MaNGOS with database (for mtmaps).

mtmaps -

without this patch your server can have limit players to 300-400 (?).

With this patch your mangos working on multi-thread. This will allow set playerlimit on 1200-2000. Without any lag.

If you would apply patch to your core, You should check this topic:

http://getmangos.eu/community/topic/15714/patch-multithreaded-packet-processing/

[Infinity]

Source: http://bitbucket.org/KingPin/trinitycore2/changeset/2e70f8737837/

Author: raczman

Rewritten by mangos, works fine.

Productivity above on 20-30 %, than at mtmaps Derex

On Quad Core:

Cpu0 : 31.7%us, 6.3%sy, 0.0%ni, 57.3%id, 0.3%wa, 0.3%hi, 4.0%si, 0.0%st

Cpu1 : 46.3%us, 6.7%sy, 0.0%ni, 43.7%id, 0.0%wa, 0.3%hi, 3.0%si, 0.0%st

Cpu2 : 39.9%us, 5.0%sy, 0.0%ni, 49.2%id, 0.3%wa, 0.7%hi, 5.0%si, 0.0%st

Cpu3 : 37.0%us, 4.6%sy, 0.0%ni, 52.1%id, 0.3%wa, 0.3%hi, 5.6%si, 0.0%st

On Dual Core:

Cpu0 : 51.5%us, 3.7%sy, 0.0%ni, 41.1%id, 2.0%wa, 0.0%hi, 1.7%si, 0.0%st

Cpu1 : 51.5%us, 4.6%sy, 0.0%ni, 38.3%id, 2.0%wa, 0.3%hi, 3.3%si, 0.0%st

Maximum online 2700ppl

patch 5.1(Fixed compil on Windows) rev 8120+ : http://filebeam.com/bf71d473c6bab04d6ca59806719ca40e

patch 5 rev 8120+ : http://filebeam.com/f1f3ca6b6321e7c714424322b2166cde

[leak]

Why is that 2700 so hard to believe and why does a multithreading maps patch _suddenly_ balance load over all cores while the very same patch never did before? Question over questions..

Ah yea before i forget. How about vmaps? A server without vmaps at least in instances -> fail.

[Infinity]

rev 5.1 fix compil on windows

2700 ppl were at switched off vmap

together with openmp is used getmap and relocation (http://getmangos.ru/forum/showthread.php?t=20619)

[leak]

Could you test what happens if vmaps are enabled in instances only?

vmap.ignoreMapIds = "369, 0, 1, 530"

[Infinity]

I cannot test with vmap, but anything terrible not will if include it in instance and bg

[leak]

More windows compile errors:

2>..\\..\\src\\game\\World.cpp(1461) : error C3001: 'task' : expected an OpenMP directive name
2>..\\..\\src\\game\\World.cpp(1518) : error C3001: 'task' : expected an OpenMP directive name
2>..\\..\\src\\game\\World.cpp(1550) : error C3001: 'task' : expected an OpenMP directive name

On mangos 8110 0.12 VC90 that is.

[Infinity]

More windows compile errors:

2>..\\..\\src\\game\\World.cpp(1461) : error C3001: 'task' : expected an OpenMP directive name
2>..\\..\\src\\game\\World.cpp(1518) : error C3001: 'task' : expected an OpenMP directive name
2>..\\..\\src\\game\\World.cpp(1550) : error C3001: 'task' : expected an OpenMP directive name

On mangos 8110 0.12 VC90 that is.

use 5.1 , i fix this http://filebeam.com/bf71d473c6bab04d6ca59806719ca40e

[leak]

Ah, my bad .. i thought that changelog is going downwards as usual.

[leak]

Your patch can't work though, you forgot:

m_configs[CONFIG_NUMTHREADS] = sConfig.GetIntDefault("MapUpdate.Threads",1);

in world.cpp

**edit

Also a crash we didn't have for ages.

http://paste2.org/p/305040

coresponding server.log output

http://paste2.org/p/305385

[Ambal]

Also a crash we didn't have for ages.

Any hint what causes this crash? :huh: Or this is a new one that you didn't have earlier? Do you use Derex's mtmaps patch and do you have similar crashes?

P.S. Infinity said holy truth about having 2700 ppl on dual-core CPU with ~160-170% utilization, I trust him because he tested my GetMap patch for more than 3 weeks. That Relocate patch is really awesome, we (me and SilverIce) just found a huge bottleneck in server performance. I have to add, that Trinity 1 used similar technique as we found out later, but it was much simpler.

[leak]

Unfortunately no good hint, i added the last few lines of the server.log to my post above though.

I searched all my crashlogs, i never had this one before and yes i was using derex mtmaps patch all the time.

I'd give that GetMap patch a shot too but that needs backporting to 0.12 and i haven't seen someone doing that so far.

As much as i can understand with google translate that relocation patch has some big troubles with stealth can someone tell a little more about that patch/issue?

This is my htop output with the 5.1 open mp patch and ~750 players online (gonna post again when there is more action):

2999 mangos    20   0 2627M 2376M  9856 S  0.0 29.8  0:21.10   1  |       `- ./mangos-worldd
3018 mangos    20   0 2627M 2376M  9856 R 12.0 29.8  2:52.65   1  |           `- ./mangos-worldd
3017 mangos    20   0 2627M 2376M  9856 S 13.0 29.8  2:54.72   2  |           `- ./mangos-worldd
3016 mangos    20   0 2627M 2376M  9856 S  2.0 29.8  0:23.62   3  |           `- ./mangos-worldd
3015 mangos    20   0 2627M 2376M  9856 S 11.0 29.8  1:59.29   3  |           `- ./mangos-worldd
3014 mangos    20   0 2627M 2376M  9856 S 12.0 29.8  2:00.23   4  |           `- ./mangos-worldd
3013 mangos    20   0 2627M 2376M  9856 S  7.0 29.8  1:59.21   4  |           `- ./mangos-worldd
3012 mangos    20   0 2627M 2376M  9856 S  7.0 29.8  1:58.65   2  |           `- ./mangos-worldd
3011 mangos    20   0 2627M 2376M  9856 S  0.0 29.8  0:00.00   2  |           `- ./mangos-worldd
3010 mangos    20   0 2627M 2376M  9856 S  7.0 29.8  1:58.71   3  |           `- ./mangos-worldd
3009 mangos    20   0 2627M 2376M  9856 S  9.0 29.8  1:59.75   1  |           `- ./mangos-worldd
3008 mangos    20   0 2627M 2376M  9856 S 11.0 29.8  2:00.26   1  |           `- ./mangos-worldd
3007 mangos    20   0 2627M 2376M  9856 S  0.0 29.8  0:00.92   2  |           `- ./mangos-worldd
3006 mangos    20   0 2627M 2376M  9856 S  1.0 29.8  0:26.46   2  |           `- ./mangos-worldd
3005 mangos    20   0 2627M 2376M  9856 R [b]35.0[/b] 29.8  7:02.41   1  |           `- ./mangos-worldd <-- main
3003 mangos    20   0 2627M 2376M  9856 R  0.0 29.8  0:00.75   4  |           `- ./mangos-worldd
3002 mangos    20   0 2627M 2376M  9856 S  0.0 29.8  0:07.83   1  |           `- ./mangos-worldd
3001 mangos    20   0 2627M 2376M  9856 S  0.0 29.8  0:01.83   1  |           `- ./mangos-worldd

It didn't really change that much, there is still much load on the main thread. That being with using vmaps in instances only (everything else is not playable imo).

Another thing: Is it really recommended to use 8 map threads? I was using 4 so far and i guess with 1100 players even that is still overkill. With 8 threads the server load goes to 3 (3 threads got no cpu time during the last schedule). I think that 1 thread per cpu rule of thumb still applies here - tell me if i'm wrong.

[ANTOMA]

leak am too test path Ambal\\SilverIce on live server...

ofcozz i have only 600+ testers.... path is plague ~30% low loading CPU

PS about stealth yes in this alfa version of path is problem, but Ambal is magician :lol:

[Infinity]

http://paste2.org/p/305040

coresponding server.log output

http://paste2.org/p/305385

This crash are caused not by patch

And the patch works fine on win and nix platforms, is checked up by many people.

[nanounico]

yes, i'm testing, about 1 hour ON right now. working fine. REV 8133 ScriptDev2 Acid, Centos 5.3 x86

thanks Infinity

[leak]

Well i never had this one before, could be from 0.12 branch too ofc.

Here load with ~960 players:

 6664 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:21.60   1  |       `- ./mangos-worldd
6681 mangos    20   0 3314M 3046M  9920 S [b]19.0[/b] 38.3 20:13.38   2  |           `- ./mangos-worldd <- net
6680 mangos    20   0 3314M 3046M  9920 S [b]20.0[/b] 38.3 20:04.87   1  |           `- ./mangos-worldd <- net
6679 mangos    20   0 3314M 3046M  9920 S  1.0 38.3  2:29.36   3  |           `- ./mangos-worldd
6678 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:00.02   2  |           `- ./mangos-worldd
6677 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:05.56   3  |           `- ./mangos-worldd
6676 mangos    20   0 3314M 3046M  9920 S [b]25.0[/b] 38.3 25:53.56   2  |           `- ./mangos-worldd <-- map?
6675 mangos    20   0 3314M 3046M  9920 S [b]29.0[/b] 38.3 25:57.04   4  |           `- ./mangos-worldd <-- map?
6674 mangos    20   0 3314M 3046M  9920 S  2.0 38.3  2:25.08   2  |           `- ./mangos-worldd
6673 mangos    20   0 3314M 3046M  9920 S [b]29.0[/b] 38.3 25:58.11   3  |           `- ./mangos-worldd <-- map?
6672 mangos    20   0 3314M 3046M  9920 R [b]72.0[/b] 38.3  1h03:01   1  |           `- ./mangos-worldd <-- main
6668 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:03.95   3  |           `- ./mangos-worldd
6667 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:42.00   3  |           `- ./mangos-worldd
6666 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:12.10   1  |           `- ./mangos-worldd

This patch behaves pretty much like raczman's original open mp patch. It doesn't really take that much load off the main thread (at least not more than derex patch does) and there is also this "one thread missing" issue. I switched to 4 map threads on this htop output.

It don't know if my partial vmaps are reponsible for this kind of behavior, yet somehow i'm far away from this nice balanced stuff in Infinity's first post.

[painejake]

I will update the mtmaps patch i have at some point. That was a good balance across all cores. However i haven't tested this one yet so i will give it a whirl after i sort out some issues on my server.

[raczman]

Well i never had this one before, could be from 0.12 branch too ofc.

Here load with ~960 players:

 6664 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:21.60   1  |       `- ./mangos-worldd
6681 mangos    20   0 3314M 3046M  9920 S [b]19.0[/b] 38.3 20:13.38   2  |           `- ./mangos-worldd <- net
6680 mangos    20   0 3314M 3046M  9920 S [b]20.0[/b] 38.3 20:04.87   1  |           `- ./mangos-worldd <- net
6679 mangos    20   0 3314M 3046M  9920 S  1.0 38.3  2:29.36   3  |           `- ./mangos-worldd
6678 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:00.02   2  |           `- ./mangos-worldd
6677 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:05.56   3  |           `- ./mangos-worldd
6676 mangos    20   0 3314M 3046M  9920 S [b]25.0[/b] 38.3 25:53.56   2  |           `- ./mangos-worldd <-- map?
6675 mangos    20   0 3314M 3046M  9920 S [b]29.0[/b] 38.3 25:57.04   4  |           `- ./mangos-worldd <-- map?
6674 mangos    20   0 3314M 3046M  9920 S  2.0 38.3  2:25.08   2  |           `- ./mangos-worldd
6673 mangos    20   0 3314M 3046M  9920 S [b]29.0[/b] 38.3 25:58.11   3  |           `- ./mangos-worldd <-- map?
6672 mangos    20   0 3314M 3046M  9920 R [b]72.0[/b] 38.3  1h03:01   1  |           `- ./mangos-worldd <-- main
6668 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:03.95   3  |           `- ./mangos-worldd
6667 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:42.00   3  |           `- ./mangos-worldd
6666 mangos    20   0 3314M 3046M  9920 S  0.0 38.3  0:12.10   1  |           `- ./mangos-worldd

You didnt notice that the main thread is the one on top, the one which spawned other threads?

[leak]

No it isn't. That is the master thread which spawns all the other threads later. It's cpu consumption time always ends with ~ 0:21.60.

The thread that does all the work on the server is what i call the main thread, if there is a better term for that feel free to share.

**edit

Another crash i never had before: http://paste2.org/p/307520

Either this patch is responsible or there is some really serious stuff wrong with all the commits from 8026 0.12 till 8128.