[New Feature] MapUpdater/per thread

[Auntie Mangos]

Im realy looking forward to test this project. Is it possible to get it from some git repo?

I've found only pastebin diff.. = /

Second question: I havent found with which mangos revision is the lastest version compatible, are you guys maintaining the compatibility with every commit?

Thanks,

Termitt (Admin@Spermik)

check post 67 : http://getmangos.eu/community/showpost.php?p=93568&postcount=67

and about second question, there was about 2-3updates here, would be cool if repo for this

[Opterman]

What bug does the patch fix? What features does the patch add?

It adds a processor map system per thread.

For which repository revision was the patch created?

8756

Who has been writing this patch? Please include either forum user names or email addresses.

Woweur && Me

This patch add a blizzlike map system processor.

Maybe blizzard use a thread pool for map processing or maybe process instead thread but i wrote this patch following the main concept of Blizzard Map System processor , so if there's a crash in 1 thread it restarts and whoever be in anothers map could continue playing..

It patch has a backtrace output, and unload system if map isnt used, a shutdown feature if there's many crashes and at last point ( maybe too important to finish with the lag ): Each map number is processed by 1 thread...

If a map is being restarted and someone is offline but is on this map and tries to login it recv a error message and if someone tries a teleport to a crashed map that is restarting it comeback at original teleport position.

This patch too change the network processor to the map if player is on World.

It means that the map process all the packets while the player is in the current map.

At teleport i change the network processor to the world thread and the same at crash and before login.

(22/11/09 Redundant crash on SendObjectUpdates fixed)

This is the patch http://pastebin.com/m74127ce3

SQL Patch for events http://pastebin.com/m25a95e35

PD: I've tested it as exhaustively as it has been possible and I havent detected errors but i'd appreciate if you test it in a server with many population

If you have a good CPU i recommend u a MapUpdater.Interval less than 30 ms

I've to say that there's 1 thing on this patch that is not blizzlike and is "ACE processing" in blizzard when there's lag in a map ( like 571 winterlag(grasp) ) you can write correctly in the chat with no lags so i think the network processing of blizzard is in a different place of map processing..

Edit: Current TODO: 2rewriting events and pools..

Thanks to Woweur for help testing and fixing crashes

Greetings, Opterman

[Wowka321]

OMG :eek:

It's very-very fine.

It's meen that instance maps per thread too?

I'll test it soon. Thx u!

[LordJZ]

Does this patch affect server perfomance somehow?

[Opterman]

Does this patch affect server perfomance somehow?

It affects to server performance yes, but i dont know the result , i've tested with no many population and works fine , but i need to know what's the result with highs population. I dont want to give you false hopes, because as i said i dont know what will happen, but i 've programmed the patch doing my best, its the only thing that i can say.

The performance result is on testing.... only there...

OMG

It's very-very fine.

It's meen that instance maps per thread too?

I'll test it soon. Thx u!

Yes ,thought in doing the patch with a thread pool, but first i want to know how works with the current method..

Greetings, Opterman

[imblizzspy]

I can confirm that its blizz-like.

Amazing patch.

/bow.

[Wowka321]

I in a shock! :eek:

Great!

BTW, do you have patch in patch-file? I have a problem with copy-paste from pastebin.com.

Edit: Ah, sorry. I found magic button download =)

[leak]

Sounds interesting, but let get this straight.

This is basicly another multithreading maps patch, right?

The others use a static thread pool and also don't have this restart capability from what i understand.

Clarifications are welcome.

[Opterman]

Sounds interesting, but let get this straight.

This is basicly another multithreading maps patch, right?

The others use a static thread pool and also don't have this restart capability from what i understand.

Clarifications are welcome.

Yes leak , is as you've said , as i've already said if it goes well i 'll think in an improving with an statical thread pool ( for instances only ( obiusly ) ).

Greetings, Opterman

[XTZGZoReX]

If you need help with implementing a thread pool, you're more than welcome to poke me (I'm Zor on Rizon) .

[Wowka321]

Just test it. World looking for players like all freeze. Nothing happen, nothing can't do Like big lag..but core don't stuck. U can teleport players by command from console. I did something wrong? :confused:

[xeross155]

Aweomse you actually figured it out, now we should all join hands and get this baby into the master branch.

Edit: I don't see how a thread pool is useful, you only need threads per map, why spawn a fixed number for it. I might be missing the point here (well it feels like I do) so enlighten me with your knowledge if possible.

[voidx]

not test for now but its a good idea.

[leak]

Aweomse you actually figured it out, now we should all join hands and get this baby into the master branch.

Edit: I don't see how a thread pool is useful, you only need threads per map, why spawn a fixed number for it. I might be missing the point here (well it feels like I do) so enlighten me with your knowledge if possible.

As much as i support that spirit, i get the bad feeling that this patch will end up like all the (fine working) mtmaps patches. I mean it's not like this patch suddenly removes all the mtmaps related issues from mangos...

[XTZGZoReX]

Aweomse you actually figured it out, now we should all join hands and get this baby into the master branch.

Not gonna happen, however. This is a totally different approach from what we really want to do (thread pool).

However, this patch have some very good ideas (like the get-map-by-thread-id) that can be used later.

[Agiko]

There is a bug in this patch somewhere... Your character name is unknown...NPC completly freezed

[Shendor]

crash at startup, on guild loading

rev 8758

Loading Guilds...
[                                                  ] 0%
[
[*                                                 ] 2%  
[
[**                                                ] 4%  
[
[***                                               ] 6%  
[[New Thread 0x416f1950 (LWP 19304)]
[New Thread 0x41ef2950 (LWP 19305)]
[New Thread 0x426f3950 (LWP 19306)]
[New Thread 0x42ef4950 (LWP 19316)]
[New Thread 0x436f5950 (LWP 19317)]

Program received signal SIGSEGV, Segmentation fault.
[switching to Thread 0x42ef4950 (LWP 19316)]
0x00000000005ba9bc in Map::Update ()
Current language:  auto; currently asm
#0  0x00000000005ba9bc in Map::Update ()
#1  0x00000000005c2a57 in MapRunnable::run ()
#2  0x00000000007c38ea in ACE_Based::Thread::ThreadTask ()
#3  0x00007f241a8c8fc7 in start_thread () from /lib/libpthread.so.0
#4  0x00007f241a1a55ad in clone () from /lib/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f241ccce700 (LWP 19300)):
#0  0x00007f241a14da4d in malloc () from /lib/libc.so.6
No symbol table info available.
#1  0x00007f241ab9d0cd in operator new (sz=2)
   at ../../../../libstdc++-v3/libsupc++/new_op.cc:52
   p = <value optimized out>
#2  0x00007f241ab9d1e9 in operator new[] (sz=2)
   at ../../../../libstdc++-v3/libsupc++/new_opv.cc:32
No locals.
#3  0x00000000007bd1b7 in Field::SetValue ()
No locals.
#4  0x00000000007badf5 in QueryResultMysql::NextRow ()
No locals.
#5  0x000000000077e3b2 in Guild::LoadMembersFromDB ()
No locals.
#6  0x00000000005ef9a7 in ObjectMgr::LoadGuilds ()
No locals.
#7  0x0000000000729a3b in World::SetInitialWorldSettings ()
No locals.
#8  0x00000000004caf63 in Master::Run ()
No locals.
#9  0x00007f241a0f41a6 in __libc_start_main () from /lib/libc.so.6
No symbol table info available.
#10 0x00000000004c91a9 in _start ()

[cyrex]

Holycow this is the best thing since pizza!! :eek:

[VladimirMangos]

Mangos recently have good progress to prepare per-map threads code run in correct way without redundent locks and unsafe case. And cleary it need still more work. So ateempt _now_ do map threading too early. It unsafe and not provide full power. It also required hack lock adding in redom lines.

Before

1) Pool system per-map rewrite

2) Creature/GO per-map guid generation/use (stuck by (1))

3) proper locks for chat/group/guid/mail code

4) proper locks for global player list

5) Make per-map thread ready BG manager code

6) maybe something more

[Ambal]

Nice catch with void Master::_OnSignal(int Signal) hooking BUT: such way of handling crashes is very dangerous, since we don't know what caused crash and simple 'restart thread and free map resources' technique is naive... I just fear that after some crashes we'll end up with infinite thread restarts :confused:

[Opterman]

crash at startup, on guild loading

rev 8758

Loading Guilds...
[                                                  ] 0%
[
[*                                                 ] 2%  
[
[**                                                ] 4%  
[
[***                                               ] 6%  
[[New Thread 0x416f1950 (LWP 19304)]
[New Thread 0x41ef2950 (LWP 19305)]
[New Thread 0x426f3950 (LWP 19306)]
[New Thread 0x42ef4950 (LWP 19316)]
[New Thread 0x436f5950 (LWP 19317)]

Program received signal SIGSEGV, Segmentation fault.
[switching to Thread 0x42ef4950 (LWP 19316)]
0x00000000005ba9bc in Map::Update ()
Current language:  auto; currently asm
#0  0x00000000005ba9bc in Map::Update ()
#1  0x00000000005c2a57 in MapRunnable::run ()
#2  0x00000000007c38ea in ACE_Based::Thread::ThreadTask ()
#3  0x00007f241a8c8fc7 in start_thread () from /lib/libpthread.so.0
#4  0x00007f241a1a55ad in clone () from /lib/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f241ccce700 (LWP 19300)):
#0  0x00007f241a14da4d in malloc () from /lib/libc.so.6
No symbol table info available.
#1  0x00007f241ab9d0cd in operator new (sz=2)
   at ../../../../libstdc++-v3/libsupc++/new_op.cc:52
   p = <value optimized out>
#2  0x00007f241ab9d1e9 in operator new[] (sz=2)
   at ../../../../libstdc++-v3/libsupc++/new_opv.cc:32
No locals.
#3  0x00000000007bd1b7 in Field::SetValue ()
No locals.
#4  0x00000000007badf5 in QueryResultMysql::NextRow ()
No locals.
#5  0x000000000077e3b2 in Guild::LoadMembersFromDB ()
No locals.
#6  0x00000000005ef9a7 in ObjectMgr::LoadGuilds ()
No locals.
#7  0x0000000000729a3b in World::SetInitialWorldSettings ()
No locals.
#8  0x00000000004caf63 in Master::Run ()
No locals.
#9  0x00007f241a0f41a6 in __libc_start_main () from /lib/libc.so.6
No symbol table info available.
#10 0x00000000004c91a9 in _start ()

Shendor the first backtrace is not the same to the second ( Full backtrace )

[switching to Thread 0x42ef4950 (LWP 19316)] /=/ Thread 1 (Thread 0x7f241ccce700 (LWP 19300))

To help you i need the backtrace of map thread

I add too that if you debug the server with debugger you 'll get it signals like SIGTERM and you should continue. It's unused map cleanup feature.

Nice catch with void Master::_OnSignal(int Signal) hooking BUT: such way of handling crashes is very dangerous, since we don't know what caused crash and simple 'restart thread and free map resources' technique is naive... I just fear that after some crashes we'll end up with infinite thread restarts

That's what i thought , so by this reason i added a option where you can specificate the number of crashes to restart the server, if there's some redundant error

I.E If i put MapUpdater.ShudownAtCrash = 6 when there's 6 map crashes ( of whatever map , not for each map.. ) then the server shudowns...

+ MapManager::Instance().IncreaseCrashCount();

+ uint16 CrashCount = sWorld.getConfig(CONFIG_MAPUPDATER_SHUT_ATCRASH);

+ if(CrashCount && CrashCount <= MapManager::Instance().GetCrashCount())

+ {

+ sLog.outError("Signal Handler: Too many crash detected. Shutting down server...\\r\\n");

+ ObjectAccessor::Instance().SaveAllPlayers();

+ exit(Signal);

+ }

Greetings, Opterman

[Ambal]

One more thing: solution like Multi-MaNGOS is needed to deal with crash handling e.g. whole process(!) restart instead of thread restart.

P.S. Btw, what about crashdump generation? Would we get them on every crash or not?

Cheers.

[Opterman]

One more thing: solution like Multi-MaNGOS is needed to deal with crash handling e.g. whole process(!) restart instead of thread restart.

P.S. Btw, what about crashdump generation? Would we get them on every crash or not?

Cheers.

If you active crash dump generation , yes , anyway the ACE Stacktrace is too poor.... so I want to read gdb for linux and another debugger on windows to make a good backtrace class and then put it on files with mapnumber and timestamp...

Aweomse you actually figured it out, now we should all join hands and get this baby into the master branch.

Edit: I don't see how a thread pool is useful, you only need threads per map, why spawn a fixed number for it. I might be missing the point here (well it feels like I do) so enlighten me with your knowledge if possible.

I think join master now isnt a good idea without certain improvings and testing...

Greetings, Opterman

[imblizzspy]

If you need help with implementing a thread pool, you're more than welcome to poke me (I'm Zor on Rizon) .

You should all guys join forces and create a perfect mtmaps patch I see a lot of great ideas coming from you.

[Ambal]

You should all guys join forces and create a perfect mtmaps patch

Mtmaps patch w/o using thread pool -> epic fail Otherwise your server would die from thread context switches