leak

Default questions: What CPUs? Vmaps enabled? (if yes, where?) What patches?

The memory is the least of the problems, you can always reduce it by disabling things. Biggest problem is the cpu usage due to being almost single threaded (yea yea i know..). Haven't seen someone running it really smooth past 1500 ppl. 500-600 shouldn't be a problem with an up to date machine.

mangos: 7483 0.12 branch sd2: 893 udb: 365 patches: anticheat, mtmaps Player killed the server: 2009-03-20 16:54:17 ERROR:GetInstance: object XYZ(1190194), typeId 4, in world 1, should be in map 562,8455 but that's not loaded yet. #3 0x0000000000757b86 in MapInstanced::GetInstance (this=<value optimized out>, obj=<value optimized out>) at ../../../src/game/MapInstanced.cpp:126 map = <value optimized out> http://paste2.org/p/169870 ---------------------- #0 Unit::InterruptNonMeleeSpells (this=0x7f313068c9b0, withDelayed=true, spell_id=0) at ../../../src/game/Spell.h:333 http://paste2.org/p/169869 ---------------------- #0 std::_List_base<Aura*, std::allocator<Aura*> >::_M_clear (this=0x13c44898) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/include/g++-v4/bits/list.tcc:76 __cur = (class std::_List_node<Aura*> *) 0x0 http://paste2.org/p/169866 ---------------------- Server freeze: http://paste2.org/p/169865 ---------------------- #0 Spell::cancel (this=0x7fd89c534fd0) at ../../../src/game/Spell.cpp:2008 http://paste2.org/p/169864 ---------------------- Player killed the server: 2009-03-23 18:49:10 ERROR:GetInstance: object XYZ(1196123), typeId 4, in world 1, should be in map 529,535 but that's not loaded yet. #3 0x0000000000757b86 in MapInstanced::GetInstance (this=<value optimized out>, obj=<value optimized out>) at ../../../src/game/MapInstanced.cpp:126 http://paste2.org/p/169862 ---------------------- Server freeze 5mins after the previous crash: http://paste2.org/p/169861

I guess this doesn't cover the "taking the flag" part of EOTS and AB?

This suggestion/question might not be all new: I'm searching some sort of API which allows executing internal mangos functions during runtime by external applications. I know the remote console but after extensive useage i noticed it is quite unreliable. Once in a while it doesn't let me log in, killed my server quite a few times (didn't really investigate this any further) and i also believe telnet isn't necessarily the proper interface for an API. If i'm not all wrong it is also not multi user capable. I think a first thing people could fine useful is the possibilty to send ingame mails from an external source. I tried to adapt the external mail patch from the trinity patch queue but i kills my server during the mail check cycle. I think using the mangos internal functions for mail/kick/ban/etc. is a better way than just trying to reimplement the very same methods into a (web)application. Does something like that exist already? Did someone already try something similar?

What happens on vmap disabled zones?

True Any chance to make git pull work without parameter for updating?

Cause that is most likely the next question coming up.

mangos: 7319 0.12 branch sd2: 893 udb: 365 patches: anticheat, mtmaps #0 SocketHandler::Select (this=0x45725bd0, tsel=0x45725bb0) at ../../../../dep/src/sockets/SocketHandler.cpp:833 in ../../../../dep/src/sockets/SocketHandler.cpp http://paste2.org/p/152085 ---------------------- #0 Unit::ProcDamageAndSpellFor (this=0x7fb4feb124a0, isVictim=true, pTarget=0x7fb4feb124a0, procFlag=16777216, procAuraTypes=<value optimized out>, attType=BASE_ATTACK, procSpell=0x7fb59780e74c, damage=0, damageSchoolMask=SPELL_SCHOOL_MASK_SHADOW) at ../../../src/game/Unit.cpp:10100 in ../../../src/game/Unit.cpp http://paste2.org/p/152086 --------------------- #0 Loot::clear (this=0x7f411c548d60) at ../../../src/game/LootMgr.h:241 in ../../../src/game/LootMgr.h http://paste2.org/p/152150 --------------------- #0 MotionMaster::DirectClean (this=0x14158db0, reset=false) at ../../../src/game/MotionMaster.cpp:120 in ../../../src/game/MotionMaster.cpp http://paste2.org/p/153208

I'm currently using the patch Arclite posted, but it has quite some flaws: Blink doesn't work on non-map surface (as mentioned above) If you can figgure the potential range right you can blink up mountains (i.e. the Warsong graveyard mountains) It still makes people fall under the map once in a while. If you need help with improving the current situation of blink - let us know.

Calm down you two guys... I don't plan on running mangos on win anyways, i just got tired of removing the mtmaps patch each time i wanna mess with the code (afterall M$ VS > rest). I have mtmaps running on gentoo with 1500 people but the spelllag is still getting very annoying past 1200 players. Just about to switch hardware from AMD to Intel, but i don't really expect any big shot from that. Those 1200-1500 numbers seem to be some sort of design limit of mangos. (and yes there are people who claim to have 1500+ lagfree, but i've also seen crazy riots on those peoples forums where the players crying about massive lag. But as usual, if someone has "the big solution" - feel free to share some knowledge) @cozzmic Thanks for the "below" that really made the difference. Who would have thought includes are that sensitive to order...

Did anyone manage to get that working?

mangos: 7066 0.12 branch sd2: 812 patches: anticheat, mtmaps #0 Creature::RemoveFromWorld (this=0x7f82b9665e50) at /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/include/g++-v4/ext/hashtable.h:596 http://paste2.org/p/132391

I don't have a repo there, the only bigger patch i use is CWN's anticheat. Few smaller game mechanic fixes like that void autosacrifice thing but nothing performance related.

Ok, i adapted mtmaps from trinity patch repository and it seems to work quite good. 4443 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:12.81 | `- ./mangos-worldd 4471 mangos 20 0 3704M 3533M 9444 R 17.0 59.3 18:12.80 | `- ./mangos-worldd <- net 4470 mangos 20 0 3704M 3533M 9444 R 17.0 59.3 18:28.55 | `- ./mangos-worldd <- net 4469 mangos 20 0 3704M 3533M 9444 S 7.0 59.3 7:07.51 | `- ./mangos-worldd 4468 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:00.09 | `- ./mangos-worldd 4467 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:05.75 | `- ./mangos-worldd 4466 mangos 20 0 3704M 3533M 9444 R 0.0 59.3 0:40.82 | `- ./mangos-worldd 4465 mangos 20 0 3704M 3533M 9444 R 69.0 59.3 1h05:39 | `- ./mangos-worldd <- main 4464 mangos 20 0 3704M 3533M 9444 R 16.0 59.3 16:28.51 | `- ./mangos-worldd <- map 4463 mangos 20 0 3704M 3533M 9444 R 19.0 59.3 16:24.00 | `- ./mangos-worldd <- map 4447 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:10.72 | `- ./mangos-worldd 4446 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:33.39 | `- ./mangos-worldd 4445 mangos 20 0 3704M 3533M 9444 S 0.0 59.3 0:16.39 | `- ./mangos-worldd There is now a little more room on the main thread, yet the problem still comes back at 1400 users. Usually all those quadcore cpus have less power on a single core. Can anyone with a larger user number post a htop, etc. output what's the mainthread is doing on such a machine?

@freghar I'm running char and realmd table in innodb with innodb_buffer_pool_size = 3072M (4GB RAM database machine), so basicly almost the full chardb should fit into the RAM. @Blub Could you also post the machine specs and the mangos rev? It seems to be a slightly older one since mtmaps stopped working at some point?

I'd like to talk a bit about the running behavior of mangos "under pressure" since there is virtually no information around about it. Running worldd on a dedicated box (AMD X2 6000+ 3Ghz per core, 6GB RAM, software raid0) and mysql/realmd on a sightly slower box crosslinked via dedicated 1Gbit connection. My observations: Mangos scales quite well with increasing number of users, there is only one issue i've noticed so far: vmaps Having full vmaps enabled at around 600 users the execution delay of spells ingame starts to increase massivly. This can be stalled by disabling vmaps for the world continents (vmap.ignoreMapIds = "369, 0, 1, 530"). Having this setting enabled i can go up to 1300 users when the same problems start again. First i thought the CPU of the worldserver hit the limit, but it is only peaking at 95% with the mainthread of mangos on one core. Wild theories: Unfortunately the mtmaps isn't at a working state so i can't really test all the stuff i wanted. I've heared other people blaming slow mysql execution for those kinda delays in spell and movement behavior, yet i think i tuned my database pretty good and rather think it is a combination of issues. My mysql server is doing about 250-300 queries/sec in average (INS,DEL, UPD, SEL) peaking up to 1000 (i assume player saves) once in a while and i've even seen it doing 4000 qps without problems. I can't really understand that mysql<->vmaps issue yet, but like the "disable vmaps for part of the world" workaround showed, there is a problem with vmaps implementation that reduces smoothness while there is still free capacity on the backend (cpu, ram, db) available. I would appcreciate if someone who has more insight into mangos innerworks would give a comment on this and if there is a "doable" approach how to solve it, without rewriting half of the core. Note: I'm not talking about the multithreading/multicomputing topic here. Imo those are higher design goals that should be approached when an application already eats all the hardware it can get (100% useage on multiple cores, etc.), which isn't really the problem of mangos yet. I'd also like to encourage other users of mangos running more than 500 players to share their impression and maybe exchange a few cfg tipps.

That's no test, those a really 1250 people. I think you can't really meassure performance by spawning bots with lets say Pseuwow. The interaction as in combat, movement, etc. are the real performance drainers.

mangos: 6991 0.12 branch sd2: 812 patches: anticheat What's going on with all those loot and item storing related crashes. They are around for ages now too. #3 0x0000000000746312 in Guild::_CanStoreItem_InTab (this=0x7f981b9118d0, tab=<value optimized out>, dest=@0x42cfadd0, count=@0x42cfaca4, http://paste2.org/p/124130 #0 0x000000000070a686 in Bag::GetItemCount (this=0x7ff945198a30, item=11829, eItem=0x0) at ../../../src/game/Bag.cpp:191 http://paste2.org/p/124132 #3 0x0000000000748f76 in Guild::AppendDisplayGuildBankSlot (this=<value optimized out>, data=@0x410c9db0, tab=<value optimized out>, slot=69) at ../../../src/game/Object.h:160 http://paste2.org/p/124133 This one is especially annoying: #0 Loot::clear (this=0x7fe4b2c46ad0) at ../../../src/game/LootMgr.h:238 http://paste2.org/p/124135

I didn't have the time for revision bump yet, but on 6909 the boss Edwin VanCleef is spawning rogues (sd2 script) which immediately kills the server with that SetInvisibilty function. Is that already covered by the 6928 fix? Gonna update asap.

I currently running ~1250 people without any bigger problems, althought the main thread of mangos is draining 100% on core most of the time and you can already notice it's getting unsmooth ingame. I also think the mtmaps patch could really help with that. There is free room on the other cpu core that could easily take 1 or 2 map threads.

Additional information for the Unit::SetVisibility crash issue: 2008-12-19 09:46:44 ERROR:Pet or controlled creature (Entry: 636 GUID: 11423) attempt targeted home 2008-12-19 09:46:44 ERROR:Pet or controlled creature (Entry: 636 GUID: 11422) attempt targeted home The same messages are appearing repeatedly on each of those crashes. Could it be a problem related to pets getting out of the range of the owner? -------------------------------------------- * 1 process 15981 0x0000000000589c92 in MaNGOS::Remove<Player> (obj=0x7f061a7132d0) at ../../../src/game/../framework/Utilities/LinkedReference/Reference.h:60 http://paste2.org/p/118976 * 1 process 18560 0x00000000006612e0 in Aura (this=0x7f527aecef10, spellproto=0x7f52f4435f44, eff=1, currentBasePoints=0x7f52a5ce572c, target=0x7f528511f000, caster=0x7f52cbdfb7e0, castItem=0x7f526e686b20) at ../../../src/game/SpellAuras.cpp:322 http://paste2.org/p/120379 * 1 process 2049 ACE_Unbounded_Queue<WorldPacket*>::Dequeue_head (this=0x7f3a8fd03f50, item=@0x42d1adc8) at /usr/local/include/ace/Unbounded_Queue.cpp:219 http://paste2.org/p/120381 * 1 process 5656 Unit::InterruptNonMeleeSpells (this=0x7f04e39c4110, withDelayed=true, spell_id=0) at ../../../src/game/Spell.h:332 http://paste2.org/p/120382

Core: 6909 SD2: 800 UDB: 364 Patches: arena, anticheat #0 Unit::SetVisibility (this=0xc44d0b8bc2a0b800, x=VISIBILITY_GROUP_STEALTH) at ../../../src/game/Unit.cpp:8747 http://paste2.org/p/117548 #0 Guild::_CanStoreItem_InSpecificSlot (this=0x90f1540, tab=1 '\\001', slot=<value optimized out>, dest=@0x434a4dd0, count=@0x434a4ca4, swap=false, pSrcItem=0x7f90df75dc50) at ../../../src/game/Object.h:160 http://paste2.org/p/117550 #3 0x0000000000708eda in Bag::GetItemCount (this=0x7f091ab6c010, item=20559, eItem=0x0) at ../../../src/game/Object.h:160 http://paste2.org/p/117551 #0 0x0000000000745d01 in Guild::_CanStoreItem_InTab (this=0x7f0914b98150, tab=0 '\\0', dest=@0x43606dd0, count=@0x43606ca4, merge=false, pSrcItem=0x7f089923fb80, skip_slot=255 'ÿ') at ../../../src/game/Guild.cpp:1833 http://paste2.org/p/117552 #3 0x0000000000758997 in MapInstanced::GetInstance (this=<value optimized out>, obj=<value optimized out>) at ../../../src/game/MapInstanced.cpp:126 http://paste2.org/p/117553 #3 0x00000000004f336b in Object::GetUInt32Value (this=0x7f6e57d64280, index=3) at ../../../src/game/Object.h:160 No locals. http://paste2.org/p/117773 #0 Loot::clear (this=0x7f2dce08ad80) at ../../../src/game/LootMgr.h:238 http://paste2.org/p/117842 **edit Omg what is going on again..?

Alright down from ~56sec to ~49sec guild loading time having mysql on an extra server with a dediacated connection. I think this is definately a step towards the right direction. Guild loading still consumes about 50% of the mangos bootup time, i hope there is some room for further improvement.

Indeed, already had network cards that required special kernel flags to be able to deal with the mass of tiny queries that mangos is doing on startup. Gonna test this one asap.