Very high CPU usage @ CentOS

[nitz]

Hello guys, I'm playing around with my own server, it's modest, a VM with one Xeon 2.8Ghz *dedicated* core and 2GB ram on CentOS 5.7 64bits and 1000mbps shared uplink.

It was troublesome to compile mangos, I had problems with TBB and ACE X CentOS plus many other stuff.

Currently my server has vmaps enabled and about 43 characters, the MySQL database is hosted in the same machine, there are rarely over 5 players online but the CPU is ALWAYS at 100%.

I was hosting this same server on a Windows machine with a Core 2 Duo 6750 with 2GB ram and Windows XP and the CPU use was pathetic.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

24633 nitz 15 0 324m 209m 10m S 98.7 17.3 659:10.84 mangosd

The database is the same and it's fairly decent comparable to the Core 2 Duo box considering it's now entirely dedicated to the server.

Server is working fine but this 100% cpu at all times worries me.

Any thoughts?

Thank you

[Xenithar]

Mine does the same thing. I have two physical CPUs on a Tyan motherboard and the mangosd process eats 100% CPU on one of them at all times, even when I am not connected from my gaming rig. I am concerned as well due to possible high temperatures on the CPU that is being stressed. My server is a shell-only install of Debian 6.0.3 with no extras. I remotely administer it using SSH (PuTTY if I am in Windows).

[faramir118]

can you compile for profiling and run gprof, and post the results?

A very simplistic example of how is here: http://cplusplusworld.com/gnugprof.html

[Xenithar]

I will try when I get home this evening. I work 9-5 though!

*UPDATE*

Alright, how do I get gprof to work during the build process with MaNGOS? Is there something I do in the makefile or what?

*UPDATE*

I edited build/CMakeFiles/CmakeCXXCompiler.cmake and added -pg to the ARGS1 var. Building now, but not sure I did it properly.

[Xenithar]

Anybody? I know how to do this with a simple project, but not with something so complex as MaNGOS. Oh, and I have never used cmake either, so that may be part of it.

[faramir118]

Try this:

https://gist.github.com/1604896

then (feel free to add other parameters)

cmake /path/to/mangos/src -DPREFIX=/path/to/install/dir -DDEBUG=1

then

make; make install

Unfortunately, it seems that gprof doesn't work on Intel Macs... I was able to verify that the profiling output is created when mangosd exits, but gprof shows no data. If I have time later, I can maybe build a virtual linux box and give it a try.

[faramir118]

maybe you also try to use callgrind, one of valgrind's tools (I haven't used valgrind before, so I won't be able to help much)

http://valgrind.org/docs/manual/cl-manual.html

[nitz]

Apparently compiling mangos with external TBB and ACE was my issue. - Wrong, see my post below

After recompiling mangos with both use external set to 0 here are the results:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1912 nitz 16 0 420m 250m 10m S 1.0 12.2 0:22.35 mangosd

Basically it's using 1% CPU and 12.2% memory with 3 players online, that's amazing.

I am in love with Mangos after trying a dozen of servers and a lot of headache, mangos just works and it's rock solid.

[Pyromaniac]

Apparently compiling mangos with external TBB and ACE was my issue.

After recompiling mangos with both use external set to 0 here are the results:

...

could you please explain how you did that? on my system mangos uses about 35% of cpu with 0 players. maybe its the same reason.

[nitz]

Ok, it wasn't TBB or ACE that I had trouble with, actually I figured out the problem is much simpler!

When using nohup it makes mangos eat 100% of CPU in background.

Since I'm administering the server through SSH once I leave the session it kills all my processes so nohup is supposed to fix it by sending the process to background and it's output to a file however it makes mangos go crazy consuming all the CPU.

I am not sure if it's a mangos or a nohup issue.

Screen works much better for this so you can leave the SSH.

[Xenithar]

I can concur. I have been using nohup as detailed in a guide I read. I normally use screen. Using screen for this fixes the problem. For some reason MaNGOS hates nohup. Glad I install screen on every Linux system I setup!

[Xenithar]

I just wanted to update this and say that after almost 60hrs (58.6 right now) the server has peaked at 7% CPU usage with three of us on from my LAN, and the RAM usage peaked at 4.9%. This is on a dual processor Athlon MP server (2.80GHz) with 4GB of ECC DDR. Steps for starting MaNGOS using screen are below, for others to use.

• Connect to your server with SSH or get to a shell if logged on locally
  
• Start a new screen session by typing screen and pressing enter
  
• Screen should show you a welcome message, just press enter to get out of it
  
• Change to the binary directory, which in my case is ~/wow-server/bin
  
• Start the realm server with ./realmd &> ./realmd.out &
  
• Start the world server with ./mangosd &> ./mangosd.out &
  
• Disconnect from the screen session by holding CTRL and pressing A followed by D
  
• Disconnect your SSH client or close your console if in KDE/Gnome
  
• Your server will continue running in the screen session until you shutdown or reboot your hardware
  

Oh and just in case you need to reconnect to the screen session int he event of a crash or whatever, get back to your shell prompt and simply type screen -r. If you are running multiple screen sessions, you must also pass the screen session ID.

[Pyromaniac]

hm, i am using "screen" too, but i have ~35% CPU usage (it's a 1,6GHz Dual-Core-CPU) and ~10% RAM (of overall 2GB) with NO player...i know there where times when mangos wasn't that hungry.

any ideas why the cpu usage could be so high on my system?

i am using the newest UDB and SD2.

AH-Bot is deactivated.

OS: Ubuntu 11.04 32bit

compiling script:

• cmake ../../sources/mangos/ -DPCH=1 -DPREFIX=/home/pyromaniac/mangos_bin
  
• make -j2
  
• make install
  

[Xenithar]

You have a 1.6GHz processor (Celeron, Xeon, C2D, what?) which is much slower than my two physical 2.8GHz chips. Also, you have half the RAM I have, so the numbers seem to match up. Also note that I have an actual server (Tyan motherboard, server chips as opposed to desktop chips, ECC, buffered RAM as upposed to standard RAM, RAID, etc) and it sounds to me like you're using an old desktop system. I personally do not see a problem with 10% RAM usage on 2GB of RAM. You also only have one physical CPU that is a little better than half the speed of mine, so 35% CPU usage seems right. What type of CPU is in your box?

[Pyromaniac]

ok, sorry, i've forgot to tell that the RAM-usage is ok in my eyes, what i'm wondering of is the CPU usage.

the cpu is an "Intel Core Duo T2300 1.66 GHz".

i know that this is not a high-power-server-system the main application for this system is a 24/7 PVR-Backend (MythTV). from time to time i am starting mangos on this system to play for some minutes/hours and after this i'm closing mangos again. so why start and close (and log in/out) it every time? because of the cpu-usage which is unnecessary.

if you look only at the CPU - Frequency (the Intel-Webpage tells me that the T2300 has really 2 cores ), than it should look like this: your (peak) 7%@2,8GHz should be 12,25%@1,6GHz on my system. but it is about three times higher than it should be. thats what i am wondering of.

but don't give you too much hassle, i'm just wondering, it's nothing that is really bugging me or something

[nitz]

That CPU is more than enough to run Mangos for a few players and a home server, do not underestimate the CPU speed.

You could try CPU Limit, it's a small utility that let you set a max percentage of a process in Linux, you might find yourself setting a limit of 5% and still perform nicely for a couple of players.

I would try a clean DB(with UBB) and see if the CPU usage is the same, I've had issues with some loop caused by something in my DB due a faulty sql update that caused high CPU load once.

These are the stats of my server running mangos rock solid using screen with about 3-8 players getting online daily:

System uptime 8 days, 2 hours, 35 minutes

Running processes 85

CPU load averages 0.09 (1 min) 0.13 (5 mins) 0.10 (15 mins)

CPU usage 0% user, 0% kernel, 0% IO, 100% idle

Real memory 2 GB total, 1.01 GB used

[Xenithar]

I know you have two cores. I have two physical CPUs. Big difference. Not only that, but you are looking at two cores at 1.6GHz as opposed to two CPUs (or cores if you want to compare them that way) at 2.8GHz. That's 1.2GHz more power. I do not expect to see a 1.6GHz box running as fast as a 2.8GHz box. On top of that you have a desktop chip with a different bus architecture than server chips. You will not see as low CPU usage on that setup.

Also, how do you know what the peak should be? That's like the red-line on street-bikes. That is the assumed damage-point. Some bikes can go past it, other may not make it that high. Depending on your build, what mods (if any) and your OS, you may get very different results. My server is using 4% CPU and 7.1% RAM right now (I am looking at the stats right now) while two friends and my girl play on my LAN. This is with a core Debian install (no X/KDE/Gnome/whatever, no extras, just shell, git, and build tools) and is great in my opinion for an older server system like mine. In fact, the server has been up for over 334hrs according to the system, which seems about right (almost two weeks). Load averages are 0.00, 0.02, and 0.00.

I still believe that your usage is normal. I will compile my stuff on my Windows box at the office Monday and check. I have a Core2Duo there at 2.4GHz. If it is above 20% CPU then I will assume yours is normal also. I still feel like you are close to being normal for your configuration though.

[nitz]

I know you have two cores. I have two physical CPUs. Big difference. Not only that, but you are looking at two cores at 1.6GHz as opposed to two CPUs (or cores if you want to compare them that way) at 2.8GHz. That's 1.2GHz more power. I do not expect to see a 1.6GHz box running as fast as a 2.8GHz box. On top of that you have a desktop chip with a different bus architecture than server chips. You will not see as low CPU usage on that setup.

Also, how do you know what the peak should be? That's like the red-line on street-bikes. That is the assumed damage-point. Some bikes can go past it, other may not make it that high. Depending on your build, what mods (if any) and your OS, you may get very different results. My server is using 4% CPU and 7.1% RAM right now (I am looking at the stats right now) while two friends and my girl play on my LAN. This is with a core Debian install (no X/KDE/Gnome/whatever, no extras, just shell, git, and build tools) and is great in my opinion for an older server system like mine. In fact, the server has been up for over 334hrs according to the system, which seems about right (almost two weeks). Load averages are 0.00, 0.02, and 0.00.

I still believe that your usage is normal. I will compile my stuff on my Windows box at the office Monday and check. I have a Core2Duo there at 2.4GHz. If it is above 20% CPU then I will assume yours is normal also. I still feel like you are close to being normal for your configuration though.

Xenithar, you mentioned earlier in this thread you were using Athlons MPs, aren't these Athlons based on the Athlon XP architecture? If yes, they're running at 2.13 GHz (named 2800+ to compete with Pentium 4 2.8Ghz). While they're server CPUs, they're identical to the Athlon XP parts except for the dual CPU support which was an extremely efficient platform versus the Pentium 4 based Xeons at the time.

You can't compare them to a Core 2 Duo, it's a completely new architecture which is much more efficient per clock. You can run down two Athlons MP running at 2.2Ghz (3200+) versus a Core i3 downclocked to 1.0Ghz and the Core i3 would still win any benchmark you throw at it. And at stock speed the new processor would be at least 5-15x faster.

I have tested the CPU usage on my old laptop with a Core 2 Duo T6500 @ 2.10Ghz, same DB as my CentOS box with the latest mangos revision and the CPU is always at 0-2% with no players so having Mangos ALWAYS at 35% is a bit weird IMO.

[faramir118]

mangos does practically nothing with no players, so CPU use should be very low.

If you can't track it down to environmental issue (screen vs nohup, external vs dep ace, etc) then you should try some of the profiling techniques I posted above so we can see what is causing the problem

[Xenithar]

Well I just tried compiling Zero here and I got 23% usage with slight variations at idle. This is connected to my regular database, so I know that isn't the issue. This is on a Core2Duo 2.20GHz chip with an ASUS P5Q Turbo board. This tells me that it is normal on a C2D to have something going on, but what I do not know. Also, this is under XP Pro x64, not Linux. Oh and my chip is an E2200 if you are wondering.

I do understand the architectural differences in the chips. AMD had a better bus back then (more data in and out at once compared to a P4, for example) but the core was still slow. Games didn't spawn extra enemy AI because you had an AMD so core speed was all that mattered unless you were running a server or heavily multitasking, in which case AMD beat Intel down. The point it though, that unless you have your server keeping all zones loaded and processing all AI, you're running essentially what I am with my server. Assume only one zone is loaded and that one zone has four mobs. That's the same amount of data to crunch each clock cycle whether you're on a 486 or an Itanium II. What makes this odd is that my exact build runs great on the old Athlon MP server while on my desktop C2D I am experiencing your exact results.

At this point I would take Faramir's advice and do some profiling. Maybe something compiled badly. This one has me because I expected it to be higher, but not like this.