Latency Tweaks

[Auntie Mangos]

I always wanted to know what variables affected latency, cpu usage and memory usage, so i thought why not make a post for this and with the community make a great research about what does what and how much does it affect this 3 problems, cpu usage, memory usage and network usage. So for this post we will be making a small guide about what variable does what and how can it improve the performance of the server.

Please when you post your variables and give help about any of the posted variables here, if you want, you can also put the hardware you have in the following order (i will start with mine):

OS: Linux Ubuntu 9.04 32Bit

CPU: Core 2 Duo 8500

RAM: 4GB

NET: Adsl 1024 (128KiBs Down / 64KiBs Up)

Cases that should be studied here are:

Case 1 - Player with no other players around

Case 2 - Players in group doing instance (Max 5 to 10 players) Casting spells, etc

Case 3 - Massive amount of players together (like a huge fight, 5vs5 arena, BG with 10vs10, etc)

VARIABLES TO CHECK:

MaxPingTime - ?

UseProcessors - Use Multiple processor for better performance (Windows Only)

ProcessPriority - Gives a higher priority to this process (Windows Only)

Compression -Compression range from 1 (No Compression) to 9 (Full Compression). Consumes a little more CPU for smaller packets send. It "may" lower latency depending of where the latency is coming from. The LOWER the compression the less cpu is needed on serve. The HIGHER the compression the more cpu is needed but it will send less packets to client which means less bandwidth use from server. In some cases the latency number in the client will rise but the game will play smoothly. This depends on distance from client. Recommended setting is 4 which gives good performance with good compression.

SaveRespawnTimeImmediately - ?

MaxOverspeedPings - ?

GridUnload - If enabled will consume very little memory usage but players will have little lag passing from one grid to another. If disabled memory usage will go higher but players will have less lag when passing from one grid to another.

MapUpdateInterval - ?

vmap.enableLOS - Consume an amount of cpu and memory for the calculations.

vmap.enableHeight - Consumes a slight cpu and memory amount.

vmap.ignoreMapIds - Depending on the maps put here, the amount of memory consume could lower or get higher. If more map ids are put here there is low memory usage, if less maps are put here there is higher memory usage.

vmap.ignoreSpellIds - ?

DetectPosCollision - Consumes more cpu and slight memory usage if enabled. Disable it and consumes less cpu and memory but positions for several spells, etc will not be 100% accurate.

TargetPosRecalculateRange - The lower the value the more cpu it needs, the higher the value the less cpu it needs. Values range from 0.5 to 5.

UpdateUptimeInterval - ?

MaxCoreStuckTime - Detects if core is frozen. Higher numbers could help in having less crashes. Recommended by default is 10. FragFrog reports 120.

Visibility.Distance.Continents, Visibility.Distance.Instances, Visibility.Distance.BGArenas - This variables create lag for players with slow connection and also makes their fps drop. The values go from 45 to 333. Recommended is between 66 and 90 (Thanks to Ambal).

Visibility.Distance.Object - This variable creates small lag for players with slow connection and also makes their fps drop a little. The values go from 5 to 333. Should be pointed that if you make it very low (about 55 or less) players will not be able to see mineral veins, herb flowers, some house, etc unless they are exactly on top of the object or very close to it. So very low values for this variable is NOT recommended.

Visibility.Distance.InFlight - This variable just works when flying on transport and it also makes lag to player and fps drop. Values go from 0 (can not see anything of ground or nearby) to 333 (can see even your grandmothers house)

Visibility.Distance.Grey.Unit ?

Visibility.Distance.Grey.Object ?

Network.Threads - ?

Network.OutKBuff - ?

Network.OutUBuff - ?

Network.TcpNodelay - Causes less latency to users far away but creates more latency to users closer. If you have many far away players then it is recommended to have it on 1, else if you are in lan it should be 0.

+ Network.TcpNoDaley updated thanks to sodbone http://getmangos.eu/community/showpost.php?p=85470&postcount=2

+ Compression updated thanks to subhuman_bob http://getmangos.eu/community/showpost.php?p=85475&postcount=3

+ MaxCoreStuckTime updated thanks to FragFrog http://getmangos.eu/community/viewtopic.php?id=10674

+ Compression updated thanks to mfour http://getmangos.eu/community/showpost.php?p=91583&postcount=23

+ Recommended settings for Compression added thanks to freghar http://getmangos.eu/community/showpost.php?p=91679&postcount=25

[Auntie Mangos]

@mfour - For me before i went to linux i used windows XP, 200 and 2003 to test connection. All of them in my PC gave me lag. Linux did give me less. After a year of using Linux i went to try Windows Again, this time Windows XP and Windows 2003. Again, where Ubuntu gave me latency times of 100ms or less for many players, XP and 2003 gave me between 300ms and 700ms. Again this is the same PC, same internet connections. So yes i will stick in the server side with Linux all the way. For the 20 players problem, i have the following configuration:

UseProcessors = 0

ProcessPriority = 1

Compression = 1

PlayerLimit = 100

SaveRespawnTimeImmediately = 1

MaxOverspeedPings = 2

GridUnload = 1

SocketSelectTime = 10000

GridCleanUpDelay = 300000

MapUpdateInterval = 200

ChangeWeatherInterval = 600000

PlayerSaveInterval = 300000

vmap.enableLOS = 1

vmap.enableHeight = 1

vmap.ignoreMapIds = "530,571,169,37,369,451,13,30,1,0"

vmap.ignoreSpellIds = "7720"

DetectPosCollision = 1

TargetPosRecalculateRange = 5

UpdateUptimeInterval = 30

MaxCoreStuckTime = 0

AddonChannel = 1

LogSQL = 1

PidFile = ""

LogLevel = 0

LogTime = 0

LogFile = ""

LogTimestamp = 0

LogFileLevel = 0

LogFilter_AchievementUpdates = 1

LogFilter_CreatureMoves = 1

LogFilter_TransportMoves = 1

LogFilter_VisibilityChanges = 1

WorldLogFile = ""

DBErrorLogFile = "DB"

CharLogFile = ""

CharLogTimestamp = 0

CharLogDump = 0

GmLogFile = "GM"

GmLogTimestamp = 1

GmLogPerAccount = 1

RaLogFile = ""

LogColors = ""

Visibility.GroupMode = 0

Visibility.Distance.Continents = 66

Visibility.Distance.Instances = 66

Visibility.Distance.BGArenas = 66

Visibility.Distance.Object = 66

Visibility.Distance.InFlight = 10

Visibility.Distance.Grey.Unit = 1

Visibility.Distance.Grey.Object = 10

I can have with my 64KiB connection almost 40 players on gurubashi with 150ms latency or less.

Also i am also testing out the Speed grid patch by Ambal here http://getmangos.eu/community/viewtopic.php?id=8912 which improves even further the connection between players. So far the server is running for a month without any crash whatsoever.

I also tested out the compression of the packet again and it gives some latency spikes every x amount of seconds. It looks like the game is stuck but then it returns to normal. This repeats itself every so often until i change the compression back to 1.

@bosit - Can the compression be in 0? wasn't 1 the minimum or am i wrong?

EDIT: Forgot to mention i am not using any additional patch except for the ambal speed grid. I have auctionhousebot once but it created much lag.

[sodboone]

Network.TcpNodelay

Just by a few bits a research of what Nagle algorithm is and then testing it...

With Nagle enabled, it causes much higher latency to where players have up to a 4 second delay with slower server connections. Closer people get about a 300 to 1000ms latency with the lag noticeable. According to Nagle notes, It is not recommended to have this enabled to where it is required for the game to have quick reactions (for example, like if the server was full of pvp, it would be horrible to try and battle one another).

The testers on my server get better response time with TCP_NO_DELAY enabled and nagle disabled (setting Network.TcpNoDelay to 1)

[subhuman_bob]

Compression - ? Not really sure what to say here for compression 1, 5 and 9. Will it create more latency when compressed? or will it create less?

You trade more CPU processing time for smaller packets. Whether this will help or hurt depends on the source of your latency. If you have latency due to network congestion, higher compression may help. If you have latency due to processor congestion, lower compression may help.

Note that it's also possible to trade one sort of latency for another (for example, to lose network latency but gain processing latency) and work out to no change.

Also keep in mind that latency will always be higher than the round-trip ping from client to server. ping+processing time=latency

In most cases, it is the router that is replying to a ping, not the actual server. Factoring in that data to/from server has this extra hop involved (x2, once inbound and again outbound) means you should add a few milliseconds on to the "baseline" value.

[freghar]

Note that the compression is also related to client machines. Maximum compression will need faster CPUs on players' machines while needing lower bandwidth, low compression speaks for the opposite.

[FragFrog]

UseProcessors - Use Multiple processor for better performance (Windows Only)

In all my tests (with the 0.12 branch) this shows no difference whatsoever. In fact, setting multiple processors on sometimes even increases lag when a loop is entered (load constant at 50%, alternating between CPU's).

Mangos isn't truely-multi-CPU under windows, at least, not as far as I have been able to test.Would make quite a difference if true multithreading is enabled at instances for example.

[bosit]

Thanks for this, I was looking for some more in depth info about the server settings.

I'd like to ask which options (apart from compression which was explained) would require less bandwidth usage from my server.

My main problem is my extremely limited upload speed, which is 256 kbps. Could anyone recommend settings that will lower the upload used by the server, even if it causes more lag to the players?

Thanks in advance

[kerbe]

My main problem is my extremely limited upload speed, which is 256 kbps. Could anyone recommend settings that will lower the upload used by the server, even if it causes more lag to the players?

It's that compression which you could tune higher. More compressed packets, less is needed to transfer. However it doesn't mean you should right away go put it at maximum, but try out various values in that setting.

Another thing you might want to tune is visibility range. Wider it is, more data is perhaps sent to client (though I'm not sure about this, but so I think.)

[bosit]

Thanks for the tips kerbe, I already set compression to 7 for some initial testing, will check out visibility too

[cyrex]

Thanks for the tips kerbe, I already set compression to 7 for some initial testing, will check out visibility too

When you finish with the testing i would like to know how it went and what you found out. Thank you.

[mfour]

On the compression issue. We have limited upload throughput as well, and I have played around with the compression a bit. Sadly there was little evidence to suggest that it really has a big impact at all to latency experienced ingame. When I say latency, I refer to the high latency/lag caused by insufficient upload throughput. We have tweaked and played with many setting, but thruth is, if you do not have enought upload capacity, you simply don't and you will experience lag.

[cyrex]

Oki after almost 18 hours i can confirm the effect that mfour says about the impact of the compress packets.

On my connection for example, i have ADSL 1024 Kbps (Which is 128 KiBs Down / 64 Kibs Up). From that connection, players have 64 kilobytes of connection. The improvement must be very VERY little since in the 18 hours that was tested different compressions, there was no actual effect, at least for slow connections like mine.

It would be good if somebody could give some help for little connections like mine. For example, in my experience, the following did in fact improve the connection:

1. Setting vmap.ignoreMapIds to a group of maps that i have a list of

2. TargetPosRecalculateRange= 5

3. Visibility.Distance less than 90 but higher than 60. I have them in 66.

For the moment, this 3 settings did improve the ping for all players, also the can be more players together without so much lag.

Any other that maybe somebody else tested and knows from experience that you get an improvement on the server, for slow connections of course.

[FragFrog]

1. Setting vmap.ignoreMapIds to a group of maps that i have a list of

2. TargetPosRecalculateRange= 5

As far as I know should these have absolutely no effect on ping if your connection is the limiting factor. From your connection speed I am guessing you host at home - do you use a dedicated machine for this, or are you using a workstation for the server also? What is your CPU load on average when lag occurs?

[cyrex]

I run it at home from a pc i have. IT has ubuntu 9.04 desktop edition (and no, server edition did not improve connection in either 8.04, 8.10 or 9.04. I did not find any difference in performance using desktop than i did with server other than the fact that it consumes less resources and it is in a neat console)

The vmap.ignore did make my connection better in the zones that i added there, i do not know how or why but for me i could see the performance gain. The targetpos was mostly to have less ping when several players were together. So after i put it in 5, i could have around 20 players on same place with less lag.

The CPU usage never goes above 10%.

When i have several players fighting, for example in the BG, the load goes to 8%. This are about 10 vs 10 players.

When i have an event that i make every now and then on gurubashi where all players gather to fight monsters and win prizes, there are about 40 players. In that moment it goes to 9 or 10%.

The memory usage also stays below 512mb. Around 300-400mb mostly.

When huge lag occurs the cpu usage stays between 8%-10%. I have never seen those huge 90+% mention by other players. I seen them wayyy back like in January when yes, i did in fact have some that got to 65%. But it went lower and lower until what i have this days.

Also on another note. I was thinking of buying to adsl 1024 connections to the same pc. Will Mangos support 2 IP address that point to the same database and same players online?

If you are asking why, then it is because of the limiting factor of my 64KiBs i have of upload. With 2 connections i would have 2 64KiBs that could help some players with lag. But i need to know if Mangos supports 2 connections to the same database without problem ( i know linux does).

[mfour]

Oki after almost 18 hours i can confirm the effect that mfour says about the impact of the compress packets.

On my connection for example, i have ADSL 1024 Kbps (Which is 128 KiBs Down / 64 Kibs Up). From that connection, players have 64 kilobytes of connection. The improvement must be very VERY little since in the 18 hours that was tested different compressions, there was no actual effect, at least for slow connections like mine.

It would be good if somebody could give some help for little connections like mine. For example, in my experience, the following did in fact improve the connection:

1. Setting vmap.ignoreMapIds to a group of maps that i have a list of

2. TargetPosRecalculateRange= 5

3. Visibility.Distance less than 90 but higher than 60. I have them in 66.

For the moment, this 3 settings did improve the ping for all players, also the can be more players together without so much lag.

Any other that maybe somebody else tested and knows from experience that you get an improvement on the server, for slow connections of course.

Interesting, we have 512 kilobits/sec upload and there is no way we can have 20 people in the same place. The server lags to an almost total standstill.

The Vis Distance is set to 66 (old default) as well. I have also applied these changes to the core in the "GridDefines.h" file, which made a huge difference:

-#define MAX_NUMBER_OF_CELLS 4

+#define MAX_NUMBER_OF_CELLS 8

I will give your numbers 1 and 2 a try, and see what happens. We use WinXP as our OS, I wonder if Linux will give less lag...?

[bosit]

Set compression to 7 from 0 and the visibility stuff to 70 and seems both latency and upload usage of the server went down

Will do further tests to measure it exactly

[mfour]

Brilliant Cyrex. I will apply your config settings and see if it helps. At the moment I have to implement a player cap of 45 (which sux), anything above that and even a 5 man instance brings the server to a halt. Thanks for pointing out Ambal's thread, I look into it for sure, and thanks Ambal

It also seems that Linux is the way to go, clearly. Thanks for that.

[cyrex]

Do not worry about Ambal's Speed Grid since i has been added in the master git with the 8524 change. See here http://github.com/mangos/mangos/commit/dd404e200fdbfa38a4e9af879829af379d21972d

So you only need to compile under Linux and enjoy the stability.

[mfour]

So you only need to compile under Linux and enjoy the stability.

That is going to be the challenge for now

PS: I have implemented your suggested config settings, and it does infact seem to have improved player latency a bit. There still appear to be some areas which suffer from more lag than others, and a couple of lag spikes evey now and then. However, our ISP has been up the creep this week, so it's hard to say exactly what is causing those spikes.

Overall it seems to have made things run a bit smoother, thank you for sharing, much appreciated.

[Gotisch]

If you have high pings due to the limits of the internet connections only optimization is to send less stuff to players.

To test out what effect increased compression really has one should do the following:

start server with compression off connect with one player and enter a highly populated area. And of course record all traffic created.

then do the same with compression set to 9. comparing the ammount of data sent to the client should be lower with compression set to 9, i wonder by how much though.

[cyrex]

Updated MaxCoreStuckTime

[henhouse]

I think I over-read and confused myself on Compression. So the lower the compression, the LESS latency, and the HIGHER the more latency, correct? Or is it the other way around?

[mfour]

I think I over-read and confused myself on Compression. So the lower the compression, the LESS latency, and the HIGHER the more latency, correct? Or is it the other way around?

I think it depends on your system specs. The way I understand it is; the higher the compression ratio, the more the packets are squeezed into a smaller packet, which in turn can de transmitted more quickly to the client, creating less lag. However, in order to do this the server PC will need more CPU power to do the compression, and could therefore actually increase your latency if the CPU is allready running at it's limits.

Hmm I could be quite wrong, but that is how I understand it.

[cyrex]

Updated compression.

[freghar]

about the compression; it isn't a bottleneck (unless you have a PIII CPU). By reducing compression to 1, you actually reduce the raw time needed by a few microseconds, but it may take more time to deliver a larger packet (or larger amount of packets) via client's slow connection.

My suggestion is to use ~4 compression level. It's very fast (about 20 times faster than level 9) and it helps a lot with traffic.

edit: not sure what compression mangos uses, but those are bench results for linux-2.6.26 tarball using gzip:

level 0 - time 0.032s - size 359342080

level 1 - time 16.677s - size 93911077

level 2 - time 17.309s - size 89806601

level 3 - time 19.449s - size 86891062

level 4 - time 23.145s - size 80465071

level 5 - time 27.902s - size 77214583

level 6 - time 37.406s - size 75974520

level 7 - time 43.751s - size 75528449

level 8 - time 64.568s - size 75242466

level 9 - time 71.764s - size 75194772

all times are "user" (ie. no disk read/write times, just userspace processing), size is in bytes

To be honest, I expected a better result.