World loop and latency on modern CPUs

[Xenithar]

Okay, I've been working on this for three days now and I am no closer to solving it beyond saying that something is broken on the Windows build. I run zero in a Server 2022 VM hosted by Hyper-V on Server 2022 on the actual system. The system has four E5-4650v2 Xeons (10-core, 20-thread each). The VM has 32GiB of RAM and four six cores. Running realmd and mangosd ONLY on the VM. The issue is latency. I get, on gigabit LAN, 80ms on average. Friends outside the LAN who normally ping sub-100 to other game servers, get 500+. The server has four gig Ethernet connections and dual SFP+ (25Gbps) connections. My WAN is only 1Gbps both ways.

I first gave the map update four threads instead of two. Then six. No change. Then I saw that the diff being reported was 130~160ms. I set the map update frequency to 150 instead of 100. Now it jumps around 20~60ms. Pings are increased. I set it to 250ms and it still deviates 0~60ms. Again, two threads or six threads makes no difference.

This indicates to me that something is broken somewhere. The entire time the server is struggling the cores it is running on are between 5% and 8% usage, so we're not pegging out the CPU. It is using less than 4GiB of RAM. Nothing is stressed. For reference, the VM sits on a RAID10 array of four 1.2TB SAS-12 disks. No bottlenecking.

Here is where it gets fun. If I build zero on my old Linux server, the one Jesus signed back in the day (E3-something, 2012 era, 16GiB RAM, single disk), it runs fine. In other words, the RC car from Radio Shack running on C batteries is absolutely demolishing a formula 1 car. All I can figure is that the Windows build is very broken somehow. The physical hardware is not being stressed in any way, it's like MaNGOS just refuses to use more than one thread or something in Windows. Is this the case? I have a Dell EMC T140 with hardware RAID I can put Artix Linux on and host from, but I'd rather have it on the big server.

Am I overlooking something or is the Windows build bugged in some way?

[madmax]

That diff is normal its expected. Servers im hosting jump between 0-70ms on the console, server .info command shows a steady world delay of 51ms which it has always been far as i can remember.

[Xenithar]

Are those Windows servers? I installed Devuan (barebones, shell-only install) on a Dell EMC PowerEdge T140 and my high on the diff has been 11ms. In addition, the server starts in under two seconds. I mean it's absolutely crazy. The T140 has an x8 PCI-E NVME RAID controller in it. I have two 256GB PCI-E 3.0 NVME SSDs in RAID1 in there. The OS and MaNGOS binaries live there. It also has a Dell PERC H730P RAID controller attached to four 500GB SAS disks (HDDs) in RAID10. This is only for the database. It is mounted with things that make ext4 commit changes instantly. The H730P has a battery backup on it. It also has 32GB of ECC DDR4 in it. The OS and MaNGOS only use around 1.8GiB of RAM though.

What I have noticed though, is that lower diff times seem to drastically affect ping. 0-11ms keeps my client ping at 20~30ms. 80ms of diff shoots it up to 80+ on the client. That's what had me wondering. Also, the T140 is far weaker than my R820 or R540 servers. The only difference is that the 2U servers run Windows Server 2022 and the T140 now runs Devuan, and it does a FAR better job despite inferior hardware.

So I only have a few remaining questions. Is the diff a measurement in milliseconds of how long it took to update the world on a single iteration? In other words, the config says 100ms and it takes 105ms so diff shows 5ms? I don't remember this from the last build I did before covid, the loss of a business, a kid, and all the other "fun" things that happened.

Also, the MaNGOS server only seems to use two cores on this server also, and only up to 20% usage on each core. Is it possible with the current build to split the world loop in some way to take advantage of multiple cores? The R820 and R540 both have 40 cores and 80 threads. The T140 is 6 cores and 12 threads. Still, if I could better use the hardware, it would be nice. Maybe a world loop for Kalimdor and a second one for Eastern?

*EDIT*

One off-topic question. I have used the GUI extractor and got dbc, gomodels, maps, and mmaps, but no vmaps. Do we sue vmaps any more? I have googled and looked around here but cannot find anything about them not being removed.

Edited September 21 by Xenithar
Added a question