[Help/Bug] High load performance bug. Could be camera system problem.

[tehavatar]

I have below a screenshot of a function monitor during high load on my server, tons of people online. During these times it seems that the server lags to death, even though it has enough resources at its disposal.

From the looks of this screenshot, there is a LOT of self-calls to the function EnsureGridCreated() and getNGrid() also takes up a considerable amount of self-time.

Self-time is the time this function takes to execute.

Any ideas?

[SilverIce]

Why you decided so?

Map::Visit(const Cell& cell, TypeContainerVisitor<T, CONTAINER> &visitor)
{
   if( !cell.NoCreate() || loaded(GridPair(x,y)) )
   {
      [i] EnsureGridLoaded(cell); [/i]
       getNGrid(x, y)->Visit(cell_x, cell_y, visitor);
   }
}

called at each grid's cell visit

Do you have another test results without camera system in core?

[tehavatar]

Im not saying its the camera system, just saw those functions and thought it could maybe be attributed to camera system.

How would I diagnose intermittent spikes?

Take a look at my bw graph:

Clearly massive spikes, core freezes every now and then =|

[SilverIce]

There is a possibility that this is caused by incorrect grid states handling

You may try this:

@@ -284,11 +284,11 @@ Map::EnsureGridCreated(const GridPair &p)
                p.x_coord, p.y_coord);

            // build a linkage between this map and NGridType
            buildNGridLinkage(getNGrid(p.x_coord, p.y_coord));

-            getNGrid(p.x_coord, p.y_coord)->SetGridState(GRID_STATE_IDLE);
+            getNGrid(p.x_coord, p.y_coord)->SetGridState(GRID_STATE_ACTIVE);

            //z coord
            int gx = (MAX_NUMBER_OF_GRIDS - 1) - p.x_coord;
            int gy = (MAX_NUMBER_OF_GRIDS - 1) - p.y_coord;

[tehavatar]

Above didnt change anything. Should I remove above changes?

Any idea how I could diagnose/test/debug these spikes?

[Forgotten]

You may revert that new system at least and test without it

[tehavatar]

Update: Compiled in Win32 mode and spiking problem dissapeared COMPLETELY. =|

Obviously not a solution to the problem, especially because of the memory limitations, but it the spikes seem to be gone.

[SilverIce]

Strange then.. seems it's platform related

Above didnt change anything. Should I remove above changes?

You may leave them

[BThallid]

If that's bandwidth spikes, I would be more concerned about how there's less activity just before a spike. The spike would be the server just sending all the packets that get queued while it was busy.

Is 31.488 Avg Self Time normal for 186 Visits for Player::LoadFromDB? Is the DB running in 64-bit or 32-bit?

[tehavatar]

Hi again,

DB is running in 64bit mode.

Let me explain the graph. The green flatline areas indicate 0 activity - green and yellow is blended DOWNLOAD bandwidth, which means the server is receiving a ton of requests from the clients but it isnt sending anything back (red)... the spikes of bw is what happens when the server regains.... consciousness so to speak, it floods upload to clients to respond to the amount of requests received.

:<

[wside420]

Same problem here x64 Windows Serer 2003. Do you unload maps?

[wside420]

Using the unload grid option Fixes This. However Unload grid has problems of its own.

The Bug happens with about 400-500 People online. Basically freezes up every min.

Server 2003, 24 Gigs of RAM, x64, Xeon CPU.

[wside420]

Maybe this is Directly Related to a Commit or Processor? Its quite the Bug non the less. Forced to Run in win32