D2X-XL Worklog

Notes on features and problems from the development of D2X-XL, newest first.

Code Cleanup: Networking

If you thought the AI code had been a mess, you hadn't seen the networking code. Just like with the AI code, I had shunned from touching it until the need to fix it forced me to tidy it up. In the end I had turned one massive 4000+ line source file into nine smaller ones I could easily handle and which, like the AI code files, actually contained what their names suggested.

Like with the AI code, I rewrote some huge, messy function consisting of 100s of code lines using a table of functions and filters for each networking state. Before even calling a networking function, the current networking state is checked against that function's filter now to decide whether the function should get called at all. That saved a tremendous amount of repetitive code, sped up the entire operation and made debugging way easier:

// packet handler type
typedef int (* pPacketHandler) (char *dataP, int nLength);

// stuff packet handlers need to know
typedef struct tPacketHandlerInfo {
   pPacketHandler packetHandler;
   char *pszInfo;
   int nLength;
   short nStatusFilter;
} tPacketHandlerInfo;

// packet handler table
static tPacketHandlerInfo packetHandlers [256];

// a little helper macro
#define PHINIT(_pId, _handler, _nLength, _nStatusFilter)
           InitPacketHandler (_pId, _handler, #_pId, _nLength, _nStatusFilter)

// initialize the packet handler table
void InitPacketHandlers (void)
{
PHINIT (PID_GAME_INFO, GameInfoHandler, 0, (short) 0xFFFF);
PHINIT (PID_GAME_LIST, GameListHandler, SEQUENCE_PACKET_SIZE,
           (1 << NETSTAT_PLAYING) | (1 << NETSTAT_STARTING));
...
}

// process network data packets
int NetworkProcessPacket (ubyte *dataP, int nLength)
{
   ubyte pId = dataP [0];
   tPacketHandlerInfo *piP = packetHandlers + pId;

if (!piP->packetHandler)
   PrintLog ("invalid packet id %dn", pId);
else if (!(piP->nStatusFilter & (1 << networkData.nStatus)))
   PrintLog ("invalid status %d for packet id %dn", networkData.nStatus, pId);
else if (!NetworkBadPacketSize (nLength, piP->nLength, piP->pszInfo)) {
   if (!addressFilter [pId])
      memcpy (&THEIR->player.network.ipx.server, &ipx_udpSrc.src_network, 10);
   return piP->packetHandler ((char *) dataP, nLength);
   }
return 0;
}

What had brought me so far had been constant connection problems when my brother and I had tried to play a coop game, occuring particulary when he tried to rejoin my game, or after I had loaded a savegame of an earlier gaming session of ours; all that with a DSL connection on both ends.

I am not gonna tell you about every skeleton I had in my networking closet , but if you're interested in some insight particulary in the process of joining a netgame, read on.

I found out that joining actually is the most delicate and error prone stage in the entire process of a Descent 1 or 2 multiplayer game. The reason for this is that quite a huge amount of data has to be transferred between game host and client in a very short time: The game host will send the current game state to the client, which besides the data of all participants includes the state of all objects in the game, as well as any open doors, destroyed lights and so forth. I found out that the most crucial issue to the process of synchronizing a new player with the game host was transferring the object information as there can be quite a lot of objects particularly in a coop game, and object data is rather big. My simply having plugged the UDP code on top of the legacy IPX code didn't exactly help this: The IPX data packet size used by Descent 2 is rather small, so a lot of separate packets had to be sent to transfer all object data for a big mine. Quite a shocking discovery was that these data packets were sent depending on the game's frame rate, meaning that a fast host machine could really flood the receiver with small data packets. No provisions were made to send the packets in a steady stream using a fixed, sufficiently big time interval.

So there were two things I could fix right away: Packet size and transmission interval. I found out that sending a data packet every 200 ms lead to a stable data transmission in my test environment (where one network adapter had to handle both outbound and inbound traffic, thus getting quite some payload). Increasing the packet size from about 1/2 KB to 8 KB lead to a reduction of the packet count from 150 to 10 packets. That's a pretty neat improvement, meaning that the entire object synchronization process could only take a little over two seconds under optimal conditions.

While digging through the connection process, another flaw caught my eye: The timeout counter wasn't reset when receiving synchronization data. That meant that if the synchronization took more than 2 seconds, the entire process was cancelled and restarted. This may work for a fast intranet, but probably not when using UDP transfer over the internet. Resetting the counter each time a data packet was received cured this issue.

The last issue on my list was handling packet loss. Until now, Descent just aborted the connection process when it detected a certain amount of packet loss. That meant that with an unstable connection and a lot of synchronization data (like in a coop game) you had no chance of ever getting connected. Descent sends a packet counter with each synchronization data packet though, so it was easy to detect when the first packet got lost. Instead of aborting or restarting the entire connection process, D2X-XL will now ask the game host to restart sending synchronization data packets, beginning with the first missing one. That means that even with a flaky connection you now have a chance to get the synchronization data in several attempts and join the game, which subsequently may run very well since significantly less data gets transferred between the players now.

The only backdraw of all these changes is that D2X-XL is not UDP multiplayer compatible with D2X-XL versions before v1.12.47 anymore. But I think that is an acceptable price for smooth internet gameplay.


News

A new D2X-XL update fixing broken screen resolution persistence is available

There is yet another D2X-XL update is available (the new Pumo Mines broke a few things ...)

Believe it or not: There's a new D2X-XL version online!

Uploaded updated D2X-XL and library source code that compiles with Visual Studio 19

After a really long time, another new DLE version is available

A new DLE version is available (hear, hear!) :D

A new Max OS X version of D2X-XL is available!

D2X-XL now offers a cartoon style render mode!

New D2X-XL and DLE versions are available

Blarget has got his own area in the level spotlight

New D2X-XL and DLE versions are available

A couple of levels have been added to the level spotlight

Hooray! We're finally having a new Mac OS X version of D2X-XL!

D2X-XL now features an observer mode for multiplayer games

A new D2X-XL version with improved multiplayer synchronization is available

A new D2X-XL version fixing out of sync problems in multiplayer games is available

A new D2X-XL version with many multiplayer bug fixes is available

Added high quality (ogg) Descent 1 and 2 music to the downloads area

A new D2X-XL version with important bug fixes is available

D2X-XL compiles and runs on Linux again!

Descent 1 and 2 high res textures are now available as complete downloads

A new DLE version with a ton of bug fixes and improvements is online

A new D2X-XL version is online

A new D2X-XL version with a much needed bug fix is online

A new DLE version is online

Published another DLE version with more bug fixes and improvements by Sirius

Published a new D2X-XL version

Published a new DLE version with tons of bug fixes and improvements by Sirius

D2X-XL now supports Oculus Rift 3D headsets!

Updated DLE and D2X-XL to support new level features requested by Pumo

Added an article about DLE's new tunnel generator to the worklog

DLE's new tunnel generator is finished

Added a new article to the worklog

Published a new DLE version

Added a new article to the worklog

Added two new articles to the worklog

D2X-XL and DLE-XP now support triangular segment sides!

DLE-XP has an OpenGL render and 1st person view!

New DLE-XP version available

Wrote a new worklog article

Check out DarkFlameWolf's level spotlight section!

New DLE-XP and D2X-XL versions are available

A new DLE-XP version is available

Added a worklog article about the way point feature

Added way point support to DLE-XP and D2X-XL

Posted a bunch of new screenshots

Posted a bunch of new screenshots

A New version of DLE-XP is available

New OS X version of D2X-XL is available!

Added article about lightmaps to worklog.

New article and video added to worklog.