Game Dev

Custom Struct Serialization for Networking in Unreal Engine

Unreal Engine has a strong networking integration which makes it a great engine for multiplayer games. At the base of its client-server communication protocol are properties replication and RPC (remote procedure call). When it comes to optimization, there are several things you can do to reduce the traffic bandwidth[a]: basically you should not send data too often for actors that are not relevant for the player. But when you have to send data, you should try to use some form of compression to reduce the bandwidth. You can leverage some quantization functionalities exposed by the engine such has Vector quantization[b][c][d][e] and Quaternion quantization[f][g]. But what if you have defined a USTRUCT that you want to use as a replicated property or for some RPC call? It turns out that each Unreal Engine USTRUCT can define a custom network serialization for its data. You can do custom compression before the data is sent to the network and decompression after the data is received.

USTRUCT NetSerialize

When you declare a USTRUCT in Unreal Engine you can add a NetSerialize method which is part of the Unreal Engine struct trait system. If you define this method, the engine will use it while serializing and deserializing your struct for networking both during properties replication and RPC.

You can find a lot of documentation about serialization in this source file:
Runtime/Engine/Classes/Engine/NetSerialization.h

Here is the NetSerialize method signature:

And here is how to add it to a USTRUCT:

Pay attention to the last part: to tell the engine that the ustruct defines a custom NetSerializer function, you have to set to true the type trait WithNetSerializer for the struct FMyCustomNetSerializableStruct. If you don’t add that piece of code, the NetSerialize method will never be called.

If you are wondering about that wibbly wobbly template thing, it is a C++ programming pattern called C++ Type Traits[h] and it is part of the wide use of C++ template metaprogramming in Unreal Engine. With this technique, class code specialization is moved from runtime to compile time. The basic idea is to resolve at compile time what is known at compile time. The result is a performance boost due to a reduced runtime overhead.

The Unreal Engine USTRUCT supports many other types of customization. You can find the complete list in the header file Runtime/CoreUObject/Public/UObject/Class.h:

NetSerialize in Action

To see an example of a struct using this method in the engine, have a look at the FRepMovement struct in EngineTypes.h, where you’ll find this code:

As you can see, the method takes a FArchive where to pack and unpack the struct data. The FArchive is a class which implements a common pattern for data serialization, allowing the writing of two-way functions. Basically, when it comes to serialization, you have to make sure that the way you serialize your data is exactly the same you use for deserialization. The best way to ensure this behavior is to write one single context-sensitive function that does both. The black magic of the FArchive lays in its overloaded << operator. This operator is at the base of the creation of two-way functions. Its behavior is context-sensitive: when the FArchive is in write mode, it copies data from right to left, when the FArchive is in read mode, it copies data from left to right. The only problem here is that the Epic guys chose to overload an operator with a strong directional meaning and at first this mechanism may result a little confusing. You can find out more about the use of FArchive in this article by Rama. For an example of struct data compression let’s have a look at this piece of code in UnrealMath.cpp:

The code above performs quantization of an FRotator Pitch, Yaw and Roll properties into byte values and vice versa. In the first 3 lines the current values are quantized from float to byte values. Then comes the two-way part of the method where for each value a bit which tells if the value is zero is written/read. If it is different from zero, then the value is written/read. The last part of the method is context-sensitive: only if the archive is in read mode, the data is decompressed and written into the float properties of the struct.

This technique can be very useful in a multiplayer networking context; for example, if you are using a struct to replicate your player’s controls, you can compress your data using a bit for each control to tell if the value is different from zero (many input controls are zero most of the time), than you can store them using byte quantization (you don’t really need float precision for an analog  user input). Here is an example:

NetDeltaSerialize and Fast TArray Replication

Unreal Engine implements a generic data serialization for atomic properties of an Actor like ints, floats, objects* and a generic delta serialization for dynamic properties like TArrays. Delta serialization is performed by comparing a previous base state with the current state and generating a diff state and a full state to be used as a base state for the next delta serialization.

As seen above with NetSerialize, it is possible to customize the delta serialization by defining a NetDeltaSerialize function in a USTRUCT.

Here is the method signature:

The source file Runtime/Engine/Classes/Engine/NetSerialization.h contains a lot of documentation about how the Unreal Engine net serialization works. It contains also commented code examples on how to implement custom struct serialization; I strongly recommend reading it.

Custom net delta serialization is mainly used in combination with fast TArray replication (FTR).

Basically, if you want to replicate a TArray efficiently, or if you want events to be called on client for adds and removal, just wrap the array into a ustruct and use FTR.  Here is what the code documentation says about FTR:

Fast TArray Replication is a custom implementation of NetDeltaSerialize that is suitable for TArrays of UStructs. It offers performance improvements for large data sets, it serializes removals from anywhere in the array optimally, and allows events to be called on clients for adds and removals. The downside is that you will need to have game code mark items in the array as dirty, and well as the order of the list is not guaranteed to be identical between client and server in all cases.

Here is the commented code example for FTR extracted from the documentation:

And here are some code examples about implementing the above “step 6 and beyond”:

As you can see above, I’m marking an item as dirty when adding or modifying it. I’m marking the whole array as dirty when removing some item.

Notes

  1. Unreal Engine Multiplayer: Performance and Bandwidth Tips []
  2. Unreal Engine: FVector_NetQuantize []
  3. Unreal Engine: FVector_NetQuantize10 []
  4. Unreal Engine: FVector_NetQuantize100 []
  5. Unreal Engine: FVector_NetQuantizeNormal []
  6. Unreal Engine: FRotator::SerializeCompressed []
  7. Unreal Engine: FRotator::SerializeCompressedShort []
  8. An introduction to C++ Traits []

Leave a Reply