wintun 0.13 driver - performance issue

Fri Jan 14 19:43:19 UTC 2022

Hi everyone !
I have this issue to share to see if it can be considered a natural limitation of the implementation or (more probably) that I am making some bad decisions.
I modified the example.c included in wintun-0.13 code to test higher rates of packet sending.
I am building Multicast UDP packets 200 bytes long that are received at a consumer application -that opens a socket to the wintun adapter registering the multicast group to the socket-.
I turned off all console logging during the test run.
I'm running on a Windows Server 2019, 8 cores, 32 GB RAM, Visual Studio Community 2019.
What I get is that the example can send the 100 Kpps (and up to ~300 Kpps) without problems, but not with the consumer application reading packets from the wintun adapter.
When the consumer application is reading packets from the wintun interface the top sending rate falls about 15-20%.
It keeps below 85 Kpps and shows some events of problems at WintunAllocateSendPacket(). I set the Rings capacity to the MAX allowed of 64 MB.
Trying to fix this behaviour I modified the ERROR_BUFFER_OVERFLOW error handling doing a wait on the Alertable state of the Receive Ring and settting the Receive.Tailmoved event when that happens.
This is the SendPackets() method:
static DWORD WINAPI
SendPackets(_Inout_ DWORD_PTR SessionPtr)
{
TUN_SESSION* Session = (TUN_SESSION*)SessionPtr;
while (!HaveQuit)
{
BYTE* Packet = WintunAllocateSendPacket(Session, 228);
if (Packet)
{
MakeIpPacket(Packet);
WintunSendPacket(Session, Packet);
++sentPkts;
spin(RATE, PPS);
}
else {
if (GetLastError() != ERROR_BUFFER_OVERFLOW) {
return LogLastError(L"Packet write failed");
}
else {
while (Session->Descriptor.Receive.Ring->Alertable != 1) {
spin(1000, FALSE);
}
SetEvent(Session->Descriptor.Receive.TailMoved);
++sendDelay;
}
}
}
return ERROR_SUCCESS;
}
and this is my spin() function:
// Packets per second sent can be defined as PPS using windows timers or with their period using a count loop
static boolean PPS = TRUE;
static LONG RATE = 115000;
static void
spin(LONGLONG rate, boolean isPPS) {
LARGE_INTEGER StartingTime, CurrentTime, ElapsedMicroseconds;
LONGLONG nano;
if (isPPS) {
nano = 1e9 / rate;
QueryPerformanceCounter(&StartingTime);
while (TRUE) {
if (HaveQuit)
break;
QueryPerformanceCounter(&CurrentTime);
ElapsedMicroseconds.QuadPart = CurrentTime.QuadPart - StartingTime.QuadPart;
ElapsedMicroseconds.QuadPart *= 1000000; // guard against loss-of-precision
ElapsedMicroseconds.QuadPart /= Frequency.QuadPart;
ElapsedMicroseconds.QuadPart *= 1000; // nanoseconds
if (nano < ElapsedMicroseconds.QuadPart) {
break;
}
}
}
else {
if (rate < 1000) {
return;
}
nano = rate;
while (nano > 0) {
if (HaveQuit)
break;
--nano;
}
}
}
Statistics reported look like this when sending to the wintun interface and without the consumer application active.
Rate of 100 Kpps is easily reached.
"Delayed" counts are ERROR_BUFFER_OVERFLOW events from WintunAllocateSendPacket().
> .\wintunprj.exe
2022-01-11 21:09:19.0371 [+] Wintun library loaded
2022-01-11 21:09:19.0377 [+] WintunCreateAdapter: Creating adapter
2022-01-11 21:09:19.0477 [+] SelectDriver: Using existing driver 0.13
2022-01-11 21:09:19.0757 [+] Wintun v0.13 loaded
2022-01-11 21:09:19.0762 [+] Starting -----------------------------
2022-01-11 21:09:19.0774 [+] Run started with PPS: 115000
2022-01-11 21:09:26.0764 [+] Elapsed time in seconds: 6.985698
2022-01-11 21:09:26.0766 [+] 757111 packets sent, 42 packets received
2022-01-11 21:09:26.0768 [+] Packets sent at 108380 pps
2022-01-11 21:09:26.0769 [+] Final Sending problem counts - delayed: 0
And the statistics look like this with the consumer application active.
Packet rate drops to 84 Kpps.
Delayed events are present but they don't look significant actually...
.\wintunprj.exe
2022-01-11 21:33:22.0198 [+] Wintun library loaded
2022-01-11 21:33:22.0204 [+] WintunCreateAdapter: Creating adapter
2022-01-11 21:33:22.0282 [+] SelectDriver: Using existing driver 0.13
2022-01-11 21:33:22.0564 [+] Wintun v0.13 loaded
2022-01-11 21:33:22.0576 [+] Starting -----------------------------
2022-01-11 21:33:22.0594 [+] Run started with PPS: 115000
2022-01-11 21:34:39.0068 [+] Elapsed time in seconds: 76.470261
2022-01-11 21:34:39.0070 [+] 6455902 packets sent, 45 packets received
2022-01-11 21:34:39.0072 [+] Packets sent at 84424 pps
2022-01-11 21:34:39.0074 [+] Final Sending problem counts - delayed: 5
Any thoughts or ideas on how is this happening and changes due will be really appreciated !
Thanks,
Alex