FForge
← Labssystems-03Systems and syscalls●●○○

UDP round trip on loopback — blocking vs epoll vs busy-poll

systems/03_udp_pingpong.cneeds: tools/bench.h, two P-cores1 steps
0/1 steps
what this lab teaches

The cost of WAITING. A client and a server exchange 50k UDP datagrams over loopback, the client timing each round trip. Both sides wait for data the same way: block recvfrom() blocks; the kernel puts the thread to sleep and wakes it epoll epoll_wait() then recvfrom(): the event-loop pattern spin recvfrom(MSG_DONTWAIT) in a loop: the thread never sleeps The network stack cost is identical in all three; the difference is entirely what happens between "packet arrived" and "thread runs".

evidence to leave with
18.5 / 19.9 / 8.7 us p50; maxima 687 / 485 / 42 us
first, get the code
run this
$ git clone https://github.com/adit4443ya/debug_labs.git && cd debug_labs
then build it
run this
$ (cd systems && ./build.sh)   -> bin/03_udp_pingpong [block|epoll|spin]
$ Client pins to CPU 2, server to CPU 4 (two P-cores). Ports 40001/40002.
walk it, one step at a time
run this
$ for m in block epoll spin; do ./bin/03_udp_pingpong $m | tail -1; done
what it printed on the machine that wrote this lab
label                       min      p50      p90      p99    p99.9      max       mean   (round-trip ns)
block                     15096    18518    23666    41822    77351   687064    20108.3
epoll                     15551    19909    23508    30649    41463   484766    20638.9
spin                       8115     8666    10383    13840    17664    41614     9070.8
how to read it

- spin: 8.7 us per round trip = two one-way trips through the loopback stack at ~4.3 us each (sendto: socket -> IP -> loopback device -> softirq -> socket queue; then the receiver's recvfrom copies it out). That is the kernel network stack's cost on this box, with nobody sleeping. p99.9 17.7 us, max 42 us. - block: 18.5 us p50, 2.1x. The extra ~10 us is two WAKE-UPS per round trip: the packet arrives, the kernel marks the sleeping receiver runnable, sends an IPI to its (idle) core, the core leaves its idle state, the scheduler switches in the thread, and it returns from recvfrom. Each is ~5 us here, and worse: the tail is 5x longer (p99.9 77 us, max 687 us) because a wake-up can hit a core that is in a deep sleep state, a timer tick, or a scheduler decision. - epoll: the same wake-up plus the epoll_wait bookkeeping: 19.9 us. The event loop is not free, but its tail was better than plain blocking in this run (max 485 vs 687 us); those maxima are single events and vary from run to run. The medians are the stable story.

what a trading system does with this

It spins. On an isolated core, a thread polling the socket (or, with kernel bypass, the NIC's receive ring directly) sees a packet in ~1-2 us of its arrival with a tight tail; a blocking thread sees it in 5-10 us with a fat tail. The price is one core at 100% and the requirement that the core really is dedicated (a spinning thread that gets preempted is worse than a sleeping one). Control-plane threads (logging, config, monitoring) keep sleeping on epoll; only the hot path spins.

the rules

N1. Waiting costs more than working at this scale: a wake-up is ~5 us with a tail to hundreds of us; the whole UDP stack traversal is ~4 us. N2. Spin only on a dedicated, pinned core; measure the tail, not the mean. N3. Loopback numbers are an UPPER bound on what the stack costs and a LOWER bound on what a real NIC costs (no wire, no DMA, no interrupt coalescing). Use them to compare wait strategies, not to predict production latency.

take it further

1. Run `spin` with both threads on SMT siblings (change the pins to 2 and 3): the two spinners fight for one core's resources. 2. Add `SO_BUSY_POLL` (setsockopt; may need CAP_NET_ADMIN for large values) to the `block` mode: the kernel spins on your behalf for N us before sleeping: a middle ground. 3. Increase the datagram to 1400 bytes: the stack cost grows slowly (copies); the wake-up cost does not change. ============================================================

read the source · systems/03_udp_pingpong.c
#include "../tools/bench.h" #include <arpa/inet.h> #include <errno.h> #include <fcntl.h> #include <pthread.h> #include <sys/epoll.h> #include <sys/socket.h> #include <unistd.h> #define ROUNDS 50000 enum { BLOCK, EPOLL, SPIN }; static int mk(int port, int nonblock) { int fd = socket(AF_INET, SOCK_DGRAM | (nonblock ? SOCK_NONBLOCK : 0), 0); struct sockaddr_in a = { .sin_family = AF_INET, .sin_port = htons(port), .sin_addr.s_addr = htonl(INADDR_LOOPBACK) }; if (bind(fd, (struct sockaddr *)&a, sizeof a)) { perror("bind"); exit(1); } return fd; } /* Wait for a readable socket in the chosen style, then read one datagram. */ static ssize_t rx(int fd, int ep, int mode, char *buf, size_t n, struct sockaddr_in *from) { socklen_t fl = sizeof *from; if (mode == BLOCK) return recvfrom(fd, buf, n, 0, (struct sockaddr *)from, &fl); if (mode == EPOLL) { struct epoll_event ev; while (epoll_wait(ep, &ev, 1, -1) < 1) {} return recvfrom(fd, buf, n, 0, (struct sockaddr *)from, &fl); } for (;;) { ssize_t r = recvfrom(fd, buf, n, MSG_DONTWAIT, (struct sockaddr *)from, &fl); if (r >= 0 || errno != EAGAIN) return r; } } static int g_mode; static void *server(void *arg) { bench_pin(4); int fd = mk(40001, g_mode != BLOCK), ep = -1; char buf[64]; struct sockaddr_in from; if (g_mode == EPOLL) { ep = epoll_create1(0); struct epoll_event ev = { .events = EPOLLIN, .data.fd = fd }; epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); } for (int i = 0; i < ROUNDS; i++) { ssize_t r = rx(fd, ep, g_mode, buf, sizeof buf, &from); sendto(fd, buf, (size_t)r, 0, (struct sockaddr *)&from, sizeof from); } close(fd); (void)arg; return NULL; } int main(int argc, char **argv) { const char *m = argc > 1 ? argv[1] : "block"; g_mode = !strcmp(m, "epoll") ? EPOLL : !strcmp(m, "spin") ? SPIN : BLOCK; pthread_t th; pthread_create(&th, NULL, server, NULL); usleep(20000); bench_pin(2); int fd = mk(40002, g_mode != BLOCK), ep = -1; char buf[64] = "ping"; struct sockaddr_in srv = { .sin_family = AF_INET, .sin_port = htons(40001), .sin_addr.s_addr = htonl(INADDR_LOOPBACK) }, from; if (g_mode == EPOLL) { ep = epoll_create1(0); struct epoll_event ev = { .events = EPOLLIN, .data.fd = fd }; epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); } uint64_t *s = malloc(ROUNDS * sizeof *s); memset(s, 0xFF, ROUNDS * sizeof *s); for (int i = 0; i < ROUNDS; i++) { uint64_t t0 = bench_ns(); sendto(fd, buf, 16, 0, (struct sockaddr *)&srv, sizeof srv); rx(fd, ep, g_mode, buf, sizeof buf, &from); s[i] = bench_ns() - t0; } pthread_join(th, NULL); bench_stats st = bench_compute(s + 1000, ROUNDS - 1000); /* drop warm-up */ bench_print_header("round-trip ns"); bench_print(m, &st); return 0; }
◇
Finish the lab: write the evidence note

A lab is done when you can say what it showed on your machine — not when you have read it.