The most expensive default in TCP for request/response traffic. A client sends each 16-byte request as TWO writes (a 4-byte header, then a 12-byte body) and waits for the 16-byte reply. Nagle's algorithm holds the second small write until the first is acknowledged; the server's delayed-ACK timer holds that acknowledgment for up to 40 ms because it hopes to piggyback it on data that never comes (the server is waiting for the body). Result: every request takes ~40 ms on a link whose real latency is ~40 us. One setsockopt fixes it.
$ git clone https://github.com/adit4443ya/debug_labs.git && cd debug_labs
$ cd systems && ./build.sh
$ ./bin/04_tcp_nodelay
label min p50 p90 p99 p99.9 max mean (request ns)
default (Nagle) 326199 41089598 42030664 43227964 44547930 44547930 41187428.2
requests slower than 1 ms: 499 of 500$ ./bin/04_tcp_nodelay nodelay
TCP_NODELAY 33648 41461 65339 189330 947555 947555 50857.6
requests slower than 1 ms: 0 of 500- Default: p50 = 41.1 ms. Not 41 us: 41 MILLISECONDS, a thousand times the loopback round trip, on 499 of 500 requests. The first request (min 326 us) escaped because there was no unacknowledged data yet. - The mechanism, step by step: write(hdr) goes out immediately (nothing in flight). write(body): Nagle sees unacknowledged data (the header) and a segment smaller than the MSS, so it QUEUES the body until the header's ACK arrives. The server has the header but needs 16 bytes, so it keeps reading; its delayed-ACK logic waits (up to 40 ms on Linux) to send the ACK with the reply. Deadlock until the timer fires. Two sensible optimizations, each fine alone, deadly together. - TCP_NODELAY: p50 41 us, p99 189 us, worst 948 us: the body goes out at once and the exchange completes at loopback speed. THE ALTERNATIVES (know all three): 1. TCP_NODELAY on the socket: send every write immediately. The standard choice for request/response and for anything latency- sensitive; every exchange gateway client sets it. 2. Do not write-write-read: build the message in one buffer and write once (or writev / sendmsg). Then Nagle has nothing to hold. Good practice regardless: fewer syscalls (lab 02), one segment. 3. TCP_CORK: the opposite tool: hold partial writes on purpose until you uncork, for bulk senders that want full segments.
It tests whether you know what the kernel does between write() and the wire, whether you can explain an interaction of two features, and whether you measured (a 1000x regression that shows up only under a specific write pattern is the kind of thing a profile does not explain: the CPU is idle the whole time).
G1. Latency-sensitive TCP: TCP_NODELAY, always, and one write per message. G2. When a request/response path is slow and the CPU is idle, suspect timers: Nagle/delayed ACK (40 ms), retransmit (200 ms+), Nagle on the peer. G3. Watch it happen: `ss -ti` shows per-connection state; `tcpdump -i lo port 40003` shows the 40 ms gap between the header and the body.
1. Combine the header and body into one write and rerun without TCP_NODELAY: fast again. The pattern, not the flag, is the bug. 2. Set TCP_QUICKACK on the SERVER socket before each read instead of TCP_NODELAY on the client: also fast (the ACK is not delayed). Which side owns the fix in a real system? 3. `tcpdump -i lo -ttt port 40003` while running the default mode: read the ~40 ms gaps. ============================================================
#include "../tools/bench.h"
#include <arpa/inet.h>
#include <netinet/tcp.h>
#include <pthread.h>
#include <sys/socket.h>
#include <unistd.h>
#define REQS 500
static ssize_t read_n(int fd, char *b, size_t n) { size_t got = 0; while (got < n) { ssize_t r = read(fd, b + got, n - got); if (r <= 0) return r; got += (size_t)r; } return (ssize_t)got; }
static void *server(void *arg) {
bench_pin(4);
int ls = socket(AF_INET, SOCK_STREAM, 0), one = 1; setsockopt(ls, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);
struct sockaddr_in a = { .sin_family = AF_INET, .sin_port = htons(40003), .sin_addr.s_addr = htonl(INADDR_LOOPBACK) };
bind(ls, (struct sockaddr *)&a, sizeof a); listen(ls, 1);
int fd = accept(ls, NULL, NULL); char buf[16];
for (int i = 0; i < REQS; i++) { if (read_n(fd, buf, 16) <= 0) break; (void)!write(fd, buf, 16); } /* 16-byte request -> 16-byte reply */
close(fd); close(ls); (void)arg; return NULL;
}
int main(int argc, char **argv) {
int nodelay = argc > 1 && !strcmp(argv[1], "nodelay");
pthread_t th; pthread_create(&th, NULL, server, NULL); usleep(20000);
bench_pin(2);
int fd = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in a = { .sin_family = AF_INET, .sin_port = htons(40003), .sin_addr.s_addr = htonl(INADDR_LOOPBACK) };
if (connect(fd, (struct sockaddr *)&a, sizeof a)) { perror("connect"); return 1; }
if (nodelay) { int one = 1; setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof one); }
uint64_t *s = malloc(REQS * sizeof *s); char hdr[4] = "HDR!", body[12] = "0123456789ab", reply[16];
for (int i = 0; i < REQS; i++) {
uint64_t t0 = bench_ns();
(void)!write(fd, hdr, 4); (void)!write(fd, body, 12); /* the classic write-write-read pattern */
read_n(fd, reply, 16);
s[i] = bench_ns() - t0;
}
pthread_join(th, NULL);
bench_stats st = bench_compute(s, REQS);
bench_print_header("request ns"); bench_print(nodelay ? "TCP_NODELAY" : "default (Nagle)", &st);
long slow = 0; for (int i = 0; i < REQS; i++) slow += s[i] > 1000000; printf("requests slower than 1 ms: %ld of %d\n", slow, REQS);
return 0;
}A lab is done when you can say what it showed on your machine — not when you have read it.
It decides which problems come first, which drills open, which labs you see and how the Learn surface is organised. Change it any time from the switch in the top bar, or press Escape to look around first.