TCP and UDP

The TIME-WAIT state is the source of one of the most common production networking issues. A server That rapidly opens and closes connections (e.g., an HTTP/1.1 server without keep-alive) accumulates Connections in TIME-WAIT on the client side. If the client exhausts its ephemeral port space, new Connections fail with “address already in use.” Solutions include SO_REUSEADDR/SO_REUSEPORT Socket options, the tcp_tw_reuse kernel parameter (Linux), or using connection pooling. The Default ephemeral port range on Linux is 32768-60999 (28,232 ports).

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Data |           |U|A|P|R|S|F|                               |
| Offset| Reserved  |R|C|S|S|Y|I|            Window             |
|       |           |G|K|H|R|N|N|                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |         Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options (variable)                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Data (variable)                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key fields:

TCP Flags:

TCP is a byte-stream protocol. Each byte in the stream is numbered. The Sequence Number in a segment Indicates the position of the first data byte. The Acknowledgment Number indicates the next byte the Receiver expects.

Sender sends bytes 0-1459 (1460 bytes, one full MSS segment):
  SEQ=0, LEN=1460

Receiver acknowledges:
  ACK=1460 (next expected byte)

Sender sends bytes 1460-2919:
  SEQ=1460, LEN=1460

The ISN is critical for security. If an attacker can predict the ISN, they can inject forged packets Into an established connection (TCP blind injection attack). Modern implementations use Cryptographically randomized ISNs (RFC 6056).

Without SACK, TCP uses cumulative ACKs: ACK=3000 means “I have received all bytes up to 2999.” If Segment 2 (bytes 1460-2919) is lost but segments 1, 3, and 4 arrive, the receiver can only ACK up to Byte 1459 (the last contiguous byte). The sender must retransmit segment 2 before sending new data.

With SACK (RFC 2018), the receiver can inform the sender about exactly which blocks of data have Been received:

ACK=1460, SACK=2920-4379, SACK=5840-7299

This tells the sender: “I need byte 1460, but I already have 2920-4379 and 5840-7299.” The sender Retransmits only the missing segment instead of waiting or guessing.

# Check if SACK is enabled
sysctl net.ipv4.tcp_sack
# Should be 1

# Check SACK statistics
cat /proc/net/netstat | awk '{print $17, $18}' | tail -1
# TCPSACKRecovery, TCPSACKFailures

Retransmission Timeout (RTO): TCP calculates the RTO based on measured round-trip times using The Jacobson/Karels algorithm (RFC 6298):

Where $\alpha = 0.125$ and $\beta = 0.25$. The minimum RTO is 200 ms (RFC 6298) and the maximum is 120 seconds. The initial RTO is 1 second.

Fast retransmit: If the sender receives three duplicate ACKs (same ACK number), it assumes the Segment was lost and retransmits immediately, without waiting for the RTO. This is significantly Faster than waiting for the timeout.

RTO vs fast retransmit:

Flow control prevents a fast sender from overwhelming a slow receiver. TCP uses a sliding window Mechanism:

• The receive window (rwnd) is advertised by the receiver in every ACK. It indicates how many bytes the receiver is willing to accept beyond the last acknowledged byte.
• If rwnd = 0The sender must stop sending data. The sender sends a zero-window probe (1 byte) periodically to check if the window has reopened.
• The sender never has more than rwnd bytes unacknowledged in the network.

The original TCP header has a 16-bit window field, limiting the window to 65,535 bytes. At high Bandwidth-delay products (e.g., satellite links, long-haul fiber), this limits throughput.

The Window Scaling option (RFC 7323) allows the window to be scaled by a factor of 2^{\mathrm{scale}Where scale ranges from 0 to 14:

With a scale factor of 14, the maximum window is $65,535 \times 16,384 = 1,073,725,440$ bytes (~1 GB).

The bandwidth-delay product (BDP) determines the minimum window size needed to keep the pipe full:

For a 10 Gbps link with 100 ms RTT: \mathrm{BDP = 10 \times 10^9 \times 0.1 = 10^9 bytes (~953 MB). Without window scaling, the 65,535-byte window would limit throughput to $65,535 / 0.1 = 655,350$ bytes/s (~5.2 Mbps) — a 1923x underutilization.

# Check window scaling on Linux
sysctl net.ipv4.tcp_window_scaling
# Should be 1

# View TCP connection details including window size and scale
ss -ti dst example.com:443

# Check socket buffer sizes
sysctl net.ipv4.tcp_rmem    # Receive buffer (min, default, max)
sysctl net.ipv4.tcp_wmem    # Send buffer (min, default, max)

Congestion control prevents the sender from overwhelming the network. This is distinct from flow Control (which prevents overwhelming the receiver). TCP’s congestion control is one of the most Important algorithms in networking.

The congestion window (cwnd) is a sender-side variable that limits the amount of unacknowledged Data in the network. The effective sending window is:

Throughput is then:

At connection start, cwnd is set to a small value ( 10 segments, or initcwnd on Linux). For each ACK received, cwnd increases by 1 MSS (Maximum Segment Size). This means cwnd doubles Every RTT — exponential growth.

RTT 1: cwnd = 10 MSS,  sends 10 segments
RTT 2: cwnd = 20 MSS,  sends 20 segments
RTT 3: cwnd = 40 MSS,  sends 40 segments
RTT 4: cwnd = 80 MSS,  sends 80 segments

Slow start continues until cwnd exceeds ssthresh (slow start threshold) or a loss event occurs. The initial ssthresh is set to a large value (65,535 or infinity), so slow start Transitions to congestion avoidance when loss is detected.

When cwnd reaches ssthreshTCP switches to congestion avoidance. Instead of doubling every RTT, cwnd increases by approximately 1 MSS per RTT (linear growth). For each ACK received, cwnd Increases by \mathrm{MSS \times (\mathrm{MSS / \mathrm{cwnd).

This is much slower than slow start but prevents the sender from aggressively probing for available Bandwidth after the initial ramp-up.

When three duplicate ACKs are received (indicating a single segment loss, not congestion collapse):

1. Fast Retransmit: Retransmit the missing segment immediately.
2. Set ssthresh = cwnd / 2.
3. Fast Recovery: Set cwnd = ssthresh + 3 * MSS (accounting for the three duplicate ACKs that have left the network). For each additional duplicate ACK, increment cwnd by 1 MSS. When a new ACK arrives (not duplicate), set cwnd = ssthresh and enter congestion avoidance.

When the RTO expires (indicating more severe congestion):

1. Set ssthresh = cwnd / 2.
2. Reset cwnd to 1 MSS (or initcwnd).
3. Enter slow start again.

This is a much more aggressive response than fast recovery because a timeout likely means multiple Segments were lost.

BBR (Google, 2016, RFC 9438) is a congestion control algorithm that models the network path rather Than relying solely on packet loss as a congestion signal. Traditional algorithms (Reno, CUBIC) Treat loss as congestion and reduce cwnd. BBR instead estimates two parameters:

• BtlBw (Bottleneck Bandwidth): The maximum delivery rate observed.
• RTprop (Round-Trip Propagation Time): The minimum RTT observed (without queuing delay).

BBR operates in cycles between probe bandwidth (sending at estimated bottleneck rate to measure BtlBw) and probe RTT (reducing cwnd to 4 segments to measure RTprop).

BBR significantly outperforms CUBIC on high-bandwidth, high-latency links and on links with random Loss (e.g., Wi-Fi). On a 100 Mbps link with 1% random loss, CUBIC achieves only ~30 Mbps while BBR Achieves ~95 Mbps. This is because CUBIC interprets random loss as congestion and reduces its Sending rate.

# Check current congestion control algorithm
sysctl net.ipv4.tcp_congestion_control

# List available algorithms
cat /proc/sys/net/ipv4/tcp_available_congestion_control

# Set to BBR
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Make permanent
echo "net.ipv4.tcp_congestion_control = bbr" >> /etc/sysctl.d/99-congestion.conf

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> LISTEN: passive OPEN (server)
    CLOSED --> SYN_SENT: active OPEN, send SYN
    LISTEN --> SYN_RCVD: recv SYN, send SYN+ACK
    SYN_SENT --> SYN_RCVD: recv SYN, send SYN+ACK
    SYN_RCVD --> ESTABLISHED: recv ACK
    SYN_SENT --> ESTABLISHED: recv SYN+ACK, send ACK
    ESTABLISHED --> FIN_WAIT_1: CLOSE (app), send FIN
    ESTABLISHED --> CLOSE_WAIT: recv FIN, send ACK
    FIN_WAIT_1 --> FIN_WAIT_2: recv ACK
    FIN_WAIT_2 --> TIME_WAIT: recv FIN, send ACK
    CLOSE_WAIT --> LAST_ACK: CLOSE (app), send FIN
    LAST_ACK --> CLOSED: recv ACK
    TIME_WAIT --> CLOSED: 2*MSL timeout
    LISTEN --> CLOSED: CLOSE, send RST
    SYN_RCVD --> LISTEN: recv RST
    SYN_SENT --> CLOSED: recv RST

TCP options are carried in the header after the Urgent Pointer. They are negotiated during the Three-way handshake.

MSS (Maximum Segment Size): Advertised during the handshake. Each side announces the largest Segment it can receive. The effective MSS is the minimum of both ends’ MSS values minus IP and TCP Header overhead. On Ethernet with no options: \mathrm{MSS = 1500 - 20 (\mathrm{IP) - 20 (\mathrm{TCP) = 1460 bytes.

Timestamps: Enable accurate RTT measurement (needed for RTTM, since only one RTT sample per Window is possible without timestamps) and PAWS (Protection Against Wrapped Sequences, which Prevents old segments from being accepted as new due to sequence number wrapping on high-speed Links).

Port numbers allow TCP and UDP to multiplex multiple connections on a single IP address. They are 16-bit values (0-65535).

Common well-known ports:

# View active TCP connections
ss -tanp

# View listening ports
ss -tlnp

# View ephemeral port range
sysctl net.ipv4.ip_local_port_range
# Default: 32768 60999

# Count connections in TIME-WAIT
ss -tan state time-wait | wc -l

# Count connections per state
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

TCP uses the 4-tuple (source IP, source port, destination IP, destination port) to uniquely Identify a connection. When a packet arrives, the kernel uses the destination IP and port to Identify the receiving socket, and the source IP and port to identify the specific connection.

UDP uses the same 4-tuple but does not maintain connection state. Each datagram is processed Independently.

A single server can handle thousands of connections to the same port (e.g., a web server on Port 443) because the 4-tuple is unique for each client.

Use TCP when: You need reliable, ordered delivery and the application does not want to implement Its own reliability layer. This covers the vast majority of applications.

Use UDP when: You need low latency and can tolerate loss, or when you are implementing your own Transport protocol on top of UDP (like QUIC or WebRTC).

QUIC (RFC 9000) is a transport protocol built on UDP that provides reliability, ordering, and Congestion control similar to TCP, but with several improvements:

1. 0-RTT and 1-RTT connection establishment. QUIC combines the transport and TLS handshakes, eliminating an RTT compared to TCP + TLS. On repeat connections with pre-shared keys, QUIC achieves 0-RTT data delivery.
2. Connection migration. QUIC uses connection IDs instead of 4-tuples, allowing connections to survive IP address changes (e.g., Wi-Fi to cellular handoff). TCP connections break when the IP changes because the 4-tuple changes and the kernel cannot match the packet to the existing socket.
3. No head-of-line blocking. QUIC delivers packets from independent streams independently. A lost packet in stream A does not block delivery of packets in stream B. TCP has a single byte-stream ordering that blocks all streams on loss.
4. User-space implementation. QUIC is implemented in user space (not in the kernel), allowing faster iteration and deployment without kernel upgrades. This is particularly important for protocol evolution.
5. Built-in encryption. All QUIC packets are encrypted (using TLS 1.3), preventing middlebox interference. TCP headers are in plaintext, which has led to middleboxes “optimizing” TCP in ways that break protocol evolution.

HTTP/3 uses QUIC as its transport layer, replacing TCP. HTTP/3 is documented in RFC 9114.

A SYN flood is a denial-of-service attack that exploits the three-way handshake. The attacker sends Many SYN segments without completing the handshake. The server allocates resources (TCB — Transmission Control Block) for each half-open connection, eventually exhausting its connection Table.

Mitigations:

1. SYN cookies (RFC 4987): The server does not allocate state for half-open connections. Instead, it encodes the connection state in the ISN of the SYN-ACK. When the client returns the ACK, the server reconstructs the state from the ACK. This trades CPU for memory and prevents resource exhaustion. SYN cookies have limitations: they cannot encode TCP options (MSS, window scaling, SACK), which reduces performance.

2. SYN cache: The server maintains a limited-size cache of half-open connections. When the cache is full, new SYNs are dropped. The cache can be much larger than the full TCB cache because half-open connections require less state.

3. SYN proxy: A firewall completes the three-way handshake on behalf of the server and only forwards the connection after it is established. This protects the server from SYN floods but requires the firewall to maintain state for half-open connections.

4. Rate limiting: Limit the rate of new connections per source IP. This is effective against distributed attacks when combined with source IP reputation.

# Enable SYN cookies on Linux
sysctl -w net.ipv4.tcp_syncookies=1

# Increase SYN backlog
sysctl -w net.ipv4.tcp_max_syn_backlog=8192

# Reduce SYN-ACK retries
sysctl -w net.ipv4.tcp_synack_retries=2

# Check SYN queue and accept queue sizes
ss -lnt | head -5
# Recv-Q is the SYN queue, Send-Q is the accept queue

# Buffer sizes (min, default, max in bytes)
sysctl net.ipv4.tcp_rmem    # 4096 87380 6291456 (default)
sysctl net.ipv4.tcp_wmem    # 4096 16384 4194304 (default)

# Congestion control
sysctl net.ipv4.tcp_congestion_control  # bbr or cubic

# Keepalive
sysctl net.ipv4.tcp_keepalive_time      # 7200 (2 hours)
sysctl net.ipv4.tcp_keepalive_intvl     # 75 (seconds)
sysctl net.ipv4.tcp_keepalive_probes    # 9 (probes)

# TIME-WAIT
sysctl net.ipv4.tcp_max_tw_buckets      # 262144 (max TIME-WAIT sockets)
sysctl net.ipv4.tcp_tw_reuse            # 0 (disable, set to 1 to enable)

# Retries
sysctl net.ipv4.tcp_retries1            # 3 (retries before giving up)
sysctl net.ipv4.tcp_retries2            # 15 (retries for established connections)

1. Confusing TCP’s byte-stream nature with message boundaries. TCP does not preserve application-level message boundaries. A single write() may produce multiple segments, and multiple write() calls may be coalesced into a single segment. Applications must implement their own framing (length prefix, delimiter) if message boundaries matter.

2. Ignoring TIME_WAIT accumulation. Servers that close connections rapidly (short-lived HTTP/1.1 connections) accumulate TIME_WAIT sockets on the client. Mitigate with connection pooling, tcp_tw_reuseOr SO_REUSEADDR. Do not set tcp_tw_recycle = 1 (removed in Linux 4.12) — it broke NAT.

3. Setting SO_REUSEADDR and SO_REUSEPORT incorrectly. SO_REUSEADDR allows binding to a port in TIME_WAIT. SO_REUSEPORT allows multiple sockets to bind to the same port, distributing incoming connections across them. They serve different purposes.

4. Misunderstanding cwnd and rwnd. cwnd limits sending based on network congestion. rwnd limits sending based on receiver buffer space. Throughput is limited by \min(\mathrm{rwnd, \mathrm{cwnd) / \mathrm{RTT. A full receiver buffer (rwnd = 0) stops the sender even if the network is idle.

5. UDP without application-level reliability. If your UDP application needs reliable delivery, you must implement acknowledgment, retransmission, and ordering yourself. QUIC is a transport protocol that does this for you, built on top of UDP.

6. Firewalls dropping ICMP. Path MTU Discovery relies on ICMP “Fragmentation Needed” messages. If a firewall drops all ICMP, TCP connections that send packets larger than the path MTU will hang (the sender never learns it needs to reduce the segment size). This is the classic “black hole router” problem. Allow ICMP type 3 (Destination Unreachable) code 4 (Fragmentation Needed) through your firewall.

7. NAT and connection tracking. NAT devices maintain per-connection state. High connection rates can exhaust NAT tables. UDP “connections” have shorter timeouts in NAT tables than TCP, which can cause premature state expiration for long-lived UDP flows (e.g., DNS, VPN tunnels). Use keepalives for long-lived UDP flows.

8. TCP_NODELAY and Nagle’s algorithm. Nagle’s algorithm (enabled by default) buffers small writes until the previous data is acknowledged, reducing the number of small packets. This interacts poorly with delayed ACKs (where the receiver waits up to 200 ms before sending an ACK), causing up to 200 ms of latency per write. Use TCP_NODELAY to disable Nagle’s algorithm for latency-sensitive applications (e.g., interactive shells, gaming).

# Disable Nagle's algorithm in Python
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# Disable delayed ACKs (application-level)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_QUICKACK, 1)

This topic covers the core concepts of tcp and udp, including underlying theory, practical implementation, and key applications.

Key concepts include:

• Big O notation and complexity analysis
• searching algorithms (binary, linear)
• sorting algorithms (bubble, merge, quick)
• graph algorithms (Dijkstra, BFS, DFS)
• dynamic programming

Understanding these concepts thoroughly is essential for both examinations and practical programming, and requires both theoretical knowledge and hands-on practice.

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.