Traversal of packet in Linux Kernel

Linux Networking Stack

• A socket either passes a packet to the user space application or receives a packet from the implementation of the transport layer protocol.
• The network interface card (NIC) forwards the packets that it receives from the receive (RX) buffer and transmits the packets read from the transmit (TX) buffer.

Socket Buffers (sk_buff)

The kernel saves packets in C structures called sk_buff. sk_buff tracks packet metadata and maintains a start and end pointer to the packet data in memory.

Egress Path

Socket Layer

• A socket that has an associated domain (eg. AF_INET, AF_UNIX, AF_XFP) has been assigned.
• A system call function like write() or sendto() enables us to send data over the socket.
• write() invokes sockt_sendmsg(). It obtains the socket struct sock from the file descriptor by the user space application.

Transport Layer

• tcp_sendmsg() for TCP and udp_sendmsg() for UDP.

TCP

• tcp_sendmsg() waits for TCP connection establishment before allocating sk_buff and enqueue into the queue.
• Ensures Maximum Segment Size.
• Appends TCP headers and writes to existing buffer. If it does not fit, creates new buffer.
• Sets the head of the sk_buff at the starting of TCP header
• It then builds the network layer protocol as specified in the family domain (eg. AF_INET for IPv4).
• tcp_write_xmit() ensures that the kernel holds the packet during congestion and also sets the re-transmission timer.
• Transfer to next queue.

UDP

• Write the user application data into the queue
• Append UDP headers, set the destination ports and other details.
• Build the network layer protocol header
• Transfer to next queue.

Network Layer

Route → Stamp → Pack → Address

• Check the route of the packet from cache (_sk_rstdst or Forwarding Information Base Table).
• If no route, drops it.
• Applies IP flags and basic security checks.
• Performs post-routing operations and size checks.
• If greater than Maximum Transmission Unit, fragments the packet.
• Checks the mac-address from cache and builds the Ethernet Layer Header. If not present, enqueue the packet till ARP reply comes back.

Ethernet Layer

Tag → Queue → Polish →Send

• Sets the mac header, and sends the packet for traffic control.
• Sends the packet into queuing discipline (qdisc) buffer.
• Perform post-processing by calculating Ethernet checksum and VLAN Tagging.
• Pushes the the packet into (TX) Ring buffer and maps the physical memory location via Direct Memory Access so that NIC can read it and send the data.

Ingress Path

Ethernet Layer

Unload → Envelope → Dispatch → Elevate

• NIC copies the raw data and copies to kernel memory using DMA, and raises an interrupt.
• Performs checksum, MAC Address validation and strips the MAC Address header. Allocates the buffer for the packet.
• The packet enters the netif_receive_skb() and VLAN tagging along with Virtual Machine.
• The rx_handler() copies the packet if the interface is meant for bridged. It is handed off to upper layer.

IP Layer

Inspect $\rightarrow$ Sort $\rightarrow$ Reassemble $\rightarrow$ Hand Off.

• Drops foreign MAC addresses, checks IP header (version, length, checksum), sets the head to transport layer
• Checks the route if the it is local_delivery ip_local_delivery() or forward ip_forward(). Continues with the processing if it is local delivery.
• If fragmented, it adds the packet into a queue and waits for other packets.
• Strips away IP layer header and passes it above.

Network Layer

TCP

• Checks the checksum and port. Checks the state of TCP connection.
• If the connection is already established TCP_ESTABLISHED, the packet is directly copied to user memory, else kernel handles the out-of-order packet, unexpected sequence number/acknowledgement number and connection status.
• Puts it into socket receive queue.

UDP

• Performs Network Layer checks and passes it into socket receive queue.

Application Layer

• read() or recv() pulls the data from the Socket Receive Buffer and frees sk_buff()

Conclusion