Notes on Linux Router

These are notes on building a Liunx router that can route without a sub net.

-- To extend the ARP time out, see: https://wiki.fysik.dtu.dk/niflheim/System_administration#kernel-arp-cache

Kernel ARP cache

If the number of network devices (cluster nodes plus switches etc.) approach or exceed 512, you must consider the Linux kernel's limited dynamic ARP-cache size. Please read the man-page man 7 arp about the kernel's ARP-cache.

ARP (Address Resolution Protocol) is the kernel's mapping between IP-addresses (such as 10.1.2.3) and Ethernet MAC-addresses (such as 00:08:02:8E:05:F2). If the soft maximum number of entries to keep in the ARP cache, gc_thresh2=512, is exceeded, the kernel will try to remove ARP-cache entries by a garbage collection process. This is going to hit you in terms of sporadic loss of connectivitiy between pairs of nodes. No garbage collection will take place if the ARP-cache has fewer than gc_thresh1=128 entries, so you should be safe if your network is smaller than this number.

The best solution to this ARP-cache trashing problem is to increase the kernel's ARP-cache garbage collection (gc) parameters by adding these lines to /etc/sysctl.conf:

1. Don't allow the arp table to become bigger than this

net.ipv4.neigh.default.gc_thresh3 = 4096

1. Tell the gc when to become aggressive with arp table cleaning.

net.ipv4.neigh.default.gc_thresh2 = 2048

1. Adjust where the gc will leave arp table alone

net.ipv4.neigh.default.gc_thresh1 = 1024

1. Adjust to arp table gc to clean-up more often

net.ipv4.neigh.default.gc_interval = 3600

1. ARP cache entry timeout

net.ipv4.neigh.default.gc_stale_time = 3600

Then run /sbin/sysctl -p to reread this configuration file.

Another solution, although more cumbersome in daily adminsitration, is to create a static ARP database, which is customarily kept in the file /etc/ethers. It may look like this (see man 5 ethers):

00:08:02:8E:05:F2 n001 00:08:02:89:9E:5E n002 00:08:02:89:62:E6 n003 ...

This file can easily be created from the DHCP configuration file /etc/dhcpd.conf by extracting hostnames and MAC-address fields (using awk, for example). In order to add this information to the permanent ARP-cache, run the command arp -f /etc/ethers.

In order to do this at boot time, the Redhat Linux file /etc/rc.local can be used. Add these lines to /etc/rc.local:

1. Load the static ARP cache from /etc/ethers, if present

if test -f /etc/ethers then

 /sbin/arp -f /etc/ethers

fi

This configuration should be performed on all nodes and servers in the cluster, as well as any other network device that can be be configured in this way.

It doesn't hurt to use this configuration also on clusters with 128-512 network devices, since the dynamic ARP-cache will then have less work to do. However, you must maintain a consistent /etc/ethers as compared to the nodes defined in /etc/dhcpd.conf, and you must run the arp command every time the /etc/ethers file is modified (for example, when a node's network card is replaced).

System administration (last edited 2007-07-13 11:11:29 by OleHolmNielsen)

   * Immutable Page
   * Info