RHEL6 udev NIC enumeration issue on VMware

CF : http://blog.hacka.net/#post64

 

RHEL6 udev NIC enumeration issue on VMWare

Out of troublePosted by Magnus Glantz 2012-05-23 21:28:44
I recently created a standardized baseline on RHEL6.2. Doing so I had some issues with enumeration of network devices on VMWare guests.

The automated kickstart installation of RHEL is as follows on VMware:
1) RHEL6 guests are installed on a specific installation NIC
2) At the end of the installation I automatically remove the installation NIC and add two new NICs: one connected to a public VLAN and one connected to a admin VLAN.

In RHEL5 I could simply run:
# kudzu

..to get RHEL5 to discover that the installation NIC had disappeared and that two new NICs had appeared. In RHEL6, kudzu is gone and you only have the underlying udev to work with.

In RHEL6 the /lib/udev/write_net_rules script creates the udev rule: /etc/udev/rules.d/70-persistent-net.rules which sets what NIC is connected to which device (ethX).

What happened on RHEL6:
As I remove the installation NIC and add the two new NICs in one async job send to VMware ESX via a broker:
1) udev would add the two new NICs as eth1 and eth2, probably because it first handled adding the two new NICs and secondly would remove the old one (though, that didn't happen out-of-the-box).
3) udev would seemingly randomly associate the two new NICs with a device, so I could not know if eth0 indeed would be associated with the NIC with the public VLAN connected and so on. This ofcourse be because of bad behavior in VMware ESX, but still.. kudzu managed fine.

So, I would end up with:
eth0 (defunct)
eth1 (NIC1 or NIC2)
eth2 (NIC2 or NIC1)

My solution:
In RHN Satellite, I created the following replacement for the default /lib/udev/write_net_rules (which writes /etc/udev/rules.d/70-persistent-net.rules which controlls what NIC ends up as which device):

---start-file---
#!/bin/bash
# Replacement for the default /lib/udev/write_net_rules script provided by Red Hat.
# Generate udev rules to order interfaces in the same order as their bus addresses.
# Credits: Skaperen @ http://www.linuxquestions.org/questions/linux-networking-3/overriding-etc-udev-rules-d-70-persistent-net-rules-903234/

if ! test -d /sys/devices ; then
echo "${0}: ERROR: /sys appears to not be mounted" 1>&2
exit 1
fi

scansys() {
find /sys/devices -type d -wholename '/sys/devices/pci*/net' -print
}

extract() {
awk -F/ '{printf "%s eth%u\n",$(NF-1),NR-1;}'
}

generate() {
echo '# these rules are for persistence based on bus device address'
# No linebreaks below!
awk '{printf "SUBSYSTEM==\"net\", ACTION==\"add\", DRIVERS==\"?*\", ATTR{dev_id}==\"0x0\", ATTR{type}==\"1\", KERNELS==\"%s\", KERNEL==\"eth*\", NAME=\"%s\"\n",$1,$2;}'
}

scansys | extract | sort | generate >/etc/udev/rules.d/70-persistent-net.rules
---end-file---

In my kickstart, after having removed my installation NIC and added the two new NICs, I did this:

# udev hackery to get installation NIC to disappear
rm -f /etc/udev/rules.d/70-persistent-net.rules
udevadm trigger --action=change
udevadm control --reload-rules

# Poor mans firstboot that ensures networks interfaces are enumerated correct
# Keep in mind that write_net_rules is the custom script above.mv /etc/rc.d/rc.local /etc/rc.d/rc.local.bak
echo "/lib/udev/write_net_rules" >>/etc/rc.d/rc.local
echo "udevadm trigger --action=change" >>/etc/rc.d/rc.local
echo "my-network-setup-script" >>/etc/rc.d/rc.local
echo "mv /etc/rc.d/rc.local.bak /etc/rc.d/rc.local" >>/etc/rc.d/rc.local
That's it.