How To Upgrade Firmware/Bios on Dell hardware in Debian
As part of some investigative work we undertook, we noticed that on Squeeze our Dell hardware was having some major issues with ocsinventory - it seemed that every time ocsinventory ran, the system dropped off the network not so long afterwards.
On further investigation the culprit was actually lspci - which is part of the pciutils package. This attempts to query some Vital Product Data on the devices in question, however the firmware for the system was so old that the system didn't recognise that it even contained VPD. This eventually caused the devices to crash and bring the system down.
Looking around the Dell community mailing lists there didn't seem to be a solution to this issue running Debian - The general consensus was to upgrade the firmware on the systems in question which would at the very least require loop-mounting an iso image and/or netbooting - and at the worst would require manually going to datacentres armed with a livecd to boot from. This was not acceptable, so a solution was cooked up.
The solution and procedure is located below. This has been tested on multiple Dell hosts of varying specs with success.
WARNING! HERE BE DRAGONS!Please note that this has been tested extensively on Squeeze - If you are wanting to do this on Lenny (or earlier) - then why not consider upgrading to an OS that is in live support with Debian and go to Squeeze first? Obviously we accept no responsibility for this, this is purely our experience with the matter. All work you undertake is at your own risk (yada yada yada etc etc)
The procedure is as follows:
vim /etc/apt/sources.list - Add in the following repositorys:
deb http://linux.dell.com/repo/community/deb/latest / # For most of the tools
deb http://linux.dell.com/repo cross-distro dell-firmware # for firmware
deb http://linux.dell.com/repo gutsy dell-software # for firmware-addon-dell and firmware-tools
There are a few choice packages that you will also need:
This was found after noting that older versions of these packages don't work and have bugs in. The reason you cannot have these packages (and others) in Debian? Ask Debian/Dell/Canonical. This should really be sorted by all parties involved.
A repository that you can use in order to get these files is
deb http://ubuntu.mirror.cambrium.nl/ubuntu/ lucid-updates main universe
Obviously you can find another mirror from:
Or if you want to add these into your own repository:
And add in accordingly.
apt-get update apt-get install rpm compat-libstdc libstdc++5
N.B If you are running amd64, you ALSO have to install:
apt-get install ia32-libs
This is due to the Dell Update Packages themselves being 32-bit only seemingly.
Then find the appropriate driver name/number from the Dell support website (at http://support.dell.com) - Make sure when you find the driver to click 'other download formats' and find the linux equivalent name. Then with a little bit of hackery, play around with Dells downloads server (located at http://ftp.dell.com) and browse to the appropriate type of driver, then amend with the name of your file.
As some examples, once you have the links - back on the server in question:
Broadcom NIC firmware - always do the NIC firmware first. There is a known bug in OMSA 6.5 that can sometimes drop the system offline otherwise.
wget http://downloads.dell.com/sas-non-raid/SASHBA_FRMW_LX_R199563.BIN - sas5i for a Dell 1950
wget http://downloads.dell.com/sas-raid/RAID_FRMW_LX_R197382.BIN - sas6 i/R on a Dell R300
Make sure that dash isn't being linked to /bin/sh (select No)
chmod +x NETW_FRMW_LX_R319248.BIN (for the Broadcom firmware in the example) ./NETW_FRMW_LX_R319248.BIN
This should take you through the standard Dell Update Package installer. Press q to close the changelog and continue on with the upgrade following the prompts as necessary. Reboot afterwards to complete the upgrade.
Once you've upgraded the NIC firmware, we're good to go ahead with the install of OMSA and the Bios upgrades. New OMSA is required in order to make the Dell Update Package work for RAID controller firmware - so it's a necessary evil, not to mention the tools themselves are very useful for hardware monitoring.
apt-get install dell-firmware-repository firmware-tools
At this point you should be able to do an 'apt-cache search system-bios-poweredge-<model number>' and find the appropriate item. For a 1950:
apt-get install system-bios-poweredge-1950 apt-get install python-libsmbios libsmbios2
And if it matches:
Once the server is back up, make sure that there are no remnants of any old version of OMSA around:
dpkg -l | egrep 'dell|srvadmin'
(ignoring your newly installed packages obviously - remove any others)
apt-get install srvadmin-base srvadmin-storage smbios-utils apt-get install snmp-mibs-downloader
Make sure that the following entries match:
SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -p /var/run/snmpd.pid'
And make sure that: /etc/snmp/snmp.conf has the mibs: option commented out.
GOTCHAIf you have issues at this point... Make SURE that you have these smbios packages installed:
dpkg -l | grep smbios
ii libsmbios2 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- dynamic library
ii python-libsmbios 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- python libraries
ii smbios-utils 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- utility binaries
Without them, stuff will fail.
And you should then be able to identify accurately your Raid controller:
/opt/dell/srvadmin/bin/omreport storage controller
Using this, you do your hackery on Dells support website (as described earlier) to find the appropriate driver for it, wget it again, chmod +x it, and run it.
chmod +x SASHBA_FRMW_LX_R199563.BIN
Reboot one last time, and now you should be running latest firmware, on a live system.
TESTINGTo test that the lspci bugs are now gone, you should be able to do:
lspci -vv | grep vpd
And nothing should be showing.
To be sure, just do another lspci and scroll through, making sure everything reports without errors.
And the acid test (if you have it):
BACKPORTED KERNEL WITH LATEST DEBIAN FIRMWAREIf there are further issues, you can always run a backported kernel..:
vim /etc/apt/sources.list, add in squeeze-backports:
deb http://backports.debian.org/debian-backports squeeze-backports main non-free
apt-get -t squeeze-backports install install linux-image-2.6.38-bpo.2-686-bigmem firmware-linux-nonfree firmware-linux-free firmware-bnx2
Obviously making sure that the kernel you want is the appropriate version for your system...