How To Upgrade Firmware/Bios on Dell hardware in Debian

‎06-02-2012

As part of some investigative work we undertook, we noticed that on Squeeze our Dell hardware was having some major issues with ocsinventory - it seemed that every time ocsinventory ran, the system dropped off the network not so long afterwards.
On further investigation the culprit was actually lspci - which is part of the pciutils package. This attempts to query some Vital Product Data on the devices in question, however the firmware for the system was so old that the system didn't recognise that it even contained VPD. This eventually caused the devices to crash and bring the system down.
Looking around the Dell community mailing lists there didn't seem to be a solution to this issue running Debian - The general consensus was to upgrade the firmware on the systems in question which would at the very least require loop-mounting an iso image and/or netbooting - and at the worst would require manually going to datacentres armed with a livecd to boot from. This was not acceptable, so a solution was cooked up.
The solution and procedure is located below. This has been tested on multiple Dell hosts of varying specs with success.

WARNING! HERE BE DRAGONS!

Please note that this has been tested extensively on Squeeze - If you are wanting to do this on Lenny (or earlier) - then why not consider upgrading to an OS that is in live support with Debian and go to Squeeze first? Obviously we accept no responsibility for this, this is purely our experience with the matter. All work you undertake is at your own risk (yada yada yada etc etc)
The procedure is as follows:
vim /etc/apt/sources.list - Add in the following repositorys:

#dell repos

deb http://linux.dell.com/repo/community/deb/latest / # For most of the tools

deb http://linux.dell.com/repo cross-distro dell-firmware # for firmware

deb http://linux.dell.com/repo gutsy dell-software # for firmware-addon-dell and firmware-tools

There are a few choice packages that you will also need:
Namely these:

libsmbios2 2.2.13-0ubuntu4.2

python-libsmbios 2.2.13-0ubuntu4.2

smbios-utils 2.2.13-0ubuntu4.2

This was found after noting that older versions of these packages don't work and have bugs in. The reason you cannot have these packages (and others) in Debian? Ask Debian/Dell/Canonical. This should really be sorted by all parties involved.
A repository that you can use in order to get these files is

deb http://ubuntu.mirror.cambrium.nl/ubuntu/ lucid-updates main universe

Obviously you can find another mirror from:
http://packages.ubuntu.com/lucid-updates/i386/smbios-utils/download
Or if you want to add these into your own repository:

wget http://archive.ubuntu.com/ubuntu/pool/universe/libs/libsmbios/python-libsmbios_2.2.13-0ubuntu4.2_all.deb

wget http://archive.ubuntu.com/ubuntu/pool/universe/libs/libsmbios/smbios-utils_2.2.13-0ubuntu4.2_i386.deb

wget http://archive.ubuntu.com/ubuntu/pool/universe/libs/libsmbios/smbios-utils_2.2.13-0ubuntu4.2_amd64.deb

wget http://archive.ubuntu.com/ubuntu/pool/universe/libs/libsmbios/libsmbios2_2.2.13-0ubuntu4.2_i386.deb

wget http://archive.ubuntu.com/ubuntu/pool/universe/libs/libsmbios/libsmbios2_2.2.13-0ubuntu4.2_amd64.deb

And add in accordingly.
Then:

apt-get update apt-get install rpm compat-libstdc libstdc++5

N.B If you are running amd64, you ALSO have to install:

apt-get install ia32-libs

This is due to the Dell Update Packages themselves being 32-bit only seemingly.
Then find the appropriate driver name/number from the Dell support website (at http://support.dell.com) - Make sure when you find the driver to click 'other download formats' and find the linux equivalent name. Then with a little bit of hackery, play around with Dells downloads server (located at http://ftp.dell.com) and browse to the appropriate type of driver, then amend with the name of your file.
As some examples, once you have the links - back on the server in question:

wget http://downloads.dell.com/network/NETW_FRMW_LX_R319248.BIN

Broadcom NIC firmware - always do the NIC firmware first. There is a known bug in OMSA 6.5 that can sometimes drop the system offline otherwise.
Other examples:

wget http://downloads.dell.com/sas-non-raid/SASHBA_FRMW_LX_R199563.BIN - sas5i for a Dell 1950

wget http://downloads.dell.com/sas-raid/RAID_FRMW_LX_R197382.BIN - sas6 i/R on a Dell R300

Then:

dpkg-reconfigure dash

Make sure that dash isn't being linked to /bin/sh (select No)

chmod +x NETW_FRMW_LX_R319248.BIN (for the Broadcom firmware in the example) ./NETW_FRMW_LX_R319248.BIN

This should take you through the standard Dell Update Package installer. Press q to close the changelog and continue on with the upgrade following the prompts as necessary. Reboot afterwards to complete the upgrade.
Once you've upgraded the NIC firmware, we're good to go ahead with the install of OMSA and the Bios upgrades. New OMSA is required in order to make the Dell Update Package work for RAID controller firmware - so it's a necessary evil, not to mention the tools themselves are very useful for hardware monitoring.

apt-get install dell-firmware-repository firmware-tools

At this point you should be able to do an 'apt-cache search system-bios-poweredge-<model number>' and find the appropriate item. For a 1950:

apt-get install system-bios-poweredge-1950 apt-get install python-libsmbios libsmbios2

Then:

update_firmware

And if it matches:

update_firmware --yes

reboot

Once the server is back up, make sure that there are no remnants of any old version of OMSA around:

dpkg -l | egrep 'dell|srvadmin'

(ignoring your newly installed packages obviously - remove any others)
Then:

apt-get install srvadmin-base srvadmin-storage smbios-utils apt-get install snmp-mibs-downloader

Then:

vim /etc/default/snmpd

Make sure that the following entries match:

export MIBS=UCD-SNMP-MIB

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -p /var/run/snmpd.pid'

And make sure that: /etc/snmp/snmp.conf has the mibs: option commented out.
Then:

/etc/init.d/dataeng restart

GOTCHA

If you have issues at this point... Make SURE that you have these smbios packages installed:

dpkg -l | grep smbios

ii libsmbios2 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- dynamic library

ii python-libsmbios 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- python libraries

ii smbios-utils 2.2.13-0ubuntu4.2 Provide access to (SM)BIOS information -- utility binaries

Without them, stuff will fail.
And you should then be able to identify accurately your Raid controller:

/opt/dell/srvadmin/bin/omreport storage controller

Using this, you do your hackery on Dells support website (as described earlier) to find the appropriate driver for it, wget it again, chmod +x it, and run it.

wget http://downloads.dell.com/sas-non-raid/SASHBA_FRMW_LX_R199563.BIN

chmod +x SASHBA_FRMW_LX_R199563.BIN

./SASHBA_FRMW_LX_R199563.BIN

Reboot one last time, and now you should be running latest firmware, on a live system.

TESTING

To test that the lspci bugs are now gone, you should be able to do:

lspci -vv | grep vpd

And nothing should be showing.
To be sure, just do another lspci and scroll through, making sure everything reports without errors.
And the acid test (if you have it):

ocsinventory-agent --debug

BACKPORTED KERNEL WITH LATEST DEBIAN FIRMWARE

If there are further issues, you can always run a backported kernel..:
vim /etc/apt/sources.list, add in squeeze-backports:

deb http://backports.debian.org/debian-backports squeeze-backports main non-free

apt-get update

apt-get -t squeeze-backports install install linux-image-2.6.38-bpo.2-686-bigmem firmware-linux-nonfree firmware-linux-free firmware-bnx2

Obviously making sure that the kernel you want is the appropriate version for your system...
And reboot.
Job done.