Introduction
In part One I got myself a Netapp DS2246 a LSI Megaraid controller card, the right SAS cable, and hooked everything up. At that point I thought that I had my storage quickly setup and running. But I was wrong. Very wrong. For some reasons the RAID controller did see the disks, which is a good thing. However the controller marked these drives as “Unsupported”.
Next step is to figure out what went wrong
In part two I already mentioned that I’m a network guy. Yeah sure, way back I once was a server dude, messing around with 24/7 clusters, fiber channel SCSI and alike. Well actually I played around with DL380 gen 1 servers a lot. Anyhow, at this point I didn’t know if it’s a problem with the disks, or with the SAS RAID controller. Luckily I have to extra 2.5″ spare disks. Since my setup is based on RAID 0 I really like to have a spare disk or two. (Well actually I always buy spare disks, just in case).
So I removed two disk from the DS2246, and swapped them out with my spare disks. And low and behold: after doing a re-scan of the disks the LSI Mega RAID controller recognized the disks, and I could configure them as RAID 0 or RAID 1 disk. So that proved to me that the DS2246 is good, the SAS cable is good, the LSI Mega RAID controller is good. Since all the disks in DS2246 are giving a green LED, I figured that the disks must be good also. But why doesn’t the RAID controller support them? Maybe firmware ?
The first rookie mistake
In part one I already mentioned that I made a rookie mistake. And that was: upgrading the firmware. I upgraded the LSI Mega raid card to 23.34.0. This resulted in a crashing WebBIOS. Once I entered the WebBIOS it just hangs. I also got a memory conflict error at startup. So at the end I could do a downgrade. To upgrade these cards, it’s just a matter of getting a tool “storcli”.
Upgrading the card of downgrading the card is done by using the command:
storcli64 /c# download file=firmwarefile nosigchk noverchk
Where # is controller card number, and firmwarefile the downloaded firmware. The firmware for this card can be found at: 9286CV-8e firmware
Well that was a luckily escape. And while I could not enter the WebBIOS the drives didn’t show up, so it was not a firmware issue.
After googling I found people mention that Netapp formats it’s disk with a different sector size. Instead of the usual 512 byte sector, Netapp uses 520 bytes. And once I read that I knew that the sector size is the problem. So how to get these drives to work? Well as it turns out, the drives can be reformatted to a sector size of 512 byes. The problem is: how to that, since the RAID controller doesn’t support the drives as is with the 520 byte sector size.
Back to the HBA mode or IT mode
In part one I talked about HBA mode or also called IT mode. In this mode the RAID controller card is in pass through mode, it just presents the disk to the OS, without any interfering. So I need to get my SAS controller card into HBA mode. Unfortunately the LSI Mega RAID 9286CV-8e card I got doesn’t support that. However the internal SAS controller card (Smart Array P420i) card in my HP DL380p gen8 supports it. The card doesn’t support it by default, but with a little trick the card can be put in HBA mode.
There is one downside however. Once the P420i card is in HBA mode, it’s no longer possible to boot from the disks. This means that I have to reconfigure my server since I boot from a RAID 0 set. However I hope it’s easy to convert back from HBA mode, to RAID mode, and can just place the disks back, without any data loss. However, since I’m not sure. And things can go wrong, I started to backup everything, just in case.
Onto the path of victory and success
Since I have two spare disks which are working with the LSI card, and the DS2246 and the possibility to put the P420i card in HBA mode I maybe could get this to work. The plan I got is this:
-
-
- Get two disk in a RAID 0 set working with DS2246 and LSI card
- Get this logical RAID 0 disk to be boot-able
- Backup the data from existing disks (2x logic drive) connected to the P420I card
- Remove the existing disks( 2x logic drive) from the P420I card
- Install Ubuntu 20.0.4 on new created RAID 0 disk
- Boot into Ubuntu 20.0.4
- Use the ssacli tool to put the Smart Array P420i card into HBA mode
- Use lssci tool and sg_scan tool to see if the Netapp drives with 520 bytes sector are accesable
- Reform the Netapp drives to 512 byte sector
- Plug the reformatted drive back into the DS2246
- Test with sg_scan if the drive works
-
To execute this plan basically I have to reinstall my server, this is a lot of work, but at the end I should end up with 24 drives working.
The key thing in this plan is to get the HP Smart Array P420i card into HBA mode. And I must be able to boot from disk connected to the DS2246.
As it turns out, getting to boot from the DS2246 was easy. Installing Ubuntu onto the disk, was also easy.
Getting the Smart Array P420i into HBA mode
The next step was to install the tool to get the Smart Array into HBA mode.To get this controller into HBA mode it needs to have recent firmware. I run 8.32 and as you will see, that works fine. Since I removed all the drives from the controller, I didn’t had to clear the configuration.
If there is older firmware installed, try to get your hands on the HP SSP for DL380p gen 8 (or what ever genaration server you got for that matter)
I followed the steps documented here
It comes down to:
Setup the repository:
I added in /etc/apt/sources.list.d/mcp.list:
deb http://downloads.linux.hpe.com/SDR/repo/mcp focal/current non-free
Next I added the HPE Public Keys:
curl https://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add - curl https://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add - curl https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
Next I updated the apt sources with:
apt-get update
And finally:
apt-get install ssacli
Next is to get the controller into HBA mode:
ssacli controller slot=0 modify hbamode=on
To check if the controller is in HBA mode:
ssacli controller slot=0 show
Which outputs:
Smart Array P420i in Slot 0 (Embedded) Bus Interface: PCI Slot: 0 Serial Number: 0014380225BD250 Cache Serial Number: PBKUC0ARH2P0SK RAID 6 Status: Enabled Controller Status: OK Hardware Revision: B Firmware Version: 8.32 Firmware Supports Online Firmware Activation: False Cache Board Present: True Cache Status: Not Configured Total Cache Size: 1.0 Total Cache Memory Available: 0.8 Battery Backed Cache Size: 0.8 Cache Backup Power Source: Capacitors Battery/Capacitor Count: 1 Battery/Capacitor Status: OK Controller Temperature (C): 48 Cache Module Temperature (C): 29 Capacitor Temperature (C): 22 Number of Ports: 2 Internal only Driver Name: hpsa Driver Version: 3.4.20 HBA Mode Enabled: True PCI Address (Domain:Bus:Device.Function): 0000:02:00.0 Port Max Phy Rate Limiting Supported: False Host Serial Number: CZ22280G56 Sanitize Erase Supported: False Primary Boot Volume: Unknown (600508B1001C83E36DFBA10AEBE3971A) Secondary Boot Volume: None
Accessing the Netapp drive from the OS
This looks good. Next I placed one of the Netapp drives into the DL380P server, and checked if I could see the drive:
ssacli controller slot=0 physicaldrive all show Smart Array P420i in Slot 0 (Embedded) Unsupported Drives physicaldrive 1I:2:1 (port 1I:box 2:bay 1, SAS, 1.2 TB, OK)
This looks good, let’s see what more we can find out:
from dmesg:
[ 565.080210] sd 3:0:1:0: Attached scsi generic sg3 type 0 [ 565.080448] sd 3:0:1:0: [sdb] Unsupported sector size 520. [ 565.080805] sd 3:0:1:0: [sdb] 0 512-byte logical blocks: (0 B/0 B) [ 565.080808] sd 3:0:1:0: [sdb] 520-byte physical blocks [ 565.081074] sd 3:0:1:0: [sdb] Write Protect is off [ 565.081076] sd 3:0:1:0: [sdb] Mode Sense: f7 00 10 08 [ 565.081452] sd 3:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 565.130243] sd 3:0:1:0: [sdb] Unsupported sector size 520. [ 565.138773] sd 3:0:1:0: [sdb] Attached SCSI disk
At this point I plugged two drives into the DL380P and lets see what we can doo with sg utilities:
sg_map /dev/sg0 /dev/sr0 /dev/sg1 /dev/sg2 /dev/sda /dev/sg3 /dev/sdb
Next lets see it we can re-format the drive:
sg_format -v --format --size=512 /dev/sg3 NETAPP X425_HCBEP1T2A10 NA01 peripheral_type: disk [0x0] PROTECT=1 << supports protection information>> Unit serial number: KZHLXDBF LU name: 5000cca01d5ac328 mode sense(10) cdb: 5a 00 01 00 00 00 00 00 fc 00 Mode Sense (block descriptor) data, prior to changes: Number of blocks=2344225968 [0x8bba0cb0] Block size=520 [0x208] mode select(10) cdb: 55 11 00 00 00 00 00 00 1c 00 A FORMAT UNIT will commence in 15 seconds ALL data on /dev/sg3 will be DESTROYED Press control-C to abort A FORMAT UNIT will commence in 10 seconds ALL data on /dev/sg3 will be DESTROYED Press control-C to abort A FORMAT UNIT will commence in 5 seconds ALL data on /dev/sg3 will be DESTROYED Press control-C to abort Format unit cdb: 04 18 00 00 00 00 Format unit has started
The format takes a while, but after some time:
FORMAT UNIT Complete
And after doing a sg_scan dmesg confirmed:
[29086.192424] hpsa 0000:02:00.0: scsi 3:0:1:0: updated Direct-Access NETAPP X425_HCBEP1T2A10 PHYS DRV SSDSmartPathCap- En- Exp=1 [29116.364679] hpsa 0000:02:00.0: SCSI status: LUN:0000000000800001 CDB:12010000040000000000000000000000 [29116.364684] hpsa 0000:02:00.0: SCSI Status = 02, Sense key = 0x05, ASC = 0x25, ASCQ = 0x00 [29116.364963] hpsa 0000:02:00.0: Acknowledging event: 0x80000002 (HP SSD Smart Path configuration change) [29116.398781] hpsa 0000:02:00.0: scsi 3:0:1:0: removed Direct-Access NETAPP X425_HCBEP1T2A10 PHYS DRV SSDSmartPathCap- En- Exp=1 [29210.540998] scsi 2:0:83:0: Direct-Access NETAPP X425_HCBEP1T2A10 NA01 PQ: 0 ANSI: 6 [29210.542399] sd 2:0:83:0: Attached scsi generic sg3 type 0 [29210.559470] sd 2:0:83:0: [sdb] 2344225968 512-byte logical blocks: (1.20 TB/1.09 TiB) [29210.577646] sd 2:0:83:0: [sdb] Write Protect is off [29210.577652] sd 2:0:83:0: [sdb] Mode Sense: f7 00 10 08 [29210.614517] sd 2:0:83:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [29211.011025] sd 2:0:83:0: [sdb] Attached SCSI disk
So now 23 disks to go…
Getting the server to boot from the HP Smart Array controller
After I formatted all the 24 disks, and checked in WebBIOS that I actually could use the disks, it’s time to revert the server back into the configuration it was before I messed around with the Netapp DS2246.
So from Ubuntu, which was booted from the DS2246, I reconfigured the Smart Array back to RAID mode with:
ssacli controller slot=0 modify hbamode=off
After that I inserted the disks back in the order that I took them out (when I took the disk out, I marked them with a sharpy).
At that point I pulled out the 2 disks of the Netapp ds2246, and rebooted the server. And sure enough: The Smart Array controller detected 2x Logical Raid 0 device. And booted without any issue the original Ubuntu installation. Even the data on the second disk was intact. That saved me a lot of time. Now all I have to do is to configured the desired raid sets, and start moving data around.
Conclusion
Getting a netapp DS2246 and getting Netapp 520 byte sectors to work, is not trivial, but a quite a learning experience. When you decide to get yourself a Netapp DS2246 or similair and it comes with disk with 520 bytes sectors, make sure to have a SAS controller card which can do at least HBA or IT mode. If you can get your hands on a card which support both, thus HBA and RAID go for it. And you will get it to work.
For the LSI controller, it works fine however, I’m not a big fan of the whole WebBIOS interface. To be honest I find it horrible. Adding a disk to an existing array is very hard, and confusing process.
