[ome-users] Storage issues

[Benjamin Schmid]

Dear all,

This question is not really related to OMERO, but maybe some of you have 
come across this before:

We have a Thecus storage system (N16000pro) that's configured as a RAID 
6 and connected via iSCSI to a machine that runs an OMERO server. The 
Thecus system provides two LUNs (it's 2 because the maximum size of a 
LUN is 16 TB). They show up on the server as 2 partitions, /dev/sdc1 and 
/dev/sdd1. LVM2 is used to combine the 2 partitions into one logical 
volume (/dev/vg0/lv0).

When I tried to reboot the server today, the logical volume wasn't 
mounted and syslog shows lots of scary error messages:
---
Sep  4 13:50:43 romulus kernel: [   20.964830] sd 8:0:0:1: [sdc] tag#0 
FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sep  4 13:50:43 romulus kernel: [   20.964842] sd 8:0:0:1: [sdc] tag#0 
Sense Key : Not Ready [current]
Sep  4 13:50:43 romulus kernel: [   20.964847] sd 8:0:0:1: [sdc] tag#0 
Add. Sense: Logical unit communication failure
Sep  4 13:50:43 romulus kernel: [   20.964852] sd 8:0:0:1: [sdc] tag#0 
CDB: Write(10) 2a 00 b8 04 06 01 00 00 01 00
Sep  4 13:50:43 romulus kernel: [   20.964856] blk_update_request: I/O 
error, dev sdc, sector 24698171400
Sep  4 13:50:43 romulus kernel: [   20.964861] Buffer I/O error on dev 
dm-0, logical block 7378305025, lost async page write
Sep  4 13:50:43 romulus kernel: [   20.966575] JBD2: recovery failed
Sep  4 13:50:43 romulus kernel: [   20.966578] EXT4-fs (dm-0): error 
loading journal
Sep  4 13:50:43 romulus mount[1690]: mount: /dev/mapper/vg0-lv0: can't 
read superblock
Sep  4 13:50:43 romulus systemd[1]: srv.mount: Mount process exited, 
code=exited status=32
Sep  4 13:50:43 romulus systemd[1]: Failed to mount /srv.
---

I checked the filesystem with fsck, and eventually let fsck everything 
it suggested. I could mount the volume again, but since then I'm getting 
lots of errors (dmesg):
[ 2822.725945] blk_update_request: I/O error, dev sdc, sector 24358432768
[ 2822.731819] sd 8:0:0:1: [sdc] tag#19 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[ 2822.731829] sd 8:0:0:1: [sdc] tag#19 Sense Key : Not Ready [current]
[ 2822.731846] sd 8:0:0:1: [sdc] tag#19 Add. Sense: Logical unit 
communication failure
[ 2822.731853] sd 8:0:0:1: [sdc] tag#19 CDB: Write(10) 2a 00 b5 84 06 00 
00 00 10 00

I'm now wondering what goes wrong. In the Thecus OS, I run the SMART 
utility on each hard-drive, and all seem to be fine. Is it on the RAID 
level where the error occurs (instead of the HDD level)? Or is it the 
LVM2 level? Is there anything I can do from here?

I'm more than grateful for any ideas or suggestions.

Again, please aplogize this off-topic question. If you know any better 
place where to put it, please let me know.

Best wishes,
Benjamin