{"id":566,"date":"2026-09-13T01:53:17","date_gmt":"2026-09-12T23:53:17","guid":{"rendered":"https:\/\/nutanix.ma\/?p=566"},"modified":"2026-09-13T02:01:20","modified_gmt":"2026-09-13T00:01:20","slug":"replace-failed-drive-nutanix-cluster-verify","status":"publish","type":"post","link":"https:\/\/nutanix.ma\/index.php\/2026\/09\/13\/replace-failed-drive-nutanix-cluster-verify\/","title":{"rendered":"How to Replace a Failed Drive in a Nutanix Cluster and Verify It Worked"},"content":{"rendered":"\n

Before anything else: if you suspect a drive has failed, open a case with Nutanix Support straight away.<\/strong> Don’t pull a disk on your own judgement. Support will confirm the drive has genuinely failed rather than been marked offline for another reason, identify the correct physical slot, dispatch the replacement, and \u2014 importantly \u2014 check the cluster with you afterwards to confirm the new drive has been properly absorbed. Pulling the wrong disk from a node that has already lost one is how a routine hardware swap turns into a data-availability incident.<\/p>\n\n

What follows is the verification discipline I apply around that process. It isn’t a substitute for the Support case; it’s what makes sure the replacement genuinely finished rather than merely appearing to.<\/p>\n\n

How the failure shows up<\/h2>\n

The platform usually tells you before anyone notices a performance problem. A failing or dropped drive typically produces a run of alerts in quick succession \u2014 metadata disk not mounted, disk marked offline, and, if the drive was one of the CVM’s own boot SSDs, a critical Boot RAID Volumes degraded<\/strong> alert:<\/p>\n\n

\"Prism
The alert that matters most \u2014 all three boot RAID volumes degraded on one CVM.<\/figcaption><\/figure>\n\n

Read the summary carefully: RAID md0, md1, md2 are degraded and not syncing in progress<\/em>. All three arrays at once is the signature of losing one of the two SSDs that carry the CVM’s own operating environment. On these nodes those two drives are partitioned to do double duty \u2014 three small partitions mirrored across both disks for the CVM’s root, boot and \/home<\/code>, plus a large fourth partition on each that serves as cluster storage.<\/p>\n\n

That’s why this alert outranks a plain “disk offline”. Cluster data redundancy is handled by the distributed storage fabric across nodes, and it protects itself. The CVM’s local boot mirror is a two-disk RAID 1 with no such luxury: one disk gone means zero remaining redundancy for the CVM itself until the mirror is rebuilt.<\/p>\n\n

Confirming the picture before the swap<\/h2>\n

From the CVM, check what the OS can actually see:<\/p>\n\n

nutanix@CVM$ list_disks\n\nSlot  Disk      Model                       Serial           Size\n1     \/dev\/sda  SAMSUNG MZXXXXXXXXXX-00005  S456NYXXXXXXXX   3.84 TB\n2     \/dev\/sdb  SAMSUNG MZXXXXXXXXXX-00005  S456NCXXXXXXXX   3.84 TB\n3     \/dev\/sdc  HGST HUS000X8XXXX       VGGXXXXX         8.00 TB\n4     \/dev\/sdd  HGST HUS000X8XXXX       VGGXXXXX         8.00 TB\n5     \/dev\/sdf  HGST HUS000X8XXXX       VGGXXXXX         8.00 TB\n6     \/dev\/sde  HGST HUS000X8XXXX       VGGXXXXX         8.00 TB<\/code><\/pre>\n\n
A hybrid node like this one has two SATA SSDs and four 8 TB HDDs. Then check the mounts:<\/p>\n\n
nutanix@CVM$ df -h\n\n\/dev\/md0        9.8G  5.6G  4.1G  59%  \/\n\/dev\/md2         40G   25G   15G  64%  \/home\n\/dev\/sda4       3.4T  1.8T  1.6T  53%  \/home\/nutanix\/data\/stargate-storage\/disks\/S456NYXXXXXXXX\n\/dev\/sdb4       3.4T  1.7G  3.4T   1%  \/home\/nutanix\/data\/stargate-storage\/disks\/S456NCXXXXXXXX<\/code><\/pre>\n\n
A slot missing from list_disks<\/code>, or a mount point that has gone unresponsive, confirms what the alerts are saying. Note the asymmetry in the last two lines above \u2014 that’s the post-replacement state, and I’ll come back to it.<\/p>\n\n
Watching the platform adopt the new drive<\/h2>\n
Once the drive is physically in, the interesting question is whether the automation picked it up properly. Hades \u2014 the hardware management service \u2014 logs the whole sequence, and grepping for the new drive’s serial gives you the story end to end:<\/p>\n\n
nutanix@CVM$ grep S456NCXXXXXXXX ~\/data\/logs\/hades.out<\/code><\/pre>\n\n
Trimmed to the lines that matter:<\/p>\n\n
INFO  disk_manager.py  Handling hot-plug event for disk \/dev\/sdb, short serial: S456NCXXXXXXXX\nERROR spdk_util.py     Could not find disk with serial 'S456NCXXXXXXXX' in the Hades proto\nINFO  disk_manager.py  Mounting partition \/dev\/sdb4 on path ...\/disks\/S456NCXXXXXXXX\nINFO  disk_manager.py  Disk \/dev\/sdb with serial S456NCXXXXXXXX is stargate usable\nINFO  disk_manager.py  Running add disk pre-checks for disk S456NCXXXXXXXX\nINFO  disk_manager.py  Add disk pre-checks passed for the disk S456NCXXXXXXXX\nINFO  disk_manager.py  Adding new disk {'disk_id': <id>, 'storage_tier': 'SSD-SATA', ...} to zeus\nINFO  disk_manager.py  Choosing S456NCXXXXXXXX for oplog\nINFO  notify.py        notification=PhysicalDiskAdd disk_serial=S456NCXXXXXXXX disk_location=2\n                       action=No Action required. Disk added to data path\nINFO  disk_manager.py  Disk \/dev\/sdb with serial S456NCXXXXXXXX is successfully added to cluster<\/code><\/pre>\n\n
Two things are worth understanding here rather than just scanning for green.<\/p>\n\n
The ERROR<\/code> lines near the top look alarming and are entirely normal. The system is checking whether it already knows this serial, and it doesn’t \u2014 because the drive is new. Those errors are the expected result of a successful first sighting. People see them, assume something went wrong, and open an unnecessary case.<\/p>\n\n
The line that actually matters is the last one: successfully added to cluster<\/strong>. Until that appears, the disk exists physically but is doing nothing for the cluster. Choosing ... for oplog<\/code> just before it confirms the new SSD has also been taken into the write-ahead log rotation, which is what you want on an SSD in a hybrid node.<\/p>\n\n
Confirming the drive itself is healthy<\/h2>\n
A drive can mount cleanly and still be bad. Dead-on-arrival units are rarer than they used to be but they still happen, and you do not want to discover it later. Pull a full SMART report:<\/p>\n\n
nutanix@CVM$ sudo smartctl -x \/dev\/sdb -T permissive<\/code><\/pre>\n\n
You don’t need to read all of it. Five things are worth checking every single time:<\/p>\n\n
SMART overall-health self-assessment test result: PASSED\n\nID# ATTRIBUTE_NAME             RAW_VALUE\n  5 Reallocated_Sector_Ct      0\n  9 Power_On_Hours             1\n 12 Power_Cycle_Count          9\n197 Current_Pending_Sector     0\n199 CRC_Error_Count            0\n\nCurrent Temperature:                    29 Celsius\nSpecified Max Operating Temperature:    70 Celsius\nUnder\/Over Temperature Limit Count:     0\/0\nPercentage Used Endurance Indicator:    0<\/code><\/pre>\n\n
    \n
  • Reallocated_Sector_Ct and Current_Pending_Sector at 0.<\/strong> Anything above zero on a brand-new drive makes it a return, not a deployment.<\/li>\n
  • Power_On_Hours and Power_Cycle_Count in single digits.<\/strong> One hour and nine power cycles is a drive fresh out of its packaging. This is also your proof that you’re reading the new<\/em> disk and haven’t fat-fingered the device node \u2014 if this shows thousands of hours, you’re looking at the wrong drive.<\/li>\n
  • Percentage Used Endurance Indicator at 0<\/strong> on an SSD. No write life consumed.<\/li>\n
  • Temperature well under the specified maximum<\/strong>, with no under\/over temperature events logged.<\/li>\n
  • Overall health PASSED<\/strong>, unambiguously.<\/li>\n<\/ul>\n\n
    Together those give you evidence the drive is fit for service, rather than an assumption based on the fact that it powered on.<\/p>\n\n
    Watching the RAID rebuild \u2014 all of it<\/h2>\n
    If the replaced drive was one of the CVM’s boot SSDs, this is the step that actually restores redundancy, and it’s the one most likely to be cut short. Check the software RAID state:<\/p>\n\n
    nutanix@CVM$ cat \/proc\/mdstat\n\nmd0 : active raid1 sdb1[3] sda1[2]\n      10475520 blocks super 1.1 [2\/2] [UU]\n\nmd1 : active raid1 sdb2[3] sda2[2]\n      10475520 blocks super 1.1 [2\/2] [UU]\n\nmd2 : active raid1 sdb3[3] sda3[2]\n      41908224 blocks super 1.1 [2\/1] [U_]\n      [========>............]  recovery = 43.7% (18336064\/41908224) finish=1.9min speed=200813K\/sec<\/code><\/pre>\n\n
    The bracketed pair is the tell. [2\/2] [UU]<\/code> means both mirror members are present and in sync. [2\/1] [U_]<\/code> means only one of the two is up \u2014 the underscore is the member still catching up.<\/p>\n\n
    Notice that in the output above, md0 and md1 have already finished while md2 is still at 43.7%<\/strong>. That’s expected: md0 and md1 are 10 GB each and resync in seconds, while md2 is the 40 GB \/home<\/code> array and takes a few minutes. It’s also exactly the trap \u2014 glance at the top of the output, see two healthy arrays, and walk away while the node is still degraded. Check all three.<\/p>\n\n
    Watch it to completion with watch -d cat \/proc\/mdstat<\/code>, and don’t stop until every array reads clean with no recovery line at all:<\/p>\n\n
    md0 : active raid1 sdb1[3] sda1[2]\n      10475520 blocks super 1.1 [2\/2] [UU]\n\nmd1 : active raid1 sdb2[3] sda2[2]\n      10475520 blocks super 1.1 [2\/2] [UU]\n\nmd2 : active raid1 sdb3[3] sda3[2]\n      41908224 blocks super 1.1 [2\/2] [UU]<\/code><\/pre>\n\n
    That is the point at which the CVM has its redundancy back. The Prism alert may well have cleared before this \u2014 which is precisely why the alert clearing is not your finish line.<\/p>\n\n
    The final pass<\/h2>\n
    One last lsblk<\/code> confirms the new disk’s partitions landed in the arrays and mount points you expect, with nothing half-configured:<\/p>\n\n
    nutanix@CVM$ lsblk\n\nsdb       3.5T  disk\n\u251c\u2500sdb1     10G  part\n\u2502 \u2514\u2500md0    10G  raid1  \/\n\u251c\u2500sdb2     10G  part\n\u2502 \u2514\u2500md1    10G  raid1\n\u251c\u2500sdb3     40G  part\n\u2502 \u2514\u2500md2    40G  raid1  \/home\n\u2514\u2500sdb4    3.4T  part          \/home\/nutanix\/data\/stargate-storage\/disks\/S456NCXXXXXXXX<\/code><\/pre>\n\n
    Three partitions in the three mirrors, and the fourth mounted as cluster storage \u2014 matching its peer drive exactly.<\/p>\n\n
    Now back to that asymmetry in the earlier df -h<\/code>. The new drive’s data partition showed 1.7 GB used of 3.4 TB<\/strong>, while its peer sat at 1.8 TB. That’s not a fault \u2014 it’s an empty disk that has just joined the cluster. The storage fabric will rebalance data onto it over the following hours. Worth knowing so you don’t chase it as a problem, and worth checking again the next day to confirm it is in fact filling up. A new disk that stays empty is telling you something.<\/p>\n\n\n\n
    <\/p>\n","protected":false},"excerpt":{"rendered":"
    Before anything else: if you suspect a drive has failed, open a case with Nutanix Support straight away. Don’t pull a disk on your own judgement. Support will confirm the drive has genuinely failed rather than been marked offline for another reason, identify the correct physical slot, dispatch the replacement, and \u2014 importantly \u2014 check […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-566","post","type-post","status-publish","format-standard","hentry","category-nutanix"],"_links":{"self":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts\/566","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/comments?post=566"}],"version-history":[{"count":4,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts\/566\/revisions"}],"predecessor-version":[{"id":570,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts\/566\/revisions\/570"}],"wp:attachment":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/media?parent=566"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/categories?post=566"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/tags?post=566"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}