A full cluster shutdown isn’t something most Nutanix admins do often, which is exactly why it goes wrong when it’s needed \u2014 a data centre power maintenance window, a planned electrical cutover, a physical move. Unlike a single node reboot or an LCM upgrade, there’s no wizard walking you through it: you’re stopping every guest VM, then Prism Central, then the cluster software, then the CVMs, then the physical nodes, in that order, and starting everything back up in the reverse order. Skip a step or do it out of sequence and you can end up with a cluster that won’t come back cleanly.<\/p>\n\n
This is the sequence I use, end to end, with the checks worth doing before you start and the gotchas on the way back up.<\/p>\n\n
Don’t shut down a cluster that isn’t in a known-good state. Run a full NCC health check from any CVM:<\/p>\n\n
nutanix@CVM$ ncc health_checks run_all<\/code><\/pre>\n\nThen confirm resiliency separately:<\/p>\n\n
nutanix@CVM$ ncli cluster get-domain-fault-tolerance-status type=node<\/code><\/pre>\n\nYou’re looking for Current Fault Tolerance: 2<\/code> (or 1<\/code>, depending on your configured RF). If NCC or the cluster is showing a rebuild in progress, stop here and wait for it to finish. Shutting down mid-rebuild is how a routine maintenance window turns into a data availability problem.<\/p>\n\nStep 2 \u2014 Power off the guest VMs<\/h2>\n
Shut guest VMs down cleanly from inside the guest OS wherever you can \u2014 that gives applications and databases a chance to close out properly, which a hypervisor-level power-off doesn’t guarantee. Once that’s done, confirm nothing is still running:<\/p>\n\n
nutanix@CVM$ acli vm.list power_state=on<\/code><\/pre>\n\nIf you’ve got stragglers and need to push it from the CVM side, this one-liner shuts down every powered-on guest VM (excluding the CVMs themselves):<\/p>\n\n
for i in $(acli vm.list power_state=on | grep -v NTNX | awk 'NR!=1 {print $NF}'); do acli vm.shutdown $i; done<\/code><\/pre>\n\nAnything that won’t shut down gracefully after that can be force powered off \u2014 but treat that as a last resort, not the default.<\/p>\n\n
Step 3 \u2014 Shut down Prism Central<\/h2>\n
Prism Central needs to come down before the cluster it’s registered to. SSH to the Prism Central VM as nutanix<\/code> and stop cluster services:<\/p>\n\nnutanix@PCVM$ cluster stop<\/code><\/pre>\n\nWait for the PCVM services to actually stop, and confirm it:<\/p>\n\n
nutanix@PCVM$ cluster status<\/code><\/pre>\n\nOnce everything shows stopped, shut down the Prism Central VM itself:<\/p>\n\n
nutanix@PCVM$ sudo shutdown -h now<\/code><\/pre>\n\nStep 4 \u2014 Stop the Nutanix cluster<\/h2>\n
Log on to any CVM in the cluster and stop cluster services:<\/p>\n\n
nutanix@CVM$ cluster stop\nnutanix@CVM$ cluster status<\/code><\/pre>\n\ncluster status<\/code> should confirm services are down before you move on.<\/p>\n\nStep 5 \u2014 Shut down the CVMs<\/h2>\n
Connect to each CVM in turn and power it off:<\/p>\n\n
nutanix@CVM$ sudo shutdown -P now<\/code><\/pre>\n\nStep 6 \u2014 Shut down each node<\/h2>\n
Two ways to do this, pick whichever fits how you work:<\/p>\n\n
Method 1 \u2014 IPMI:<\/strong> log on to the IPMI web console for each node, go to Remote Control > Power Control<\/strong>, and select Power Off Server \u2013 Orderly Shutdown<\/strong> for a graceful shutdown.<\/p>\n\nMethod 2 \u2014 from the hypervisor itself:<\/strong><\/p>\n\nroot@AHV# shutdown -h now<\/code><\/pre>\n\nStep 7 \u2014 Powering everything back on<\/h2>\n
Bring the network up first. Power on your TOR switches, and if you’re running a stacked switch setup, make sure the stack members boot together and re-stack successfully before you go any further.<\/p>\n\n
Power on the nodes \u2014 either the physical power button or through IPMI \u2014 then give it a few minutes (ten or less is usually enough) before you start checking connectivity. Ping every CVM and AHV host to confirm they’re all reachable before you try starting the cluster.<\/p>\n\n
Once everything responds, SSH to any CVM and verify all CVMs are actually up:<\/p>\n\n
nutanix@CVM$ svmips\nnutanix@CVM$ cluster status | grep -v UP<\/code><\/pre>\n\nThe second command should return nothing \u2014 that means every service on every CVM reports UP. Once it’s clean, start the cluster:<\/p>\n\n
nutanix@CVM$ cluster start<\/code><\/pre>\n\nAnd confirm every service came up:<\/p>\n\n
nutanix@CVM$ cluster status<\/code><\/pre>\n\n(cs<\/code> works as a shorthand for the same thing.)<\/p>\n\nStep 8 \u2014 Bring production back in order<\/h2>\n
Power sequencing matters here as much as it did on the way down, just reversed. Start Prism Central first. Then bring your VMs up in dependency order \u2014 infrastructure services before the applications that depend on them: domain controllers, NTP, ADCS and the like first, then your application tier \u2014 databases, web servers, and so on.<\/p>\n\n
Worth repeating<\/h2>\n
The two things that actually cause outages during a planned shutdown aren’t exotic \u2014 they’re skipping the health check before you start, and powering things back on out of order. Confirm fault tolerance and rule out an in-progress rebuild before you shut anything down, and respect the dependency order in both directions. If anything looks wrong at any step \u2014 services that won’t stop, a node that won’t rejoin, resiliency that doesn’t come back \u2014 stop and get it checked rather than pushing through it.<\/p>\n\n