How to safely shutdown a Nutanix AHV Cluster

A full cluster shutdown isn’t something most Nutanix admins do often, which is exactly why it goes wrong when it’s needed — a data centre power maintenance window, a planned electrical cutover, a physical move. Unlike a single node reboot or an LCM upgrade, there’s no wizard walking you through it: you’re stopping every guest VM, then Prism Central, then the cluster software, then the CVMs, then the physical nodes, in that order, and starting everything back up in the reverse order. Skip a step or do it out of sequence and you can end up with a cluster that won’t come back cleanly.

This is the sequence I use, end to end, with the checks worth doing before you start and the gotchas on the way back up.

Step 1 — Confirm the cluster is healthy before you touch anything

Don’t shut down a cluster that isn’t in a known-good state. Run a full NCC health check from any CVM:

nutanix@CVM$ ncc health_checks run_all

Then confirm resiliency separately:

nutanix@CVM$ ncli cluster get-domain-fault-tolerance-status type=node

You’re looking for Current Fault Tolerance: 2 (or 1, depending on your configured RF). If NCC or the cluster is showing a rebuild in progress, stop here and wait for it to finish. Shutting down mid-rebuild is how a routine maintenance window turns into a data availability problem.

Step 2 — Power off the guest VMs

Shut guest VMs down cleanly from inside the guest OS wherever you can — that gives applications and databases a chance to close out properly, which a hypervisor-level power-off doesn’t guarantee. Once that’s done, confirm nothing is still running:

nutanix@CVM$ acli vm.list power_state=on

If you’ve got stragglers and need to push it from the CVM side, this one-liner shuts down every powered-on guest VM (excluding the CVMs themselves):

for i in $(acli vm.list power_state=on | grep -v NTNX | awk 'NR!=1 {print $NF}'); do acli vm.shutdown $i; done

Anything that won’t shut down gracefully after that can be force powered off — but treat that as a last resort, not the default.

Step 3 — Shut down Prism Central

Prism Central needs to come down before the cluster it’s registered to. SSH to the Prism Central VM as nutanix and stop cluster services:

nutanix@PCVM$ cluster stop

Wait for the PCVM services to actually stop, and confirm it:

nutanix@PCVM$ cluster status

Once everything shows stopped, shut down the Prism Central VM itself:

nutanix@PCVM$ sudo shutdown -h now

Step 4 — Stop the Nutanix cluster

Log on to any CVM in the cluster and stop cluster services:

nutanix@CVM$ cluster stop
nutanix@CVM$ cluster status

cluster status should confirm services are down before you move on.

Step 5 — Shut down the CVMs

Connect to each CVM in turn and power it off:

nutanix@CVM$ sudo shutdown -P now

Step 6 — Shut down each node

Two ways to do this, pick whichever fits how you work:

Method 1 — IPMI: log on to the IPMI web console for each node, go to Remote Control > Power Control, and select Power Off Server – Orderly Shutdown for a graceful shutdown.

Method 2 — from the hypervisor itself:

root@AHV# shutdown -h now

Step 7 — Powering everything back on

Bring the network up first. Power on your TOR switches, and if you’re running a stacked switch setup, make sure the stack members boot together and re-stack successfully before you go any further.

Power on the nodes — either the physical power button or through IPMI — then give it a few minutes (ten or less is usually enough) before you start checking connectivity. Ping every CVM and AHV host to confirm they’re all reachable before you try starting the cluster.

Once everything responds, SSH to any CVM and verify all CVMs are actually up:

nutanix@CVM$ svmips
nutanix@CVM$ cluster status | grep -v UP

The second command should return nothing — that means every service on every CVM reports UP. Once it’s clean, start the cluster:

nutanix@CVM$ cluster start

And confirm every service came up:

nutanix@CVM$ cluster status

(cs works as a shorthand for the same thing.)

Step 8 — Bring production back in order

Power sequencing matters here as much as it did on the way down, just reversed. Start Prism Central first. Then bring your VMs up in dependency order — infrastructure services before the applications that depend on them: domain controllers, NTP, ADCS and the like first, then your application tier — databases, web servers, and so on.

Worth repeating

The two things that actually cause outages during a planned shutdown aren’t exotic — they’re skipping the health check before you start, and powering things back on out of order. Confirm fault tolerance and rule out an in-progress rebuild before you shut anything down, and respect the dependency order in both directions. If anything looks wrong at any step — services that won’t stop, a node that won’t rejoin, resiliency that doesn’t come back — stop and get it checked rather than pushing through it.

If you’ve got a shutdown coming up and want a second pair of eyes on the plan, get in touch.