{"id":525,"date":"2026-09-07T22:53:16","date_gmt":"2026-09-07T20:53:16","guid":{"rendered":"https:\/\/nutanix.ma\/index.php\/2026\/09\/07\/restore-node-stuck-in-phoenix-lcm-firmware-upgrade-failure\/"},"modified":"2026-09-07T22:53:16","modified_gmt":"2026-09-07T20:53:16","slug":"restore-node-stuck-in-phoenix-lcm-firmware-upgrade-failure","status":"publish","type":"post","link":"https:\/\/nutanix.ma\/index.php\/2026\/09\/07\/restore-node-stuck-in-phoenix-lcm-firmware-upgrade-failure\/","title":{"rendered":"How to Restore a Node Stuck in Phoenix After an LCM Firmware Upgrade Failure"},"content":{"rendered":"

Every so often an LCM firmware upgrade doesn’t come back. The node was booted into Phoenix \u2014 the small Linux environment LCM uses to flash firmware \u2014 the update failed or stalled partway, and now the node is sitting at a Phoenix prompt with its CVM and hypervisor both in maintenance mode. Prism shows a host down, the cluster is running on one fewer node than it should be, and the temptation is to start clicking things.<\/p>\n

Don’t. The recovery is well-defined and mostly a matter of doing four things in the right order:<\/p>\n

    \n
  1. Reboot the node out of Phoenix and back into the hypervisor.<\/li>\n
  2. Take the CVM out of maintenance mode.<\/li>\n
  3. Take the hypervisor host out of maintenance mode.<\/li>\n
  4. Confirm cluster services are back and metadata has settled.<\/li>\n<\/ol>\n

    This is the sequence from a recent case on a six-node AHV cluster, with screenshots. Customer identifiers, addresses and serials are redacted throughout.<\/p>\n

    Step 1 \u2014 Make sure nothing is still running<\/h2>\n

    Before touching the node, confirm LCM isn’t still mid-operation. Rebooting a node while LCM believes it owns it makes the situation worse, not better. From any healthy CVM in the cluster:<\/p>\n

    nutanix@CVM~$ lcm_upgrade_status<\/code><\/pre>\n
    You want to see No task found for kLcmUpdateOperation<\/code>. Then check for an inventory task as well:<\/p>\n
    nutanix@CVM~$ ecli task.list include_completed=false operation_type_list=kLcmInventoryOperation<\/code><\/pre>\n
    That should return no tasks either. If either command shows something in progress, wait \u2014 or if it’s genuinely wedged, that’s the point to involve Nutanix Support rather than force it.<\/p>\n
    In Prism, the Hardware table will show the affected host flagged. In this case NODE1 was carrying the critical alert while the other five were up.<\/p>\n
    \"Prism
    Hardware table \u2014 one host flagged, five healthy.<\/figcaption><\/figure>\n
    Step 2 \u2014 Confirm the node is actually in Phoenix<\/h2>\n
    A node that isn’t responding on the network could be in Phoenix, or it could be hung, powered off, or stuck in BIOS. They need different fixes, so look before you act. Open the KVM console from the node’s IPMI web interface. A node in Phoenix looks exactly like this \u2014 a bare [root@phoenix ~]#<\/code> prompt and nothing else:<\/p>\n
    \"IPMI
    The Phoenix prompt in the IPMI console. This is the state we’re recovering from.<\/figcaption><\/figure>\n
    If you see this, the node isn’t broken \u2014 it’s parked. If you see something else, stop and diagnose that first.<\/p>\n
    Step 3 \u2014 Boot back into the hypervisor<\/h2>\n
    From the Phoenix prompt in the console, run the reboot script that Phoenix ships with:<\/p>\n
    phoenix \/ # python phoenix\/reboot_to_host.py<\/code><\/pre>\n
    The node reboots normally, through the Nutanix BIOS splash and into AHV. Give it time \u2014 a full boot with a CVM starting up is several minutes, and the host won’t appear healthy in Prism until the CVM is fully up.<\/p>\n
    \"IPMI
    Back through BIOS and into the hypervisor.<\/figcaption><\/figure>\n
    Step 4 \u2014 Take the host out of maintenance mode<\/h2>\n
    LCM put the CVM and the host into maintenance mode before it started, and because the operation failed, it never took them out. You do that manually. First find the host ID:<\/p>\n
    nutanix@CVM~$ ncli host list<\/code><\/pre>\n
    Then, from a healthy CVM \u2014 not the one on the node you’re recovering<\/strong>, clear maintenance mode:<\/p>\n
    nutanix@CVM~$ ncli host edit id=<host-ID> enable-maintenance-mode=false<\/code><\/pre>\n
    Verify it took:<\/p>\n
    nutanix@CVM~$ acli host.get <hypervisor-IP-address>\nnutanix@CVM~$ cluster status | grep -v UP<\/code><\/pre>\n
    The acli<\/code> output should show the host no longer in maintenance mode, and the second command should return nothing \u2014 meaning every service on every CVM reports UP. Anything it does print is a service that hasn’t come back, and worth a second look before moving on.<\/p>\n
    Step 5 \u2014 Re-enable the metadata store if the node was out for long<\/h2>\n
    Here’s the part that catches people out. If the node was in maintenance mode for more than about 30 minutes, the cluster will have detached it from the Cassandra metadata ring to protect itself. Bringing the host back doesn’t automatically re-attach it \u2014 you have to enable the metadata store on that host explicitly.<\/p>\n
    In Prism, open the host’s details from the Hardware table and use Enable Metadata Store<\/strong>:<\/p>\n
    \"Prism
    Enable Metadata Store on the recovered host.<\/figcaption><\/figure>\n
    \"Prism
    Prism confirms the metadata store is enabled on the host.<\/figcaption><\/figure>\n
    Then watch the ring rebuild. From any CVM:<\/p>\n
    nutanix@CVM~$ nodetool -h 0 ring<\/code><\/pre>\n
    Immediately after enabling, the recovered node shows as Forwarding<\/strong> while the others are Normal<\/strong>. That’s expected \u2014 metadata is being redistributed back onto it.<\/p>\n
    \"nodetool
    One node in Forwarding \u2014 the ring is rebalancing.<\/figcaption><\/figure>\n
    Keep re-running nodetool -h 0 ring<\/code> until every node reads Normal. Don’t start the next maintenance task, and don’t retry the LCM upgrade, until it does. Depending on cluster size and metadata volume this can take a while; on this cluster it was a few minutes.<\/p>\n
    \"nodetool
    All nodes Normal \u2014 the metadata ring is whole again.<\/figcaption><\/figure>\n
    Step 6 \u2014 Confirm the cluster is healthy<\/h2>\n
    Back in Prism, the home dashboard should show Cluster Resiliency OK and “Data resilient as per configuration.” Expect warnings to linger for a while \u2014 in this case, incompatible-AHV-version alerts from the interrupted upgrade \u2014 but no critical alerts, and hosts, VMs and disks all green.<\/p>\n
    \"Prism
    Resiliency OK. The cluster is back to a normal state.<\/figcaption><\/figure>\n
    Only now should you go back and look at why<\/em> the firmware upgrade failed, and whether to retry it.<\/p>\n
    Two rules worth repeating<\/h2>\n
    First: always run the maintenance-mode commands from a healthy CVM<\/strong>, not from the one on the node you’re recovering. Its services may not be fully up, and you want the cluster’s view of the host, not the host’s view of itself.<\/p>\n
    Second: if any step produces something you don’t expect, stop and contact Nutanix Support.<\/strong> This procedure is for the clean case \u2014 a node parked in Phoenix after a failed firmware flash. If the node won’t boot, the ring won’t converge, or cluster status<\/code> keeps showing services down, that’s a different problem and pushing through it on your own is how a one-node outage becomes a data problem.<\/p>\n
    If you’ve hit this and want a second pair of eyes before you act, get in touch<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Every so often an LCM firmware upgrade doesn’t come back. The node was booted into Phoenix \u2014 the small Linux environment LCM uses to flash firmware \u2014 the update failed or stalled partway, and now the node is sitting at a Phoenix prompt with its CVM and hypervisor both in maintenance mode. Prism shows a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-525","post","type-post","status-publish","format-standard","hentry","category-nutanix"],"_links":{"self":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts\/525","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/comments?post=525"}],"version-history":[{"count":0,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/posts\/525\/revisions"}],"wp:attachment":[{"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/media?parent=525"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/categories?post=525"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nutanix.ma\/index.php\/wp-json\/wp\/v2\/tags?post=525"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}