adding a Cisco ASA back into the cluster circa 8.5
Simple job, one of the cluster members was unwell and hung on a reboot and was like that for months, me being dutiful decided to fix it,
visited the DC, disconnected its sync and data cables, rebooted it, rebooted into an older image, binned the broken image, copied across the same as on the live one & booted it,it came up, i checked the to make sure it would not become live when i rejoined the cluster, connected the sync and data cables and observed on the console the formerly broken ASA copying its out of date config to the running ASA & remained in back up mode.
Massive Oh $%^&, luckily i had taken a copy of the original live ASA config before i started work and plugged in its serial cable, the console session was still live and i pasted the former live config in. Write mem and the config from the once broken ASA over wrote it again. I then pulled the other ASA's sync and data cables, pasted in the config to the live box again, wr mem, consoled into the other ASA, wr erase and reload, booted and brought it back into the cluster.
Never seen that before or since, it was the kind of unbelievable event you'd expect from a newb. Luckily i resolved the issue quickly enough that no one noticed, turned out the ASA was broken before the cluster was moved from another DC and was installed faulty. The new DC had the ports rearranged so it was connectivity was definitely broken as the old config had the wrong port assignments.
Biting the hand that feeds IT
