OpenVZ
We have a pair of OpenVZ servers running the containers from common NAS (meaning it's a matter of seconds to push a container from one to the other).
I wrote the following function:
clusterfuck () {
running=`vzlist| grep running| awk '{print $1}'`;
if [ -z "$*" ]; then echo $running;
else for i in $running; do
echo "### $i ###";
vzctl exec2 $i "$*";
done;
fi
}
(named because of what happens if you're careless with it), and made one of the OVZ boxes AptCacher proxy. All the updates get picked up, but only the template container (DAMP) updates automatically, so if an update causes problems the Nagios starts screaming about the template.
Updating all the containers on a box is a matter of
root@VZserver# clusterfuck "aptitude update && aptitude safe-upgrade && aptitude clean"
Updating the boxes themselves requires shoving the containers to the other box, updating, rebooting if there's a new kernel, checking everything, and bringing the containers back.