SMART + ZFS/RAIDZ + WRV + LsyncD + CopyFS
To quite a fantastic film: "It's not whether you're paranoid - it's whether you're paranoid _enough_."
Disks are atrociously, mind-boggingly unreliable. This is just a fact of life in 2012. Plan accordingly.
The way I protect my data from loss involves:
1) Monitoring SMART attributes of disks in cacti
1.1) Actually making a point of checking this monitoring data at least daily
2) Running short SMART checks daily
3) Running long SMART tests weekly
4) Running zpool scrubs weekly
5) Having Write-Read-Verify enabled on all disks that support it. Sadly, very few do (mainly Seagates). I wrote a patch for hdparm to add this feature which has been rolled into the release some months ago, you may want to look into upgrading to latest hdparm and using it if your disks support WRV.
6) Running lsyncd on everything to monitor all files and copy them to the warm-spare server, and to the backup server after each close following a write.
7) The backup server target location runs on CopyFS backed by ZFS with dedupe and compression enabled. so every version of a file that ever existed can be preserved (weekly cron job prunes the most ancient, most churned over of the files, dedupe and compression keep data growth relatively minimal).
Needless to say, the backup server is not at the same site as the primary and the warm spare.
Despite the relevant mentioned precautions I still had occurrences in the past of enough disks failing in a single array to hose the whole pool. But the warm spare and the near-real-time versioned backup server has always kept me out of serious trouble.
Disk are cheap and unreliable. Data is expensive and irreplaceable. Act accordingly.