Over writing files is hard
File systems no longer decide where data is stored on a disk-drive [0].
Gone are the days of data being stored in a known physical location on a disk (cylinder, head, sector). CHS has long since been replaced by Logical Block Address - LBA. Here a separate computer, embedded in the disk-drive[1] is now responsible for deciding where on the disk-drive[2] the data should actually reside.
By allowing the disk-drive[3] controller to map logical blocks to physical storage various magics can be achieved, namely:
o Faster performance - data blocks may span multiple physical entities (i.e. multiple heads of a spinning disk or several FLASH devices)
o Even Faster performance in Solid State Disks - Blocks of FLASH need to be erased before they can be re-written, by dynamically mapping Logical blocks to the next 'empty' physical block there is no need to wait for the physical block to be erased before it can be re-used. Old physical blocks can then be erased by the controller when host has finished accessing the device. this feature is also known as dynamic wear levelling, because it has the affect of moving the physical storage around all available blocks, and therefore increasing the service life of the device when the same Logical Block address is repeatedly written.
o Improved production yield - Production of storage is not perfect, sometimes not every physical block works reliably when manufactured. By over-provisioning physical blocks bad blocks may be mapped out whilst still maintaining the advertised capacity of the disk-drive[4].
o Improved longevity - Over provisioning capacity by more than statistically necessary to ensure a satisfactory yield means spare blocks are now available when a physical block fails [6]
o Increased longevity - Mostly pertaining to solid state devices, but applies to a lesser extent to spinning disks, each block only has a finite life expectancy (SLC FLASH lasts longer than TLC which lasts longer than MLC), so wear static levelling is applied. Unlike dynamic wear levelling described above (as a performance enhancement feature), here the controller copies blocks of data that have low erase counts into blocks that have higher counts in an attempt to even out the service life of the drive so that all blocks age at a similar rate.
Of cause if your disk-drive AND file system support the TRIM command then deleted files can be simply put back into the pool of free physical blocks and the disk-drive's controller will erase the physical data when it has some spare time... However this will not erase old fragments of a file that has been updated but there are un-errased physical blocks waiting in the dirty queue.
I believe there are commands now available to force erasure of the dirty queue as well as a secure erase of the entire drive, but these enhancements came about after I stopped developing SSDs
/Rattus
[0] Disk-drive - Generic term for any Non-Volatile / persistent storage device
[1] Disk-drive - In days of old 1's and 0's were stored on spinning platters of rust called disks, consisting (in some part) of ferrous metal, information could be written or read to these disks by use of electro magnets. Whilst little of this steam driven technology remains today, the naming convention, and underlying protocols still persist.
[2] Disk-drive - Device for storing and retrieving patterns of 1's and 0's curated by a controller that is responsible for the actual storage. Storage is presented to the host system as a contiguous series of blocks starting at block address 0 and working upwards, however the controller is responsible for mapping Logical Blocks to Physical storage anywhere it likes.
[3] Disk-drive - this sub-definition exists only to continue the series of sub-definitions whenever disk-drive[4] is mentioned
[4] Disk-Drive - It may come as a surprise but bad block management is not something unique to FLASH based disks, this applies to spinning disks as well. Yields of multi-layer FLASH (MLC) are worse than that of Two Layer FLASH (TLC) which in turn is worse than that of single Layer FLASH (SLC), however density (and hence cost) of MLC is > TLC > SLC by large margins, thus making MLC financially most attractive even if a relatively greater amount of over provisioning is required.
[5] Disk-drive - recursion of disk-drive[5] achieved
[6] see SMART - this reports statistics on the available unallocated physical blocks available, as well as tracking the number of erase cycles blocks have been put through and therefore projected remaining life expectancy of the disk-drive[5]
Biting the hand that feeds IT
