Here is a related article….

Data Recovery and/or Hard Drive Recovery is not always possible in all scenarios but in the majority of cases significant recovery……

Making an encrypted and compressed backup of your files onto DVDs « su-root.eu…

[...]Here is a Great Blog You Might Find Interesting that we Encourage You[...]……

I saw this really good post today.Thank you!…

Compression: The thing that got me on that point was that it seemed for naught, as you probably didn’t save a disk. ~14 gig down to ~12 isn’t that good a compression ratio in any case. I’m wondering if much of the data is already compressed, as are many media formats (image/audio/movie) and most tarballs, or is pre-encrypted individual files, as you won’t get much savings off of that. If that’s 90% of your data, no wonder the compression ratio was so poor, even for gzip.

Sort-by-size: du helps here and is command-line so may be useful in your scripts. =:^)

You don’t mention your DE, but I find the kde-based filelight (radial view) and konqueror’s filesize view (nested rectangular) very helpful. Solutions I haven’t used personally but that I found with a quick package manager based search plus more listed on those homesites include xdu and xdiskusage (generic X-based), gt5 (console, produces a text table in the form of an html file browsable in links/lynx), disk-usage-analyzer/baobab (part of the gnome-utils package, radial similar to filelight, baobab is a type of tree!), treesize (gtk, table view, helfully listed several of the others as alternatives), and pysize (ascii, ncurses, gtk, rectangular).

Using one/several of these tools should help quite a bit in dividing up your data domain into dvd-sized chunks. I used them here when designing my partition layout and continue to use them for admin tasks where getting a quick picture of what ate the several gigs of space I THOUGHT I had available in a particular partition (see below) can be incredibly helpful.

Encryption chunksize: Individual encryption of large numbers of files will, as you suggest, have quite the overhead. A compromise might be to encrypt whole directory trees, at the level you include them in the image, or perhaps a level or two below that, but certainly not all the way out at the leaf/file level.

Memory-based block device (rd)? Something that /might/ be useful, depending on the amount of memory you have (I have six gigs here, so an entire 4.7 gig dvd image in RAM isn’t out of the question), would be various methods of creating an image out of ram and encrypting it, then burning that. I’m not an encryption or block device guru by a long shot, but it seems to me that creating a 4.7 gig rd/ramdisk and layering LUKS or etc on that has potential. After filling up whatever filesystem on top of LUKS, a direct dd from the underlying rd to dvd would then, in theory at least, allow you to mount the dvd as a LUKS encrypted whatever filesystem.

Cryptoloop? Another interesting solution would be a cryptoloop mounted ISO, both for pre-burn creation and for recovery, direct off the DVD.

One more note: LVM: Definitely YMMV on this one, but I did run LVM on top of one big system-level RAID (except for /boot and /, since I wanted to avoid an initr*) for awhile. However, I eventually removed it, in favor of multiple RAID-1s on partitioned physical media, with some of the RAID-1s themselves partitioned. There were several reasons for this. This isn’t the time or the place to have a debate about that, but take it from someone who has had to do real recovery (thus some of the hard-earned recovery wisdom, above), if LVM, or RAID, or for that matter LUKS, was unable to find its config info and load properly, are you confident in your ability to restore the damaged config in-place (by telling LVM to use one of its backup info caches, for instance, or by using a GPT based partitioner to recover the backup partition table that unlike MBR, it keeps, checksummed) as well as to recover from backup, if necessary? Especially under the increased stress and limited resources available in a recovery situation, could you do it?

When I asked myself that question, I was able to answer in the affirmative for the RAID level and the device and partition layers, but was uncomfortable with my own answer about LVM. If you know how to recover LVM’s backup management data, without referring to documentation that won’t be available under that scenario, good for you! If you don’t, I urge you to really think about whether LVM is making you job as admin over your system simpler, or more complex. The answer for me was that I was better off without it, so I made it so. I sleep better, now. =:^) (FWIW, there’s also recovery and re-sync benefits at the RAID level, of having multiple RAIDs on physical device partitions, with selected bits of the system or backup on each RAID, as opposed to one big RAID. That’s why I did it that way when I reorganized my layout during the LVM removal.)

@Gerard Lally – Although I could write a script to do that I think my back up method (as mentioned by Duncan) could do with improving.

@Duncan – You raise several good points

1) You’re right that by compressing my data I’m increasing the chance of failure. I probably should of thought of that before I started. I also should of played with xz, I forgot that tar had built in support for it.

2) There’s no way to easily recover a single file. Yeah that’s not great either. I have quite a lot of free space on my drives so scratch space currently isn’t a problem but others may not.

3) Loosing a single disc means all data is lost. Yeah that is probably the worst point of my method. I did it the way I have done it to preserve the directory structure that I have because it took me quite some time to organise my files in this way. Sorting my data by size into new directories manually is a major hassle. I am however quite up to the challenge of writing a shell script to do the work for me. I’ll look at it when I get time.

You’re right that doing a separate tar ball for each disc would be better. It is also tempting to encrypt each file individually to make it even easier to access to individual files. I imagine though that encrypting/decrypting each file would add considerable overhead in my case as gpg would be invoked many times instead of once for every tar ball.

I have a fairly complicated hard drive setup at the moment. I have md/kernel RAID-1 on two drives and those make up an encrypted LUKS partition. On top of that is a LVM physical volume which makes a single volume group. That volume group is then used to make my logical volumes ( /, /home, /boot).

I then have a similar set up using another two drives for backup. The DVD backups I was making was for backing up my most important files rather than everything backed up on my backup drives.

Thank-you for the valuable comments.