There is a desire in computer science, just as in physics, for the “grand unified theory”; so you will see people argue that there should be one product that could solve all problems efficiently; whether that is a file system, or a relational database, or a key/value databases. It used to be, for example, that people thought no matter what your problem was, the answer was a relational database. Oracle even tried to convince people that an Oracle database could be the basis of a general purpose file system. That idea died quickly once people discovered how awful Oracle was at being a file system. Similarly, we are now seeing non-relational databases pop up in Amazon, Google, and many other distributed systems because it turns out relational databases really suck at scaling out.

So I really get my dander up when people say, “the file system is essentially a database”. I suppose it is, in the sense that any computer program can be transformed into a Turing Machine. But that doesn’t mean that it is an efficient or sane thing to do for a production system….

Hard linking backup solutions is not a silver bullet. However, there this approach has some advantages.

One possibility is that you could limit the history (number of snapshots) the hard linking backup solution is generating and you could then use a backup tool to push files to tape from the latest archive. The advantage of such an is the ability to restore for disk if you do not need to go back very far. If it is no longer available on disk then you could order in the tapes to go back further.

Finally, you may want to look though the list of projects listed on the LBackup about page.

Deleted files are recorded and not restored[1] (don’t understand how other incremental backup tools fail at this point). Dar doesn’t use rsync-like algorithm to detect changed files, just looks at the time. It writes its own archive format.

[1] by default

Hi, it would be nice to have all this comments/suggestions summarized in a new blog post… of course with your conclusions.

great blog

One particular usage scenario I am considering a solution with de-duplication for (whereas I’m using rdiff-backup everywhere else) is for backing up a digital photo archive. The vast majority of the files should not change at all, and if they have changed, it’s probably due to bit errors or corruption somewhere in the hardware/software chain. De-duplication would hopefully catch this (and backup the modified/subtly corrupted photo to a different hash). Whether or not it would report it adequately I don’t know. (It’s also possible some dumb photo management software could corrupt my photos, e.g. a tool which does lossy JPEG rotate).

Whilst the files would not change, they may well move around a lot as I try different management schemes. I currently have a scheme whereby I file them at disc/YYYY/MM/DD/photo , where disc is an arbitrary separator that helps me split the files into DVD-R manageable chunks. A naive backup system (such as I think, unfortunately, rdiff-backup) could consume an enormous amount of disk space if it re-represented a chunk of files that moved from one location to another.

With respect to Bacula, you can extract data from the tapes without the database. See bextract, bls, etc.

The Catalog is stored in the database to make certain tasks easier. For example, what backups do I have of /etc/openvpn.conf between last month and today? By no means do you NEED the Catalog. It is a very convenient tool

I go back in and manually remove the incrementals that are less useful; e.g. for each month, I keep the incrementals for the 1st, 11th, and 21st day. More recent months keep more daily incrementals. This lets me tune up the space allocation while still giving me easy restores (max 3 passes: full, mid, and latest incremental) and access to many recent versions of any file.

This has the great advantage that I can remove the backup disk and stash it in a safe place (i.e. where no computer can write on it! When did the write-protect switches/jumpers disappear?), then insert an empty drive for many months of subsequent incrementals. Ultimately I must retain the drive(s) containing fulldumps and middumps until I’ve recycled all the incdumps that depend on them. But I’m free to discard any incdump (or drive full of incdumps), and can discard any middump or fulldump that has had a subsequent fulldump (unless I want it for archival purposes, which I often do).

What this lacks is: a database of what’s backed up where (only needed when you want to restore and don’t know which incrementals it might be on),
and automation of restores. You need to be comfortable editing shell scripts
as well. It doesn’t automatically manage its disk space consumption. It doesn’t feed the cat. I have to use “tar” to back up filesystems that dump doesn’t understand (like MSDOS and Windows stuff).

The idea of *depending* on a database for my backups strikes me as foolhardy. It sure wouldn’t work for archival purposes — y’mean I need to get a copy of this 30-year old database program running before I can even read the 30-year-old backups from the machine I had at the time? No thanks!

I have done plenty of restores from these backups, and I trust ‘em. (Except just this week I was restoring a filesystem with many directories containing a hundred thousand files each — a set of MH mail folders full of spam. The older copy of restore that I was running burned CPU time forever, without writing to the disk, because it kept and searched a singly linked list of all files in a directory. I upgraded to the latest version, which uses hash buckets if you ask nicely, fixed a few things, and it’s doing fine at restoring my filesystem.)

By the way: Every backup disk I make contains a “tools” directory that has both binaries and source code for every tool used to make the backups. And before I remove a drive to keep it in offline safe storage, I remove the journal so that anything later that can read ext2 can read it. (If there was a more popular filesystem that could do the job, I’d switch to it.)