Dear Lazyweb,
I’m looking for recommendations for Open Source backup solutions which track incremental backups using a database, and which do not use hard link directories. Someone gave me a suggested OSS backup program at UDS, but it’s slipped my memory; so I’m fairly sure that at least one or more such OSS backup solutions exist, but don’t know their names. Can some folks give me some suggestions? Thanks!
There are a number of very popular Open Source backup solutions that use a very clever hack of using hard link trees to maintain incremental backups. The advantage of such schemes is that they are very easy to code up, and it allows you to easily browse incremental backups by simply using “cd” and “ls”.
The disadvantage of such schemes is that it creates very large number of directories blocks which must be validated by an fsck operation. As I’ve discussed previously, this causes e2fsck to consume a vast amount of memory; sometimes more than can be supported by 32-bit systems. Another problem which has recently been brought home to me, is how much time it can take to fsck such file systems.
This shouldn’t have come as a surprise. Replicating the directory hierarchy for each incremental backup is perhaps the most inefficient way you could come up with for storing information for an incremental backup. The filenames are replicated in each directory hierarchy, and even if an entire subtree hasn’t changed, the directories associated with that subtree must be replicated for each snapshot. As a result, each incremental snapshot results in a large number of additional directory blocks which must be tracked by the filesystem and checked by fsck. For example, in one very extreme case, a user reported to me that their backup filesystem contained 88 million inodes, of which 77 million of them were directories. Even if we assume that every directory was only a single block long, that still means that during e2fsck’s pass 2 processing, 77 million times 4k, or 308 gigabytes, worth of directory blocks must be read into memory by e2fsck’s pass 2. Worse yet, these 308 GB of directory blocks are scattered all across the filesystem, which means the time to simply read all of the directory blocks so they can be validated will take a very, very, long time indeed.
The real right way to do implement tracking incremental backups is to use a database, since it can much more efficiently store and organize the information of what file is located where, for each incremental snapshot. If the user wants to browse an incremental snapshot via “cd” and “ls”, this could be done via a synthetic FUSE filesysem. There’s a reason why all of the industrial-strength, enterprise-class backup systems use real databases; it’s the same reason why enterprise class databases use their own data files, and not try to store relational tables in a filesystem, even if the filesystem supports b-tree directories and efficient storage of small files. Purpose written-and-opimzied solutions can be far more efficient than general purpose tools.
So, can anyone recommend OSS backup solutions which track incremental backups using some kind of database? It doesn’t really matter whether it’s MySQL, Postgresql, SQLite, so long as it’s not using/abusing a file systems’ hard links to create directory trees for each snapshot. For bonus points, there would be a way to browse a particular incremental snapshot via a FUSE interface. What suggestions can you give me, so I can pass it on to ext3/4 users who are running into problems with backup solutions such as Amanda, Backup-PC, dirvish, etc.?
Update: A feature which a number of these hard-link tools do right is that they do de-duplication; that is, they will create hard links between multiple files that have the same contents, even if they are located in different directories with different names (or they have been renamed or their directory structure reorganized since they were originally backed up) and even if the two files are from different clients. This basically means the database needs to include a checksum of the file, and a way to look up to see if the contents of that file have already been backed up, perhaps with a different name. Unfortunately, it looks like many of the tape-based tools, such as Bacula, assume that tape is cheap, so they don’t have these sorts of de-duplication features. Does anyone know of non-hard-link backup tools that do de-dup and which are also Open Source?
No related posts.
January 12th, 2009 at 6:20 pm
http://code.google.com/p/brackup/ comes to mind.
January 12th, 2009 at 6:30 pm
I’m not too sure, but perhaps bacula, http://www.bacula.org/en/, is what you are looking for?
January 12th, 2009 at 6:34 pm
Would the hardlink approach work better if it could also hardlink directories? Git repositories have that property: identical subtrees simply reference the same object. Thus, a commit changing one file needs only one blob object and one tree per directory between that file and the root.
January 12th, 2009 at 6:38 pm
I’m a happy user of backup2l (available in Debian).
It doesn’t use use a database or allows browsing of snapshots – it stores it’s meta information in textfiles and puts with each incremental backup all new and changed files into one tarball.
No fancy GUI or cool features, the main features are:
- define in a configfile which dirs to include and exclude from your backups
- make a backup
- restore files/directories as they were at some specified time
It doesn’t match the wording of what you are looking for, but if someone wants a simple backup solution that supports incremental backups it’s an option.
January 12th, 2009 at 6:46 pm
@2: Wilfred,
Yes, thanks, Bacula was the one that was suggested at the Ubuntu’s Developer Summit. I haven’t examined it in detail, but a number of people at UDS seemed to think it was a good alternative to look at. A quick examination of its web pages seems to suggest there isn’t a FUSE viewer for examining Bacula backups, but that’s just a Small Matter of Programming.
January 12th, 2009 at 6:49 pm
The hardlink approach isn’t as awful when directories are allowed, as Anonymous suggests, which is how Apple’s Time Machine works. It’s still a bit of a hack, though.
January 12th, 2009 at 6:51 pm
@3: Would the hardlink approach work better if it could also hardlink directories?
Yes, if we could create hardlinks for directories, that would largely solve this problem. The problem with hardlinks and directories is that the ‘..’ link gets screwed up. It might be possible to use symlinks for directories, but the semantics for symlinks are quite different. For example, if A is a hard link to B, and you delete B, A is still valid. If A is a symlink to B, and you delete B, A becomes a dangling link. So life gets a bit more difficult if you want to garbage collect old backups once you start using symlinks to directories.
Git repositories have that property: identical subtrees simply reference the same object. Thus, a commit changing one file needs only one blob object and one tree per directory between that file and the root.
Yep. And using git as a way of managing incremental backups would work; unfortunately, git is designed as an SCM, not as a backup system, so it doesn’t easily support deleting old snapshots. You could use something like git filter-branch, but it would get messy awfully fast….
January 12th, 2009 at 7:22 pm
Just wanted to throw my two cents in for Bacula as well. I, and others, have found it to be a very solid backup system. It’s personally saved my bacon on several occasions (RAID is great, until every drive in the array dies at the exact same time…).
As for the FUSE option, you get a vague approximation of it when you perform a restore job. When you specify the backup you want to restore from, you are dropped into a pseudo-shell where you can cd and ls around to see what files are contained.
A full blown FUSE plugin, with the ability to actually read files, would be rather tricky, though. To start with, Bacula is typically used with collections of tapes, which give utterly horrid performance for typical random access operations (think waiting 15 minutes to read a file). If that’s not enough, the fact that the database catalog is stored separately from the tapes means you will quite often have access to metadata via the database about data stored on tapes that are currently offline in a cardboard box somewhere. Reading those tapes would be quite tricky =)
January 12th, 2009 at 7:34 pm
Recent versions of Gnu tar have a interresting -g option. See http://www.gnu.org/software/tar/manual/html_chapter/Backups.html#SEC88
January 12th, 2009 at 7:39 pm
I’d also recommend Bacula, I’ve been a very happy user for several years now. I like the sound of the FUSE viewer, I’ll be interested to see what happens with that.
January 12th, 2009 at 7:58 pm
rdiff-backup [1] does not use a relational database (it keeps one full mirror and then compressed deltas for incremental backups), but there is a FUSE filesystem for it [2].
1: http://www.nongnu.org/rdiff-backup/index.html
2: http://code.google.com/p/archfs/
January 12th, 2009 at 7:58 pm
I would suggest rdiff-backup (http://www.nongnu.org/rdiff-backup/). I’ve been using it on my /home directory for a year now and it does incremental backups wonderfully. My backup size is only ~10% larger than the source /home.
Google for a rdiff-backup fs based on fuse. There is one out there, I think it’s on google code, it’s name escapes me.
Rdiff-Backup doesn’t use a database but it also doesn’t use hard links. Instead, it keeps the most current copy of the filesystem as a plain old rsync-esque mirror but then with a rdiff-backup-data directory that contains all of the binary deltas.
January 12th, 2009 at 8:10 pm
@tytso, in response to comment 7:
Yes, I realize that hardlinks to directories make life painful for filesystems. Furthermore, you’d also need real garbage collection rather than reference counting, to handle cycles.
Regarding “..”, though, I’ve often thought that logic just doesn’t belong in the kernel in the first place; userspace libraries could handle “..”, and for that matter probably relative paths in general.
As for using Git for incremental backups: you just need to make each backup an independent commit with no parent, and tag each commit. When you want one to go away, just delete its tag and then garbage collect. (Make sure you turn off reflogs, or they’ll preserve backups longer than you intended; alternatively, just use a single ref and let the reflogs track old backups.)
January 12th, 2009 at 8:24 pm
@8: A full blown FUSE plugin, with the ability to actually read files, would be rather tricky, though. To start with, Bacula is typically used with collections of tapes, which give utterly horrid performance for typical random access operations (think waiting 15 minutes to read a file).
Sure, but if you are doing disk-to-disk backups, presumably it shouldn’t be that bad. Bacula hopefully has the option to write the backups to files instead to a tape device, doesn’t it? In the ideal case, hopefully a backup volume could either be on disk, or later on, moved off to a tape or DVD or other off-line storage medium.
January 12th, 2009 at 9:32 pm
Responding to @14, yes, Bacula does indeed let you perform backups to file based volumes as well as tape based ones. The catalog still frequently tracks data that’s offline, though, so it would certainly be…. interesting =)
January 12th, 2009 at 11:24 pm
I heartily recommend “duplicity”, found here: http://duplicity.nongnu.org/
This program combines librsync, GNU tar/gzip and GnuPG and provides a simple command line interface on top.
It does full backups and incremental backups. The incremental backups will be done on a block level (I think) which potentially saves a lot of space compared to hardlink-based incremental backups. In addition the archives are compressed which saves even more space. The archives are GPG-encrypted compressed tar files.
The archives can be stored in a local directory or sent to a server using a number of different network protocols.
Restore is very easy, too.
January 12th, 2009 at 11:43 pm
Aha, found it:
ArchFS:
http://code.google.com/p/archfs/
I’ve used it on my backup archives, it really does “just work” which is quite nice coming from complicated backup solutions.
January 12th, 2009 at 11:56 pm
A feature which a number of these hard-link tools do right is that they do de-duplication; that is, they will create hard links between multiple files that have the same contents, even if they are located in different directories with different names (or they have been renamed or their directory structure reorganized since they were originally backed up) and even if the two files are from different clients. This basically means the database needs to include a checksum of the file, and a way to look up to see if the contents of that file have already been backed up, perhaps with a different name. Unfortunately, it looks like many of the tape-based tools, such as Bacula, assume that tape is cheap, so they don’t have these sorts of de-duplication features. Does anyone know of non-hard-link backup tools that do de-dup and which are also Open Source?
January 13th, 2009 at 1:01 am
@18 Unfortunately, it looks like many of the tape-based tools, such as Bacula, assume that tape is cheap, so they don’t have these sorts of de-duplication features.
The premise that Bacula assumes tape is cheap is not true. Deduplication with tape backups is problematic. Consider that you need to restore. If you need to mount N tapes because the original files are on many different tapes, that’s just annyoying and time consuming. Then consider retention periods and having to keep some tapes longer because they contain unique data.
Fair easier to do no-dups for disk-backups.
I think no-dups is a very narrow niche, based upon the results I’ve seen from potential savings, combined with the number of people asking for this feature.
January 13th, 2009 at 1:01 am
I’ll echo the recommendation for duplicity. http://duplicity.nongnu.org/
I especially like how it can do remote encrypted incremental backups
so that I can backup to a less-than-fully trusted host.
(I think it may have S3 support as well now?)
January 13th, 2009 at 2:48 am
I think incremental backups for the small user are just a bad fit. Incremental/differential might work well when you have tons of money to spend on an enterprise backup solution. With a disk pool, tape library, and embedded oracle database saving state.
The problem (which I’ve seen happen) is that now the database can become its own failure point.
Believe it or not. Dump works fine for backing up machines. Just do a sync during off hours and dump. I work for a fortune 500 where we backup all 1000 servers in our de-militarized spaces using dump.
Incremental backup systems usually have the same flaws:
Run as a privileged role or user
Run as persistent daemon
Add a database.
Data and metadata are obscured to the user.
Instead, you can have 1 or more backup servers using plain unix tools and go poll your other servers and dump their filesystems back to them. Unless you have a highly volatile filesystem, sync/dump works fine. And even then you can lvm snaphost/dump if its really that crazy/volatile.
I just think the requirements here are a little offbase. We have bad backup ideas that the vendors have set us to expect. Strategies based on the assumption of tape media and incrementals.
If you want true peace of mind that can restore a machine to a point of time. Dumping the filesystems is about the only way to have it. Nothing is hidden or obscured. Boot from a rescue image, configure the network adapter, and you can restore those dumps back to the machine without worry.
Having worked with incremental backups before, and having done a disaster recovery sim to an offsite facility across the country, I can tell you that is frankly negligent to assume that a restore from incremental backups to tape will actually be successful. Tapes break in transit. Even with lots of foam. It happens and I’ve seen it.
I think the only fans left of these types of backups are the tape vendors and the backup software vendors. If people realized that the tools they need to take perfect backups of their systems were built into the OS……
Ted. I have faith in dump.ext2fs. Linus may not be a big fan of having a user with read access to the device reading it while its read/write mounted, but in practice it works fine. Its also the absolute most secure way of running backups that I have seen versus software that opens listening ports or runs with superuser credentials.
January 13th, 2009 at 4:54 am
Perhaps you like ftplicity(http://www.heise.de/ct/06/13/links/216.shtml), it’s a wrapper for the tool duplicity (http://www.nongnu.org/duplicity/).
You can copy anywhere where rsync is able to copy to, crypt your packed backups with gpg and make incremental backups ….
January 13th, 2009 at 7:16 am
I would also suggest rdiff-backup. It’s “local/remote mirror and incremental backup” based on rsync protocol. The nice feature is that the most current copy is accessible as normal files. You can use SSH to backup to a remote machine, etc. IMHO very simple and useful tool.
January 13th, 2009 at 8:49 am
@19: The premise that Bacula assumes tape is cheap is not true. Deduplication with tape backups is problematic. Consider that you need to restore. If you need to mount N tapes because the original files are on many different tapes, that’s just annyoying and time consuming. Then consider retention periods and having to keep some tapes longer because they contain unique data….
I think no-dups is a very narrow niche, based upon the results I’ve seen from potential savings, combined with the number of people asking for this feature.
Dan,
Given the sheer number disk-to-disk backup systems that work using hard links and directory hierarchies (i.e., Amanda, dirvish, Backup-PC; the list goes on), it’s obvious that disk-to-disk systems is hardly a narrow niche. And that shouldn’t come as a surprise; the problem with tape-based backup systems is that (a) they are slow, and (b) the initial setup cost for the actual tape drive is very high; a single LTO-3 tape drive well over $2000. (And tape libraries start at $7000 and very rapidly climb up to $20,000 and more.) In contrast, you could get six terrabyte SATA drives, configure them using RAID-6, and you can get 4TB of storage for $600 or so; and hot-swap SATA is not a big deal; heck, there are 1U servers that have room for 8 hot-swap SATA bays. For many home systems, that’s more than enough, and it’s a hell of a lot cheaper and faster than tape.
I’ll grant that de-dups functionality may not be that useful for tape-based systems for reasons other than “tape is cheap”, but it’s likely that anyone who cares about doing disk-to-disk backups will take one look at Bacula, and decide that it is too tape-centric, and move on to another solution. If you are doing disk-to-disk backups, and you have a fixed set of disks giving you 4TB of space, being able to do de-dup is important. The problem is that solutions such as Amanda, Dirvish, Backup-PC have surprising scaling limits, and it would be nice if there was a solution that was nicely positioned for the space between where these hard-link directory-tree solutions live, and a full-blown enterprise tape backup system where people don’t mind shelling out $20k or $30k for a tape storage library device.
January 13th, 2009 at 9:24 am
Interesting how people could conclude that Bacula is too Tape centric. Bacula treats tape and disk the same and calls them both Volumes. If they make that conclusion, they aren’t taking a very long look.
I think our communication issue is one of terminology.
I didn’t say back to HDD is a narrow niche. I’m claiming the requirement for de-dup is narrow.
If you’re doing FULL backup after FULL backup, I can see how de-dup is relevant. In such cases, you want to do only incremental backups. That way, you never store the same file twice. That eliminates dups within one client. Bacula has a project for this: See ‘Merge multiple backups (Synthetic Backup or Consolidation)’ at http://bacula.org/en/?page=projects
de-dup amongst all clients requires a diferent type of solution involving comparing each file (an MD5 or similar is sufficient for this). Then you need to do a reference count to make sure you keep the unique dups long enough.
I’d like to see some stats on the savings of no-dup based on both scenarios from above.
In addition, Bacula also has a project called base level backups, which is also related to the de-dup scenario.
January 13th, 2009 at 9:42 am
I’ve been using Bacula for about half a year now to back up /etc and /home on two different machines. I haven’t had to do any restores, thank goodness, but it does give me peace of mind.
As a home user, I don’t have a tape system. I use an automated disk-to-disk backup, as in http://www.bacula.org/en/dev-manual/Automated_Disk_Backup.html . Bacula automatically creates files such as Full-0001, Diff-0003, Inc-0002, and Default-0001 to hold my backups. The backups are stored on an encrypted external hard drive.
It’s true that there is waste due to duplicated files, but on my system the number of such files is small; it would mostly consist of files that I’ve moved from one directory to another.
January 13th, 2009 at 9:50 am
Dan,
I suspect the issue very much is semantics. If you talk to the Amanda, Dirvish, and Backup-PC folks, they view their solution as taking “incrementals”; not in the traditional Backup sense of the word, where you only take backup the a file that had been previously backed-up if the mtime/ctime/size has changed, perhaps, but the net effect is the same; they are making a copy of any file that has changed.
Their storage mechanism also allows them to perfectly handle files that have been deleted (something which Bacula doesn’t handle, although it is defined as a “most people don’t need this”, although I note there’s a project to handle that as well). In addition, their storage mechanism makes it trivially easy to browse the data and contents of any particular daily snapshot — at the cost which is arguably less then Bacula’s incremental system, they don’t have to backup files which are moved or renamed, and they can avoid duplicate backups of files stored on multiple machines.
Yes, it does mean that you have to MD5 the file, or use various other heuristics — for example, if there is no other file with the same size and extension that has been backed up, you can skip the MD5 checksum.
I do see quite a few ext3/4 users who are using these tools, and some are backing up 30-40 client machines using this tool, and they rely on the de-dup so that they only have to store one copy of distribution-installed files (i.e., /usr/bin/firefox, et. al). Presumably it’s less work from a configuration point of view, and that way they can be guaranteed to have a backup of any file that their users might have put on the machine, even if their users stashed it in a “system directory” like /usr/bin. For these users, de-dup is obviously a major win. You can say that they are stupid for wanting to backup the whole disk, including distribution files, but its arguably not a bad strategy, especially given there are tools that do this quite well.
January 13th, 2009 at 10:03 am
Ahh, those ext3/4 users want Base Level backups (the Bacula term; not yet implemented).
Personally, I backup only data files, not system-installed files. I can easily recreate the OS.
A previous post mentioned dumps. Dumps are not portable, yet ideal for some scenarios.
January 13th, 2009 at 10:10 am
Dan,
Does Bacula support hard links and sparse files correctly? Or will it backup a hard linked file N times, and “fill-in” sparse files? Bacula appears to be client OS agnostic, so I could imagine that it might consider these to be Unix-specific hacks. But it makes a huge difference for people who, for example, are using directory structures and hard links to organize pictures for example (i.e., create hard links so they can easily find pictures by subject, or type, or location). Using hard links is a perfectly valid way of organizing photo archives, and it sounds like Bacula will not handle such cases particularly well. (It’s very much related to the de-dup problem.)
Also for people who are doing certain types of virtualization, the thought of taking a sparse file that had consumed only 100 megabytes of space, even though its apparent file size was 1gigabyte, gets singularly unfortunate if when the file is restored, the restored file requires ten times as much as space as it did when it was originally backed up.
January 13th, 2009 at 10:47 am
hardlinks=yes|no
When enabled (default), this directive will cause hard links to be backed up. However, the File daemon keeps track of hard linked files and will backup the data only once.
sparse=yes|no
Enable special code that checks for sparse files such as created by ndbm. The default is no, so no checks are made for sparse files. You may specify sparse=yes even on files that are not sparse file. No harm will be done, but there will be a small additional overhead to check for buffers of all zero, and a small additional amount of space on the output archive will be used to save the seek address of each non-zero record read.
The above is taken from http://www.bacula.org/en/rel-manual/Configuring_Director.html
January 13th, 2009 at 10:58 am
FWIW, Bacula started off on unix. There are clients for many different OS, but if it’s unix-like, the client should just work. Mac, having unix underneath, also works.
Windows.. is a different story. It has its own client.
January 13th, 2009 at 12:33 pm
Venti from Plan 9. It is even available in Plan 9 port, as an application on UNIX.
Venti is an append-only system which identifies data blocks by sha1 hashes, so dedup happens ‘for free’. Using vacfs (also in plan9port) and either 9pfuse or Linux’s own v9fs module, you can mount archives like file systems.
January 13th, 2009 at 2:53 pm
Amanda doesn’t do deduplication and doesn’t use hard links. It does:
* support incremental backups, of the “backup only newly altered files” variety
* use tar, so it does backup hard links
January 13th, 2009 at 3:32 pm
I’m afraid not – I’ve been after one for a while. One other reason for usind a DB and not massively hardlinked tree is if you want to replicate your backups over to another site. Even rsync has huge issues.
January 13th, 2009 at 7:45 pm
tytso, I suspect you modestly underestimate the good work you did with ext3
I do live backups with backuppc (so, hardlinks & al.) over ext3, on
a cheap machine (sata raid5, with filesystem not well stripe aligned)
for a dozen linux serveurs (for about 700 GB backup pool (compressed,
deduplicated)), and it does perform well enough for such a small
setup. Especialy with backuppc 3.1 (and additional IO::Dirent perl
module). Yes, last boot took 1h30 because of fsck, but for some people
that’s still ok, provided that a backup server do not require high
availability. In my smallish setup, the lack of a complete windows
support is more problematic. But even if it mess up with filesystem
and presure them with I/O load as you said, that’s still usable (on
ext3) for modest needs.
Some people above suggested bacula, I used it too an can support their
voice : this is a good, robust tool, and it keeps indexes on database
(though if you require deduplication, it won’t be the one for you, plus
it is very hard to configure).
January 13th, 2009 at 8:20 pm
I’ll add another vote for brackup, particularly for its pluggable backend support. I do my home backups both to a server on the LAN and to Amazon S3, for instance.
I’ve used both bacula (to tape) and rsnapshot at $dayjob to do medium sized backups (~1TB of data). Bacula was pretty good, but is a reasonably complex beast to maintain, and seems to generate a lot of data of its own for backups of that size.
rsnapshot was a fail for us – we have pretty big directory trees (lots of small files) and it just couldn’t seem to walk the trees fast enough to be usable. An incremental backup over 400GB was taking about 10 hours to run, for instance, which was too long a window for us.
We’ve now implemented brackup and it does incremental backups of our ~1TB of data in about 2 hours to a local server, which we then rsync offsite for disaster recovery. Once up and running, requires almost no care. And it’s just filesystem based, so no db required.
January 13th, 2009 at 9:04 pm
@35: Benpi,
Yes, at the scale which you are running, BackupPC seems to be OK for you — although many people I know would not consider 1hr 35minute fsck times to be acceptable, or at least on the very bleeding edge of what would be considered acceptable.
The biger problem is that past a certain size, BackupPC causes e2fsck to fall off a cliff, because of the memory usage (especially on 32-bit platforms, where userspace is limited to 3GB, and shared libraries reduces the available space for large memory arrays even further). So I hear about it when people complain about e2fsck times or e2fsck not working at all due to memory problems.
In addition, I have a friend who has run into hardware problems, and so he wants to move his set of hard-linked directory hierarchies to another set of disks — and he’s running into memory usage problems due to cp and rsync needing vast amounts of memory and large amounts of time to figure out the hard links. We’re hopeful that he’ll be able to use dump/restore to move his backup hierarchies, but it’s increasing my unease about this particular technique of using large number of hard links. I expect that the problem with fsck speed, at the very least, will be true regardless of which filesystem people use, because you can’t get around the fact that you need to read all of those directory blocks, and they will be scattered all over the disk.
So the question is what advice do I tell them? “Sorry, you’re f*cked?” That’s why I was hoping for some kind of alternative to BackupPC and Dirvish that was functionally equivalent, but which used a database to track which files were where.
The suggestion of Vbackup, Venti, and Fossil, seems interesting, but some quick web searches returned phrases such as “research prototype” and “some assembly required”, so I suspect some polishing work is needed before it could be used as a production backup system.
It might be amusing try abusing git into a backup system, but new code would definitely be required; git’s index is great for a source code management system, but it would be disastrous from an overhead perspective for a backup system. Also, git doesn’t store all of the filesystem metadata required for a backup system. Still, it would be entertaining to steal various modules from git and reuse its object storage system as a way of storing file contents, and then developing a new method for storing directory and inode metadata contents. But this would be even more an case of “some assembly required”, although we could start with a pretty good toolkit.
January 19th, 2009 at 4:46 am
What I’ve been looking into is separating information objects from the layout or presentation. I, for my home use, am not using traditional backup for majority of data. For normal text documents and other stuff, normal tar-based incremential backup over network share is just fine.
But then there is a large bulk of material that I really care for (family photos, videos) and stuff that is nice to keep but loss of it is not a disaster (like recording of TV programs). That stuff I manually replicate, the first one to several geographically and logically diverse locations (some off-line), and the second every now and then to external cheap usb disks.
For saving the information, it is just sufficient to know that there are at least two copies (for very important stuff more copies) available and if one is lost or corrupted, the other copy could be replicated to maintain redundancy.
Then there is a lot of data, that is just a different representation of same information. For example, a file may be stored as-is, compressed with gzip, stored inside of zip, or tar.bz. I’ve playing with deduplication script that extracts archives and looks for files inside. The filesystem with metadata and archive metadata are just attributes along with “container identifier” i.e. sha1 value of bit stream. So, even if you loose all copies of docs.tar.gz, you are able to reconstruct it as long you have all members of archive in other archives or separate files and needed metadata.
You can take this even further: you may have original MPEG-2 sequence of DVB or DVD video, but then you have e.g. MKV version with h.264 compression, and maybe avi version. To go back to MPEG-2 you may loose some information, but in many cases that is not critical. You couls use higher replication need for the original and use other formats as additional backups. One can live with that. Same goes for edited versions of family photos: I typically fix colors and make other minor editing – something that could be easily played back as long you have original and instructions that generated derivative document.
So, at end one could have list of document digests, and instructions how to generate one sha1 value from another. You can apply encryption too, you can say that document inside of this encrypted file has this digest without revealing what the file is (unless some other person has the very same file). For a filesystem backup, there would be list of files with fs-specific attributes.
This could use some out-of-box thinking and end up with quite efficient data preservation, even shared with multiple users.
January 20th, 2009 at 7:03 am
Check out also boxbackup ( http://www.boxbackup.org/ ).
It does not use a database, but uses its own (efficient) packed format. A useful characteristic is that the client initiates the backup so it is very easy to setup in situations where the client is behind NAT or firewall.
January 20th, 2009 at 10:16 am
Prasinos: without a database, how does one get my file from two Fridays ago on server ABC? Or, for that matter, get a list of all the backups of that file so I can see on what day it got messed up?
January 20th, 2009 at 1:34 pm
StoreBackup is another backup utility that makes use of these items. What has actually been interesting me is building a FS that itself handles internal deduplication of data. In my development environments I store the same file a number of times (one per environment obviously) in an identical manner which wastes space since for the most part its read operations on the files anyway. Something that could then handle COW style semantics so that it creates a new file from the old one automatically and schedules it to be deduplicated if possible (e.g. I update one of my environments to a different version that I have in a different dev environment the disk usage increases temporarily until a daemon comes through to re-examine the files and reduce them automatically; files with only one copy wouldn’t be marked for COW instead handling the operation directly).
January 21st, 2009 at 5:32 am
Give inSync a try – http://www.druvaa.com/products/insync.html
It offers data de-duplication capabilities (saves single copy of duplicate content) … and saves upto 90% bandwidth and storage.
Uses portgreSQL internally to manage meta-data …
Jaspreet
(Part of Druvaa Team)
January 21st, 2009 at 8:31 am
@42: Jaspreet,
inSync is a proprietary product, and it is not available as Open Source, it looks like. Fairly pricey, too — a perpetual license for the server is $750, and each client is $75. It might be a fine choice for corporate users, but at that level there are a number of proprietary enterprise backup tools, many of which will do de-dup.
As I stated in the first line of my post, though, I was looking for Open Source backup solutions which handled de-dup and file system access (via FUSE or equivalent) to daily snapshots without using hard-link directory hierarchies.
January 21st, 2009 at 8:42 am
Yup, its pricey because it offers unmatched features. It uses the database, just as you mentioned later in the code.
inSync is FREE for opensource projects and startups.
PLease let me know if you are interested.
For opensource, try backupPC … not sure what stage it is in right now.
January 21st, 2009 at 10:00 am
@44: Jaspreet,
BackupPC uses hard-link trees, which runs into the scalability problem I described in this blog post.
inSync might be free-of-charge for open source projects, but (a) you don’t get source code, and (b) it’s not clear what “for open source projects” means in the context of backup software. My open source code is relatively small, and is well backed up using distributed SCM tools like git. I have back up needs like everyone else, but much of what I want to back up on my laptop — e-mail, mp3’s, video, etc., aren’t “open source projects”. And of course, what about corporate open source development groups, such as Intel’s Open Source Technology Center or IBM’s Linux Technology Center? Sounds to me like the assertion that inSync is free for “open source projects” is more marketing pablum than anything else, for what is essentially a proprietary product.
(For a proprietary SCM, like BitKeeper, saying that it was free-as-in-beer for open source project makes more sense, but given the pain that was suffered by the development community when the proprietor of BitKeeper suddenly decided to take his marbles and go home, I suspect there are many in the open source development community who would be suspicious of free-as-in-beer-but-we-don’t-give-you-source-and-reserve-the-right-to-change-the-license-and-revoke-access-at-any-time offers of proprietary software…. Heck, given that kind of uncertainty, I’d probably choose either (a) to pay for it outright, or (b) find an open source tool that will meet my needs, or (c) do without.)
January 21st, 2009 at 10:39 am
@TyTso,
Take your call
You are right, there are no guarantees on when the company might pull the plug. All i can say is that its a good and light weight product which meets your requirements and there are a few evaluation options – http://www.druvaa.com/products/insync_evaluate.html
But, overall i agree with what you just said.
Jaspreet
January 21st, 2009 at 10:43 am
Jaspreet: First sentence of this page: “I’m looking for recommendations for Open Source backup solutions “.
How does the product which you are selling meet that requirement?
January 21st, 2009 at 10:47 am
@Dan,
Please don’t start a flame war here
Not selling anything here. Not the market, i sell in. If you later read my comments I wanted to offer it Free .. just wanted to help.
Jaspreet
January 21st, 2009 at 11:07 am
I’m not trying to start a flame war at all. And since you didn’t answer my question, I’ll be more direct:
Your product does not meet the requirements stated in the first sentence of the page.
disclosure: I have written code for Bacula.
February 6th, 2009 at 1:31 am
[...] while back, Ted Ts’o asked for a incremental backup solution that used a database. It reminded me of the talk at the 2009 MySQL Conference & Expo, titled Build your own MySQL [...]
February 14th, 2009 at 6:27 am
Someone mentioned archfs earlier, a fuse-powered filesystem frontend for rdiff-backup directories. I found a stale Debian ITP and have took it on board.
February 20th, 2009 at 10:58 pm
I do disk-to-disk backups with “dump” and “restore” and a small set of shell scripts. Being raised in the mainframe era, I do a “full dump” periodically,
do “incremental dumps” nightly in cron, and do a “middump” whenever the
incrementals get so big that they’re eating too much disk space. I tune the dump script for each filesystem to compress it or not; and can queue the compression until after the dumps, to get fast snapshots of filesystems.
I go back in and manually remove the incrementals that are less useful; e.g. for each month, I keep the incrementals for the 1st, 11th, and 21st day. More recent months keep more daily incrementals. This lets me tune up the space allocation while still giving me easy restores (max 3 passes: full, mid, and latest incremental) and access to many recent versions of any file.
This has the great advantage that I can remove the backup disk and stash it in a safe place (i.e. where no computer can write on it! When did the write-protect switches/jumpers disappear?), then insert an empty drive for many months of subsequent incrementals. Ultimately I must retain the drive(s) containing fulldumps and middumps until I’ve recycled all the incdumps that depend on them. But I’m free to discard any incdump (or drive full of incdumps), and can discard any middump or fulldump that has had a subsequent fulldump (unless I want it for archival purposes, which I often do).
What this lacks is: a database of what’s backed up where (only needed when you want to restore and don’t know which incrementals it might be on),
and automation of restores. You need to be comfortable editing shell scripts
as well. It doesn’t automatically manage its disk space consumption. It doesn’t feed the cat. I have to use “tar” to back up filesystems that dump doesn’t understand (like MSDOS and Windows stuff).
The idea of *depending* on a database for my backups strikes me as foolhardy. It sure wouldn’t work for archival purposes — y’mean I need to get a copy of this 30-year old database program running before I can even read the 30-year-old backups from the machine I had at the time? No thanks!
I have done plenty of restores from these backups, and I trust ‘em. (Except just this week I was restoring a filesystem with many directories containing a hundred thousand files each — a set of MH mail folders full of spam. The older copy of restore that I was running burned CPU time forever, without writing to the disk, because it kept and searched a singly linked list of all files in a directory. I upgraded to the latest version, which uses hash buckets if you ask nicely, fixed a few things, and it’s doing fine at restoring my filesystem.)
By the way: Every backup disk I make contains a “tools” directory that has both binaries and source code for every tool used to make the backups. And before I remove a drive to keep it in offline safe storage, I remove the journal so that anything later that can read ext2 can read it. (If there was a more popular filesystem that could do the job, I’d switch to it.)
February 21st, 2009 at 1:02 pm
The idea of *depending* on a database for my backups strikes me as foolhardy. It sure wouldn’t work for archival purposes — y’mean I need to get a copy of this 30-year old database program running before I can even read the 30-year-old backups from the machine I had at the time? No thanks!
With respect to Bacula, you can extract data from the tapes without the database. See bextract, bls, etc.
The Catalog is stored in the database to make certain tasks easier. For example, what backups do I have of /etc/openvpn.conf between last month and today? By no means do you NEED the Catalog. It is a very convenient tool
March 3rd, 2009 at 9:46 am
Hello again!
One particular usage scenario I am considering a solution with de-duplication for (whereas I’m using rdiff-backup everywhere else) is for backing up a digital photo archive. The vast majority of the files should not change at all, and if they have changed, it’s probably due to bit errors or corruption somewhere in the hardware/software chain. De-duplication would hopefully catch this (and backup the modified/subtly corrupted photo to a different hash). Whether or not it would report it adequately I don’t know. (It’s also possible some dumb photo management software could corrupt my photos, e.g. a tool which does lossy JPEG rotate).
Whilst the files would not change, they may well move around a lot as I try different management schemes. I currently have a scheme whereby I file them at disc/YYYY/MM/DD/photo , where disc is an arbitrary separator that helps me split the files into DVD-R manageable chunks. A naive backup system (such as I think, unfortunately, rdiff-backup) could consume an enormous amount of disk space if it re-represented a chunk of files that moved from one location to another.
March 6th, 2009 at 8:38 pm
I’m currently trying out Gibak. It de-duplicated and compresses really nicely, and it’s also basically a shell-script around a git-repository, so you can use regular git commands if you want. Very cheap on space and fast transmitting.
March 23rd, 2009 at 12:35 pm
This isn’t what tytso is looking for, but since some folks have mentioned git and keeping important personal stuff, I’ll throw out flashbake (http://bitbucketlabs.net/flashbake/). Flashbake is aimed at not loosing work in progress. (Disclaimer: One of my friends runs this project.)
April 14th, 2009 at 6:01 pm
@tytso
Hi, it would be nice to have all this comments/suggestions summarized in a new blog post… of course with your conclusions.
great blog
September 13th, 2009 at 5:48 pm
I use dar (http://dar.linux.free.fr/) for incremental backups to disk. Tested restore, too.
Deleted files are recorded and not restored[1] (don’t understand how other incremental backup tools fail at this point). Dar doesn’t use rsync-like algorithm to detect changed files, just looks at the time. It writes its own archive format.
[1] by default
December 3rd, 2009 at 7:56 pm
The file system is essentially a database. Personally, when restoring a file I like the flexibility of tools available for copying files from a file system.
Hard linking backup solutions is not a silver bullet. However, there this approach has some advantages.
One possibility is that you could limit the history (number of snapshots) the hard linking backup solution is generating and you could then use a backup tool to push files to tape from the latest archive. The advantage of such an is the ability to restore for disk if you do not need to go back very far. If it is no longer available on disk then you could order in the tapes to go back further.
Finally, you may want to look though the list of projects listed on the LBackup about page.
December 3rd, 2009 at 11:22 pm
The file system is a database, sure; but it’s not a very good general-purpose database. It’s optimized for the workloads that are typically experienced by file systems, which are quite different from those that might be seen by most relation databases, for example.
There is a desire in computer science, just as in physics, for the “grand unified theory”; so you will see people argue that there should be one product that could solve all problems efficiently; whether that is a file system, or a relational database, or a key/value databases. It used to be, for example, that people thought no matter what your problem was, the answer was a relational database. Oracle even tried to convince people that an Oracle database could be the basis of a general purpose file system. That idea died quickly once people discovered how awful Oracle was at being a file system. Similarly, we are now seeing non-relational databases pop up in Amazon, Google, and many other distributed systems because it turns out relational databases really suck at scaling out.
So I really get my dander up when people say, “the file system is essentially a database”. I suppose it is, in the sense that any computer program can be transformed into a Turing Machine. But that doesn’t mean that it is an efficient or sane thing to do for a production system….
March 25th, 2010 at 3:06 pm
Currently investigating Lessfs http://www.lessfs.com/ and NILFS http://www.nilfs.org/ with a view to FUSE-ing lessfs on to NILFS and seeing how that combo behaves. (Out of curiosity as Data Domain/EMC has a de-duping log-structured FS under the hood).
Lessfs stores various metadata in a database but it may not be of the sort you require.
April 14th, 2010 at 5:39 am
Here’s the problem you have:
“The disadvantage of [backup solutions that use a very clever hack of using hard link trees to maintain incremental backups] is that it creates very large number of directories blocks which must be validated by an fsck operation. As I’ve discussed previously, this causes e2fsck to consume a vast amount of memory; sometimes more than can be supported by 32-bit systems. Another problem which has recently been brought home to me, is how much time it can take to fsck such file systems.”
Why do you think that the correct response to this problem is to use some database backend? If the *only* problem is that e2fsck is too slow, then perhaps you should consider some other filesystem for your backup filesystem? Not all filesystems have problems with high number of directories (performance wise).
April 24th, 2010 at 12:15 pm
@62: Mikko.
It’s not a performance problem, it’s a memory utilization problem. And it only applies to fsck, and it’s going to be true for pretty much any file system. The problem if you are trying to make sure the links count (i.e., the value returned by stat(2) in the st_nlinks field) is accurate, you need to count all of the directories that reference a particular inode. This means keeping an in-memory array so you can well, keep the count. Pretty much any Unix file system has this requirement, since it’s enforced by POSIX, and it’s needed so you know when the last directory has removed its link to the inode, so that the inode and the blocks associated with it can be released. If the link count is too low, you might release the inode early, and that would result in data loss.
Now there are optimizations you can use, such as only allocating memory for counting the ref counts for files that are referenced by more than one directory (i.e., they have at least one additional hard link beyond the directory entry created when the file was first created). E2fsck does that. But if you use a backup system which utilizes huge numbers of hard links, then you need lots of memory. And that’s a solution too. And if you don’t have enough memory, you can use an on-disk database to deal with the fact that you don’t have enough memory to store all of the refcounts in memory. And e2fsck has that too.
Or you could decide that the “clever hack” of using hard link trees to maintain incremental backups has costs which are too great for the benefit they bring, and in fact using a real database would be cheaper past a certain scaling point. Which is basically what I tried to argue, perhaps not clearly enough.