- \n
- When a file is created, it automatically gets the archive bit set.<\/li>\n
- When a file has been changed, the operating system (especially Windows) automatically sets the archive bit of that file, if it is not already set.<\/li>\n
- The incremental backup software checks if there are files which have the archive bit set. If it is set, this file is considered as being changed and thus it is stored in the current backup set. Once the backup has been executed successfully, the archive bit of all files backed up is reset\/cleared.<\/li>\n<\/ul>\n
Although at first glance this looks as a very efficient and appealing algorithm, it is not. There is one major drawback which I found during a short simulated backup restore drill: What about deleted files? They simply do not have archive bits!<\/p>\n
Versioning and Backup Software<\/h4>\n
In my case it became evident that the backup software used exactly the algorithm stated above (and BTW this was the reason why it was so fast, one of the advertised “features” of this product). However, the non-existance of a file can have semantics, too. In our use case, the application for which the data was backed up uses a dedicated file for each object to indicate a distributed lock (also not a very intelligent approach, but that’s another story). Restoring all the incremental backups thus caused all old locks “reappearing”. Fortunately there was an automatism inside the application to detect stale locks, so no severe impact (besides an initial bad performance when the system cleaned up the mess) was observed. Shortly afterwards, some very fancy, inconsistent behaviour proved previous rumbles out of my stomache: Some time before (during the time frame of the incremental backups) the application moved files from one directory to another. As usual, a move was implemented as a copy from the source to the destination with a clean up afterwards at the source location. Thus, the archive bit was set properly (at the target position). As the backup ignored deletions, the “old version” at the point in time when the move was executed “reappeared”. Whenever the implementation of the application looked at the “previous location” (for example in case of compatibility) it “fell over” the old version which did not fit to the rest of the data it had read before. In short, looking at the backup application, this is an excellent example for the situation where non-completeness of a feature may cause significant inconsistency which may happen to be detected “too late”.<\/p>\n
Versioning and Subversion<\/h4>\n
Some time ago I experienced similar issues on consistencies with another program (there it was not on files, but on directories). It was Concurrent Versions System<\/a> or better known by its acronym CVS. At the end of this story I decided to switch the program and move to the next generation versioning tool called Subversion<\/a> (svn). Subversion is an application which is exactly built for the purpose of versioning and uses the paradigm of database transactions to ensure ACID<\/a>-like operations. By this it can even deal with versioning of directories<\/a>. It belongs to the group of revision control<\/a> programs (more resource on this tool you can find the additional resource section at the end of this post). When I saw the similarity of those two approaches I wondered if it would not be possible to reuse Subversion also for providing historical information on backup strategies. Out of that experience I knew that svn did not suffer the same “stale file” issue as the backup program above had. And even better: Subversion has a very compelling diff algorithm for storing deltas between versions in a very compact manner — even in the case of binary files (even better: the application mentioned above mostly uses text files for data storage, so this was not a key differentiator). After all it provides an open-source license which makes it quite cost-efficient especially on acquisition. Yet, there were some minor obstacles.
\n<\/p>\nAdded and Removed Files\/Directories<\/h4>\n
Out of experience I knew that it was necessary to “register” and “deregister” files at subversion before it can do its “magic”. To be more precise, there are commands for<\/p>\n
- \n
- adding files\/directories,<\/li>\n
- removing files\/directories and<\/li>\n
- committing the changed state to the server (changed content of files is detected automatically during this call)<\/li>\n<\/ul>\n
As the third bullet item did solve the problem of changed files, it opened up the “can of worms” for both new and deleted files. At first glance this appears to be more step back than forth. However, there is support for this situation: the command line tool
svn<\/code> allows to provide details about the current state of the files. One may callsvn status<\/code> to query for a list of changes made locally. For each file the name and its status is printed. For example the values “?” indicates that a file (or directory) is available locally, but has not been uploaded\/added to the list of files to add on the next commit to the server. Similarly, the “!” indicates that a file has been deleted locally, but the deletion has not been queued to be sent to the server yet.
\nThe resolution of this matter was a simple batch file which looks as follows (derived from svnautocommit<\/a>):<\/p>\nfor \/f \"tokens=2*\" %%i in ('svn status %1 ^| find \"?\"') do svn add \"%%i %%j\"\r\nfor \/f \"tokens=2*\" %%i in ('svn status %1 ^| find \"!\"') do svn delete \"%%i %%j\"<\/pre>\n(whereas
%1<\/code> denotes the directory where the local copy is being stored) To state it already right out of the box: This isn’t the most efficient way of doing so, but in my case it already did the job. Adding the command<\/p>\nsvn commit -m \"Automatic Commit\" %1<\/pre>\n
afterwards around enabled this batch file to be executed automatically with the help of Windows Task Scheduler<\/a>.<\/p>\n
Adoption to the Usecase and Long Lasting Commits<\/h4>\n
Having figured out this part of theory, I sprang into action: It ever rankled me that I did not copy the data “out” of the WinS server. As stated before the backup data was stored on a DVD+RW, but this media had to always remain in the drive at the server system (for the next backup). A better approach is to make sure that the data leaves the system and gets copied to some foreign location (in case of fire ideally this would also be another room \/ location which is not possible in my case, but even using another computer at least is better than nothing). Seeing a chance to change that I installed an Apache Webserver plus the subversion server package on the LinS. On the WinS I installed the Subversion command line tool<\/a> and uploaded the data to backup from the WinS to the LinS (it would have been better to use the
svn import<\/code> command, though I used thesvn add<\/code> command). During this activity I hit another unexpected problem: When apparently preparing to upload a large file to the HTTP repository during executing thesvn commit<\/code> command, the client (here, the WinS) required some time to calculate the difference which should be sent to the Subversion server via the Apache Webserver. In my case, however, the KeepAlive option of the webserver was enabled in \/etc\/httpd\/httpd.conf (which is a good idea as such), but the KeepAliveTimeout parameter there was set to 15 seconds. If the client took more than this leap time to send the next file to the server, the webserver closed the connection (due to this KeepAlive timeout) which in turn caused the client to quit uploading with a cryptic error message. I came to this idea when I thought about this behaviour having read through this article<\/a>. In my case I changed the standard value for KeepAliveTimeout from 15 seconds to 150 seconds which did the trick – since then I never experienced commit stops anymore. Please note that this might be a bad idea, if your webserver also serves “real clients” such as normal web pages – in my scenario almost the sole purpose of the webserver is to deal with the Subversion requests, so I could make that change globally without risking anything.
\n<\/p>\nTagging<\/h4>\n
To make my life easier and to find out which version was valid at which point in time I also implemented a
tags<\/code> subdirectory. After each backup commit to the Subversion server, I automatically store a tag directory of the current version by using the remote copy functionality of the subversion client (for details refer tosvn copy --help<\/code>, use case “URL -> URL” there) knowing that this is not necessary with subversion. This is only for my personal convience as I do not like to specify dates when checking out data from a versioning system like Subversion.
\nPlease note that using the local copy function (use case “WC -> WC”) is a bad idea in this case as on execution of the commit command also will copy the entire data locally on your filesystem, which will eat up your entire disk space over time quite quickly. Using the remote version makes the Subversion server creating a reference copy (lazy copy) and thus only takes some kilobytes for storing a reference in a transaction file, even if you have a large number of huge files stored in the repository. The downside of this is that this cannot be done together with the commit, but is executed as a seperate transaction on the Subversion server. However, this should not pose a major issue as the numbers of transaction is not limited practically.<\/p>\nThe Script<\/h4>\n
If you want to set up a similar scenario you may benefit from the following references: <\/p>\n
- \n
- Download this file:
![\"\"](\"http:\/\/blog.schmoigl-online.de\/wp-content\/plugins\/wp-downloadmanager\/images\/ext\/zip.gif\" "\\"\\"")
Subversion Backup Scripts<\/a><\/strong> (1.3 KiB, 1,214 hits)<\/p> (License<\/a> information for these files). It contains my set of scripts and a collection of subdirectories which might be handy for you. In any case make sure that you adjusted the repostory’s path in file backup_run.cmd<\/code>.<\/li>\n- To run these programs you will need the Windows Subversion command line tool<\/a> installed on the system.<\/li>\n<\/ol>\n
Further Aspects<\/h4>\n
- \n
- If you do not have a linux box for backup at hand but another Windows system, you might find this blog on installing WAMP with a Subversion server<\/a> useful.<\/li>\n
- Please note that you might have a problem more now which you did not know before: How do you get rid of old versions? I will eloberate on this topic in a seperate posting.<\/li>\n
- Whereever there is light, you also will find shadows. The dark side of this approach is that the entire data is stored twice on the backupee system: once for the original and possibly changed data and once again in the corresponding .svn subdirectories. This is a known behaviour of the subversion client which enables it to efficiently compare what has been touched before committing only the data which is necessary. In our case this, however, implies that you double the required disk space.<\/li>\n
- besides the required disk space for those .svn subdirectories there is another drawback: In some rare cases the existance of this additional .svn subdirectory in each directory of your backup directory structure may pose problems or cause a different behaviour of the application. This may happen, for example, if the application wants to delete a otherwise empty directory, but unfortunately cannot because this .svn directory still is there. Thus those applications need to be programmed intelligent enough to ignore this automatically.<\/li>\n
- A very similar approach could also be achieved via the recent “Previous Version” functionality in the latest NTFS version. Microsoft refers to it as Volume Shadow Copy<\/a>. In my case it was not subject of discussion, because the version of the Operating System of the Windows Server is too old. Thus, I also did not play around with it yet.<\/li>\n<\/ul>\n
Additional Resources<\/h4>\n
- \n
- If you are interested in a quick tutorial on subversion, you might want to read http:\/\/aymanh.com\/subversion-a-quick-tutorial<\/a><\/li>\n
- For a discussion about the drawbacks and impact of using Subversion as a backup server you can also read http:\/\/stackoverflow.com\/questions\/61888\/using-subversion-for-general-purpose-backup<\/a>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
Surprisingly well Subversion servers can be used also for implementing incremental backup strategies. As it incorporates a good difference handling algorithm and provides magnificant support on investigating versions, it automatically provides features which are ahead of any standard backup program. This post explains how I got to that view and describes the scenario in which it is in productive use.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[],"class_list":["post-19","post","type-post","status-publish","format-standard","hentry","category-subversion-svn"],"_links":{"self":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/19","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=19"}],"version-history":[{"count":72,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/19\/revisions"}],"predecessor-version":[{"id":195,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=\/wp\/v2\/posts\/19\/revisions\/195"}],"wp:attachment":[{"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=19"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=19"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.schmoigl-online.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=19"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- For a discussion about the drawbacks and impact of using Subversion as a backup server you can also read http:\/\/stackoverflow.com\/questions\/61888\/using-subversion-for-general-purpose-backup<\/a>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
- To run these programs you will need the Windows Subversion command line tool<\/a> installed on the system.<\/li>\n<\/ol>\n
- Download this file: