This show has been flagged as Clean by the host.

What Is The Indie Archive?

I'm Hairy Larry and you're listening to the Plain Text Programs podcast.

The Indie Archive is a archival solution for indie producers. Since most indie producers run on a shoestring budget it's important that the Indie Archive is inexpensive to install and run. It's especially important that monthly expenses are minimal because a reasonable expense most months will sometimes be more than an indie producer can afford during some months.

The first major constraint is cost. So I'll be talking about prices a lot in this podcast and get more technical in future podcasts about The Indie Archive.

Indie Archive is an archival system which is different than a backup system. If you don't have a backup system do that first. My backup system uses the same tools as Indie Archive, rsync and rsnapshot. My brother uses the online backup service Carbonite. There are many other options.

A good backup system runs automatically to back up everything frequently and preserve version history. It's also good to have backups offsite.

An archival system, like Indie Archive, keeps multiple redundant copies across several hard drives on several systems in multiple locations. An archival system also checks file integrity as protection against file corruption or user error.

When you have a project you really never want to lose, like a finished novel, a music album, a video, or any other major effort that involves significant work, that's when you need an archival system.

So The Indie Archive does not automatically backup your projects every day. That's what your backup system should do.

The Indie Archive is an archival system where the producer of the content decides what needs to be archived and when it needs to be archived and then manually moves a directory containing the files onto the Indie Archive carefully preserving the file's metadata during the transfer. Then these files are propagated over at least 7 hard drives on 4 different systems in three locations.

File integrity checks are run daily comparing the files and reporting discrepancies.

Two of the systems are kept in the studio where the content is produced. I call them the primary and secondary systems. They have a boot drive and two data drives each.

One of the systems is kept offsite at a nearby location. I call it the remote system. It also has a boot drive and two data drives. If you have a more distant location where you can put a second remotes system you can have remotenear and remotefar systems. Otherwise ...

The final system is somewhere in the cloud provided by a professional data storage provider. It has a single copy of the data and usually some additional data retention. The provider makes the backups of this data. This is the part that might involve a monthly bill. So, depending on the size of your file set, it could be free or it could cost so much a month. There are a lot of options for cloud storage providers.

But first I'm going to discuss the three systems, primary, secondary, and remote, and how they function.

As far as the hardware goes the systems are the same. Now, I"m a Linix guy and I do all my production work on Linux so I'm using Linux. I want to test the system on several versions of Linux and with BSD.

I'm not a Mac guy or a Windows guy so I won't be going there. The software is open source and the required programs run on all three platforms so I'll let a Mac or Windows programmer test The Indie Archive for their systems. My guess is that the Mac fork will be easier than the Windows fork because of the file metadata. It might even be possible to add Mac folders to The Indie Archive running Linux but I'll let someone who actually has a Mac figure that out.

I don't think the same is true for Windows. Windows file metadata is different and so if you want to preserve the metadata you will probably have to install The Indie Archive on Windows systems.

So, I'm developing and deploying on Linux and I will also test on BSD. So far I have tested Debian, Ununtu, FreeBSD, Midnight BSD, and Xubuntu and The Indie Archive works fine all of these operating systems.

So, back to the hardware. Pretty much any older system that will support at least three sata drives will work. I'm using older business class desktops, Dell and HP. I pulled mine out of storage but they are very inexpensive to buy if you're not like me with a shed full of old computer stuff. I just bought a Small Form Factor HP Desktop on ebay for $30 including tax and shipping.

To clarify, it's best if the primary system supports 4 sata drives. The secondary and remote systems do not need an optical drive so they should support three sata drives but they can be run on two sata drives if you boot from the primaryfile drive. I am currently testing a remote system with two sata drives running Midnight BSD.

The Dell desktops made a big deal about being green. I am open to suggestions on what would be the best energy efficient systems for The Indie Archive, because of both the cost of electricity and the impact on the environment.

There are three drives on each system, a boot drive and two data drives. The boot drives can be SSD or spinning hard drives and need to be big enough to hold the OS comfortably. The data drives need to be large enough to hold the files you want to archive and they should be high quality spinning drives, I use the multi terrabyte HGST drives and I am also looking at some Dell drives made by HGST.

There will be a data drive and a snapshot drive on each system. If they are not the same size the snapshot drives should be larger. I am testing with 3 terrabyte data drives and 4 terrabyte snapshot drives. Besides the main data set that is being archived the snapshot drives also hold the version history of files that have been deleted or changed. So, that's why they should be the larger drive.

So my primary system has a primaryfiles directory with a 3 terrabyte drive mounted to it and a primarysnapshots directory with a 4 terrabyte drive mounted to it. Same for the secondary and remote systems.

Now, so far I only had to buy one drive but generally speaking the six data drives will be the major expense in assembling the systems.

So a good bargain on six 4 terrabyte drives could be $120 used or $270 new. And this is the most expensive part.

I install used HGST drives all the time and rarely have problems with them. I have worked for clients who won't buy used, only new. Since the file integrity checks should give early warning on a drive failure and since there is a seven drive redundancy on the data files, if I were buying drives for The Indie Archive I'd go with six used 4 terrabyte HGST drives for $120.

There is no reason not to use drives all the same size as long as the snapshot drives are large enough. The size of data drives you need depends on the size of your projects and the time it takes to do a project. Look at your hard drives on your working systems. Think about what directories you would like to see in archival storage. What is the total size of these directories? Check how many gigabytes these projects have consumed in the last year. Think forward a few years. Assume you will use more disc space in the future than you are now. Do some quick arithmetic and make a decision.

Like I said I only had to buy one drive so far because I'm weird and I had a bunch of 3 terrabyte drives available. If I had to buy drives I probably would have tried to start larger. I am sure that at some point in the not to distant future, when I am running The Indie Archive and not developing it, I will have to upgrade my drives.

The primary system is the console for The Indie Archive. When you copy a project onto The Indie Archive the directory goes into the primaryfiles directory. From there it is propagated out to the primarysnapsh...

HPR4312: What Is The Indie Archive?

Powered by Wave AI