Опасная зона

Опасная зона

Wednesday, March 25, 2015

Ideas for Safe and Secure Long Term Data Storage

Over time I have gathered a quite substantial amount of data I want to keep in a safe place, such as raw data from work, personal data, collections of rare music etc.

Image is totally unrelated to this post and was just added for decoration purposes.

The question arise, how to store these data safe and recoverable and prevent data rot (some call this bit rot, apparently a confusion of terms) which may flip single bits. A single bit flip in a .tar.gz file might be catastrophic, so some preventive strategy is needed.

In a nutshell, this is what I came up with:
  • Use multiple non-identical HDDs for backup.
  • Use BTRFS file system for primary storage, and backup. On one secondary HDD use a filesystem other than BTRFS.
  • Use LZIP for compressing your tarballs.
  • Use par2 for extra recovery possibilities of your tarballs.
  • Use rsync for copying/backup files exclusively, and always check the output by a dry run (option -n) first.
Here goes the reasoning: Regarding the physical storage media, my backup media will mostly spend its life in a drawer, with only few occasional uses per year.
  • Media such as CDs/DVDs are not reliable for more than a few years of storage, and highly unpractical in handling.
  • USB memory sticks may retain data for 5 or 10 years, but I simply would not trust my data to this medium, no idea of what quality the memory chips are.
  • HDDs are convenient to use, but may fail catastrophically without any notice ahead after a few years. I never experienced this, but often had HDDs which started making weird sounds, creating bad blocks etc, sometimes already after 2 years of use.
  • SSD drives have no real track record yet, and some suggest even that their life is even shorter than that of HDDs. I do not really trust this.
  • Various tape decks may have good life-time, but I think these are clumsy to handle, and the more clumsy gadgets get, the less likely I will use them. Also again, I have no idea which vendor to choose, who has a proven track record, etc.
  • RAID1 (or higher) systems may give false sense of security. Data are stored in one place, and accidental use rm -rf is still fatal.
The only viable solution I could come up with, is based upon the expectation that at least one or two simultaneous possible point of failure will happen, and then I still want to be able to recover my data. 

Another utterly unrelated image to cheer this post up.


I decided to go with three external USB HDDs for backup/storage. Since I expect an average HDD lifetime of no more than five years, these will mirror each other. In particular:
  • One disk will act as a primary storage and accessed frequently. The two secondary HDDs are used to backup the primary disk.
  • One of the secondary HDDs is from a different vendors than the other two HDDs.
  • The two HDDs from the same vendor were not bought at the same time, in order to avoid batch errors from this particular vendor.
  • All three HDDs are not kept at the same place, so even if a meteor strikes, a copy of data should be safe. (OK, depending on the size of the meteor, of course.)
Next consideration is the file system. Most Linux PCs use ext4 as default, but I decided to store all my precious data on a BTRFS system, since this file system has intrinsic error recovery (switched on by default) which prevents data rot.
  • On my desktop PC, the partition which holds precious data, is formatted BTRFS
  • Two of the backup HDDs were formatted with the BTRFS file system: the primary HDD, and one of the two secondary HDDs.
  • One of the secondary backup HDDs is formatted ext4, just in case some grave bug in BTRFS should turn up and render my partitions unreadable.
  • All disks are encrypted using LUKS. This is probably the weakest point, if LUKS breaks for some reason (or I forget the passphrase), I will be in trouble.
Finally, I have added some software level redundancy:
  • Instead of compressing large archives to .tar.gz, I compressed them to .tar.lz using the lzip compression algorithm. Unlike gzip and xz, which can only detect errors, lzip is capable of recovering from a few bit flips. Tar supports lzip natively:
tar cvfa FOOBAR.tar.lz FOOBAR/
should do the trick.
  • I keep particular important directories also in uncompressed form.
  • On top of this, I use par2 (see https://github.com/BlackIkeEagle/par2cmdline - it is also in the Debian/Ubuntu repository) with the tar.lz files, which adds 5% redundancy using Reed-Solomon error recovery. Apart of data rot, this may also catch (and recover from) errors which occurred during copying of data from one disk to another. My desktop PC has no error correcting (ECC) RAM, so in principle one bad cell can make havoc on my data.
  • When copying data, I mostly use rsync. Rsync always checks if the file was properly reconstructed on the receiving side. Good thing!
  • I always make a dry run first with rsync -van, and check what files will be overwritten or possibly deleted.
  • In case of copying with other tools than rsync, I can test whether the data actually are the same using the --checksum option in combination with the -n option. This takes very long time, and is unfeasible for routine use. I also use --checksum -n  when I suspect problems for other reasons, which is very rare.
  • Just in case everyone will forgets how LZIP, TAR and PAR2 works, I added the uncompressed tarballs to the disk.
The three backup disks are arranged in a hierarchy, where I make regular backups from my Desktop/Laptop to the primary backup disk, with is formatted with the BTRFS filesystem. A few times per year, I then propagate the data from this primary disk to the two secondary BTRFS and ext4 HDDs.

Safe and secure storage of your data may require proactive work from your side.
To date I do not see any viable set-and-forget solutions. (Picture entirely unrelated to this post.)


Advantages of this method are obvious:
  • all 4 HDDs (the one in the Desktop/Laptop and the three backup HDDs) must catastrophically fail at the same time, before I suffer data loss. 
  • Rsync should prevent propagation of errors due to bad RAM during copying.
  • Even if single bits are flipped, both par2 and lzip should be able to recover the medium.
  • Along with rsync dry runs, the delayed propagation to the secondary disks and physical separate storing of the HDDs, I have multiple points where I can detect fumble fingered actions, like accidentally erasing important data and propagating this to other disks.

Hope this may inspire someone. Suggestions are more than welcome. :)

Sunday, July 27, 2014

No Visa Required - Part 3: Transnistria - Tiraspol

During our stay in Chişinău (see previous post) we took a one-day trip to the rouge nation of Transnistria, also called Приднестровская Молдавская Республика or simply Приднестровье [Pridnestrovie]. Often it is abbreviated as ПМР [PMR].

Border checkpoint of Transnistria: Hammers and sickles. (Source: flickr.com, as I did not dare to make photos at the actual checkpoint myself.)

Not recognized as a country as such by the United Nations, this area between Moldova and Ukraine is still claimed by Moldova as part of its sovereign territory, but can be regarded as a region which Moldova lost control of. It is recognized by the also non-recoginized countries of South OssetiaAbkhazia and the republic of Nagorno-Karabakh (yeah, never head about those before either).

Needless to say, you will not find Transnistria on any standard map. It is simply marked as Moldova. Under normal conditions, you can enter the country freely, but as far as I heard, if you want to stay over night, you may have to bribe your way out of the country again.

It is truly a strange country, a weird mix of smuggler state run by an ex-KGB gang with a former Soviet theme omnipresent as socialist realistic artwork.
Here, Soviet Russia has not ceased to exist, but transformed into something much more bizarre, going way beyond the fateful year of 1989. Thus, in Transnistria, they have their own hammer and sickle passports and own hammer and sickle minted currency. And none of this is recognized outside Transnistria, of course.

Our target was the capital city of Tiraspol. There is a fancy propaganda promotion video worth watching, and as it is with propaganda promotion videos it looks like a cheerful place. Titled "I love you, Tiraspol...":



Preparing for the trip, we left all credit cards in our hotel in Chişinău, and just took a limited amount of hard currency with us. I think I merely carried 30 US$ and 20 € along.

Daily bus connections exist form Chişinău bus station to Tiraspol, first task is to find the proper ticket booth, which is different from the others.


Chişinău bus station.
Gilbère, my experience travel companion you may recognize from previous posts, has done this trip earlier. After a slight moment of orienting himself, he found the "Casa de Peron", which was nicely build round 2 cut down trees.

Here you can get bus tickets for Bender and Tiraspol.
We got two tickets for like 3 or 4 €, very good price for a full day for the commie open-air museum. A very caring and service minded bus stewardess (truly relieved that we spoke a few words of Russian) directed us to the proper bus and handed us the forms which we needed to fill out to enter Transnistria as a foreigner.




Our passports where collected, and Gilbère scored additional bonus points from the bus stewardess for filling out the documents properly in first attempt. Way to go, Gilbère!

Form for entering Transnistria, front page.

Form for entering Transnistria, back page. Note the PMR hammer and sickle stamp to the lower right.

There was nothing unusual about the documents themselves, but being confronted with a stamp featuring a hammer and sickle in 2014, felt to me as a joke - or a preparation for entering a cold war museum. It should turn out they are not making fun at all.
This is the real thing.

After almost 1 - 2 hours of driving, we arrived at the unofficial border to Transnistria. First we zigzagged without stopping through a Moldovan cordon, and then we reached the Transnistrian checkpoint we had to get off as the only foreigners on the bus.

First was there was a small booth with a passport scan. Our friendly and service minded stewardess presented us to the female officer (all in Russian): "Look at these beautiful two faces, they want to enter. :-D".
We put on our best smiles.
Officer: "Journalist?"
Me: "Niet, Tourist".
Officer: "Camera?"
This was a tricky question. I think I got at least three cameras with me, two of them in the bus, an other in the jacket I carried. So what would be a proper answer here?
Me: "Da."
Officer: "Pokazhite." [ = "show me"]
Bummer, what to do now... I simply just grabbed the small video camera I had in my coat and silently passed it to her. She took it and after a quick inspection: "O.K."

Then we were taken to the second checkpoint, a few meters further down, where we had to turn in our filled out documents, to register for a one-day stay. We politely queued up in the line, until the officer told us that we can skip and come to the desk for immediate processing. Which was cool, felt like the fast-track lane for frequent fliers at the airports. Very friendly.

After a few minutes he handed us back our papers, duly signed and with the latest exit time marked, which was 22:07. Probably this means we can stay there 10 hours, judging from our "check in" time. While handing over our documents he politely wished us "Good luck!", whatever that meant...

After passing a city which was named after the robot from the Futurama series (Bender), we arrived at the train station of Tiraspol. Note the funny detail, railway station in Moldovan (or Romanian) is "Gara", and in Russian it's "Voksal", here both are written with Cyrillic letters, i.e. also the Romanian word. I learned that people in Transnistria are forced to use Cyrillic letters, even for the Romanian language.

Tiraspol railway station. "Gara" written in Cyrillic letters.

As mentioned before, Transnistria is run by a rouge ex-KGB gang. One part representing this is the local Sheriff conglomerate, which owns all petrol stations and supermarkets, a mobile network, a Mercedes-Benz dealer, a football club FC Sheriff Tiraspol, a 200 MUS$ stadium and much more you need to run a country.

As the Sheriff name suggests, here police forces are at work here, upholding the law.

Truth is, it is one big mafia-style money laundry. Wikipedia has more details.

Another major industry and the largest exporter is the KVINT factory, producing brandy like spirits, in particular Cognacs from the region. Note the funny way they mirrored the V, for a serif font it should be the other way round.


Strolling down the Lenin street, we passed the ministry of justice, again nicely decorated with the Transnistrian hammer and sickle logo.

Ministry of justice, at the "Uliza Lenina" (Lenin street).

Gilbère tried again to blend in with his polyethylene bags, quite successfully this time, I'd think. Note also the extremely clean streets, no trash found anywhere! However pavements was not always trolley friendly.



Reaching closer downtown, we passed the embassy of South Ossetia and Abkhazia. At this point, Gilbère was getting slightly more nervous, yet I managed to take a few pictures of it. Note in particular the leather jacket guy in the picture below, just in front of the embassies:

Embassy of South Ossetia and Abkhazia.

.. that guy, stopped us 50 meters earlier, asking us in Russian, what time it is.
Now, think of that, in the age of smart phones and cheap watches, a guy asking us what time it is!

Honestly, this is when I went paranoid, I seriously doubt he was asking for the time, but possibly testing our language skills. My Russian is not good enough to answer him, so I merely lifted my arm exposing my watch, and he thanked and backed off.

Possibly he now thinks we are ultra professionals. This way, we did not expose our nationality by our accent.

We stopped by for lunch at Andy's Pizza, (0.30 € for a vodka shot) which looked very much like any other Pizza place. Reaching closer to the center, the propaganda was more frequent in form of massive LED billboards showing tank processions and marching people along with its Transnistrian version of the red soviet flags (which is the red soviet flag with a green banner added - following the Moldovan SSR).


Close to the parliament we encountered a group of young people, practicing for their national day, by marching in front of what Gilbère thinks was the local house of culture.

Young citizens marching, preparing for the upcoming national day, presumably.
We stopped some locals to take a picture of us:


The Arstotzka T-shirts are absolutely intentional. Gilbère bought them in particular for this place in honor of the computer game "Papers Please" by Lucas Pope. Tiraspol is probably one of the last places which still endorses the true Arstotzkan spirit. Glory to Arstotzka!

Glory to Arstotzka!
I was slightly hesitant here, the idea of explaining our T-shirts and our posing in front of the parliament to a Transnistrian officer did not comfort me. Gilbère seemed very pleased though.

My ongoing paranoia was fueled even more, when this car stopped. The guy inside clearly watched us and talked into mobile phone for some time, possibly a colleague of the leather jacket guy who didn't know what time it was.


This was the moment, when we though of a discrete escape, and a natural spot would be the local museum just next door. Three very friendly ladies (only Russian speaking) welcomed us, and one gave us a dedicated tour (there where no other visitors) with long explanations (still, all in Russian).

Слава Арстотцка!
Later Gilbère pointed out we paid 10-fold tourist price, like 20 rubles instead of 2 (= 1 € instead of 0.1 €). How did they figure out we where not natives? Even Gilbère's polyethylene bag did not make an impression.

Another thing I encounter often during my trips to Russia, is the glorification of local heroes in front of e.g. educational institutions, such as this:


A few more impressions of the Tiraspol:

The tidy roads of Tiraspol.



Our trip concluded at the local KVINT booze-shop, with a full exchange booth inside, where you could change your hard currency into local units.

Exchange bank in a KVINT shop.
Those local units can then immediately be exchanged into booze. Note all this is an irreversible operation, it is not possible to change the local units back to hard currency.

Selection of spirits, just next to the currency exchange booth.

Finally we took our bus back from the train station, and passed by several checkpoints without any issues. At one point at the Dniester river, we saw a camouflaged Russian guard post, there should be several Russian troops in Transnistria, but we hardly saw any of them.

Things went very smooth, we did not have to leave the bus. Some officer with a big soviet hat (hammer and sickle) simply picked up our slips, and that was it.

No stamp, no signature. There is no proof in our passports that we have been in Transnistria. Sadly.

One week later, due to the escalating conflict Ukraine spilling over to Odessa, we found this notice on Wikitravel, which generally said that Transnistria was closed for foreigners:



Possibly the border was only closed from the Ukrainian side, but we are not sure. Nonetheless, this is the third time something dramatic happens where ever we go, seems to be pathologic. All planned well in advance, always immediately after booking, something strange happens, like invasion, civil war like conditions, planes shot down, etc. If Gilbère is up to open a travel agency (he surely got talent for making bookings and arranging stuff) he might need more luck here.

Full album here. Enjoy!

No Visa Required - Part 2: Kiev - Chişinău

This post describes the best train part of our "No Visa Required" trip from Kiev to Chişinău, i.e. going from Ukraine the beautiful country of Moldova.

The main purpose of this trip was to experience the famous 3TE10 trains and derivatives. Any fan of Russian diesel trains appreciate its distinct sound, though it is not up to date with respect to the European emission standards:



My travel companion Gilbère recommended a special route, which goes straight trough Moldova, 10 hours of train ride for less than 300 km. Yes, this is going to be a very slow ride.


Again, as usual with the travels which Gilbère books, some dramatic accident happenes quite closely to our trip. In this case this particular train derailed and crashed into a freight train a few weeks after we took it.

From Reuters.

So, at 1:30 in the morning we were waiting at the railway station at Kiev. We had a shared compartment for four people and no idea who the other two would be. Looking around me at the platform, I mostly saw various versions of dodgy training suit wearing youngsters, ceaselessly spitting on the ground, I feared the worst.

Things turned out very positive though, a very nice Moldovan couple from Israel were our companions. The guy immediately offered us vodka and bread with cold meat. After a few shots, the mood was better and the lady told him off to keep it low.

After a nice sleep-in there was breakfast:


In the morning, we were also approaching the Moldovan border, and officers with very cute looking dogs entered the coach.



As an officer I would really feel my authority compromised by all the passengers addressing the cute dogs.

We crossed the Dniestr river, which separates Ukraine from Moldova. Housing and villages looked more shabby in Moldova I must admit. Ukraine seems to be better off.

Dniestr river.

This was also when the electrification of the tracks stopped, and a 2TE10 was pulling us across the country side. Beautful!

Unfortunately windows could not be opened, except for the one in the toilet, where I could stick out a hand to capture this scenery.

2 or 3TE10 pulling us uphill.

At a first stop in Ocniţa, I tried to blend in with the locals, meaning wearing plastic slippers, unfortunately I did not bring a training-suit.

"Technical stop" (Yes, those slippers are intentional.)

A scent of coal hangs in the air, since the coaches are all coal fired.

Coal furnace for the coach.

During the stop, babushkas try to sell their goods. In particular suspect where the shrimps, far away from any natural sources.

Shrimps, anyone?

Scenery was epic, though. A bit fantasy like, possibly close to Ungheni, the mountains at the horizon is possibly already Romania.


A stop at the train station of Ungheni was only 300 meters from the Romanian border. I felt tempted to run there, but the stop was too short. Our diesel trains was changed to a 3TE10 but only 2 units. Gilbère notes that Moldovans tend to mix all the train sub-units up.



We will make a decent video of the entire trip later, hopefully.


And the ride goes on and on and on...




And finally in the evening (19:35) we arrived in Chişinău!


Chişinău, possibly the least known capital of all European countries. Gilbère was talking for ages about the blessings of the "Hotel Cosmos". This wonderful marvel of a reminiscent communist era, is a functioning time machine. As soon as you enter the main entrance you are taken 30 years back in time.


An astonishing lift took us to our rooms (checkout Gilbère's Facebook profile for more stunning pictures of the lift at Hotel Cosmos.)


Hallways of Hotel Cosmos could probably provide a suitable location for various gory movies.

Redrum. Redrum. Redrum.
The interior reminded me of wallpaper artwork from Windows 3.1.


During our trip in Ukraine, I noted a systematic absence of toilet paper at most lavatories. Therefore, as a rule, we always kept some rolls with us in stock. Staggering with the superior quality of the rooms, along with the very affordable price tag per night (I somehow suspect a money laundry here), I momentarily abandoned this fundamental law of traveling in eastern Europe.
After checking in and entering my room, I indeed found a very decent toilet, which was a very pleasant surprise. Cheerful and enthusiastic of this marvelous hotel Gilbère has discovered, I found comfort at this wonderful spot to do my serious and urgent business.

Now take a close look at the picture below. What is missing here?


After a proper formulated text message ("Where is the f*cking toilet paper!") Gilbère organised a rescue team which eventually resolved my plight.

But Chişinău is not just Hotel Cosmos, yes it is hard to believe, but another gem is the Hotel Chişinău, located just 100 meters away from the Hotel Cosmos.

Hotel Chişinău, "Stalin's style hotel".

You simply got to love their commercial video!
In the opening scene they blatantly boast of being a "Stalin's style hotel" , and the personnel are really making a genuine effort to act natural and make you feel at ease. The movie itself is truly a master piece on multiple levels, a hybrid of Stalin and Gheorghe Zamfir. Highly recommended! Five hammers and sickles!

Watch it and give it thumbs up, they surely deserve it:




Our commie architecture genes were truly pleased when passing by the interesting reminiscences of ancient times.




True Asberger people will notice a little break in the symmetry in the top right, where a barrier between two rooms where removed.


One part of the so called city gates of Chişinău. Concrete on concrete.

The gate of Chişinău. Or at least one side of it.
We did not stay long in Chişinău. For dining we highly recommend the "Pegas" restaurant. Excellent!

Not much to tell here, but coming up: Part 3 - Tiraspol, an open air museum for commie lovers!

---