150

Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in some kind of viral marketing scheme. As we don't allow ServerFault to be abused in such a way, the original question has been deleted and the answers merged with this question.


Here's a an entertaining tragedy. This morning I was doing a bit of maintenance on my production server, when I mistakenly executed the following command:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

I didn't spot the last space before / and a few seconds later, when warnings was flooding my command line, I realised that I had just hit the self-destruct button. Here's a bit of what burned into my eyes:

rm: cannot remove `/mnt/hetznerbackup': Is a directory
rm: cannot remove `/sys/fs/ecryptfs/version': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/inode_readahead_blks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_max_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/delayed_allocation_blocks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/max_writeback_mb_bump': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stream_req': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_min_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stats': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/trigger_fs_error': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/session_write_kbytes': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/lifetime_write_kbytes': Operation not permitted
# and so on..

I stopped the task and was relieved when I discovered that the production service was still running. Sadly, the server no longer accept my public key or password for any user via SSH.

How would you move forward from here? I'll swim an ocean of barbed wire to get that SSH-access back.

The server is running Ubuntu-12.04 and hosted at Hetzner.

Jonas Bylov
  • 1,613
  • 49
    Restore from backups. Honestly, this is one of those no-easy-way-back scenarios. – MadHatter Apr 07 '14 at 06:45
  • 322
    How do you even type --no-preserve-root accidentally?! :-o – ThatGraemeGuy Apr 07 '14 at 06:46
  • 151
    Greame, the keys are like right next to each other. – MadHatter Apr 07 '14 at 06:47
  • 41
    Tuesday work: Look for new job ;) Take it as a lesson why backups are needed. – TomTom Apr 07 '14 at 07:00
  • 2
    Reach for your backups and DR plan - you have both don't you ? – user9517 Apr 07 '14 at 07:20
  • 5
    Haha, thanks for all the lovely encouragement. Thankfully backups are safe. – Jonas Bylov Apr 07 '14 at 09:05
  • @TomTom He never said he had no backups; quite to the contrary. – DBedrenko Apr 07 '14 at 09:40
  • 1
    Thanks. Nice to see someone with enough brain to make backups - too many people trust in that (guild of that myself). Gratulations. Now you have the bad work of rescuing the system - but at least you ahve data. – TomTom Apr 07 '14 at 09:44
  • 2
    @TomTom this is actually not why backups are needed. Because, "distilled stupidity" [1] has the potential to ruin backups just the same. Backups are there for cases of accident or emergency. ([1] re-using with permission) – sehe Apr 07 '14 at 10:06
  • 3
    @user22394: Is what we have here not an accident? Like when I almost cut off my thumb was an accident out of stupidity, but still an accident? – phresnel Apr 07 '14 at 10:40
  • 9
    an entertaining (and eye opening) read, if you don't have better tools at hand than the existing system and a few backups: http://www.ee.ryerson.ca/~elf/hack/recovery.html – Olivier Dulac Apr 07 '14 at 11:08
  • 1
    @phresnel In this analogy "--no-preserve-root" would be like a protective cover that automatically reverts back into protective position right after every single maintenance operation do when operating the saw, or after n seconds idle. In that case, yes it's really surprising that --no-preserve-root could be in effect. Unless, of course, it's hard coded somehwere. And that's stupidity, indeed – sehe Apr 07 '14 at 12:13
  • 46
    This sure seems like trolling to me. You can't accidentally type --i-really-mean-delete-my-whole-root. – psusi Apr 08 '14 at 01:35
  • 3
    I guess you referred this – iceman Apr 08 '14 at 08:21
  • 1
    :-(( Ouch! On the bright side, you're truly an Unix-administrator now, since this is a rite of passage we all (have to?) go through... of course bonus-points are given for having taken back-ups. – Baard Kopperud Apr 08 '14 at 10:38
  • 2
    First off what the hell were you doing deleting stuff on a production server, while the service is running? Why do you even have access to it? This should have been done on a staging server, tested, and then pushed onto the production server. Also if you mounted the volume containing the backups (assuming NFS share here?), why are you even working at all on the production server? Shouldn't you have been on a workstation with the backups share mounted there instead? –  Apr 08 '14 at 05:05
  • Second, as many have said, backups backups backups. A full backup each month and incremental each day (or even every hour in the case of high transaction volume) is usually the safest bet. If you had a staging server, you wouldn't even have needed them anyways. And on the off chance something awful and unexpected happened on the production deployment but everything worked on the staging server, you could have just imaged it and copied it over in about 15 minutes. –  Apr 08 '14 at 05:05
  • @Baard: yeah, but I did it when I was 11, on an IBM PS/2 running Red Hat 4. – nomen Apr 09 '14 at 04:52
  • 1
    Always use -v or alias rm='rm -v', so you can have more chances to press Ctrl-C quickly enough if something is going wrong on the screen. – kenorb Apr 22 '15 at 19:15
  • 2
    If you don't have backups what are you even trying to recover? Everything is gone. Data recovery tools are your best bet but I wouldn't count on them. – André Borie Apr 10 '16 at 22:29
  • 14
    If you really don't have any backups I am sorry to say but you just nuked your entire company. – André Borie Apr 10 '16 at 22:42
  • 9
    I feel sorry to say that your company is now essentially dead. You might have an extremely slim chance to recover from this if you turn off everything right now and hand your disks over to a reputable data recovery company. This will be extremely expensive and still extremely unlikely to really rescue you, and it will take a lot of time. – Sven Apr 10 '16 at 23:07
  • 6
    Backups need to be offsite, offline, and incremental. That you could delete them from your main server means they weren't what I would call backups. – Tim Apr 10 '16 at 23:09
  • 18
    You're going out of business. You don't need technical advice, you need to call your lawyer. – Michael Hampton Apr 10 '16 at 23:10
  • 5
    I'm curious about how the command succeeded though - a simple rm -rf / should fail (or at least it does fail on my personal server) unless the --no-preserve-root option is provided. – André Borie Apr 11 '16 at 03:25
  • 2
    Could we answer instead with a reference checklist of how this should have been prevented? – Houston Fortney Apr 11 '16 at 03:38
  • 1
  • Tape backup 2) Dropbox synchronization script 3) Manually copy your incremental backups to a hard drive 4) BitTorrent Sync etc
  • – Tim Apr 11 '16 at 06:49
  • 2
    He must be trolling , can't rm -rf / without --no-preserve-root – Phyo Arkar Lwin Apr 11 '16 at 09:56
  • And if there is --no-preserve-root , you must just be doing that on purpose. – Phyo Arkar Lwin Apr 11 '16 at 10:04
  • @PhyoArkarLwin You can do rm -rf /*. Or you could be running an old OS; --no-preserve-root wasn't brought into GNU rm until 2006, IIRC. – Jenny D Apr 11 '16 at 10:50
  • Take look at this link : https://unix.stackexchange.com/questions/80270/unix-linux-undelete-recover-deleted-files My2cents –  Apr 11 '16 at 11:20
  • 2
    I don't know if this helps, but this story is entertaining as well. In this story, "rm -rf /" was allowed to run accidentally for a bit before being interrupted. The author tells of some very creative engineering using tools in unusual ways to reconstruct "/etc" and then "/etc/passwd", etc. Unix Recovery Legend – Kevin Wright Apr 08 '14 at 18:34
  • Next time, in Bash: set -o nounset – Luchostein Apr 17 '16 at 13:15