66

A disaster just occurred to me after I ran the command yum remove python and now I can't boot the server up anymore.

How it happened: I tried updating some apps via yum on my CentOS 5 VPS and the command was failing due to some weird python 2.4 error. I noticed that my version of python was old and I tried reinstalling it by first removing it, and so I did yum remove python.

After that it asked me something about removing dependencies and it looked like nothing I could miss so I clicked Y.

So the aftermath of that was that I was unable to run any command what so ever. I even tried cd /var/www but it said something like "command does not exist in /usr/bin". When I used tab to see folder navigation suggestions, the file structure still seemed to be there (at least the /var/www bit which is really important to me). After that I tried restarting the vps (from the admin panel since reboot command did not work) and now it doesn't boot anymore.

Now my question is: how can a command like that possibly destroy my server like this?

chicks
  • 3,819
  • 10
  • 28
  • 36
tadoman
  • 773
  • what kind of error do you have at boot ? – olivierg Apr 15 '17 at 20:40
  • 33
    Incidentally, apt-get remove apt works. I haven't tried dpkg --remove dpkg yet. I bet that's really bad. – joshudson Apr 16 '17 at 03:58
  • 19
    You do have a backup, right? – vasin1987 Apr 16 '17 at 17:55
  • 21
    @joshudson I just spun up a Debian 8.4 live CD in a VM and tried it. Result: # dpkg --remove dpkg spits out dpkg: error processing dpkg (--remove): this is an essential package; it should not be removed. If I add --force-all to dpkg's command line, dpkg spits out a whole bundle of warnings and proceeds to remove itself, along with breaking about two dozen other packages that depend on dpkg. On a real system, I'm pretty sure you'd have some trouble recovering from that, but you probably could (there's little magic to .debs); CentOS may or may not be similar in this regard. – user Apr 16 '17 at 20:35
  • 6
    @joshudson It works because there's a copy of apt running in memory when you issue the command... as soon as it's finished (and the program terminates), you won't be able to use apt any longer. – SnakeDoc Apr 16 '17 at 21:11
  • Would yum still be able to remove packages if not run by root? Don't most package management commands usually have a "read-only" mode to test run them with? – samus Apr 17 '17 at 13:53
  • 2
    Yum itself is implemented in Python I believe... So removing Python shoots your own foot pretty well! – bobflux Apr 18 '17 at 13:29
  • 1
    @SnakeDoc I think joshudson's point is that apt doesn't provide protection against this sort of mistake, the way dpkg apparently does. – Kyle Strand Apr 18 '17 at 23:48
  • This was more or less like exploding the System32 folder from a Windows box... – T. Sar Apr 19 '17 at 19:22
  • @KyleStrand Probably because apt is really just a convenience utility; a very useful convenience utility, mind you, but perfectly possible to get along without in a pinch. The real trouble begins when you remove dpkg. That said, as I alluded to above, because .deb files are really just compressed archives of program binaries plus some metadata and scripts, with some work but not too much difficulty you probably could get back to a working system manually without needing to run dpkg, at least to the point that you can run dpkg. – user Apr 20 '17 at 07:28
  • On Gentoo - emerge --ask --unmerge portage gives Not unmerging package sys-apps/portage since there is no valid reason for Portage to unmerge itself. – VL-80 Apr 20 '17 at 12:52
  • @peufeu So, if we can't remove python, then is there a way to upgrade the default python version to 3.5.x? I tried upgrading but after upgrading I got this error when running yum -bash: /usr/bin/yum: /usr/bin/python: bad interpreter: No such file or directory – Muaaz Khalid Aug 07 '17 at 10:15
  • http://www.keepcalmstudio.com/gallery/poster/1BLAEVH yea I made few bucks due to this. – AKS Mar 13 '19 at 23:04

6 Answers6

107

Frankly, because you did something you didn't fully understand. Python is an essential part of the OS and the things you considered unimportant are very important. Restore from backup.

When you removed Python, yum showed you a long list of packages that would also be removed. This list contains such essentials as yum itself, coreutils, net-tools and others. You confirmed to yum that you know what you are doing and want to proceed anyway. The result of this is a non-working system. This shouldn't be surprising.

For the record, on newer CentOS version this isnt't possible anymore, as certain packages are now marked as protected and can't be removed, only reinstalled or upgraded. And since CentOS 5 is now EOL anyway, this is a good time to upgrade to a newer version.

Sven
  • 99,533
  • 15
  • 182
  • 228
  • 3
    Haha well that escalated quickly (yum remove python == just wipe whole machine :D ). Do you reckon whether there is any way I can at least restore the files that were in /var/www or is that completely foobard as well? – tadoman Apr 15 '17 at 20:02
  • 1
    Restore is something you need to consider before it's necessary. Do you have backups? –  Apr 15 '17 at 20:13
  • Of course I have backups. this did not answer by question so I'm still interested – tadoman Apr 15 '17 at 20:20
  • 8
    @tadoman: If you can boot into some kind of rescue system, you will be able to mount the disk and recover data from it (this should be all of your user data, including config files). You have to contact your provider to discuss what is possible in their environment. – Sven Apr 15 '17 at 22:30
  • Files in /var/www are not dependencies of anything so they should still be there. – reinierpost Apr 16 '17 at 08:42
  • Are you saying that nothing other than python requires coreutils,net-tools, and the like, or that yum remove simply doesn't know or care about those other dependencies? – Russell Borogove Apr 16 '17 at 15:09
  • 6
    @RussellBorogove: I don't say either of this. yum works by walking down the dependency graph and it removes packages until all dependencies are satisfied. If package A depends on package B and package B depends on package C, yum will also remove package B and in turn, package A to satisfy all dependencies if you try to remove package C. For central packages (like python), this can result in a large number of removed packages that are seemingly unrelated. – Sven Apr 16 '17 at 15:20
  • 1
    Ahhh, I get it -- coreutils requires python, so without the latter there's no point in leaving the former installed. – Russell Borogove Apr 16 '17 at 16:25
  • 4
    @RussellBorogove: Yes, exactly, although coreutils doesn't depend on python directly in CentOS 5, but via one or more intermediate packages - the result is the same though: Removal. – Sven Apr 16 '17 at 16:32
  • 4
    The number of intermediate packages to coreutils appears to be two. python -> cracklib -> pam -> coreutils The dependency exists because there are python bindings to cracklib, pam was built with cracklib, and that su is integrated with pam. Of course it took out lots of important things too, like yum to install any more packages... – John Mahowald Apr 18 '17 at 05:50
69

I'm sincerely sorry: I can feel the pain to have a server unbootable/unserviceable.

However, I'm lost when reading that:

After that it asked me something about removing dependencies and it looked like nothing I could miss so I clicked [Y]

The list of to-be-removed packages surely was really huge, as python is an essential part of RHEL/CentOS. You should never confirm some warning message you don't really understand.

The best thing you can do, as already suggested, is to boot via a recovery media (ie: livecd), extract the required data files, and reinstall your machine with a newer CentOS release (and as CentOS 6 is quite old, I strongly suggest you to rebase on CentOS 7).

shodanshok
  • 50,565
  • 2
    For the future: build servers with the OS on LVM LVs (which ISTR is the RHEL default). Before anything that has even the slightest chance of breaking the system, create snapshot(s). If it does indeed break the system, you can then revert to the snapshot in minutes. Otherwise when all is OK you can remove the snapshot. https://unix.stackexchange.com/questions/18913. NB snapshots are not backups. You still need backups for situations where the snapshot cannot save you. – nigel222 Apr 18 '17 at 12:48
  • On Debian/Ubuntu, he could boot recovery image(e.g. finnix), unpack base system with debootstrap, chroot and apt-get install ubuntu-desktop. No idea whether centos/rhel have debootstrap equivalent. – Edheldil Apr 19 '17 at 08:56
  • 1
    CentOS 6 is supported until November 2020. There's no rush to upgrade (though going to 7 does have plenty of advantages). – psmears Apr 19 '17 at 09:10
15

You did something without fully understanding the consequences

That install is irrecoverable, would require a lot of work to reinstall centos5. And that is a bad plan because

  1. CentOS 5 is End of Life, and therefore has no updates. This is extra serious given it sounds like a webserver serving content on the public internet and that you use panel apps to control it.
  2. CentOS >5 would have stopped you from doing this update and killing the box. That's a nice airbag to have.
  3. CentOS 7 claims to support major version upgrades in place. I've never used it, but being able to jump from 7 to 8 when it releases will be very good. Debian's had this since forever, but Redhat always required a reinstall for major version jumps.

Solution

Your best bet is to create a new VPS, fresh install CentOS7, and then reattached the old centos5 disk volume and mount it read-only. Then work to copy (not move) your data from the old drive to the new.

Note this would be my method using AWS. If your VPS provider can't attach disks to different VMs then you'll have to tweak the plan.

No matter what you do, please consider setting up automated backups in the future. It wouldn't save you, but it would make recovery somewhat more flexible. Right now you need the data on that disk in a new working server. Don't loose the existing disk.

Criggie
  • 2,320
  • 14
  • 27
9

How could this happen? Well, quite simple: By removing parts that were critical to your server.

Next steps for you: re-deploy a fresh OS and restore your data from backups.

EEAA
  • 109,904
  • 1
    Uninstalling a package should never ever remove data. There really shouldn't be a need to restore anything, unless some package maintainer royally screwed up. – Jörg W Mittag Apr 16 '17 at 11:04
  • 6
    If the OP restores a fresh OS like I recommended, restoration of data and configuration will most assuredly be required. – EEAA Apr 16 '17 at 11:06
  • @JörgWMittag I think you might have misunderstood the use of the word "restore." EEAA means that the system will have to be reconfigured and the existing data placed somewhere on the new machine. "Restore" might not be the best word for that, although I could see someone using database restore functionality for that piece. – jpmc26 Apr 19 '17 at 02:42
  • @JörgWMittag Oh, his data is still there, most likely. It's just that with his system nonfunctional he has no way of accessing it. – Shadur-don't-feed-the-AI Apr 20 '17 at 04:54
4

As pointed out by dragon788 and others in the comments, in Gentoo, the developers also maintain a set of tinderbox packages which are just pre-built, binary versions of a set of core packages of the OS for just such situations. If you lose a core package, you just get the system booted to the LiveCD/DVD, mount the OS drive of the broken server and unpack the tinderbox package(s) to the filesystem, unmount, reboot and, if it boots up correctly, rebuild the packages back to your server's specs and configs.

So, to perform something similar in CentOS, I think you would need to find the right versions of the RPMs that were uninstalled, then boot to a LiveCD/DVD, mount the OS drive and chroot in (perhaps... if you are familiar with how to use the "--relocate" flag for rpm, you might not need to chroot), then reinstall those packages, unmount and reboot.

Of course, since support for CentOS 5 ended last month, after you get the system rebooted properly, you can update it to a current version.

HTH.

B.Kaatz
  • 179
  • 4
0

You can typically boot from the install media and then chroot to run commands or enter into the current install and recover files or perform a reinstall of packages.

dragon788
  • 836
  • 4
    You won't be able to chroot into a root directory with no useful binaries in /usr/bin, etc. It would give the same result as booting into the dead system. Chroot also isn't necessary because once you have the filesystem mounted, you can copy out the files that you want to rescue from /var/www. – qris Apr 16 '17 at 11:45
  • Definitely true, I'm recalling installing Gentoo/Arch where there are usually some useful binaries available. I've never uninstalled everything first. ;) – dragon788 Apr 16 '17 at 13:04