4

I don't even know where to start debugging this one ...

I have a Debian 9 VM (Linux 4.9.0-7-amd64 on x86_64) with four hard drives -

  1. / (RootFs) 30Gb 26% used raw
  2. /mnt/data1 2TB 91% used qcow2 } same physical HDD
  3. /mnt/data2 2TB 88% used qcow2 }
  4. /mnt/data2 2.5TB 73% used raw (did not exist when this story starts)

A few days ago I tried to copy a file over to drive 3 using Samba and the server locked up, I rebooted and the server locked up again immediately after the GRUB screen. I used a recovery USB and edited fstab to not mount the non-root, then rebooted again. That worked and got the server back up and running. I manually mounted drive 2 and everything was ok, but when I mounted drive 3 it immediately locked up again.

I rebooted again, uncommented drive 2 in fstab, tried manually mounting drive 3 again and got the same outcome (surprise surprise). Figuring it was a drive failure I used gddrescue to copy drive 3 to a brand new drive, drive 4. I left that overnight and by morning it had completed - with zero errors. I rebooted, and tried to mount drive 4 and that worked fine. I then extended drive 4 using gparted to fill the entire drive and added an entry to fstab, rebooted and again that worked fine (can read from the drive, no problem).

However, when I tried to copy a file to drive 4 using Samba the same thing happened, the entire OS locked up and I had to stop the VM. I tried to copy a file locally from drive 1 to the drive 4 and that locked it up too.

I've looked in the /var/log/syslog, debug, messages and kern log files, and there's nothing remotely curious before the start-up entries which might explain what happened just before the lock-up.

Vitani
  • 163
  • 1
    Sounds like the virtualization has overcommitted disk space, and when you fill up the disk it stops the VM until disk space becomes available. – wurtel Oct 05 '18 at 14:55
  • Good thought, I just checked my host and it does say Usage 100.00% (3.58 TiB of 3.58 TiB) for the disk shared by 2 & 3. I'll try deleting drive 3 (seen as it's copied now) and see what happens. – Vitani Oct 05 '18 at 15:50
  • Yep, that seems to have solved the issue, I will take better care of my virtual drive sizes in the future. If you want to add it as an answer, I'll accept it. – Vitani Oct 05 '18 at 18:24

1 Answers1

2

I have experienced in the past that in the case that disk space is overcommitted across VMs, when the underlying storage fills up, a VM will be blocked from running until disk space ie freed up so that the VM can complete its IO operation. This matches what you describe is happening.

Free up some space on your VM host, and the VM should be able to run.

wurtel
  • 3,914
  • 14
  • 16