0

We recently moved the data directory of our database to a new hard drive because the old one was full.

I needed to clean up some space in order to install something, so I deleted the old data directory.

Now postgres doesn't start anymore, presumably because I deleted the postgresql.config

There are no logs in /var/log/postgres/....

The data directory we copied to the new hard drive doesn't contain a postgresql.conf, i don't know why though.

Deleting the old data directory also didn't free any space even though the folder was 21GB, so I cannot use initdb to create a new config and point it to the new data directory because the disk is full.

Is there any way I can get the system back up and running? We are using postgres 12, if that matters.

C5H8NNaO4
  • 103

1 Answers1

2

Postgres will indeed refuse to start if the postgresql.conf file does not exist, although I'm also not sure why your data directory doesn't include one. Are you sure you didn't accidentally delete it?

Regardless, your Postgres package should have come with sample configuration files, including a postgresql.conf.sample. If you built it from source with the default options, you can find it in the /usr/local/pgsql/share directory. The Arch Linux package manager install puts these "architecture-independent support files" in the /usr/share/postgresql directory (this is what the pg_config utility calls them).

You can see where your particular installation was configured to put them by querying the pg_config utility with the --sharedir option. The server does not need to be running, and I temporarily renamed my own postgresql.conf file to see whether I could still query pg_config, and I could, so I don't know whether this is a viable option for you, unfortunately, sorry about that.

To actually be able to create a configuration file and initialize the postmaster daemon though, have you tried runninglsof +L1 | grep deleted to see if any active processes still hold a reference to the old data directory? This always seems to be the problem when unlinking an inode (i.e., deleting a file/folder) doesn't actually free up the corresponding space.

Jose Fernando Lopez Fernandez
  • 231
  • Thanks a lot. I managed to create a new postgres.config/pg_hba. I copied the config to the new data directory and am able to start postgres using sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl -D /mnt/data/main -l /var/log/postgresql/tmp.log start. Everything seems to be working again. The only problem I have now is that i can't use systemctl nor service to start the database. I have to use pg_ctl. – C5H8NNaO4 Oct 02 '20 at 14:10
  • I tried lsof +L1 | grep deleted to find locks on the file. I even rebooted the server. The old directory is definitely gone, but the space doesn't free up. – C5H8NNaO4 Oct 02 '20 at 14:11
  • Did you edit the postgresql service file? You can use sudo systemctl edit postgresql to override settings from the original unit file, that way the changes don't get overwritten when you upgrade the postgres package. In particular, I would look at overriding both ExecStartPre=... and ExecStart=.... Regarding the file system space, that's definitely weird. I'm not sure how a reboot wouldn't have helped there. I might still try lsof +D /var/lib/postgres, but make sure to run it as root. If systemd or anything is opening a directory, stream, block, etc., that will find it, but I'm stumped – Jose Fernando Lopez Fernandez Oct 02 '20 at 14:24