MySQL accessing a lot of deleted temp files

Question

MySQL version: mysql Ver 8.0.25-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))

I have a server that hosts a MySQL database and Apache Airflow. It was working fine for a couple of months. For a couple of days, my services are constantly crashing.

I see that the error is due to insufficient Disk Space: These are the errors that I see when I try to run my Airflow Service.

This is happening even though I see there is disk space left:

I figured out that this could be due to a MySQL bug, https://community.oracle.com/tech/developers/discussion/4267115/mysql-daemon-not-releasing-deleted-temp-file

But I have a newer version of MySQL installed so this should not happen.

Tried restarting the server a couple of times.

Tried killing the MySQL services manually, but it always goes back to the same state.

So I am not even able to get it fixed temporarily.

Not able to understand how to fix the error permanently, any help will be appreciated.

Does this answer your question? Disk full, du tells different. How to further investigate? — Michael Hampton, Jun 21 '21 at 20:40
Lower max_connections in my.cnf to only 30. Is tmpdir being set in the config file? — Rick James, Jun 22 '21 at 05:53
Yes, @MichaelHampton, Inodes were 100% used, I increased the disk space. So seems it got solved temporarily.
Mysql is still accessing those deleted files.

So I meant, MySQL accessing deleted temp files is unrelated. — ofnowhere, Jun 22 '21 at 15:05
I did that a couple of times, the PID changes, but it keeps accessing those files. — ofnowhere, Jun 22 '21 at 15:10
100% inodes used -- sounds like lots of tiny files on the file system. In my several decades of using *nix, I don't remember running out of inodes before running out of space. I wouldn't even know what OS parameter to change to increase the ratio between inode and data blocks. — Rick James, Jun 22 '21 at 15:45
@RickJames 100% inodes were because of Airflow, Each dag execution in Airflow creates a log directory(multiple jobs are running every couple of minutes), we have logrorate to delete the log files, but directories remain here.
I suppose that is what caused it. — ofnowhere, Jun 22 '21 at 19:03
I will write a cron to clean up empty directories and then try that. — ofnowhere, Jun 22 '21 at 19:03
Not sure if it is a bad, but a bad implementation of a feature. — ofnowhere, Jun 23 '21 at 16:57

score 1 · Answer 1 · answered Jun 22 '21 at 15:55

Out of inodes? Check with df --inodes

A quick, but temporary, fix would be to toss files in .Trash and Download

Maybe there are lots of unnecessary empty files: https://www.interserver.net/tips/kb/solution-running-inodes/ https://askubuntu.com/questions/231585/running-out-of-inodes

Maybe "compressing" a few little-used disk trees would help. (You will need a few free inodes before trying this.) Or tar up some trees that you don't need.

(etc)

MySQL accessing a lot of deleted temp files

1 Answers1