35

I'm having issues deleting a directory on my macOS Mojave 10.14.3 system.

The filesystem is APFS and these are the symptoms:

$ rm -rf strange/
rm: strange/: Directory not empty

$ sudo rm -rf strange/
Password:
rm: strange/: Directory not empty

$ ls -lia strange/
total 0
786499 drwxr-xr-x   3 kk  staff    96 Feb 26 18:56 .
430961 drwxr-xr-x+ 49 kk  staff  1568 Feb 26 21:50 ..

$ ls -lid strange/
786499 drwxr-xr-x  3 kk  staff  96 Feb 26 18:56 strange/

Notice that the link count for the directory is is 3. A freshly created empty directory has a link count of 2.

No other file or directory on the system has the same inode number (this was checked using sudo find / -inum 786499), and I believe that directories on APFS filesystems can't have additional hard links anyway (like they can with HFS+).

Putting the directory in the trash and emptying it returns a dialog with the text

The operation can’t be completed because the item “strange” is in use.

However, running sudo lsof "$HOME/strange" (after returning the directory to my home directory) returns nothing, and rebooting the machine changes nothing. Using sudo lsof +L1 (listing open files with a link count of zero, i.e. deleted files kept open by some application) also does not reveal any paths under the strange directory.

The directory does not have any ACLs or extended attributes:

$ ls -lde@ strange/
drwxr-xr-x  3 kk  staff  96 Feb 26 18:56 strange/

Making a copy of the directory using the Finder creates strange copy. This copy directory has a link count of 2 and can be deleted. Curiously though, the "Get Info" dialog in Finder says that each directory (original and copy) contains an item (it usually reports zero items for freshly created empty directories).

$ ls -la strange*
strange:
total 0
drwxr-xr-x   3 kk  staff    96 Feb 26 23:57 .
drwxr-xr-x+ 49 kk  staff  1568 Feb 27 10:36 ..

strange copy:
total 0
drwxr-xr-x   2 kk  staff    64 Feb 26 23:57 .
drwxr-xr-x+ 49 kk  staff  1568 Feb 27 10:36 ..

enter image description here

Running "First Aid" using the macOS "Disk Utility" does not change anything (returns "Successful" at the end). This regardless of whether this is done on the live system or in Rescue Mode.

Running fsck_apfs -y on the disk in Rescue Mode mode does produce a number of warnings on one of the filesystem snapshots of the following type:

warning: Cross Check : Mismatch between extentref entry reference count (1) and calculated fsroot entry reference count (0) for extent (0x236654a + 4)

This does not seem to be repaired by fsck_apfs though.

Any clues as to how to go about deleting this directory?

For those who wonder, the directory was initially found as a subdirectory of a Steam game (Factorio, release 0.17.0). The original path of the directory was ~/Library/Application Support/Steam/steamapps/common/Factorio/factorio.app/Contents/data/base/campaigns/demo/locale/ru. The part from demo downwards caused the game to not start properly (it was, apart from the directories, empty, but the game tried to pick up some non-existing file from the demo directory and crashed). Moving the directory away made the game playable, but I later discovered that the bottom-most ru directory could not be deleted. This is the directory that I then renamed strange and that I refer to throughout this question.

I have submitted feedback to Apple about this, but I'm no Apple developer so I can't submit a formal bug report.

Kusalananda
  • 412
  • 2
    could it be an open deleted file? – Rui F Ribeiro Feb 26 '19 at 22:37
  • @RuiFRibeiro Shouldn't lsof have shown it? – Kusalananda Feb 26 '19 at 22:39
  • Yeah, it should. But then there is that pesky hidden file Finder creates in all directories, that you see when you mount a Mac folder in Linux....while strange Finder might have it open. try defaults write com.apple.finder AppleShowAllFiles YES – Rui F Ribeiro Feb 26 '19 at 22:41
  • @RuiFRibeiro And those files are easily deletable and visible when looking at them in a terminal. – Kusalananda Feb 26 '19 at 22:47
  • 2
    yeah, ls -la should normally show a .DS_Store – Rui F Ribeiro Feb 26 '19 at 22:48
  • Search the filesystem for any other entities using the same inode? – DopeGhoti Feb 26 '19 at 22:58
  • @DopeGhoti There is none. – Kusalananda Feb 26 '19 at 23:04
  • 3
    Seriously file a bug report, Apple will give you test builds to sort it out. – user1133275 Feb 26 '19 at 23:21
  • 3
    I concur with @user1133275: APFS is a relatively new filesystem, and so somewhat more likely to contain new bugs. You may have hit a corner case that is not covered by the Disk Utility's First Aid. Examples of real-life error situations (in the form of bug reports) help in improving the Disk Utility and making the new filesystem more stable. Also, try sudo lsof +L1 to explicitly list any deleted-but-still-open files system-wide: a basic lsof targeted to a specific directory will only report on files that are there, and a deleted file does not quite qualify. –  Feb 27 '19 at 09:37
  • just read the manual: chflags – Thomas Dickey Feb 27 '19 at 09:56
  • @ThomasDickey I can see no extended attributes, flags, ACLs or whatever on this directory (I already mentioned this). Setting nouchg, nohidden or noschg with chflags on the directory and its contents does absolutely nothing. I will try in single user mode though. – Kusalananda Feb 27 '19 at 10:07
  • Are you running Time Machine backups? TM can add hard links to directories, but clearly it's not supposed to cause problems like this. – Mark Plotnick Feb 27 '19 at 12:50
  • 1
    @MarkPlotnick Yes, Time Machine can do that, but this is not a Time Machine backup disk and one can't create hard links to directories on an APFS volume. – Kusalananda Feb 27 '19 at 12:53
  • 2
    I'm guessing that the file is named the null character. I would file a bug report with Apple but that doesn't mean that Apple will contact you with an answer/fix. – fd0 Feb 27 '19 at 13:53
  • 1
    Hmm I am trying to replicate this issue. How was the directory created? – chrisdahfuh Feb 27 '19 at 15:07
  • 1
    @chrisdahfuh I have added this info. I did not create the directory manually. It was found as part of a steam game. The game was misbehaving (did not start), and removing the directory (which was seemingly empty) and its immediate two parents solved that issue. – Kusalananda Feb 27 '19 at 15:49
  • @Kusalananda ohh ok got it! so removing its parent directory removed the ru folder. That is really strange. – chrisdahfuh Feb 27 '19 at 17:00
  • 1
    @chrisdahfuh No, removing the parent of ru solved another issue (playing the game). When I later tried to actually delete the ru folder, I failed. Hence this question. I have clarified this in the question. Thanks. – Kusalananda Feb 27 '19 at 17:18
  • This may be wholely unhelpful, but a few years ago I had a similar issue that I think was related to HFS+ and how it deals with NULL characters. Maybe my question, troubleshooting process and comments/answers might nudge you in the right direction: https://unix.stackexchange.com/q/253932/3488 – Harv Feb 28 '19 at 16:52
  • @Harv There's nothing wrong about the name of the folder itself. I can rename it to whatever I want (and have done so). The issue is that it seems to contain an invisible and undetectable file that can't be deleted. – Kusalananda Feb 28 '19 at 17:06
  • 1
    Yeah ... I wish I could reproduce this on my end, I'd love to try and figure this out. – Harv Feb 28 '19 at 17:23
  • Would trying Apple's File Manager API help? Try this:

    '#!/usr/bin/python
    from Foundation import NSFileManager
    fm = NSFileManager.defaultManager()
    fm.removeItemAtPath_error_('/foo/bar', None)'

    Create a python script file and chmod +x it, then run it in terminal. It's possible this higher level API might do the trick, if not, worth a shot. Replace foo/bar/ with your file path. It should look like this: https://imgur.com/a/G82XWhA

     – chrisdahfuh Mar 01 '19 at 21:12
  • 1
    @chrisdahfuh Thanks! The code runs, produces no output, exits with a zero exit status, but does not seem to do anything to the directory. It's able to delete an ordinary empty directory though. – Kusalananda Mar 01 '19 at 21:27
  • @Kusalananda...darn it. – chrisdahfuh Mar 01 '19 at 21:27
  • @Kusalananda I have the exact same problem! I put an old version of Xcode in the trash a while back and when I tried to empty my trash yesterday, it wasn't able to delete two deep directories. They seem to have the exact same properties that you're describing: can't delete as root, no locks to be seen, link count of 3, first aid does nothing. Did you file a bug report with Apple? Is there a link I can follow? – jchitel Mar 12 '19 at 17:46
  • @jchitel I can't file a proper bug report as I'm not a registered Apple Developer, but I submitted "feedback", which you can do as an ordinary user. – Kusalananda Mar 12 '19 at 18:16
  • Can you rule out that the system has been rooted in some fashion? If ls isn't ls, so to speak, you've got a bigger problem. For reference: [MacBook-Pro:/bin]% shasum -a 256 -b ls c832fe1dc2c8ea1c871f5994916582ba735a56505d476ea5f16d1a8a8534724b *ls – Bill Horvath Mar 14 '19 at 17:54
  • 1
    @BillHorvath Same checksum. I'm not the paranoid type, but I am security conscious and know about not running random programs, unnecessary services etc. I'm also fairly well acquainted with what should be running on the machine and how it should behave. I have seen nothing suspicious. I suspect a slightly broken APFS directory entry that fsck_apfs simply isn't able to repair. – Kusalananda Mar 14 '19 at 18:00
  • Does stat reveal anything useful? stat -f "%N: %HT%SY" ./strange/* – Bill Horvath Mar 14 '19 at 18:11
  • @BillHorvath stat: ./strange/*: stat: No such file or directory. There is nothing to expand the * glob to, and there's no file in the directory that is in any way visible, not even with ls -a. – Kusalananda Mar 14 '19 at 18:22
  • This one had me very curious, so I did enough reading to believe that you did exactly the right things (#duh), that fsck_apfs should have fixed it (assuming that the disk was unmounted when it was run), and that the folks who suggested filing a bug report with Apple are right. It also appears that you may be able to fix it by wiping and reformatting the drive, reinstalling the OS, and restoring off a Time Machine backup, but that's a rather onerous process. (One other suggestion was formatting it as HFS+ and having the Mojave installer change the formatting. Ugh.) Any other ideas? – Bill Horvath Mar 15 '19 at 14:17

3 Answers3

21

I filed a bug report with Apple and this seems to finally be fixed now in Mojave 10.14.4 (18E226).

To fix, update your OS to the latest version. Then reboot into recovery (CMD-R) and run disk first aid.

It should show one or more error: directory valence check: directory (oid 0x123456): nchildren (1) does not match drec count (0) and / or error: nchildren of inode object (id 12345678) does not match expected value and repair them.

Once it is done, reboot into OS X and you should finally be able to delete the folder normally!

Adam
  • 276
  • Ah, yes. I updated the machine just recently but didn't get back to this issue. Thanks! Running a disk repair in recovery mode does seem to solve it now and I was finally able to delete that pesky directory. Much thanks! – Kusalananda Mar 27 '19 at 17:12
0

Adam's answer is correct and my previous answer itself not that helpful, but I include this from the comments section as another thing to try:

Try deleting your APFS snapshots:

  • tmutil listlocalsnapshots
  • sudo tmutil deletelocalsnapshots SNAPSHOT_DATE
Old Pro
  • 6,132
  • 3
  • 29
  • 71
  • Thanks for your effort, but I still only find two entries anywhere under / with that inode number. The strange directory entry itself and the . entry inside it. – Kusalananda Mar 17 '19 at 19:56
  • The three names you mentioned are not three hardlinks to the same file, so it is kind of obvious that your find command doesn’t find the inode they represent. And they represent directories anyway, which usually can‘t be hardlinked. – nohillside Mar 17 '19 at 20:16
  • 1
    @Kusalananda that is not entirely unexpected, but I thought it worth a try. Next thing I would try is deleting your APFS snapshots: tmutil listlocalsnapshots / and sudo tmutil deletelocalsnapshots. @nohillside if you look at the output of ls you will see that the 3 names all refer to the same inode. – Old Pro Mar 17 '19 at 20:24
  • @OldPro Still have to do some further check but I had the same issue as above (even worst as my disk was actually full) and I fixed it with tmutil listlocalsnapshots / and sudo tmutil deletelocalsnapshots SNAPSHOT_DATE. Thank you! You saved my day. – Alessandro Benoit Sep 06 '19 at 15:22
0

In a nutshell:
1) update OSX to the latest version
2) start in recovery mode
3) start disk first aid (which should fix stuff)
4) restart

Memmo
  • 109
  • The existing accepted & bountied answer already says this. – Tetsujin Mar 18 '20 at 15:47
  • 1
    @Tetsujin I totally agree with you. But since yesterday I had the same problem, I could not understand the solution using only the accepted answer and therefore I had to cross-reference multiple answers. So I said to myself "Why not rewrite the answer in a more schematic way so that if he were to ask himself the problem again in the future, I know exactly what to do?" I believe that stack exchange responses also have the purpose of being used as very useful documentation to the problems we have faced, not just once... – Memmo Mar 19 '20 at 08:30
  • ... @Tetsujin, so save them to be able to quickly consult them again in the future, I think it's the right thing to do. – Memmo Mar 19 '20 at 08:30