diskDedupe 4+

Great for deduplication on APFS

This utility is extremely fast and flexible. Unlike other duplicate file finders, this one has a few unique features including: ability to scan only files above or below a minimum size, and a simplified deduplication process that doesn't require you to pick a canonical original.

Under the hood, the speed comes from the APFS file cloning capability, which is sort of like really efficient hardlinks on HFS but allows copies to diverge efficiently without clobbering one copy or the other if a cloned copy is modified or deleted. This takes the guesswork out of deduplication, and for its use case beats the pants off of any other duplicate file utility I know of.

The only real limitation of this utility is that it ONLY works on APFS file systems. You can't use it on HFS disks. However, it *does* work reliably on mounted APFS disk *images* (including sparsebundles) even if they're located on HFS or other filesystems, so it's a great way to get the benefits of APFS cloning and deduplication even if you have spinning disks or RAID systems that don't do APFS well.

One sparsebundle was reduced from 9 TB to only 1.4 TB after deduplication with this utility followed by hdiutil compaction. Your data patterns may vary, but this was pretty compelling for me.

In short, I highly recommend it.

UPDATE: This WORKS well, Responsive developer, addressed issues promptly

I recently wrote a review of this cautioning people to not run this unless they knew recovery techniques, the developer contacted me and I explained the issues I encountered and they were prompt in responding and taking care of these issues.

A responsive developer and addressed concerns and issue quickly is great. I have not had much time to verify all the fixed, but everything seems to work correctly. Integrity and responsiveness in developers is rare so I highly recommend this product and the developer as well!

Developer Response ,

FIXED WITH Version 1.601
Thank you for your detailed feedback. Since in all our tests the deduplication of executables worked without any issues, we would appreciate understanding more about the specific problem you encountered. We would be happy to provide you with a test version that includes the changes you suggested, as they certainly make sense from a safety perspective. It would be greatly appreciated if you could send us an email to feedback@diskdedupe.com, allowing all users to benefit from your suggestions.

Great app!

This app works perfectly! I would only advise against using it on files in iCloud Drive, because iCloud Drive storage acts differently than any other "normal" APFS drive. When I used it on iCloud Drive, iCloud preserved all of the original files I "deduplicated", as well as all of the copy-on-write duplicates that this app created, so I essentially ended up with two copies of each of the files (the original plus the new copy-on-write file). Other than that, the app works great on any APFS drive I've tested it on!