I just realised yesterday that the size of my media library was growing significantly faster than anticipated. The culprit seems to be that hard linking doesn’t work in radarr and sonarr because I messed up the folder structure of my docker containers, so they’ve been copying everything to my media folders instead.
Now, the real issue is that my media folders are a mix of and older collection (manually sorted) and the new parts that has been added through radarr/sonarr, so I can’t just nuke that and re-import everything from my torrents folder once I’ve fixed the folder structure. I don’t want to nuke the torrents folder because I want to keep seeding all my downloads (most through private trackers, so I need the download credit it creates), but the imported library in my media folders has all the files renamed and many rearranged.
What’s the easiest way to fix this? There’s about 8tb of actual data, without duplicates, I have 6tb of unused disk space left.
fclones is fast and supports hardlinking/softlinking of duplicates instead of removing them.
I’ve used it successfully to deduplicate my documents folder (and “archive”).
As its quite the amount of data, I recommend using the
--cacheoption to make subsequent runs way faster, if you want to dial in the options. This directory can be deleted at any point and isn’t necessary.https://github.com/pkolaczk/fclones
+1 for fclones
Btw. Always wondered if the name possibly comes from Fucking Clones
I’m using this in my unraid. Scheduled weekly, just in case.
#!/bin/bash #noParity=true fclones group --one-fs --follow-links --min 256MB --name '*.mkv' --name '*.mp4' --cache /mnt/user/data/torrents/ /mnt/user/data/media/ --max-prefix-size 128MiB --max-suffix-size 64MiB --skip-content-hash > /tmp/dupes.txt fclones link < /tmp/dupes.txt