7zip on several terabytes, drive already compressed

Question

so, I have a drive that is NTFS. it has thousands of graphics/video files that totals to about 6TB. I wanted to compress and send this info to someone.

What is my best option for compressing? I've also noticed that the drive has compression set on file explorer: "compress this drive to save disk space"

When I start to use 7zip on the data, the total is larger than what I anticipated, like 8TB or 9TB which I'm guessing is because drive compression is on.

I've tried Keka on mac but it's going slow as hell. What are my options here to ensure the recipient doesn't get corrupted data? Currently, I'm trying using 7zip format and 'store' for compression. Thanks!

How are you planning to transfer the data? If it's over the internet, I would suggest using rsync with enabled compression. That should do as well as possible without re-encoding the files themselves. — confetti, Aug 07 '18 at 00:40
Lossless data compression algorithms cannot guarantee compression for all input data sets. In other words, for any lossless data compression algorithm, there will be an input data set that does not get smaller when processed by the algorithm, and for any lossless data compression algorithm that makes at least one file smaller, there will be at least one file that it makes larger. Since multimedia files are already compressed, they're very hard to compress further and is the set that most likely expand when you compress again — phuclv, Aug 07 '18 at 04:17

score 0 · Answer 1 · answered Aug 07 '18 at 00:37

0

Compression algorithms have very little effect on media files.

The reason for this is that media files are already using as little space as possible when they are encoded.

Compression algorithms (for the most part) work by finding similar patterns in a file and putting these in a way that is more compact without having to alter the original file. For media files this is a very hard task, since there usually aren't many repeating patterns.

I've recently tried out a lot of algorithms (zip, gzip, xz, bzip, lzma - all on --best) on my media library (movies, tv shows, music) and it did barely any effect at all. For some files the size was even increasing.

The only really effective way for you to shrink the file sizes is by re-encoding your media. Depending on the media type this would mean reducing the resolution, frames per second, quality (e.g. colorspaces) or reducing the bitrate. In some cases you can also change the container during re-encoding. For example: A 100MB FLAC audio file might only be 5MB when MP3-encoded with 120Kbit/s and a variable bitrate.

answered Aug 07 '18 at 00:37

confetti

2,455

1

changing the codec also has effect, for example H265 is twice better than H264, which in turn was vastly better than MPEG2 in DVDs. But of course you'll lose some quality when re-encode the files – phuclv Aug 07 '18 at 04:28
With H265 you can remain great quality at a much lower filesize, however H265 isn't that well supported on every device and consumes a LOT more CPU usage during playback. – confetti Aug 07 '18 at 04:30
Hey guys, OP here, my concern is that the archive arrives without corruption, so should I just do one giant 6 TB .zip or .7zip File or should I split it up into chunks, such as several hundred gigabytes or 1 to 2 TB a piece. – Don Mick Aug 08 '18 at 05:42
That seriously depends on how you are planning to send it. I'd still suggest rsync with enabled compression (-z), no zipping or packing at all. Using --partial flag you can resume filetransfer in case it fails, so I'd suggest to use that one too. – confetti Aug 08 '18 at 06:38
it would be by proprietary remote/cloud based delivery at my company. I would be sending it over a file-transfering service that is pretty reliable. I did it once with a 2TB file and the client had no issues. – Don Mick Aug 08 '18 at 17:47
It's up to you, splitting would probably be safer. – confetti Aug 08 '18 at 22:49

7zip on several terabytes, drive already compressed

1 Answers1