NTFS compressed drives for storing plots

Windows has a “compression” option, when formatting drives via the command line:
c:/windows/system32/format.com

Running…:
format/?

…will show help and options for the command.
One of the options is /c
The above option will compress data written to the drive.

format-compression

Has anyone ever used the compression option?
Does anyone know if the plot files are compressible?
If they are compressible, by how much? If it is only 2%, then it is probably not worth the trouble. But if it is 20% or more, then that would seem like a worthwhile option, to make the most out of your storage drives.

Also, the compression will slow the access times down. But by how much? If it is only a fraction of a second, then it probably does not matter.

What I am trying to determine is if I should compress my storage drives, by determining what percentage of additional plots will become available, and if the performance hit will not matter to the farming and harvesting.

Thank you.

I don’t know for certain, but I am guessing there would be no space savings. The plots contain encrypted data, which is usually either non-compressable or actually grows in size due to the overhead of the compression format laid on top of the uncompressable data.

On way to check, I think, would be to make a plot with a size of 1 (which would not be usable for farming but should be the same contents wise, but small enough to compress in a reasonable amount of time) 7zip that and compare the file size to the he uncompressed plot. My guess is that it would not help.

Also I have no idea how windows manages compressed volumes but I think the overhead might be crippling with 101GiB files

2 Likes

Chia plots are random data so largely incompressible, this will only slow you down.

1 Like

Because I was curious I created a K25 plot (the smallest the GUI allows - god, small plots are so much faster!) and ran it through 7zip. As you can see below, about a 1% reduction in size, which is better than I thought it would get actually. However 7zip compression is much more aggressive than windows volume compression, which is built for speed not size reduction, so it would have even less impact AND would definitely have a speed impact since when farming your system has to look through the filtered plots to find your proofs and respond within 30 seconds, something which some setups are already having trouble with (mostly involving slower NAS drives or complex sets of NAS drives). I think adding in this extra variable would probably not be worth the very minor reduction in size. As @codinghorror said, you would just be slowing yourself down for no real gain

2 Likes

Thanks for performing the test.

The space savings would have to be enough to squeeze additional plots onto the drive.
Clearly, even with aggressive compression, it looks like no additional plots would fit on the drive. So there is no point in considering this any further.

We have our answer.

1 Like

Purely out of curiousity, and knowing that plots are effectively randomized data, with supremely careful placement of k32 plots, and compressing the entire drive, I was able, on occassion to squeeze 148 k32’s onto a 16TB NTFS drive instead of the usual 147. It wasn’t worth the hassle, because across my entire array of drives, I calculated, without actually performing the work, I would squeeze a total of 7 extra plots in to a 1PiB+ array which frankly just wasn’t worth the extra effort it would take. Across my entire array of (uncompressed) disks I have under 100GB of slack space, not even enough space to squeeze one more plot.

E.G.

3 Likes

I wish you had done a full plot check and farmed the plots you put on an NTFS compressed disk. The real question after establishing it actually does save space is does it affect plot checks, or slow down access times for challenges?

One extra plot for selecting an option is still one more plot. I’m in the process of replotting so I’m going to enable compression on a USB drive full of OG plots to see for myself since I haven’t found an answer to this question.

I run all my HDDs with NTFS compression turned on at the drive level. I have always done that. With Chia, I tested and noticed a “compressed” plot and a not compressed plot take up largely the same amount of space. Compression, at least in my setup does nothing to increase the number of plots I can put on a drive. I can’t recommend it if it isn’t turned on, but if it is turned on, there isn’t a need to turn it off.

1 Like

I tried it once some time ago. As I remember, the compressed file tock slightly MORE space than the uncompressed file. Big nothingburger that.

I have seen a compressed drive (at the root level) have more space used fully compressed than not compressed. This has to do I think with compression info needing to be stored somewhere. I don’t know if compression info is stored in the file/folder etc or if it is just part of the drive and how it is set up.

It probably does no harm, but since the data in a plot is already compressed you are not going to see any reduction in size.

The best thing I have seen for fully taking advantage of all your space is to pool the extra <106 of GIB that remain after filling your disks into a virtual disk that contain a few more plots. Though you should be careful doing so - at least one person here on the forum accidentally formatted a drive full of plots trying to set that up. There is thread about doing this on the forum:

1 Like