Wondering if any of you guys have enabled BTRFS compression on your farming drives? Has it affected your lookup times, or transfer speeds at all?
I’m a windows dude right now!!! This is mostly for Linux correct?
Yeah. BTRFS is a Linux based filesystem and ZSTD is a compression standard for that filesystem. Just wondering how compressable plots were and if the higher compression levels affected lookup times and the sort.
NTFS offers drive compression…not sure how it compares to BTRFS ZSTD’s efficiency. But I suspect that if a file is hard to compress, then all compression algorithms will not do much in that department.
Here is a thread from when I inquired about NTFS’s drive compression for plots:
There is really not that much benefit of having a compression for those plots (if at all). You can think about those plots as already compressed. Yeah, chia is working on further compression, but that is nothing like normal, rather a plot layout, etc. Still, from what was recently said by them, it will bring just single digit compression results.
The second thing is that as much as btrfs is a good file system, it doesn’t add anything of value to those drives. I would rather stick with ext4 and disable journaling, as those files are basically static so there is nothing to journal.
Although, I assume that you are not running any RAID on multiple drives (would imply RAID0, but that should be no go for chia).
1 Like
I figured they were probably already highly compressed. Just thought I’d ask if anyone had tried it out yet.
zstd compression of a K-29 plot saves about 1% (about 110MiB of 11GiB). That is however on the command-line - a btrfs zstd compression block is much smaller than the buffer used by command-line zstd, so the space saved by btrfs compression will be less than 1%.
RAID0 enables writing to 2 HDDs at approximately 2*100 MB/s without disturbing proof lookup times of plots already stored on the two HDDs in the meantime (2*18 TB HDDs can store more than 300 K-32 plots). Without RAID0, the write bandwidth limit is about 100 MB/s, taking into account the fact that the maximum write speed on most 7200 RPM HDDs near the end of the disk is about 130 MB/s (and on a 5400 RPM HDD it might be lower than 130 MB/s).
Sure, the same thing can be done with a simple script, and a separate plotter. On the other hand, one shake on that RAID0, and you need to start replotting. I guess, it is a trade off, and I went for a script and a plotter.
What exactly do you mean?