Hi all, I thought I’d just share my experience which I think is somewhat different to the norm, in the case that it helps someone else.
So I’ll start with what I’ve got, what speed I get and then hopefully we can talk about what further optimisations we could make.
AMD Threadripper 1950x, Asus X399 Prime M/B, 128GB 2666MHz RAM (not overclocked), 4 x Intel DC S3520 240GB SSD’s, a second machine that acts as the node (this one is the plotter) which is connected over a Gigabit network.
With this I get 29Min Plots. 27 with a few tweaks that I don’t apply.
I personally think that’s pretty good considering the slow SSD’s that I’m using, my ram is not that fast, the CPU is old and of a slower MHz than modern standards (though plenty of threads).
To achieve this I’ve done a few things. Primarily, configured the SSD’s into a ZFS Stripe, enabled LZ4 compression and set a 1MB recordsize. On top of that atime off, xattr = sa, redundant metadata = most. I haven’t yet looked at primary /secondary cache changes which may increase things further. Anyway, I can explain that if you’re interested. I’ve done no bucket optimisation (running at standard 256), I use a 110G RAMDisk of course, have applied all 32 threads. There are some remaining ZFS tweaks that increase the speed (sync and checksum) but I haven’t applied them as I don’t really care about 1 more minute at the expense of any potential data quality issue.
After each plot completes it is copied over the network to my NAS / node (unraid) which happens in the background while the next plot is running. This is currently running on Ubuntu.
Nothing is overclocked. I’ve been experimenting a bit with the best settings because understanding how databases work with tables and stripe sizes, it’s all about iops and matching the disk structure up to the kinds of reads and writes that the application needs.
I nearly get this speed on a single INTEL D3-S4510 I have, which just goes to show how bad these 4 Intel M.2’s are unless striped. If I had more of the faster INTEL SSD’s, I’d expect something even more favourable. In fact, the 4x3520’s have a much slower write iops and a much slower sustained throughput for read and write, but once striped, the reads are SO fast that overall it outperforms the 4510.
I’d encourage anyone with a few drives lying around to try a decent raid tech for the plotting portion to increase iops and see how well it goes.
Also note that I tried this with some supposedly awesome NVME drives called Seagate Firecuda 520 (500MB) versions. They’re PCie 4.0 and I only have 3.0 but nevertheless the plot time was an incredulous 50minutes. On top of that the two runs I did on them reduced their health down to 85% so that are not as durable as the say. Compare that to the INTEL that is still at 99% for literally hundreds of runs.
So my conclusion is that IOPS is extremely important. And that ZFS is awesome because you can essentially set all these settings on the fly and for an individual folder, rather than having to reformat the whole array. I look forward to hearing if anyone else has done anything similar! And the way ZFS does record sizes across a stripe is pretty fancy too.
What do you think?