Despite being interested in Chia late last year, it wasn’t until the introduction of madMAx’s plotter and official pooling that I felt comfortable making a modest investment. So I’m still quite new but wanted to post a few observations.
Fortunately, I already had an X570-based system in my workshop with a U.2 port, 2x2TB 860 EVOs, etc. running Ubuntu 21.04. I upgraded the CPU to a 3900X, which is ideal for madMAx because it’s the least expensive Ryzen CPU with two CCDs—and therefore twice the memory write bandwidth of e.g. a 3700X or 5800X. I also found good deals on a 128GiB kit of DDR4-3200 CL16 and a 280GB Optane 900P.
The 3900X is not overclocked, per se, but I have the usual PBO, Fmax enhancer, etc. enhancements turned on. The 900P is formatted with f2fs and mounted with
nodiscard (Optane does not need to be trimmed); I haven’t tried
fsync_mode=nobarrier yet but I don’t expect it to make much of a difference. The 860 EVOs are striped to quickly receive and hold the final plots until they can be copied over to a harvester.
Here are my best times (
-r 12 -K 2 -u 256 -v 256):
- Phase 1 took 653.426 sec
- Phase 2 took 374.992 sec
- Phase 3 took 336.229 sec, wrote 21877233523 entries to final plot
- Phase 4 took 50.7809 sec, final plot size is 108835864169 bytes
- Total plot creation time was 1415.47 sec (23.5912 min)
- Copy to
/chia-plots/plot-k32-<snip>.plotfinished, took 100.213 sec, 1035.73 MB/s avg.
Watching the system during plotting, I’ve noticed that there are some periods of time when the CPU is significantly under-utilized (i.e. <50%) and the 900P isn’t busy reading or writing. My assumption is that the tmpfs is the bottleneck during these moments. My next step is to try a mild RAM overclock to 3600 CL18 to try to alleviate this bottleneck somewhat. This should also help speed things up between the CCDs, CCXs, and the I/O die thanks to the higher FCLK.
The other issue I’m dealing with is that any other I/O during plotting absolutely kills performance. Unfortunately, this includes copying the plots over to the harvester. It makes perfect sense… the NICs and SATA controllers hang off the chipset, along with the U.2 port, so they’re all drinking from the same bucket. If the RAM overclock doesn’t help, I might try moving the 900P so it’s directly attached to the CPU, but I use this system for other things and want to keep it workable when I’m not actively plotting.
I also have an X299-based system with a 7960X at my disposal, so if push comes to shove I can move the RAM and the 900P over there to take advantage of its additional memory channels and PCIe lanes. I don’t see a lot of folks plotting on X299 so that might be interesting to try regardless.
If you made it this far, thanks for reading, and please let me know if you have any questions or suggestions!