I’m getting wacky, and welcome experiments to try on my rig!
Materials:
Upcycled Dell PowerEdge R720 (a BRIGHT YELLOW one)
with a hardware RAID
200gb SSDs x4,
1tb HDs x 14
and a giant pile (270gb) of RAM.
No NVMEs. 'cause I’m cheap.
I’ve got the SSDs RAID-0 into a FAST drive, and the HDs RAID-0 to a BIG drive.
But… this wasn’t great. I could plotman squeeze in 3 plots in parallel into the SSDs. Meh. So I started digging around to find out if the RAM or SSDs could be used to scale, instead of directly.
Throwing more RAM at each plot didn’t do much AFAIK. The .6 max seems reasonable.
Messing with “write back” cache did seem to help a bit! But I don’t have hard numbers.
Time to get more crazy. I installed bcache because of their marketing:
Hard drives are cheap and big, SSDs are fast but small and expensive. Wouldn’t it be nice if you could transparently get the advantages of both? With Bcache, you can have your cake and eat it too.
I love cake! But I am worried - caching gets tricked by chia plotting. LRU caching “falls off” the end of the cache during plotting because it is just too much data.
So right now, I’m trying a “12 in parallel” test using bcache, with the SSD as the cache layer, the HDs as the backing layer, and the ram… gets to chill.
I don’t like letting the ram chill. And I’d love suggestions on stuff to try! Different file systems? btrfs vs ext4 vs zfs? ZFS for fun and profit?
Problem I see is that the RAM cache will not help much as the single plot working space is 290GB, and I think the cache hit are going to be minor as the phases go mostly sequentially through the data. You can split up your array into 2x2 disks though which I suspect would improve on the 1x4. Maybe you could also test plotting 64 buckets as you have the RAM anyways.
Using some RAM as a pure write cache would be beneficial as well
I get the same feeling - the process chugs though, and everything spools off the end of the cache.
I’m trying 64 buckets now - so far, I don’t see a HUGE speedup, but (shrug), worth trying.
I’d love to do more of that, assuming more write cache (write back cache?) = more bucketing of writes. But I’m not sure how to get that to work.
Salamander:
Can you try to break up the hdd-raid? It could be worth a shot to have as mutch hdd-volumes as parallel plots (maybee 10 single-drives, 2 x2 miniraids) and put the ssd-cache in front of it.
if some plots are “random-fetching” blocks from a 14x1 Raid the whole raid-througput should be affekted and the cache-write to disk should slow down.
As differents plots in differents phases of calculation generates different io-patterns separating the raidvolumes + the cache could give you more performance.
I do not think bcache is a good match for Chia workloads. You will be far better off just using those SSD in a raid 0 as a temp drive where you could try plotting up to 3 plots at a time on that, and using the disks not raided as slow but steady 1 plot each plotting temp drives chugging out 14 more plots in parallel.
I’ve got a 720 (with dual 2650 v2s) and 320gb of ram, tried a 270gb ramdrive and plotted one plot.
It still took maybe 16 hours. Certainly doesn’t finish in 1/8th of the time it takes to plot directly to 8 simultaneous sas drive as seperate drives without any SSDs, as I do not want to destroy any hardware in this process. Since I am only throwing 4 threads at this one plot and it seems more isn’t going to improve things then I am not sure what to try next without adding ssds in to the mix. I might try PrimoCache as some people think it helps and allocate a big chunk of my ram to it, this may help all the simultaneous plots. I might also attach maybe 4-8 more USB drives and plot to them simultaneously too, as I am not CPU or ram limited plotting 8 plots at a time. I bought the extra 256GB of ram to see if it would be useful but my conclusion is not really, I had 64GB to begin with which would have been good for 15 parallel plots anyway. 15 plots every 16 hours is nearly 23 plots a day on old recycled hardware and secondhand drives. I’m also using a R515 as a farmer but that can manage about 4-6 plots/day also, and a few old laptops and other old pcs churning a plot or two per day out also.
Sorry for all the edits, its just how my brain works, but I guess if I had a much faster processor and used the ramdrive to plot one plot, using 4 threads from some uber-processor would result in a pretty fast plot, I guess 4 threads with my slow processor just isn’t all that stunning. Its a shame the Chia software doesn’t allow you to throw 32 threads at one plot but then if it did it would become a big power drain like bitcoin, and be all about fast processors rather than having a lot of drives.
