I’m not sure exactly how you were using numactl, but keep in mind that, for instance, -t and -2 should be in the same numa node to maximize benefits. I haven’t tested without it recently, and I don’t have any more disks to plot on at the moment, but I recall a pretty big gain from 1x madmax to 2x. I don’t recall specifically what my results were from running 2x mm with and without numactl, however. I’'ll check that again when I get some more disks.
The faster the ram, the better gain you get from bladebit. With E5 v2 xeons, you will struggle to get a single plot much under 30 min with mad max, but with bladebit I’ve seen some v2 users getting around 18 minute plots.
Plotting entirely in ram, I make plot every 1400 seconds. Plotting twice with -2 tmpfs and -t nvme I make a plot every 1000 seconds. I’m pretty sure I could get over 100 plots per day, if I had 512 mb of ram, and if I could move the plot to 3rd level storage before plotting runs out of ram. The unfortunate thing about this platform is the ram speed dropping from 1866 to 1600 to 1066 as you got 256/512/768 gb of ram, so i’ll always need that 3rd tier for 2x ram plotting, I’m guessing…
Incidentally, I’m not sure what V2s were used for the 18 minute plot times you saw. If they were also 2697s, then extrapolating my current time with parallel mm jobs, mm would indeed be faster than 1 bb, as I speculated. If they were the 10- or 8-core variety, then bb is probably significantly faster.
When considering cost, you could likely find a dell r520/720 systems with 256gb ram, add in some cheap NVME and possibly upgrade the CPUs for $600-700 and make 70 plots/day by parallel plotting with mad max. A similar system for bladebit would cost around $1100-1200 and only increase output by a small amount. The newer the system (like an r730) the higher the output will be, but also at significantly more cost. It would cost me around $1500 to upgrade my v3 system to 512gb RAM. So not really cost effective to buy a bladebit capable system, but if you’ve already got the hardware it will be more efficient.
My 12 core V2s are making at least 80 plots a day right now (I said 1000 seconds before, but it’s more like 1050), but it depends very much on the nvme I’m using for -t. Cheaper older ones can add 50%, samsung EVO 970s and 980s work well. As I said before, I’m pretty sure I can get over 100 plots a day with pure ram plotting.
They outperformed my OCed TR1950X pretty handily, though the TR has only 16 cores, and the dual xeons have 24 combined.
You make a good point about not getting much benefit from the extra ram needed to run bb. That said, if I’m still plotting a few years from now, the cost of those NVMe will add up. Eventually, it will become more expensive than plotting in RAM. Also, you’re amortizing that initial investment the entire time, and it will have some resale value in the end (junked nvme obviously wont).
When I get more disks, I’ll upgrade one of my servers to 512 so I can test 2xmm vs 1xbb.