Plotting System with 256GB RAM

Chia Plotting
1

Morning all,

I’ve searched the forum and found some references to having a 110GB RAM disk for plotting with MadMax. I’m looking at hardware to do more plotting more quickly (I’ve previously plotted on SSD drives, not RAM disks) and find the sweet spot for hardware in terms of cost is with 256GB of RAM.

What would be the best settings with MadMax on a 256GB system running Windows (10/2016;/2019) ? What size RAM disk would be best ? Do I need to allocate just 110GB for the temp2 folder ? What about temp1 folder ?

In the original chia plotter the plot process would use potentially up to 260GB IIRC, so this would not fit into a RAM disk on a 256GB system. Presume MadMax has a lower requirement ?

Found bits and pieces of info scattered around the forum but not a ‘definitive’ guide as such, but if there is one and someone knows where it is, please let me know.

Thanks.

2

Don’t run Windows for ramdisk plotting, it utterly sucks.
Maybe there are some ok ramdisk programs for Windows, but I don’t know about them. I tried several and the results was increased plot times from 26 minutes in Ubuntu to 44 minutes in Windows.

1 Like
3

Interesting thread tangent, but related I guess. What RAM disk programs have you used on Windows ?

4

I don’t remember which ones I tried, but remember had difficulty to get them to work.

I settled with OSFmount in the end, easy to use and it worked. But like I said plotting time sucked.
Now that I no longer use the same system for farming and plotting, I switched the plotter back to Ubuntu.

1 Like
5

Well that’s one I’ve never heard of until now.

I’ve used SoftPerfect’s RAMDisk, Romex Software’s PrimoRAMDisk, IMDisk and some native MFC tools in the past with great success in some application instances (but not for chia plotting).

6

IMDisk or PrimoRam Disk are fine in Windows. Depending on your memory speed, and/or your SSD speed, memory may not end up being faster however.

7

What kind of CPU / RAM you have?

I have one box i9-10900 / 128 GB RAM DDR4. In Win using ImDisk I get ~31 mins / k32 plot. I also installed Ubuntu on this one, and got ~28-29 min / plot. Not a big advantage, but counts when more is to be plotted. That was using MM with t1 on a single NVMe, and 112 GB RAM for t2. I tried to do NVMe RAID0, but that was somehow slowing down plotting time (not much, but slowing). Maybe the problem was with using host based RAID0, not through mobo/BIOS. (People using AMD processor report more luck with NVMe RAID0 arrays, as those processors have more PCIe lines.)

I also have a dual Xeon E5-2695 v2 / 256 DDR3 box. On Win 11, it was taking ages to pull a plot. On the other hand, using Ubuntu, it does one plot per CPU in ~40 mins (effective 20 min plots) running 2x MM. I have tried to run completely from RAM, but it was slower. My understanding is that RAM has bandwidth issues at DDR3 rates, especially when using MM. BB would be better, as it is not using RAM as RAM drive, but rather RAM directly, but I would need to upgrade to 512 GB, what I didn’t (stopped plotting at that time). For those two CPUs, I have two NVMes each sitting on a PCIe slot connected to a given CPU, and each CPU has its own 128 GB RAM. I didn’t try to use NUMA control, what should speed it a bit more.

1 Like
8

I don’t have a system with 256GB yet - was considering re-purposing one that’s surplus from a local IT recycler around the corner. It’s a Dual Xeon E5-2660, 256GB of which I think is DDR3.

How long is ‘ages’ please ? What configurations did you try as regards RAM disk/NVMe etc ? And what RAM disk software (if you used anything…) ? How much slower when plotting completely in RAM and what size RAM disk ?

To be honest, 20mins per plot I’d be more than happy with as my older systems were doing one plot every 6 hours (although each system had 4 running in parallel, so typically 4 plots per day, and I had six systems, so 24 plots/day).

Sorry for all the questions, just trying to see if this system is worth a punt… :slight_smile:

9

Running that box with just one CPU in was giving ~50 mins plots (under Win ). Once the second CPU was added, I was killing it around 1+ hours, as there was no point to continue that agony. My understanding is that Win 11 Pro just cannot bite NUMA settings (multiple CPU). Once I switched to Ubuntu, all was over.

I settled down on ImDisk (on that i9 box). I was trying Romex, some other RAM disk utils, but ImDisk was free, and if not faster, it was equal to other utils. Although, when I had just 64 GB RAM on that i9 box, I used Romex’ Primo Cache (giving it ~55 GB RAM, and pointing it at t2). With that setup, it was taking ~50% hits off of the t2 folder (a second NVMe).

I have tried on that box (Ubuntu) to run a single instance of MM all in RAM, a single instance with t1 as NVMe / t2 as RAM, but all those various settings were slower than running 2 MM instances in parallel, each having it’s own NVMe, and 128 GB RAM sitting on each CPU.

My DDR3 is running as 1886 MHz (max on that motherboard).

What you need to also consider is the power draw of that box. During plotting, my box draws ~450W. The TDP for 2695 is 115W, where 2660 95W. However, 2695 has a couple of extra cores, more cache (30MB vs 25MB), and runs at slightly higher clock rates (2.4 vs 2.2). So, I would really get 2695 / 2697 (higher clock rates, higher TDP).

Also, you should use NUMA control, as that speed things up a bit more (just search this forum for NUMA, as there are a couple of great threads about it being used with parallel MM - of course for Linux). Also, some people reported that MM was faster with CentOS or Debian rather than Ubuntu on those dual CPU boxes (maybe on single CPU as well, don’t recall it).

By the way, I got Dell Precision t7610 workstation for ~$250. Each CPU was ~$100. RAM is the most expensive part. I got 32 GB 1886 LRDIMM ECC sticks, but some people were getting 1600 (a bit less expensive), and overclocking those to 1886 without any problems. Also, I had to switch to water cooling, as that motherboard really sucks (hot air exhaust from CPU1 is 1" in front of CPU2 intake, so the stock fans were making jet noise, plus the box was temp throttling).

If you have plenty of plots to do, maybe getting that box with 512 RAM and using BB would be a better route for you (effective plotting times closer to 15 mins, or so).

I think that those dual / quad e5-2600 v2 boxes are potentially the most cost efficient for medium size farms.

When you mentioned that you were running 4 plotters in parallel, were those MM or rather Chia plotters?

1 Like
10

Thanks for the update, useful info. I’m aware of process affinity and often script things to stick on certain core(s) or nodes.

By the way, I probably won’t use Windows 10/11 but rather Server 2016/2019.

I was running 4x standard chia plotters in parallel in the early days (starting back in May 2021). Before MM became available.

I’ve grown my farm to almost 600TiB now, but I still have 6x 16TB drives to fill. Also, the local Dell server (the dual Xeon E5-2660) has the ‘benefit’ of 72TB of onboard SAS disks, and the price is a relative bargain, being worth it just for that capacity alone, so I may just take a punt.

Power costs are not too much of a concern as I won’t be plotting forever, but I do intend to get to 1PiB at least.

Thanks again. :+1:

1 Like
11

My 2xE5-2650v2 with 512RAM (DDR3-1600) and turbo mode off takes 30min/plot using BB plotter (Windows) and ~300W power.

2 Likes
12

I was running a Dell T5810 with a E5 2699v3, which is 18 cores, and I had 256GB of DDR4 2300Mhz ram.

I had 4 x S3710 200GB SSD’s in raid 0 and the ram disk, and it would turn out 1 K32 plot every 22 minutes.

I wouldn’t use Windows, not sure if a server edition would make any difference, but Linux is way faster. I’m a Windows man, but managed to get it working on Linux. I’m selling the T5810 (only64GB ram) as I now have a T7910, with dual E5 2699v3 CPU’s and 512GB of ram. BladeBit takes 12 minutes, but the disk transfer slows it down, and I think if I run two Madmax in parallel it will do two plots every 36 minutes, but for some reason I’ve not saved those stats.

1 Like
13

I do all of my plotting on Windows server 2022 with BladeBit. I got a bunch of Dell R820s w/ 1TB of RAM for free. When I was first plotting, I used MadMax with OSFMount’s RAMdisk and I would get around 39-40 minute plots. With BladeBit, I’m able to usually do 23-24 minutes. Not quite as good as people can get on Linux, but I’m fine with it as I only have so much space to fill at any 1 time.

I originally ran this on 2019 server, but there are some improvements in 2022 server that speed up file transfer on 10Gb networks. (probably 10-15% only, though) If you can’t get the plots off of the plotter fast enough to make room for creating more, then being able to plot fast is only going to do so much.

If you go for 2022 server, you can download an eval version that is good for 180 days for free. And there are some sites where you can buy actual licenses for pretty cheap. I got mine for $9.30 at gamers-outlet.net. Yeah, it’s an odd website for server licenses, but it worked just fine for me. I bought 3 licenses from them without issue. And it turns out you can use the same license on 2 systems. Not sure if that’s intentional or just some odd bug that seems to work.

1 Like
14

IF you want to use windows. use WLS

15

Do you mean WSL?
If yes, then why?

16

It’s just faster than doing it on regular Windows. Not as fast as 100% Linux but still faster.

17

And where is the data to back it up?

18

Agreed, can’t see how it can be when there’s an additional layer to consider.

19

is that with copying or pure plot creation?

20

see you did lots of experiments. Would you be so kind to share some info?At the time I was rather using Swar manager at that time, three nvme SSDs, 64GB RAM and a 5900x and I was able to output over 40 plots a day with copying them to the HDDs. Sure, it was an investment, but I will sell most of the stuff. The power was no more than 150W if I recall. And that was on Windows.
Your setup is plot every 30 minutes is 48 plots a day. Is that with copying them to the HDD? I am asking because temporarly I have a dual 2699v3 server with 256GB RAM and 2 slots for SSDs there (1U - so USB transfer to the HDDs as two pcie slots will be taken by nvme, that sucks but… well, its what I have) and I want to build 2-3 super fast boxes with mimumum investment to replot fast… definately one cheap GPU plotter for less than 400 USD (i have the 3060ti)… would you still recommend the dual 2699v3? what would you change?