RAMdrive vs SSD

I’m curious if plotting bottleneck is CPU constrained or SSD constrained.

If you setup a system with 512GB of RAM, and created a RAMdrive for temp files, would it be faster generating plots in serial vs. a system with a lot of cores and parallelism and SSD for temporary files?

From what I have seen, plotting appears to be CPU constrained, as I have seen a 12 core system max out it’s CPUs but the Enterprise U.2 SSD bandwidth is still fairly low, at approximately 300 MB/s total.

1 Like

100%, just check out the topic I already made about this.

The disk thing was wildly overstated at the beginning of the project, perhaps because they optimized the plotter so much. You can go from a NVME SSD to a SATA SSD and lose, like 20% speed max. If disk speed was so earth-shatteringly important, you’d see more like a 2× slowdown.

Also see

3 Likes

Thanks for that feedback!

Just want chiming in here with my experience: not worth it.

I set up a 380GB instance for 1hr on Azure, which is pretty cheap. An entire plot would have cost me about $18. However, the problem is raw temporary space: no matter how fast RAM is, you can’t outpace parallel plotting.

4 Likes

Well yeah especially in the cloud! Cloud is expensive as heck relative to rolling your own servers. You could potentially get an old decommissioned server with a lot of extra RAM though… but… you’d quickly hit CPU limits.

1 Like

Set up two separate 350-ish GB RAM disks (thanks NUMA architecture! /s) on my workstation and plotted all the way up to 1.0.3 with two plots running in parallel, and produced almost 500 plots. Tweak the overclocking on the CPU so that it was fast but stable, crank out 15 plots per day. 19 if I got the stagger “just so.”

Talked to a guy doing 100+ plots per day on SSD on comparale h/w and figured I’d give it another shot which is how I had started out initially but became paranoid about wear on my drives. Sure enough moving to SSDs my plotting went from 15 (sometimes 19) a day on RAM disks to 80+ and still made the machine more than usable for my daily work. I was fretting about wearing out SSDs, and when I looked up the specs, realised each of the drives had 60PBW expected lifespan.

I also have a six-core/12-thread i7 (that runs the farmer) from 2011 that turns out 20 plots per day by plotting straight to 10 separate HDDs.

If I had started a year earlier with my plotting, I would have most likely stuck with RAM disk, but I only started plotting about a month prior to mainnet launch and was very much learning the ropes.

If you had 12TB of RAM and were offering plotting-as-a-service it might be worth it. Otherwise, stick with SSD or HDD.

1 Like

What’s the exact model of this CPU, I am curious? Which generation? 20 plots per day, how many seconds per plot?

Paired it with 64GB DDR3 on an abused Rampage IV Extreme (also 2011) and the LSI HBAs (host-bus-adapters) to run the HDDs. The motherboard, CPU & RAM were sat on my “obsolete h/w shelf I cannot bring myself to throw out yet.”

Technically I sold this project to my wife of “I can build you a new VR gaming machine but I need $15,000 worth of hard drives to do it.” There is a SATA DC SSD for the -f step, and the plots are moved from the -f to their final resting place on an available HDD. I have a script that packs the plots on to HDDs using a naive 1D knapsack algorithm which as you know is a fancy way of saying “find the drive with the least amount of space that will accommodate the largest plot I have to copy.”

Running 10 parallel plots direct to HDD with no exotic configuration and default parameters (two threads, default RAM & buckets), 60 minute stagger to mitigate thread over-allocation:

Time for phase 1 = 15886.560 seconds. CPU (130.790%)
Approximate working space used (without final file): 286.709 GiB
Final File size: 101.393 GiB
Total time = 35500.069 seconds. CPU (102.350%)

Thread over-allocation still gets you, but other factors start to come in to play then, e.g. blowing out every cache line in your CPU.

If I run one plot, the time is below 24,000 plotting to HDD. I don’t have a single plot at a time log file kicking around so I cannot verify that number, but it sounds about right. With parallel plotting on that CPU I suspect RAM bandwidth and L1/L2 CPU cache are the limiting factors followed by possible SAS bus saturation. I haven’t looked in to that too deeply because it’s just the farmer, and I don’t want to over-optimise on it, cause instability or have the wife go “WTF?!?” in the middle of Beat Saber.

3 Likes

That’s a very good number; so a single plot on an i7-3960x is 6.6 hours? Considering it’s a 3rd gen Sandy Bridge circa 2011 that’s very good. This does lend credence to the theory that going back pretty far in Intel generations doesn’t hurt you very much; it’s all about cores … cc @instrumentality

Unfortunately Intel was very very stingy on cores until AMD came along and kicked their ass :boom::leg: on that front … more cores for the people … which was well deserved! Hit 'em right in the Xeon jewels, where it really hurts. That and ECC for everyone!

1 Like

Yes. I’ve been an Intel die-hard for a long time (not a fan of Intel, but use them for their compatibility, stability and virtualization capabilities and also working for them at one point) but it was about time someone came along and kicked them in the fork. If it didn’t involve junking the m/b and expensive CPUs I’d certainly consider the higher-end AMD as my next CPU but I upgraded from the i7 3960X to dual Xeon Skylake 28-core CPUs at the tail end of 2019 so I don’t really have much choice to stick with what I have right now. Maybe if XCH hits BTC levels I can justify it. My wife said I cannot touch our BTC but she didn’t say jack about the XCH. :wink:

2 Likes

I was absolutely floored by the plot speed someone got out of the $850 Intel NUC that was on Chia Decentral the other week. “Wait, why am I wasting my time on kitting out the workstation when I could just buy three NUCs and achieve the same performance?!?”

2 Likes

Yeah they did a heck of a good job optimizing the plotter. It also does seem to favor Intel over AMD a bit in my opinion. Maybe it uses all the AVX magic?

1 Like