Hello. I need some help. I have HP DL380 g8 with 2x Xeon E5-2450L, 32 gigs of RAM and 4 TB Samsung 870 QVO SSD + 13 HDDs. My plotting is extremely slow - one plot is about 16 hrs. I started 12 plots some time ago and it is disaster - almost 40hrs for plot. My CPU is about 25% max and RAM used is below 25 GB. I use Ubuntu Server 20.04. Where can I find any solution for it?
How did you connect that SSD?
I put it in the bay using 3.5" → 2.5" adapter trough ServeRAID M5110 I belive
I’ve heard some people using this server type doing ok, but the CPU in this one at least is pretty slow. Single core speed matters a lot for plotting. So I’m not sure it’s an SSD problem.
What kind of plotting settings are you using?
It’s k32, 3389 RAM, 2 threads. But now I realised that my SWAP i 100% used. And what is strange, it shows 31G total RAM, 20G used, but 300M free. I will disable SWAP, make one plot and check it.
And I think it isn’t better. I will leave it now but it’s over 500 secs for phase 1, table 1
If I understand this you are running 12 simultaneous plots with 32 GB of RAM to a SATA connected SSD. I would say your plot times are about right. This is not enough RAM to be running this many simultaneous plots and a SATA connected SSD does not have nearly enough bandwidth for this many simultaneous plots. Your limiting factor is probably the SATA bandwidth, I wouldn’t try more than 3 simultaneous plots to a SATA connected SSD.
2 Likes
Ok, but now I run one plot and it is over 9 hrs. And beginning of phase 3. I made dd on that SSD - write is about 500MB/s so it’s ok. What else can I benchmark? I think issue with memory is strange:
total used free shared buff/cache available
Mem: 31Gi 4.0Gi 704Mi 3.6Gi 26Gi 23Gi
All of today’s SSDs are very fast. However, most are very fast, only due to their gigabytes of cache.
Once that cache is full, then you hit the actual speed of the SSD’s NAND fabric (and most are slow).
There are various types of NAND fabrics used. From fastest to slowest:
SLC, MLC, TLC, QLC (single, multi (as in two), triple, and quadruple). I forget what the “LC” stands for.
A 4TB drive probably has between 25GBs to, perhaps, 100GBs of cache. And for nearly any other use, the SSD will fly, because you never exceed its fast cache (which is probably made form SLC or MLC fabric).
Chia will quickly fill the largest caches, and then you are running at TLC or QLC speed, and they are probably 1/10th the speed of the cache, or slower.
SSDs made from 100% SLC are very expensive (and very fast, no matter how much you pound on them). MLC is also very fast, and will not slow down. They will also last longer than TLC and QLC SSDs.
Datacenter and Enterprise SSDs are very fast, never slow down, and will last a very long time.
Cheers!
1 Like
Unfortunately you picked one of the worst possible SSDs for this type of workload. This QLC drive is highly dependent on its 78GB SLC cache, and performance drops to the floor after that is exhausted. (below is review for the 8TB version, which has a slightly larger cache)
"Samsung’s 870 QVO writes at a rate of 490 MBps for 84GB, neck in neck with the competition until the before degrading to slower folding speeds. That 84GB of cache capacity actually slightly exceeds Samsung’s specs. After the TurboWrite exhausts, the SSD’s after-cache write performance degrades to an average of 172 MBps.
As we noticed with the 1TB model, the 8TB model’s cache recovery mechanisms work similarly. After letting the drive rest at idle for 30 seconds, the 870 QVO gains back 6GB of its cache. It recovers fully with 5 minutes of idle time. "
9h sounds about right for 1 plot.
I am using E5 2680 v4 cpus and the best I can get for a single plot is just under 7 hours (on windows) but the 2688 v4 has almost double the single thread performance to the 2450L.
When I run 10 in parallel (with cpu affinity fixed and 4G ram each) they complete in around 10 hours each but I am also using 2 nvme in pcie for that.
On other machine I make one plot for about 8h on Linux and 10h on Windows. It has Xeon e3-1220 v3, 8G of RAM and 500G Samsung 860 EVO SSD. It make it strange for me that second machine with much more power is slightly slower
Not sure I follow what you mean, which is your “second machine”?
Your E3-1220v3 is way more powerful on single thread than the E5-2450L.
(according to cpubenchmark data)