New rig - Need some assistance

Chia Tools & Scripts
1

Hi,

I’ve just finished building my new rig and the results are awful.

Rig specs:

HP Z420
CPU - I’ve upgraded the CPU to the E5 4657L (2.4 GHz, 12 cores, 24 threads)
RAM - 64 GB of 1866 MHz
Disk - 2TB PNY gen3 (3500/3000 RW speeds)

I’m running windows 10 (installed on a dedicated SSD)

I’m using plot manager and started yesterday morning.
Until now I’ve created only 1 (!!) plot.
I can’t seem to find my bottleneck.

These are the definitions from my config.yaml

(max_concurrent 9)
 - name: Plotter
    max_plots: 999
    farmer_public_key:
    pool_public_key:
    temporary_directory: Z:\Plotter
    temporary2_directory:
    destination_directory: E:\Plots
    size: 32
    bitfield: true
    threads: 2
    buckets: 128
    memory_buffer: 4000
    max_concurrent: 1
    max_concurrent_with_start_early: 10
    initial_delay_minutes: 0
    stagger_minutes: 45
    max_for_phase_1: 4
    concurrency_start_early_phase: 4
    concurrency_start_early_phase_delay: 0
    temporary2_de`Preformatted text`stination_sync: false
    exclude_final_directory: false
    skip_full_destinations: true
    unix_process_priority: 10
    windows_process_priority: 32
    enable_cpu_affinity: false
    cpu_affinity: [ 0, 1, 2, 3, 4, 5 ]

This is how my view looks like:

plotman1
plotman11536×864 19 KB

I’m seeing some really weird behavior like my external seagate drive writing with 10-28 MB/s (it’s usually around 80-100 MB/s)

I would put the system performance screenshot but as a new user it only allow me to have 1 picture.
Anyway - CPU is really low (~ 20%)
RAM is really low (~ 24%)
NVME is 100%
external HDD is 100%

Can someone please guide me in order to find the bottleneck?

Thanks,
Diver

2

My guess is that your bottleneck is your PNY SSD.

You need to determine which NAND cell fabric it uses. From fastest to slowest:
SLC (single layer cells), MLC (multi layer cells – as in double), TLC (triple layer cells), and QLC (quadruple layer cells).

Forget the speed stats that are on the box.
Forget the speed stats that are on the manufacturer’s web site and amazon or wherever you purchased it.

Those fast speeds are only for the SSD’s cache, which is probably between 25GB to 100GB.
As long as you do not fill the cache (which is made from either SLC or MLC), then you get the very fast speeds.

However, Chia is not a typical user of your SSD. It quickly fills the cache, and then uses the SSD’s native NAND cells, which probably run slower than a mechanical hard drive.

I just tested a Samsung T7, and it sucks. So back it goes.
It is taking over 24 hours, and yet it is rated at something like 1GB per second. That rating is for its cache. The rest of it is slow as molasses.

It is hard to find the makeup of SSDs. The type of NAND cells used are not easy to find, and are almost never on the manufacturer’s web site.

If the SSD was inexpensive, compared to other SSDs of the same storage capacity, then that is a sign that the inexpensive SSD was inexpensive because it was made from the less costly QLC cells.

I have a Samsung T5 that is somewhat fast (nothing to brag about – but it is not bad). It takes roughly 10 hours to complete a pair of plots (could be a bit longer – I cannot remember). So I figured that the T7, which is advertised at twice the speed of the T5 would be faster. Forget it. The T7 sucks. The T7 takes more than twice as long as the T5, and the T7 is doing only 1 plot.

Maybe for general use the T7 is okay. But not for Chia. And your PNY SSD is probably not good for Chia, too.

1 Like
3

Thanks for your reply.
On my main PC I’m plotting with the 1TB Samsung 970 Evo plus and with i9 9900 can produce 11-12 plots a day before tuning it. What is the recommended M.2 to use?

I also guess 2 of 1TB will give better performance than 1 2TB?

Is my assumption correct?

4

I am using the Samsung 980 Pro NVMe drives.
I am getting 7½ hour plot times, running plots in parallel.

I read that the 970 Pro NVMe drives are faster. But they max out at 1TB capacity (I believe).

However, since the 980 Pros do not keep my CPU cores waiting, they are fast enough.
I believe that even with a RAM drive, the bottleneck would be the speed of each CPU core – but I am no expert.

Once you verify that your PNY SSD is the bottleneck, based on its NAND cell makeup (not its cache makeup), then the Samsung Pro will probably no longer be an SSD bottleneck.

As to how many plots you should run per SSD, well, I find that running 6 plots gives me 7½ hour plot times, and running 7 plots takes me close to 9 hours for each plot. So the plot time increased with 7 parallel plots, but for the extra hour and a half, I get another plot done.

On the other hand, if I run only 6 plots, I can get them all restarted sooner than if I run 7 plots. This math makes my head hurt.

With your 64 GB of memory, you could run as many as 14 or 15 plots (maybe more). That would be fine for as it pertains to your memory. But if you run more than 11 plots (I am guessing), then you will probably have poor performance due to not having enough cores.

So let’s say you can safely run 11 plots with a fast NVMe SSD. Well, that would be too much for a single NVMe drive, even a 2TB NVMe drive.

My guess it that you will probably get the best performance out of no more than 6 plots, per 2TB NVMe drive.

And since you should probably run no more than 11 plots, then to do so effectively, you would need two NVMe drives, and run 5 on one and 6 on the other (maybe try 6 on each?).

And to run 5 or more (maybe 4 or more) plots on an NVMe drive, then that drive must be more than 1TB capacity. So you would need two 2TB NVMe drives.

Lastly, I did little research in choosing my Samsung SSDs.
They were the best/fastest SSDs that the store had, so that is what I purchased.

There are faster SSDs. I do not have a list. But a duckduckgo search will turn up answers.
But I do not think that faster than the Samsung Pro SSDs will reveal much improvement. So if you see test results that show some other SSDs are faster, then go with the lowest cost SSDs. But make sure that the test results are not for BS, ordinary use, where the cache is not exceeded. You need to see results where the drives are pounded on with 500GB or more, non-stop writes, and no slowdowns.

Good luck.

5

Thank you for your tips.
On My i9 9900 pc I can run 5 plots on a 1tb Samsun 970 Evo Plus M.2.
I think my max is 11 plots parallel (11*2 cores = 22 cores, 2 cores will serve the OS).
So maybe going for 2 1 TB SSD which will work in parallel will give me better results.

Another thing is that my motherboard only support PCI-E gen 3 so the fastest M.2 are not relevant for me.

I’ll do a test of running 5 in parallel on the PNY 2TB and see how it works.
In case it will give me close enough results to the i9 9900 PC I’ll realize that having 9 in parallel like I tried is not a good idea.

Thanks again.

6

Your bottle neck is the external seagate. If you open up Device manager and click on Disk Drives / Your external drive. Right click on it and go to the Policies tab and turn ON Enable Write Caching on the device. Your write speeds will increase. Might also increase your delay time between plots to account for the longer write times to an external drive.

7

Thanks Jeff,

I’ll definitely check your recommendation, my Seagate writes from 25 to 100 MB/s. I think it can reach higher speed once implementing your tip.

I think I’ve found the root cause for my issue. Apparently the cpu_affinity was indicating only 6 threads while I can use 22.

I’ve added the rest of the thread numbers and gave it another try. I’m now out of home, I’ll check if that’s working when I’m back. For now I’ve noticed that once the second parallel plot started the CPU utilization did increased. I consider that to be a good sign.

Yesterday’s plots took over 20 hours. I now try only 5 in parallel and if that goes as plan I’ll raise it to 8 or 9.

1 Like