Upgraded from SSD to M2 NVME - no performance improvement. Help needed

Chia Plotting
1

Folks, need help finding bottlenecks. I had a system - intel i7 10700 (8cores, 16 threads), 24 gb DRR4, Asus ROG B460-H motherboard and 2 Sk Hynix 1 TB SSDs in Raid0 and 14TB external hdd. OS is Centos 8 with kernel 4.18.

With this setup I was running 4 plot processes in parallel with -k 32 -b 4000 -r 4 -u 128 and delay between starts of 1h. That was giving me plotting times of about 5.5 hours per plot. Despite having more RAM and CPU threads for a few more plots in parallel, I wasn’t trying to run them until I got myself an M2 NVME. Finally got much advertised 2TB Samsung PM983, and replaced SSDs with it, hopping that I will be able to run 5-6 plots with approximately same performance, and cannot get even 5 plots to run in parallel without unreasonable drop in performance - plotting times shoot up to 7.5 -8 hours per plot for 5 plots, and 10-11 hours for 6 plots. Memory and CPU seems to be underloaded:
$ top

`top - 19:05:22 up 1 day, 7:54, 1 user, load average: 7.11, 6.95, 7.22`
`Tasks: 281 total, 3 running, 278 sleeping, 0 stopped, 0 zombie`
`%Cpu(s): 28.4 us, 2.3 sy, 0.0 ni, 68.1 id, 1.2 wa, 0.0 hi, 0.0 si, 0.0 st`
`MiB Mem : 23730.1 total, 303.5 free, 12180.3 used, 11246.3 buff/cache`
`MiB Swap: 12024.0 total, 11930.5 free, 93.5 used. 11155.6 avail Mem`
`PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND`
`5476 user1 20 0 5059192 1.9g 10252 D 100.0 8.1 298:59.09 chia`
`5542 user1 20 0 5629364 1.9g 10320 R 100.0 8.3 275:57.04 chia`
`5647 user1 20 0 1657292 1.2g 10220 R 100.0 5.0 247:13.72 chia`
`5788 user1 20 0 5233864 3.6g 11572 S 100.0 15.4 126:51.86 chia`
`5720 user1 20 0 5233864 3.1g 11308 S 93.3 13.2 189:26.65 chia`
`5992 user1 20 0 65520 4932 4016 R 6.7 0.0 0:00.01 top`

NVME and CPU temperatures are around 50 C.

Dstat doesn’t look worrisome (but maybe Im reading wrong):

    `$dstat`
    `You did not select any stats, using -cdngy by default.`
    `----total-usage---- -dsk/total- -net/total- ---paging-- ---system--`
    `usr sys idl wai stl| read writ| recv send| in out | int csw`
    `55 2 38 5 0| 201M 357M| 70 642 | 0 0 | 12k 2704`
    `48 2 46 4 0| 139M 273M| 70 322 | 0 0 | 11k 2382`
    `39 3 53 5 0| 278M 346M| 70 338 | 0 0 |9845 3043`
    `25 2 66 6 0| 315M 299M| 70 338 | 0 0 |7927 3280`
    `20 2 64 14 0| 329M 574M| 70 338 | 0 0 |7479 4505`
    `28 2 68 2 0| 482M 48k| 70 338 | 0 0 |8724 3439`
    `26 1 68 4 0| 538M 371k| 70 338 | 0 0 |8561 3969`
    `28 1 68 3 0| 544M 32k| 70 338 | 0 0 |8974 4007`
    `29 1 68 1 0| 419M 56k| 70 338 | 0 0 |8223 2660`
    `29 2 68 0 0| 200M 70M| 70 338 | 0 0 |6863 1192`
    `46 2 51 1 0| 186M 266M| 70 338 | 0 0 |9928 2289`
    `55 2 38 5 0| 229M 315M| 70 338 | 0 0 | 12k 3094`
    `53 2 39 6 0| 204M 361M| 70 338 | 0 0 | 11k 2870`
    `53 2 40 5 0| 178M 305M| 70 338 | 0 0 | 11k 2541`
    `38 3 51 8 0| 227M 399M| 70 338 | 0 0 |9793 3354`
    `27 2 62 9 0| 221M 503M| 70 338 | 0 0 |7827 3684`

Would anyone suggest what else should I check or possible root cause of my performance problems?

2

Not an expert but you will want to make sure you format the new drive with the correct file system xfs and block size 4K

2 Likes
3

I was following Build a Budget Chia Cryptocurrency Plotting Rig - , so formatted NVME according to it with a default settings:

mkfs.xfs /dev/nvme0n1
mount -t xfs -o discard /dev/nvme0n1 /mnt/ssd

so my block size on NVME is 512:

$ sudo blockdev --getbsz /dev/nvme0n1
512

but, it’s 4k for the hdd which is permanent storage.

I will try to reformat it tomorrow, do you by chance have a link handy that mentions formatting temp drive with 4k block size?

4

Actually you are correct, there is no mention of using 4K block size for temp ssd. My mistake sorry!
I followed the instructions given at NUC Small Form Factor Chia Plotting Build -

5

OK I ran 1 more test, to check NVME write speeds and it looks to be really good:

$ sudo dd if=/dev/nvme0n1 bs=1M count=102400 of=/dev/null
102400+0 records in
102400+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 39.1009 s, 2.7 GB/s

I cannot think of what else to look for to root cause it.

6

Are you turning on SSD TRIM? That’s a big gotcha on Linux systems. It is on by default for Windows, but will cause big perf problems in Linux unless you enable it.

3 Likes
7

I mount NVME with the discard option:

mount -t xfs -o discard /dev/nvme0n1 /mnt/ssd

It enables continuous trim, right? Or there is something else that needs to be turned on?

1 Like
8

Ok one more data point, seems that bottleneck is in the memory:

Bucket 82 uniform sort. Ram: 3.679GiB, u_sort min: 1.500GiB, qs min: 0.479GiB.
Bucket 83 uniform sort. Ram: 3.679GiB, u_sort min: 1.500GiB, qs min: 0.479GiB.
Bucket 84 uniform sort. Ram: 3.679GiB, u_sort min: 1.500GiB, qs min: 0.479GiB.
Bucket 85 uniform sort. Ram: 3.679GiB, u_sort min: 1.500GiB, qs min: 0.479GiB.
Bucket 86 QS. Ram: 3.679GiB, u_sort min: 0.750GiB, qs min: 0.276GiB. force_qs: 1
Second computation pass time: 780.857 seconds. CPU (63.120%) Sun May  2 21:03:54 2021
Wrote 3712421169 entries
Total compress table time: 1800.337 seconds. CPU (77.750%) Sun May  2 21:03:54 2021
Compressing tables 6 and 7
	Bucket 0 QS. Ram: 3.246GiB, u_sort min: 0.750GiB, qs min: 0.375GiB. force_qs: 1
	Bucket 1 QS. Ram: 3.246GiB, u_sort min: 0.750GiB, qs min: 0.375GiB. force_qs: 1
	Bucket 2 QS. Ram: 3.246GiB, u_sort min: 0.750GiB, qs min: 0.375GiB. force_qs: 1
	Bucket 3 QS. Ram: 3.246GiB, u_sort min: 0.750GiB, qs min: 0.375GiB. force_qs: 1
	Bucket 4 QS. Ram: 3.246GiB, u_sort min: 0.750GiB, qs min: 0.375GiB. force_qs: 1

Algorithm periodically switches to QS sort (which is slower, right?) despite me allocating 4500 MiBs of memory today, and system having 24 GBs of memory. On this particular node I don’t run absolutely anything, not even farmer, harvester - plotting only, and top shows around 2GBs buff/cache minimum at any time. Any suggestions on what to change here?

2 Likes
9

Hey @useful_Yak , have you found a solution to your nvme bottleneck and qs ram issue?

I guess i am having the same issue and see qs a lot in my logs.

10

I updated my Kernel to 5.12 and allocating 5gb of ram per plot. See minor improvements - around 30 min less per plot on average. But iostat shows worse values for the NVME than for my other machine where I have just 2 1tb SSD in raid0:

NVME:

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme0n1        2401.39    318988.09    311125.69 191086649504 186376757039

SSD:

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
md0           13125.60    315233.33    433776.23 188859365028 259879574852

So I guess while there are improvements, NVME is still the issue.

2 Likes