Plotting on linux with tmp NVMe drive too slow

Hi guys,

I turned old gaming hardware into a GPU mining rig with hiveOS installed and upgrade this rig with some harvester SSDs/HDDs:

MB: Gigabyte AX370 K7
CPU: 1800x
RAM: 32GB DDR4 @ 2666MHz JEDEC
NVMe: Micron 7300 Max 1.6TB U.2 Interface
HDD: Seagate Ironwolf 4TB
OS SSD: Patriot 60GB SATA

I started some plotting but unfortunately the NVMe can’t keep up with my win10 setup with a WD_Black 1TB SN750 SSD: while my main win10 machine finishes a plot in about 6 hours, the linux machine with the U2 NVMe takes about 14 hours.

I already managed to upgrade the NVMe firmware to the latest version 95420260 but no success. I found that the block sizes where formatted like 512/512bytes. I reformat the NVMe now fdisk shows 4096/4096bytes.

I don’t see what’s the problem here tbh. I guess wether I did something wrong in formatting, linux settings or maybe even the 3rd party U.2 cable I ordered from amazon.

anybody have any hint for me?
thanks in advance

What makes you think it’s the SSD that can’t keep up?

How many plots are you running at the same time?

I tried one single plot and 5 parrallel plots, no difference.
I did a hdparm -t /dev/nvme0n1 that never goes above 200MB/sec and as fare as Im aware this should go up to 3000MB/s according to the nvme specs

In regards to cores: I use 4 threads per plot. Otherwise I can’t imagine that the difference between 1800x and 3700x (win machine) and 2666MHz vs 3200MHz RAM clockspeeds leads to +8h per plot.

Im unfamiliar with nvme configuration under linux, especially when enterprise level with U.2 interface, and this drive plus cable is new so I tend to search here

1 Like

That drive should be really good for plotting.

I am plotting on 3900x with sn750 getting about 30 plots/day. Another guy here with the same cpu but those micron drives was getting 40/day.

CPU single core performance matters quite a lot and the difference between 1800x and 3700x is quite big i suppose.

But…14 hours is too slow, I get that on my old Athlon x4 845running 1 plots on an external ssd so something else must be going on somehow if you get that kind of time with just a single plot running

Trim settings could be a thing in Linux

1 Like

thanks for the trim hint, will check that tomorrow morning. Besides can we verify the blocksize settings? I am still unsure about this since I wanted to get something like logical size 4096 and physical size 512 (or other way around?)

first time I had 512/512 and after fw upgrade and format with nvme-cli tool from micron I get 4096/4096. If that is wrong, how can I reformat this correct?

I’m actually just experimenting with this myself and don;t know too much about it, so no answers yet. My drive is formatted to 128K atm
That’s in windows btw.

I think 4096/4096 is the standard though

Lol i replied to the wrong thread

Although hdparm is a pretty bad way to benchmark a drive 200mb/s is indeed very slow for an nvme ssd. Are you sure nothing else is using the drive?
The fact that 5 parallel plots take the same time per plot points to something else than being io bound though. If you’re io bound on a single plot I’d expect running 5 plots in parallel would take significantly more time per plot.
But that’s all guessing. You can check if you’re io bound by running a plot and then looking at the iowait % of the system. It shouldn’t go above a few percent or you’re certainly io limited. Or run a single plot using plotman, it gives the total iowait time of the entire plot. It shouldn’t be more than a few seconds up until phase 4, it can go up to a few minutes at the end of phase 4 if the destination drive is a slow hdd.

And Ryzen is actually pretty dependent on RAM speed. On a threadripper 1920x going from 2666Mhz to 3600 takes an hour of plot time. Also the IPC difference between the first ryzen series and 3000 series is 25% according to AMD.
Only phase 1 of plotting uses multiple threads, the other 3 phases are single threaded, so single core performance of the CPU is pretty important.

How would you benchmark such a speedy nvme then?
I did some research today about trim: It seems like my nvme supports it according to nvme-cli output.

on google most people check trim functionality with hdparm --I /dev/nvme0n1 but I get the message ‘HDI0_DRIVE_CMD(identify) failed: Inappropriate ioctl for device’ even with latest hdparm 9.9

I also saw in chiaforum a thread with similar problems and that the other user was using xfs instead of ext4 so I reformat with ‘mkfs.xfs -f -s size=4096 /dev/nvme0n1’. there is also an option for trim but it should have been enabled by default.

Now when I mount the partition with ‘mount /dev/nvme0n1 /NVMe’ and have a look at ‘fstrim -v /NVMe’ I see 35MiB trimmed on that partition so I guess trim is working.

After reformatting to xfs the nvme reaches about 300MB/s with hdparm. so there is an increase but unfortunately still not what I was expecting

@ Ryzen RAM dependency: yeah Im aware of that but still even if the 3700x is 30% faster and you gain 10% by ram clocks, it should finish a plot under 10h imo. It feels more like your mentioned external ssd in terms of speed

I found this: Optimize performance for SSD (NVMe) on Linux · GitHub
and applied only the part to add to the grub boot default params, since I cant even see my nvme @ fstab ?! after reboot I reached 750MB/s with hdparm -t

I tried to add an entry to fstab for the nvme with the options stated in the link above but that resulted in OS not booting anymore.

I reinstalled HiveOS and set up everything again except for the fstab entry and now cant reach 750MB/s anymore with hdparm :frowning:

can someone init a plot session and have a look with iostat -m 5 what are your peak writing/s ?

I foudn this: [SOLVED] Poor SSD Performance, Low Buffered Disk Reads / Kernel & Hardware / Arch Linux Forums

I wanted to have a look at my alignment but I already fail to check the output of fdisk -l it gives me:

fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.5 TiB, 1600321314816 bytes, 390703446 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

how can I check the alignment of the nvme?

I had another try of editing fstab with options like noatime to test. however, I failed again when reboot, mounted the drive on win and edited fstab but I had no other choice but to reflash the OS drive.

anybody has a clue what is going wrong on this fstab?

# /etc/fstab: static file system information.
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
# <file system> <mount point>   <type>  <options>       <dump>  <pass>

LABEL=HIVE /hive-config ntfs-3g errors=remount-ro,fmask=0133,dmask=0022,noatime,remove_hiberfile,nofail 0 2

UUID=b4b60f60-cd34-49c7-859b-53f802e8659c / ext4 errors=remount-ro,noatime,commit=120 0 1
UUID=d66fba70-4746-4ef8-a5d6-ac74dbec5398 /NVMe xfs noatime,errors=remount-ro 0 2

#tmpfs /tmp tmpfs defaults,noatime,size=100M,mode=1777 0 0
tmpfs /var/tmp tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0
tmpfs /var/log/apt tmpfs defaults,noatime 0 0
tmpfs /var/log/journal tmpfs defaults,noatime 0 0

after replugging the cable into ssd and mainboard i reach the desired transfer rates :grinning_face_with_smiling_eyes: