I turned old gaming hardware into a GPU mining rig with hiveOS installed and upgrade this rig with some harvester SSDs/HDDs:
MB: Gigabyte AX370 K7
CPU: 1800x
RAM: 32GB DDR4 @ 2666MHz JEDEC
NVMe: Micron 7300 Max 1.6TB U.2 Interface
HDD: Seagate Ironwolf 4TB
OS SSD: Patriot 60GB SATA
I started some plotting but unfortunately the NVMe can’t keep up with my win10 setup with a WD_Black 1TB SN750 SSD: while my main win10 machine finishes a plot in about 6 hours, the linux machine with the U2 NVMe takes about 14 hours.
I already managed to upgrade the NVMe firmware to the latest version 95420260 but no success. I found that the block sizes where formatted like 512/512bytes. I reformat the NVMe now fdisk shows 4096/4096bytes.
I don’t see what’s the problem here tbh. I guess wether I did something wrong in formatting, linux settings or maybe even the 3rd party U.2 cable I ordered from amazon.
I tried one single plot and 5 parrallel plots, no difference.
I did a hdparm -t /dev/nvme0n1 that never goes above 200MB/sec and as fare as Im aware this should go up to 3000MB/s according to the nvme specs
In regards to cores: I use 4 threads per plot. Otherwise I can’t imagine that the difference between 1800x and 3700x (win machine) and 2666MHz vs 3200MHz RAM clockspeeds leads to +8h per plot.
Im unfamiliar with nvme configuration under linux, especially when enterprise level with U.2 interface, and this drive plus cable is new so I tend to search here
I am plotting on 3900x with sn750 getting about 30 plots/day. Another guy here with the same cpu but those micron drives was getting 40/day.
CPU single core performance matters quite a lot and the difference between 1800x and 3700x is quite big i suppose.
But…14 hours is too slow, I get that on my old Athlon x4 845running 1 plots on an external ssd so something else must be going on somehow if you get that kind of time with just a single plot running
thanks for the trim hint, will check that tomorrow morning. Besides can we verify the blocksize settings? I am still unsure about this since I wanted to get something like logical size 4096 and physical size 512 (or other way around?)
first time I had 512/512 and after fw upgrade and format with nvme-cli tool from micron I get 4096/4096. If that is wrong, how can I reformat this correct?
I’m actually just experimenting with this myself and don;t know too much about it, so no answers yet. My drive is formatted to 128K atm
That’s in windows btw.
Although hdparm is a pretty bad way to benchmark a drive 200mb/s is indeed very slow for an nvme ssd. Are you sure nothing else is using the drive?
The fact that 5 parallel plots take the same time per plot points to something else than being io bound though. If you’re io bound on a single plot I’d expect running 5 plots in parallel would take significantly more time per plot.
But that’s all guessing. You can check if you’re io bound by running a plot and then looking at the iowait % of the system. It shouldn’t go above a few percent or you’re certainly io limited. Or run a single plot using plotman, it gives the total iowait time of the entire plot. It shouldn’t be more than a few seconds up until phase 4, it can go up to a few minutes at the end of phase 4 if the destination drive is a slow hdd.
And Ryzen is actually pretty dependent on RAM speed. On a threadripper 1920x going from 2666Mhz to 3600 takes an hour of plot time. Also the IPC difference between the first ryzen series and 3000 series is 25% according to AMD.
Only phase 1 of plotting uses multiple threads, the other 3 phases are single threaded, so single core performance of the CPU is pretty important.
How would you benchmark such a speedy nvme then?
I did some research today about trim: It seems like my nvme supports it according to nvme-cli output.
on google most people check trim functionality with hdparm --I /dev/nvme0n1 but I get the message ‘HDI0_DRIVE_CMD(identify) failed: Inappropriate ioctl for device’ even with latest hdparm 9.9
I also saw in chiaforum a thread with similar problems and that the other user was using xfs instead of ext4 so I reformat with ‘mkfs.xfs -f -s size=4096 /dev/nvme0n1’. there is also an option for trim but it should have been enabled by default.
Now when I mount the partition with ‘mount /dev/nvme0n1 /NVMe’ and have a look at ‘fstrim -v /NVMe’ I see 35MiB trimmed on that partition so I guess trim is working.
After reformatting to xfs the nvme reaches about 300MB/s with hdparm. so there is an increase but unfortunately still not what I was expecting
@ Ryzen RAM dependency: yeah Im aware of that but still even if the 3700x is 30% faster and you gain 10% by ram clocks, it should finish a plot under 10h imo. It feels more like your mentioned external ssd in terms of speed
I found this: Optimize performance for SSD (NVMe) on Linux · GitHub
and applied only the part to add to the grub boot default params, since I cant even see my nvme @ fstab ?! after reboot I reached 750MB/s with hdparm -t
I had another try of editing fstab with options like noatime to test. however, I failed again when reboot, mounted the drive on win and edited fstab but I had no other choice but to reflash the OS drive.
anybody has a clue what is going wrong on this fstab?
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
LABEL=HIVE /hive-config ntfs-3g errors=remount-ro,fmask=0133,dmask=0022,noatime,remove_hiberfile,nofail 0 2
UUID=b4b60f60-cd34-49c7-859b-53f802e8659c / ext4 errors=remount-ro,noatime,commit=120 0 1
UUID=d66fba70-4746-4ef8-a5d6-ac74dbec5398 /NVMe xfs noatime,errors=remount-ro 0 2
# LOGS OFF START
#tmpfs /tmp tmpfs defaults,noatime,size=100M,mode=1777 0 0
tmpfs /var/tmp tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0
tmpfs /var/log/apt tmpfs defaults,noatime 0 0
tmpfs /var/log/journal tmpfs defaults,noatime 0 0
# LOGS OFF END
