Are you saying you have used ntfs on your nvme for plotting on linux?
If yes, you should really try xfs without crc or f2fs for your ssd!
1 Like
yes, that is correct. since linux mint was reading the ntfs volume fine, i didn’t bother to format it (in case I was going back to the windows build). I am going to format it after this plot is done and report back . thank you for the input.
Dont forget about trim f2fs has it enabled by default, other you might want to set it.
No idea if thats the problem but doesnt hurt to try
1 Like
My best ~ 30min with 10 years old server
2 x E5-2670v2, 256GB ram DDR3 1333 for temp1 and temp2, ubuntu 20.04
Multi-threaded pipelined Chia k32 plotter - 9e649ae
Number of Plots: 1
Crafting plot 1 out of 1
Process ID: 13227
Number of Threads: 40
Number of Buckets P1: 2^9 (512)
Number of Buckets P3+P4: 2^8 (256)
Working Directory: /mnt/ram/
Working Directory 2: /mnt/ram/
[P1] Table 1 took 14.1196 sec
[P1] Table 2 took 134.439 sec, found 4294780592 matches
[P1] Table 3 took 157.217 sec, found 4294538533 matches
[P1] Table 4 took 158.531 sec, found 4294053743 matches
[P1] Table 5 took 151.793 sec, found 4293206428 matches
[P1] Table 6 took 147.434 sec, found 4291517986 matches
[P1] Table 7 took 118.355 sec, found 4288158439 matches
Phase 1 took 881.928 sec
[P2] max_table_size = 4294967296
[P2] Table 7 scan took 8.01158 sec
[P2] Table 7 rewrite took 41.048 sec, dropped 0 entries (0 %)
[P2] Table 6 scan took 27.1367 sec
[P2] Table 6 rewrite took 39.3715 sec, dropped 581693813 entries (13.5545 %)
[P2] Table 5 scan took 25.5788 sec
[P2] Table 5 rewrite took 36.3715 sec, dropped 762504584 entries (17.7607 %)
[P2] Table 4 scan took 25.3137 sec
[P2] Table 4 rewrite took 36.1546 sec, dropped 829246116 entries (19.3115 %)
[P2] Table 3 scan took 25.5184 sec
[P2] Table 3 rewrite took 36.1304 sec, dropped 855345584 entries (19.9171 %)
[P2] Table 2 scan took 24.9063 sec
[P2] Table 2 rewrite took 36.8134 sec, dropped 865750947 entries (20.1582 %)
Phase 2 took 382.976 sec
Wrote plot header with 268 bytes
[P3-1] Table 2 took 48.2943 sec, wrote 3429029645 right entries
[P3-2] Table 2 took 28.2337 sec, wrote 3429029645 left entries, 3429029645 final
[P3-1] Table 3 took 48.2911 sec, wrote 3439192949 right entries
[P3-2] Table 3 took 29.4568 sec, wrote 3439192949 left entries, 3439192949 final
[P3-1] Table 4 took 49.0913 sec, wrote 3464807627 right entries
[P3-2] Table 4 took 29.183 sec, wrote 3464807627 left entries, 3464807627 final
[P3-1] Table 5 took 49.6753 sec, wrote 3530701844 right entries
[P3-2] Table 5 took 30.446 sec, wrote 3530701844 left entries, 3530701844 final
[P3-1] Table 6 took 47.9859 sec, wrote 3709824173 right entries
[P3-2] Table 6 took 31.8798 sec, wrote 3709824173 left entries, 3709824173 final
[P3-1] Table 7 took 48.9686 sec, wrote 4288158439 right entries
[P3-2] Table 7 took 33.8413 sec, wrote 4288158439 left entries, 4288158439 final
Phase 3 took 481.707 sec, wrote 21861714677 entries to final plot
[P4] Starting to write C1 and C3 tables
[P4] Finished writing C1 and C3 tables
[P4] Writing C2 table
[P4] Finished writing C2 table
Phase 4 took 69.9645 sec, final plot size is 108742549503 bytes
Total plot creation time was 1816.67 sec (30.2779 min)
2 Likes
@juppin it looks like my NVME filesystm was definitely the issue. I formatted it to f2fs and getting under 2400s now using 16 threads + the default U/V of 256. I’m running through other settings for U/V now. But, wow, that was surprising (somewhat). Still need to figure out the cpu usage …
I may try xfs without crc like you mentioned as well later this week.
I use f2fs from first day, but I’m more limited by cpu than ssd throughput.
Don’t forget to mount your xfs nvme fs with options defaults,noatime,discard, for example in your /etc/fstab
...
/dev/nvme0n1p1 /mnt/tmp/00 f2fs defaults,noatime,discard 0 2
For f2fs discard is enabled by default, but a defaults,noatime couldn’t hurt and should improve i/o performance at least a bit.
“noatime,discard 02” would this work for ext4? are you getting better results on f2fs than ext4?
I cobbled together a system from leftover/old parts and I’m pretty impressed by the results. Did buy the NVMes new though. Total outlay about $500AUD
Ryzen 7 1700X on a ASUS Crosshair VI board, overclocked to 3.9GHz.
32GB of old RAM, non-matching sticks
2x 1TB Crucial P5 NVMe drives for temp 2
1x 500GB WD Green NVME for temp 1
Running Ubuntu 20.04 on windows subsystem for linux.
chia_plot -n 5 -r 12 -2
Around 55 - 60 minutes per plot.
oddly … it looks like something with the NVME file system being ntfs was also holding back my cpus from reaching 100% utilization. ever since I formatted the nvme to f2fs, cpu looks normal now that it is at 100% utilization during some of the phases.
Yes, will also work on ext4…
My single test with ext4 without journaling and the mentioned moint options where almost identical to f2fs. I’m not sure if my plotter was more limited by cpu, but I thought f2fs would probably be better for my ssd lifetime…
Seeing some weird behavior on Linux. So, yesterday, after formatting the NVME to f2fs, I was happily plotting along at about 2300-2600 seconds per plot (good improvement from 3200 on Windows).
At night, I scripted and went back to testing the U/V values … this morning, I noticed they were all super slow (4000+ seconds). So, I thought it may have been TRIM on the f2fs because I had the ‘discard’ option set. No luck running fstrim (nothing to trim) … so, I recreated the file system as f2fs and even xfs. Still no luck. I even tried un-mounting and re-mounting the ramdisk (didn’t try to fully recreate it with whatever commands I had used).
Nothing changed until I rebooted (xfs still in use for the nvme). Anyone can think of the oddity with this?
So it became better again after rebooting?
Correct, after reboot, it went back to a much quicker speed.
plot1 - 2535
plot2 - 2575
plot3 - 2622
plot4 - in progress but i’m presuming it will probably be about 2705 seconds
pattern would seem to match the degradation I saw last night after it had been plotting all day then I went to test U/V values late at night.
i created the ramdisk initially like this (ignore the size values as it’s proper at 111GB):
mount -t tmpfs -o size=1024m myramdisk /tmp/ramdisk
/etc/fstab (myramdisk /tmp/ramdisk tmpfs defaults,noatime,size=111G,x-gvfs-show 0 0)
will try another way after plot4 finishes re-mount it:
nodev,nosuid,noexec,nodiratime,size=1024M 0 0
odd indeed.
I’m very new to Linux, but my experience so far is not that different form Windows. Weird shit happens and then you need to reboot to fix it 
I installed standard Ubuntu 20.04, mounted the ramdisk by copying the line in the madmax github guide and mounted nvme by pressing the button that says “mount” in the disk manager.
Plotted a full 12 TB disk and didn’t see any slowdown. Times vary a bit but in the long run they are pretty consistent.
looking at my syslog, I noticed my usb drives were constantly dismounting and mounting (as the harvester was requesting read). in my /etc/fstab, I had some options that were disconnect on idle/etc that I had used from a site. Took those out and it stopped then.
ended up changing the tmpfs to a ramdisk with different mount options. prior to reboot, noticed the ramdisk was not properly being allocated memory or something because chia_plot kept getting into like 2-5 minutes into phase 1 by linux low memory management (some critical error). rebooted and it’s all been fine since … getting about 2450s per plot. going to change the xfs back to f2fs this weekend and see if I remain stable.
there is a setting in Ubuntu for USB drives (like sleep timers or standby thing…) you can disable.
How much faster will be the plotting process with a ramdisk as tmp2 instead of 2x970 pro?
Currently getting 36 min on 5800x and 2x970 pro
With cori 7 10700
16gb 3600
And 970 evo plus 500 gb
I cant get it under 75 min
I know there is something wrong
I know someone who can get 60 min with 14 thread
With 0.0.6 madmax
I think it doesn’t use my ram correctly
It use 5 gb of 16 gb
Can anyone help me?
Guys can you tell how parallel plotting on mad max works? If i put -n 3 for exp does it wait for plot one to finish and then start the second plot or what ?
Thankyou