Are they plotting slower than you would like, than you expect - it sounds like the problem is not the drives but just in case, have you benchmarked the drives together and separately - do you know that they are not sharing PCIEe channels - i don’t have a threadripper so i don’t know how to verify they have individual bandwidth available
The drives do not share bandwidth there are tons of lanes(128 on the tr pro) and the X4 card is split x4x4x4x4. The other 3 drives(7 total in each machine) are in m.2 built in slots. The drives measure fine its just SOOO strange that the weaker speed drives(firecuda) plot the same speed as the faster drives(970/980 pros)…
Has anyone used Swar to plot on different drives where the drives plotted at different speeds?
TBH, and Ive not done testing in enough detail, it seems like they plot to the slowest drive speed…its strange…Its almost like the iowait is global not per drive, could that be? So if you bottleneck 1 drive you bottleneck them all
I have similar issues with 3900x( 1tb- wd750, hp EX920 and MX500) Things I believe to be somewhat true(windows 10 pro) b
single thread limits/ poor code. I get terrible average throughput (MB/s in Hwinfo64)
-highest throughput looks to be stage 1. The more threads the higher MB/s… But it limits how many plots you can run in stage 1.
i added 2 HDD to try get some extra usage of cpu cycles… I only hover around d 50-65%x cpu most of the time. Even with 4 jobs in phase 1 at once. It slowed all the other plots way down. I ended up killing one of the HDD drives plot jobs. I tried again with one HDD ( with various buckets) and it dragged everything down.
It appears to hold cpu until a request is completed. So a thread latency issue. I would have cpu activity dips that correlate with HDD increase activity. Where as if i just use the nvme, it us more of a steady saw tooth cou activity graph
My hwinfo64 avg. Write throughput on a each ssd;
Wd Sn750 1tb - 125-145MB/s
HP EX920 1TB (Intel 760 is same hardware) 110-125MB/s
Crucial MX500 - 99MB/s
The WD and HP numbers are on the lower side when running all 3 at once. And higher when running nvme only
To add to my last comment, if I run a Crystal Diskmark bench while drives are loaded rynning 8+ plots(swar staggered), I still have 50% of max write bandwidth/score.
Yea again, im no expert, but it has to do with something like iowait affects all drives if one is backed up or something of that nature is the only thing I can think of…that does not make sense though
With 3 nvme only, I have almost zero i/o queue . Only dud the Hdd plotting drive would spike between 0-6 queue
But individual threads are maxed while others have almost nothing for activity. Windows seams to pin it to a thread (probably for ccx/ccd reasons…might not be true… I just remembered that my i7 5820k is got pinned single threads too)
Seems to be cpu per thread bottleneck or i/o latency bottleneck.
If you have multiple drives in a RAID0 array then yes, speed is limited by the slowest drive in the array.
Say you want to write 512Kb to a 4 drive RAID0 array then the data is broken up into 4 128Kb pieces (stripe size) and each is written to a single drive. The 512Kb is consideren written when the slowest drive has finished. That’s why you generally dont want an array of mixed drive types.
Iowait % is indeed a global figure. If you run something like
iostat -m 5
you get a global iowait % for the system and then you have to look at the iops and bandwidth per drive to determine which drive is having a problem.
If you use plotman and mean the iowait column, that’s the amount of time each plot spent waiting for IO.
Thinking more, its probably both cpu per thread bottleneck combined with individual disk latency.
Reasoning, sata (mx500) has more latency than nvme. And it performs the worst of the ssds. (i have seen ppl with low cpu frequency not see any difference between Sata and nvme because the latency on sata is shorter than the cpu latency.
I am not really drive bandwidth restricted by a factor or 4-5x( for sata,). I only have peaks of 1500-1750 on the NVME drives even. All drives are bursts of i/o between 0 and 500MB/s usually. (lots of time at 0…so something cpu/infinity fabric related and core speed)
If I only run 3 jobs… Even on one nvme my clocks are 100-150mhz higher… And plots are way faster.
If I run same or less jobs per disk… But more disks… Cpu clock sags more… And longer plots.
Also suggested disk latency bottlenecking: ppl that use Intel optane Nvme have the records for fastest per plot speed. Those drives have super low latency.
