Well, you’ve convinced me – it is possible to achieve 50 plots/day on a 16c/32t machine. It is surprisingly tricky though. My simple approach of just throwing a bunch of 4t/6gb parallel plots and tons of I/O bandwidth (six full bandwidth nvme ports which is crazy overkill) at the problem didn’t really work.
My gut feeling says the specifics of the stagger must be important, the timing?
(Also I’ve read that the 970 pro is actually one of the best plotting drives there is. It’s incredibly fast on my i9-9900ks … I get 4.5hr plot times even running 3 parallel)
I have since removed 2 of the drives (so it has 4 now, instead of 6) and am moving on to throwing more machines in the mix.
That doesn’t make sense, we are not using a single plotter! We always use plotters in parallel.
The absolute fastest I’ve seen a plot finish myself, on hardware I own, is 270 minutes (4.5 hours), that is 5.3 plots per day. With that machine we’d need 9 parallel processes, and no speed degradation as we increase the number of plotters… fat chance of that! So it’s a question of
I read that 8 threads is actually worse than 4, so I am puzzled by a lot of this @harris… perhaps there’s some magic in the 8 threads when everything is under full load? I always do 4 threaded plots, except on my 4c/8t machines where I do 2.
I’ve hit 51 (now 52 after the queue levelled out) plots a day (12 parallel) on my Linux 5950X machine (not overclocked)
So 12 parallel to hit 50… means each of those 12 must be doing 4.16 plots per day, or 346 minutes per plot, under 21000 seconds per plot or 5h46m?
On another Ryzen 5950x I just set up, I see 21000 seconds or under for the very first plot (low contention from other plotters), but 25000, 26000 seconds for the subsequent plots. To achieve 50 per day, at 12 plotters, all of the plotters must be doing 21000 seconds or better.
Yes but this possible, I have very little storage bottleneck left, it only becomes a thing in phase 4 which is basically, I can’t write to my USB3 connected disks any faster.
This is a 5900x with up to 12 plots going at the same time.
I think most important you are asking about would be:
tmpdir_stagger_phase_major: 2
tmpdir_stagger_phase_minor: 1
# Optional: default is 1
tmpdir_stagger_phase_limit: 5
# Don't run more than this many jobs at a time on a single temp dir.
tmpdir_max_jobs: 12
# Don't run more than this many jobs at a time in total.
global_max_jobs: 12
# Don't run any jobs (across all temp dirs) more often than this.
global_stagger_m: 30
What you’re saying makes sense to me and your observations about plot times and speed degradation is consistent with what I see. But I’m not too concerned with individual plot time.
My thinking is that our total throughput will be limited by our stagger time in an otherwise unconstrained system, correct? For example, if the stagger time is set to 1 hour; it doesn’t matter if each plots finish in 10 seconds or 10 hours, the system will not exceed 24 plots in a 24 hour period.
If we are targeting 50/day, any stagger value greater than 28.8 minutes guarantees that the system will fall short of our target.
Does that make sense, or perhaps I am misunderstanding something?
No, I’m not following; the stagger only affects the start time for the first plot, all subsequent plots kick off immediately afterwards. I think we have different understandings of the word “stagger”. For me it is a one-time value, that determines how offset the plotters are against each other, and it is only set once, forever e.g. “plotter 5 will be on phase 1, while plotter 6 is on phase 3 – they’ll always be four hours apart”.
There might be ways of doing stagger every single time every plotter begins a plot, but I personally only use stagger values when starting the plotter initially.
Hmm now I’m confused about how you’re staggering plots. Do your plots start in a way that is consistent with Quindor’s screenshot from post #112? Or are there times when you have more than 1 plot starting at once?
In particular, the “wall” column in plotman reports the time a plot has been running. So for plots # 0-5 we can see his system was unconstrained and each plot is offset by exactly 30 minutes, his stagger time. Plots 6-10 have some additional variability in the timing offset, presumably due to some secondary plot limit (indicating a minor system constraint).
So in this case, with a stagger of 30 minutes, Quindor would be able to achieve 48 plots/day as an absolute maximum. In practice it looks like he would land just shy of 48 because some plots (ie plot 9) are having to wait up to 35 minutes before starting.
That is not correct, it will always wait until it’s stagger time is up and then check the other values if it’s allowed to start a new job. If not it waits until it’s allowed to and then stagger starts counting again. Check my stream shot again, you see the stagger time is 1024s/1800s
I see, so we’re using different versions of “stagger”. I should probably say “offset”; I let everything constantly plot, just offset from everything else by {x} minutes.
It’s possible I’ve been using the word wrong! Let’s clarify here
Excellent data! Thanks for sharing! I just got my 4x 970 Pros in. I can only use 3 for now until I get NVME to PCIE card. I will be testing 4 once that arrives. If I can plot 15 in parallel on 4 x 1TB, I’ll stick with that, but if not, I’ll do 5x 1TB. May do 5x 1TB anyway just to get more IO bandwidth.
I just started using the 970 Pros but they already seem to be ~25-30% faster than the 2TB Firecuda 520s I’ll be sending back. I will have to see the final results when the plots finish, and will post the results here.
Hi all, im looking for an ideal config for my setup
its currently Ryzen 5900x with 64gb DDR4 3200mhz, on a x570 board
Ive installed Ubuntu 20.4 and running Swar plot manager
My SSD is
2tb xpg gammix nvme
2tb xpg gammix nvme
1tb msi crucial nvme
400gb intel ssd
400gb intel ssd
(2tbs are in m.2 slots on the MB and 1tb is in a pci adapter)
Intels i would like to raid using controller in future as i get more of them and cables to the controller.
1)i would be glad if you could point me out what is the best way to config that on 2tb and 1tb drives
2)My intel ssds, maybe there is a point of using them as temp2 directory for 2tb nvme?
@harris we also have confirmation that Linux is 10% faster at plotting than Windows, so that explains inability to hit 50 in Windows as well. 10% of 50 is 5!
