Here are my results for a single queue.
QUEUE 1, PLOT 1:
Starting phase 1/4: Forward Propagation into tmp files… Sat Jun 5 18:17:52 2021
Time for phase 1 = 5406.763 seconds. CPU (233.050%) Sat Jun 5 19:47:59 2021
Time for phase 2 = 4087.494 seconds. CPU (98.600%) Sat Jun 5 20:56:06 2021
Time for phase 3 = 7022.806 seconds. CPU (99.290%) Sat Jun 5 22:53:09 2021
Time for phase 4 = 467.572 seconds. CPU (99.920%) Sat Jun 5 23:00:57 2021
Total time = 16984.636 seconds. CPU (141.720%) Sat Jun 5 23:00:57 2021
Copy time = 877.871 seconds. CPU (11.110%) Sat Jun 5 23:15:37 2021
QUEUE 1, PLOT 2:
Starting phase 1/4: Forward Propagation into tmp files… Sat Jun 5 23:15:41 2021
Time for phase 1 = 5494.400 seconds. CPU (233.490%) Sun Jun 6 00:47:16 2021
Time for phase 2 = 3999.716 seconds. CPU (99.570%) Sun Jun 6 01:53:56 2021
Time for phase 3 = 7049.941 seconds. CPU (99.540%) Sun Jun 6 03:51:26 2021
Time for phase 4 = 471.267 seconds. CPU (99.770%) Sun Jun 6 03:59:17 2021
Total time = 17015.325 seconds. CPU (142.800%) Sun Jun 6 03:59:17 2021
Copy time = 862.080 seconds. CPU (11.690%) Sun Jun 6 04:13:42 2021
What stagger time would you suggest for parallel plotting queues?
Many thanks
My understanding is that you want phase 1 complete on plot 1 before beginning plot 2. So you took 5600 seconds to get to phase 2 so it’s 95 minutes or so.
Since phase 1 uses 4 threads as a rough optimum but drops to 1 thread when complete.
I guess if you have a lot of threads available you might start multiple phase 1s?
DjDemonD, thanks for your replay.
I do have 24 threads (Ryzen 9 3900x).
I am however hesitant to start 6 queues at the same time (6 x 4 threads = 24).
I imagine the most i/o intensive phase is the one we would want to avoid running in parallel.
If this is phase 3, does that mean I should stagger after 7023 seconds (1.95 hours)?
If I do this, I start having overlapping phase 3s as soon as I do 3 queues.
You do not really get an advantage by using 4 threads per plot - the per thread speed increase is overwhelmed by the increase that you get in running more parallel plots. You mostly need to be careful about exceeding your temp space (you can typically run 4 plots in phase 1 and 8 total plots on a 1TB drive at once).
I recommend taking a look at a plot manager, such as Swar, that actually monitors the logs to determine what phase your plots are in, rather than trying to figure out second timings for phases, which will change constantly as you add more plots and have plots in various different phases at the same time.
1 Like
I actually see quite a good improvement for phase 1 times from 2 to 4 threads and even from 6 or 8.
The thing is that you don’t need to worry to much about the total number of threads on your cpu. In any case multi thread is only used in phase 1, this is also by far the most cpu heavy part.
But like edgore said, the most important thing is figuring out how many you can run at a time, and get optimal output per day.
I have the same CPU and am running 9 plots at a time.
3-4 in phase 1, all 8 threads.
So in total I am running 30-37 threads at a time on 24 thread cpu. ’
stagger is 40 minutes.
This is now doing 32 plots/day using Swar on windows with 3x 1TB disks
1 Like