I was actually just watching a plot as it was finishing phase 3.
At the end the tempfile size on the second temp drive quite suddenly jumped from 84GB to 105GB, and then phase 3 was done.
@Voodoo hi
did you manage to run some timing tests with your temp2 drive?
can you see a time benefit from adding a temp2 ?
current benefits are :
A. reduce IO on temp1 for phase 3 & 4
(I perpetually have 8 plots in phase3/4/5. If temp2 is slower than temp1 will I see reduction in plot times? )
B. reduce wear on temp1 and create wear on temp2
(basically wear is spread over more drives that you have to spend money for)
An experiment with temp 2
Here I have 11 plots, 10 in phase 3, 1 in phase 4
plot id k tmp dst wall phase tmp pid stat mem user sys io
f2e7d477 32 /chia/plot-temp1 /chia/farm-vol8 6:34 3:1 235G 82394 RUN 10.2G 5:49 0:19 0:31
213a9622 32 /chia/plot-temp1 /chia/farm-vol8 7:04 3:1 219G 81339 RUN 9.6G 6:06 0:20 0:38
21fdbdc1 32 /chia/plot-temp1 /chia/farm-vol8 7:34 3:2 203G 80405 RUN 9.6G 6:22 0:21 0:47
11a45cec 32 /chia/plot-temp1 /chia/farm-vol8 8:04 3:2 182G 79487 RUN 9.6G 6:40 0:22 0:57
571053f5 32 /chia/plot-temp1 /chia/farm-vol8 8:34 3:3 161G 78575 DSK 9.6G 6:58 0:24 1:09
7c7e2d57 32 /chia/plot-temp1 /chia/farm-vol8 9:04 3:4 142G 77529 RUN 9.6G 7:16 0:25 1:23
0e9f3cdf 32 /chia/plot-temp1 /chia/farm-vol8 9:34 3:4 119G 76599 DSK 9.6G 7:34 0:26 1:39
a7f7d4aa 32 /chia/plot-temp1 /chia/farm-vol8 10:04 3:5 98G 75085 RUN 9.6G 7:52 0:27 1:58
837d608c 32 /chia/plot-temp1 /chia/farm-vol8 10:34 3:5 80G 73862 RUN 9.6G 8:13 0:28 2:14
d3fc073b 32 /chia/plot-temp1 /chia/farm-vol8 11:04 3:6 93G 72970 RUN 9.6G 8:26 0:30 2:41
6cc40c31 32 /chia/plot-temp1 /chia/farm-vol8 11:34 4:0 56G 72026 RUN 9.6G 8:43 0:31 3:04
And here is the output for sar -p -d
temp 1: nvme3n1
temp 2: nvme10n1
Farm: nvme6n1
What I see is that temp1 is the most used in terms of IOPS even in phase 3 but its almost ALL read!
temp2 gets written once a block is ready in the memory which is not that frequent.
ubuntu@ip-172-31-31-55:~$ sar -p -d 1 20 | egrep -i 'nvme3n1|nvme10n1|nvme6n1'
01:08:47 nvme3n1 2617.00 342472.00 0.00 0.00 130.86 2.92 1.12 100.00
01:08:47 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:47 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:48 nvme3n1 2856.00 345584.00 38540.00 0.00 134.50 3.28 1.15 100.00
01:08:48 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:48 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:49 nvme3n1 4787.00 381052.00 447860.00 0.00 173.16 7.05 1.47 100.00
01:08:49 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:49 nvme10n1 1227.00 0.00 305460.00 0.00 248.95 2.25 1.83 29.60
01:08:50 nvme3n1 3478.00 445020.00 0.00 0.00 127.95 3.85 1.11 100.00
01:08:50 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:50 nvme10n1 854.00 0.00 214504.00 0.00 251.18 1.60 1.87 13.60
01:08:51 nvme3n1 3305.00 422520.00 0.00 0.00 127.84 3.67 1.11 100.00
01:08:51 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:51 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:52 nvme3n1 2434.00 311036.00 0.00 0.00 127.79 3.28 1.35 100.00
01:08:52 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:52 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:53 nvme3n1 3052.00 403448.00 0.00 0.00 132.19 3.56 1.17 100.00
01:08:53 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:53 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:54 nvme3n1 6669.00 622328.00 529140.00 0.00 172.66 9.32 1.40 100.00
01:08:54 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:54 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:55 nvme3n1 8030.00 538496.00 1023244.00 0.00 194.49 17.79 2.22 100.00
01:08:55 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:55 nvme10n1 1813.00 0.00 456880.00 0.00 252.00 6.72 3.71 66.80
01:08:56 nvme3n1 4180.00 570744.00 577.50 0.00 136.68 7.69 1.84 100.00
01:08:56 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:56 nvme10n1 2357.00 0.00 598459.00 0.00 253.91 11.98 5.08 100.00
01:08:57 nvme3n1 3999.00 550520.00 0.00 0.00 137.66 7.50 1.87 100.40
01:08:57 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:57 nvme10n1 2454.00 0.00 618496.00 0.00 252.04 11.98 4.88 100.40
01:08:58 nvme3n1 4979.00 681376.00 0.00 0.00 136.85 6.56 1.32 99.60
01:08:58 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:58 nvme10n1 728.00 0.00 180848.00 0.00 248.42 3.78 5.19 31.60
01:08:59 nvme3n1 4215.00 576660.00 0.00 0.00 136.81 4.71 1.12 100.40
01:08:59 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:08:59 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:00 nvme3n1 3679.00 465148.00 20316.00 0.00 131.96 4.06 1.10 100.00
01:09:00 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:00 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:01 nvme3n1 5040.00 486640.00 348324.00 0.00 165.67 6.80 1.35 100.00
01:09:01 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:01 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:02 nvme3n1 3824.00 509564.00 0.00 0.00 133.25 4.81 1.26 100.00
01:09:02 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:02 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:03 nvme3n1 4050.00 539540.00 0.00 0.00 133.22 4.88 1.20 100.00
01:09:03 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:03 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:04 nvme3n1 3774.00 504316.00 0.00 0.00 133.63 4.27 1.13 100.00
01:09:04 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:04 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:05 nvme3n1 3734.00 499328.00 0.00 0.00 133.72 4.10 1.10 100.00
01:09:05 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:05 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:06 nvme3n1 7322.00 513788.00 885760.00 0.00 191.14 11.35 1.55 100.00
01:09:06 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:09:06 nvme10n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: nvme3n1 4301.20 485479.00 164688.08 0.00 151.16 6.07 1.41 100.02
Average: nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: nvme10n1 471.65 0.00 118732.35 0.00 251.74 1.92 4.06 17.10