2nd Temporary Directory Experiments

I was actually just watching a plot as it was finishing phase 3.
At the end the tempfile size on the second temp drive quite suddenly jumped from 84GB to 105GB, and then phase 3 was done.

@Voodoo hi
did you manage to run some timing tests with your temp2 drive?
can you see a time benefit from adding a temp2 ?

current benefits are :
A. reduce IO on temp1 for phase 3 & 4
(I perpetually have 8 plots in phase3/4/5. If temp2 is slower than temp1 will I see reduction in plot times? )
B. reduce wear on temp1 and create wear on temp2
(basically wear is spread over more drives that you have to spend money for)

https://chiaforum.com/t/plotting-process-measurements/4788/5?u=gladanimal
Here is comparison.

An experiment with temp 2

Here I have 11 plots, 10 in phase 3, 1 in phase 4

plot id    k                tmp               dst    wall   phase    tmp     pid   stat     mem   user    sys     io
f2e7d477   32   /chia/plot-temp1   /chia/farm-vol8    6:34     3:1   235G   82394    RUN   10.2G   5:49   0:19   0:31
213a9622   32   /chia/plot-temp1   /chia/farm-vol8    7:04     3:1   219G   81339    RUN    9.6G   6:06   0:20   0:38
21fdbdc1   32   /chia/plot-temp1   /chia/farm-vol8    7:34     3:2   203G   80405    RUN    9.6G   6:22   0:21   0:47
11a45cec   32   /chia/plot-temp1   /chia/farm-vol8    8:04     3:2   182G   79487    RUN    9.6G   6:40   0:22   0:57
571053f5   32   /chia/plot-temp1   /chia/farm-vol8    8:34     3:3   161G   78575    DSK    9.6G   6:58   0:24   1:09
7c7e2d57   32   /chia/plot-temp1   /chia/farm-vol8    9:04     3:4   142G   77529    RUN    9.6G   7:16   0:25   1:23
0e9f3cdf   32   /chia/plot-temp1   /chia/farm-vol8    9:34     3:4   119G   76599    DSK    9.6G   7:34   0:26   1:39
a7f7d4aa   32   /chia/plot-temp1   /chia/farm-vol8   10:04     3:5    98G   75085    RUN    9.6G   7:52   0:27   1:58
837d608c   32   /chia/plot-temp1   /chia/farm-vol8   10:34     3:5    80G   73862    RUN    9.6G   8:13   0:28   2:14
d3fc073b   32   /chia/plot-temp1   /chia/farm-vol8   11:04     3:6    93G   72970    RUN    9.6G   8:26   0:30   2:41
6cc40c31   32   /chia/plot-temp1   /chia/farm-vol8   11:34     4:0    56G   72026    RUN    9.6G   8:43   0:31   3:04

And here is the output for sar -p -d

temp 1: nvme3n1
temp 2: nvme10n1
Farm: nvme6n1

What I see is that temp1 is the most used in terms of IOPS even in phase 3 but its almost ALL read!
temp2 gets written once a block is ready in the memory which is not that frequent.


ubuntu@ip-172-31-31-55:~$ sar -p -d 1 20 | egrep -i 'nvme3n1|nvme10n1|nvme6n1'
01:08:47      nvme3n1   2617.00 342472.00      0.00      0.00    130.86      2.92      1.12    100.00
01:08:47      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:47     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:48      nvme3n1   2856.00 345584.00  38540.00      0.00    134.50      3.28      1.15    100.00
01:08:48      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:48     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:49      nvme3n1   4787.00 381052.00 447860.00      0.00    173.16      7.05      1.47    100.00
01:08:49      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:49     nvme10n1   1227.00      0.00 305460.00      0.00    248.95      2.25      1.83     29.60
01:08:50      nvme3n1   3478.00 445020.00      0.00      0.00    127.95      3.85      1.11    100.00
01:08:50      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:50     nvme10n1    854.00      0.00 214504.00      0.00    251.18      1.60      1.87     13.60
01:08:51      nvme3n1   3305.00 422520.00      0.00      0.00    127.84      3.67      1.11    100.00
01:08:51      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:51     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:52      nvme3n1   2434.00 311036.00      0.00      0.00    127.79      3.28      1.35    100.00
01:08:52      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:52     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:53      nvme3n1   3052.00 403448.00      0.00      0.00    132.19      3.56      1.17    100.00
01:08:53      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:53     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:54      nvme3n1   6669.00 622328.00 529140.00      0.00    172.66      9.32      1.40    100.00
01:08:54      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:54     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:55      nvme3n1   8030.00 538496.00 1023244.00      0.00    194.49     17.79      2.22    100.00
01:08:55      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:55     nvme10n1   1813.00      0.00 456880.00      0.00    252.00      6.72      3.71     66.80
01:08:56      nvme3n1   4180.00 570744.00    577.50      0.00    136.68      7.69      1.84    100.00
01:08:56      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:56     nvme10n1   2357.00      0.00 598459.00      0.00    253.91     11.98      5.08    100.00
01:08:57      nvme3n1   3999.00 550520.00      0.00      0.00    137.66      7.50      1.87    100.40
01:08:57      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:57     nvme10n1   2454.00      0.00 618496.00      0.00    252.04     11.98      4.88    100.40
01:08:58      nvme3n1   4979.00 681376.00      0.00      0.00    136.85      6.56      1.32     99.60
01:08:58      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:58     nvme10n1    728.00      0.00 180848.00      0.00    248.42      3.78      5.19     31.60
01:08:59      nvme3n1   4215.00 576660.00      0.00      0.00    136.81      4.71      1.12    100.40
01:08:59      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:08:59     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:00      nvme3n1   3679.00 465148.00  20316.00      0.00    131.96      4.06      1.10    100.00
01:09:00      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:00     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:01      nvme3n1   5040.00 486640.00 348324.00      0.00    165.67      6.80      1.35    100.00
01:09:01      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:01     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:02      nvme3n1   3824.00 509564.00      0.00      0.00    133.25      4.81      1.26    100.00
01:09:02      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:02     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:03      nvme3n1   4050.00 539540.00      0.00      0.00    133.22      4.88      1.20    100.00
01:09:03      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:03     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:04      nvme3n1   3774.00 504316.00      0.00      0.00    133.63      4.27      1.13    100.00
01:09:04      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:04     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:05      nvme3n1   3734.00 499328.00      0.00      0.00    133.72      4.10      1.10    100.00
01:09:05      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:05     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:06      nvme3n1   7322.00 513788.00 885760.00      0.00    191.14     11.35      1.55    100.00
01:09:06      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:09:06     nvme10n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      nvme3n1   4301.20 485479.00 164688.08      0.00    151.16      6.07      1.41    100.02
Average:      nvme6n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     nvme10n1    471.65      0.00 118732.35      0.00    251.74      1.92      4.06     17.10