Yeah. Splitting up the files into subfolder more or less fixed my problem, at least it is no longer timing out at 1194 seconds! Example log file entries:
2021-04-24T11:34:03.394 harvester chia.harvester.harvester: INFO 11 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 1.61972 s. Total 3882 plots
2021-04-24T11:34:07.122 harvester chia.harvester.harvester: INFO 8 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 1.35923 s. Total 3882 plots
2021-04-24T11:34:39.006 harvester chia.harvester.harvester: INFO 8 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 23.23485 s. Total 3882 plots
2021-04-24T11:34:51.613 harvester chia.harvester.harvester: INFO 5 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 24.16673 s. Total 3882 plots
2021-04-24T11:35:04.239 harvester chia.harvester.harvester: INFO 10 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 25.23309 s. Total 3882 plots
2021-04-24T11:35:30.203 harvester chia.harvester.harvester: INFO 6 plots were eligible for farming 857f51b97c... Found 0 proofs. Time: 38.57434 s. Total 3882 plots
Breaking that time out… I grabbed the last 650 harvester times and slapped them in a spreadsheet, got average 87s, min 0.8s, max 279s, median 65s. So it’s still too slow. I will note that in the Chia 1.0.0 release notes they said:
Weight proof request timeout was increased to 180 seconds.
However, “weight proof is when you ask another node to prove to you that they have a higher block, before you sync up to that block. Harvester 30 seconds is set in stone in consensus. No way to change that now.”
I need to identify which network device is the slow one, so he said to run
chia plots check -n 100 -g {filename}
on a file from each NAS to determine if one of them has performance issues, so here goes. I picked a random file on each individual NAS and executed that command; below is the output. The 3 lines I will highlight are:
- found the plot
- loaded / started to test the plot
- proof result for the plot
(load times, I have been told, are irrelevant so I’ve removed those… what matters is the time it takes for the -n 100 check)
NAS A (192.168.1.222)
2021-04-24T14:04:22.928 chia.plotting.plot_tools : INFO Found plot \\192.168.1.222\public\chia-plots-3\plot-k32-2021-04-02-14-42-0bed5fdf88d808ef6bec5883eba6340e6e26dd59359190c968f71b44af2ce895.plot of size 32
2021-04-24T14:04:22.944 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09897997698135441 TiB, in 1.1403071880340576 seconds
2021-04-24T14:06:19.598 chia.plotting.check_plots : INFO Proofs 113 / 100, 1.13
14:04:22 → 14:06:19 = 1:57
NAS B (192.168.1.97)
2021-04-24T14:08:59.865 chia.plotting.plot_tools : INFO Found plot \\192.168.1.97\public\chia-plots-3\plot-k32-2021-04-13-13-46-88ff07fd3b7c1c954ff2cf5213bf8b720211c0982ad230cf419beef71d4d2250.plot of size 32
2021-04-24T14:08:59.881 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09899792094620352 TiB, in 4.687199354171753 seconds
2021-04-24T14:10:36.520 chia.plotting.check_plots : INFO Proofs 98 / 100, 0.98
14:08:59 → 14:10:36 = 1:37
NAS C (192.168.1.114)
2021-04-24T14:11:48.006 chia.plotting.plot_tools : INFO Found plot \\192.168.1.114\public\chia-plots-3\plot-k32-2021-04-03-21-43-fc9dcc050b5f9e44c00302b682e98870997687d85b42f11d3d663e43718f79cd.plot of size 32
2021-04-24T14:11:48.006 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09896503558775294 TiB, in 2.671677350997925 seconds
2021-04-24T14:14:09.267 chia.plotting.check_plots : INFO Proofs 107 / 100, 1.07
14:11:48 → 14:14:09 = 2:21
NAS D (192.168.1.223)
2021-04-24T14:15:06.287 chia.plotting.plot_tools : INFO Found plot \\192.168.1.223\public\chia-plots-3\plot-k32-2021-03-28-00-35-62c642e68df8e144946224f189a844f18c03d12ba5fffbcc160c14bde4c43b2f.plot of size 32
2021-04-24T14:15:06.303 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09902664939090755 TiB, in 1.6250286102294922 seconds
2021-04-24T14:17:19.467 chia.plotting.check_plots : INFO Proofs 113 / 100, 1.13
14:15:06 → 14:17:19 = 2:13
NAS E (.126)
2021-04-24T14:18:20.975 chia.plotting.plot_tools : INFO Found plot \\192.168.1.126\public\chia-plots-2\plot-k32-2021-04-11-11-28-f258f57480c978e5025d026485775cced27a47c64ff305b42ec9a1c07de7f326.plot of size 32
2021-04-24T14:18:20.990 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09895394780960487 TiB, in 5.437449932098389 seconds
2021-04-24T14:19:37.305 chia.plotting.check_plots : INFO Proofs 92 / 100, 0.92
14:18:21 → 14:19:37 = 1:16
AMD machine (JBOD, 192.168.1.16)
2021-04-24T14:20:27.943 chia.plotting.plot_tools : INFO Found plot \\192.168.1.16\chia-final-z\plot-k32-2021-04-24-02-50-a3a9d0b52561ddbde4914da7eeb4aad8a6f226709d30e96ecaea5f78bf119231.plot of size 32
2021-04-24T14:20:27.943 chia.plotting.plot_tools : INFO Loaded a total of 1 plots of size 0.09897387777891709 TiB, in 6.140592098236084 seconds
2021-04-24T14:21:15.157 chia.plotting.check_plots : INFO Proofs 81 / 100, 0.81
14:20:28 → 14:21:15 = 0:47
I’m… really not sure what to do with this information? Oh and @ryan I checked pings to these IPs and they were all 1ms.