I recently decided to do an overhaul of my Chia Environment. Currently my setup is as stated below I have 3 Nodes (Farmers) with a different wallet for each node, multiple harvesters connected to each node.
I upgraded all my nodes and harvesters to version 1.7.1 then 1.8.1
Node 1 and Node 3 upgrade was fine everythign is harvesting as it should
Node 2 - after upgrading to 1.7.1 stopped harvesting, would only harvest entering the command “chia start farmer-only -r” on the node and would stop harvesting after the plot refresh occured on each harvester @ 1200 second intervals. Was recommended to upgrade to 1.8.0 to see if issue would resolve but it did not. Now submitting here for support
Nodes are all running on 1.8.0 on Ubuntu CLI.
All Harvesters are running on headless Ubuntu. Also all updated to 1.8.0
Really would appreciate some help, please let me know if some logs are needing to be pulled.
I have also posted in chia GitHub as well but not finding much luck on what is going on.
I actually upgraded from 1.3.5 to 1.7.1 and then to 1.8.0. Same issue with the one node. I have been on the chia discord to ask for assistance and they have asked me to delete plots that were un-readable or had issues. But each time ive deleted them more pop back up. I’ve deleted about 20 or so plots. Each same problem after, im afraid if i keep doing i will have no plots left farmable.
Like I said, its weird that it worked on an older version and when upgrading to a newer version it stopped. And it isn’t the whole environment, its just one of the three nodes that have an issue. Sorry if I seem a bit desperate but its about 80+PB of plots and so far the only answer I have gotten is to delete bad plots.
As stated above all the issues started happening after i updated to 1.7.1 from 1.3.5. After version 1.7.1 didn’t work I was asked to update to 1.8.0, afterwards that still did not work. The only thing that allows the harvesters to start farming is entering the command
chia start farmer-only -r as recommended by a chia staff member. When you enter that command in it will start showing up “xx plots are eligible for farming” it will keep farming until after the plot refresh then will not harvest any longer.
This only happens on one node, which has been replaced and I have replaced one harvester connected to the same node. All of this was farmable on version 1.3.5
The error logs for each harvester always displays text stating
Blockquote
2023-05-16T07:38:05.489 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d9ct-78/plot-k32-2021-10-22-19-27-74cd04048a5ec7faebd43e7bdc9ee27d5147ab8c5d8ae686c2f42da1d2eeb232.plot. Invalid file /mnt/d9ct-78/plot-k32-2021-10-22-19-27-74cd04048a5ec7faebd43e7bdc9ee27d5147ab8c5d8ae686c2f42da1d2eeb232.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d9ct-78/plot-k32-2021-10-22-19-27-74cd04048a5ec7faebd43e7bdc9ee27d5147ab8c5d8ae686c2f42da1d2eeb232.plot
2023-05-16T07:38:05.489 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d9ct-78/plot-k32-2021-10-27-21-42-b722e393718f9c6ac54658a2e46e972524c510c0477d20cc7126b3917d3acbae.plot. Invalid file /mnt/d9ct-78/plot-k32-2021-10-27-21-42-b722e393718f9c6ac54658a2e46e972524c510c0477d20cc7126b3917d3acbae.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d9ct-78/plot-k32-2021-10-27-21-42-b722e393718f9c6ac54658a2e46e972524c510c0477d20cc7126b3917d3acbae.plot
however i was recommended to delete these plots out of my harvester. I have removed about 20 plots on my harvester and these error messages are still popping up. One interesting fact is that when I run the command to delete the plot "rm xxxxxxxxxxxxx.plot i always get a permission denied error, I have to run include “sudo rm” for it to completely remove it. Not sure if this makes a difference but when I have to remove folders or node keys i do not have to SUDO RM to remove that folder, just specifically with these plot files.
Please let me know if you guys need anything else.
obviously either logical error with file system, or hdd is failed. stop deleting plots since its not helping. can you read your plots with another software, like just reading your files to NULL? if yes, could you also check those plots with chia plots check -g command?
I will try that, But I believe you gave me some inspiration in the sense that why am I loading every plot up, why don’t I load up a few drives and see.
The only weird thing is it just stopped working the update, and also now im seeing weird behavior on my harvesters where it shows the “xx plots eligible for farming… 0 Proofs found” now after each cycle it says its finding multiple proofs now, is that normal behavoir?
even mounting 20 drives, I still get below error, It happens across most of those harvesters that are offline
tail -f .chia/mainnet/log/debug.log
2023-05-17T00:16:54.950 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot. Invalid file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot
2023-05-17T00:16:55.037 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot. Invalid file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot
2023-05-17T00:19:22.908 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot. Invalid file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d5xf-1/plot-k32-2021-08-12-18-08-7010a186de0767a7b6733ba16a5138050870288f6dbbcc93a9c1eff434b4d85b.plot
2023-05-17T00:19:22.977 harvester chia.plotting.manager : ERROR Failed to open file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot. Invalid file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot Traceback (most recent call last):
File “/home/bigdreams/chia-blockchain/chia/plotting/manager.py”, line 274, in process_file
prover = DiskProver(str(file_path))
ValueError: Invalid file /mnt/d5xf-22/plot-k32-2021-08-18-23-07-6d4427d9d4e3060417bc32e55d2b25a1ca95b8b4ed678aad7e4cba6fc516b56e.plot
I can’t put my finger on it but there is no way that all 7 racks of jbods all crashed at one time. Call me skeptical but I don’t think that is the case. Any ideas anyone?
