No XCH since moving to the multiple harvester model

Chia Farming & Harvesting
1

TLDR; I was farming XCH every 5-7 days when using a single farmer. I switched to a farmer <–> harvester model and haven’t farmed any XCH since… about ~21 days (but ~23 days since last XCH). I would like to know if I’ve done something wrong or if I’m just in a period of bad luck.

Prior to May 10th, I was using a single farmer/harvester on windows. At the time I was farming XCH every 5-7 days (yea, I know, not everyone has farmed some) with ~2400 plots. I was receiving warnings in my log about some of the lookups taking too long, every couple hours. The warnings indicated they would take ~5 seconds to ~10 seconds. Nothing close to the 30 seconds.

Given I had a lot more storage to fill I figured I would move some of the storage off my farmer and setup some harvesters.

After I was done I had a setup like this…

chia_architecture
chia_architecture1265×657 17.4 KB

All of the harvesters are running Ubuntu on bare metal except for harvester04. I have a large TrueNAS instance with a lot of available space, so I spun up a VM (harvester04) to run a chia harvester against plots stored on the NAS. This is Ubuntu also.

Since moving I have grown to about 4500 plots. On each of my harvesters I am seeing messages like this (below). I’m not going to bother posting one for each harvester (or the farmer), but I have confirmed each one seems to be communicating with the farmer and each harvester and the farmer are responding to about 6 challenges per minute.

2021-06-01T09:34:07.707 harvester chia.harvester.harvester: INFO     6 plots were eligible for farming f4b8e9459c... Found 0 proofs. Time: 0.23596 s. Total 1496 plots
2021-06-01T09:34:07.707 harvester harvester_server        : DEBUG    -> farming_info to peer 10.233.44.101 5fde4dff5bbb0165657027d544c461a25d9e1dc54f4607c497927f7704a99198

I’m also seeing the plots in the farmer gui in the “Last Attempted Proof” section from my harvesters.

Also, I’m using the chiaharvestgraph, they all pretty much look like this…

image
image909×437 16.2 KB

As you can see from my diagram I am also a GPU miner. These are just the GPU rigs I have plotting. I have others, just not plotting.

I realize the netspace has grown a little faster than I have grown, but it just seems weird that my rewards have stopped when I switched to this model.

7 Likes
2

I think as long as each harvester is getting the challenges, and responding, you’re good? I can’t think of anything that could be wrong with evidence of the proof checks happening in the logs…

Excellent and very detailed post btw

3

I think so, but my expected time to win according to chiacalculator.com is 8 days… It was 7 or 8 when I last got some Chia… so I have grown with the network. I realize that it is all statistics and when looking at the last 23 days or so you can’t count on what the calculator says.

1 Like
4

Can you switch back to the other architecture, as an experiment? I’m really not sure what to check, because the best test I know of is the harvester proof checks and those all look correct in the logs you shared?

5

I actually switched to using only LAN shares from my plotters and farmers primarily because I can’t stand it that I can’t see my total plots and the challenges, and the estimated win time, in one place. Seeing the challenges get more and more hits is easier on the mind when its takes a month to win. Its worked for me with wins. (800 plots here). I don’t have any reason to doubt the built-in harvester functionality, but I couldn’t help but wonder its its better or working right. My response times are almost always 0.5s or less so I can’t see why not do it this way. Most of the drives are on the farmer, but a few are over 1G LAN, and I carefully manage what’s going on on those connections, mainly be wiring up multiple cables between farmer and plotters.

6

There is surely a need to be able to reassure farmers, a check button or command that can test the entire farm and ensure its valid and farmable. A lot of effort went into plotting we ought to be able to check that its all farmable easily.

6 Likes
7

I wish it could do this easily, but it would mean adding another ~35 disks to this windows machine. I could do it, but it would be a pain in the butt…

8

The only ERROR in my debug.log is this one message…

2021-06-01T17:44:42.715 full_node full_node_server        : ERROR    Exception Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x0000019FCACD3C18> at 51.15.216.154, exception Stack: Traceback (most recent call last):
  File "chia\server\server.py", line 248, in incoming_connection
  File "chia\server\ws_connection.py", line 59, in __init__
ValueError: Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x0000019FCACD3C18> at 51.15.216.154

No WARNING items.

These are all over the place in the logs so it seems like it is getting data from the harvesters.

2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 127.0.0.1 fb4ca402e346c0efd414452cd304ef767de26de6b6103a38c230a85332664437
2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 10.233.44.106 36f9f3b8ac444ad02bf6c9e92c58637569b04ff4433b3845f262355e8accd94e
2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 10.233.44.108 596efe8ecc1fe7a76a70885f950ea1b2dfe7117cdcec70eb0459d856bc59f680
2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 10.233.44.109 6fe1fedf31362701690c5340fa9b43ee7b77fe3686beabf05013b75c362c4b45
2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 10.233.44.110 43b3c9657a95f9f05916ebd740b6afdcdffefe633ea0719ca490db9e3ae7f3ef
2021-06-01T17:45:07.913 farmer farmer_server              : DEBUG    -> new_signage_point_harvester to peer 10.233.44.105 8f2677a3fb41692e83734cf1d7a43c50fa1ff644124402c97d9f0f786929fb59
9

I suspect that the multiple harvesters model is worse than having 1 harvester probably because the farmer doesn’t know the actual netspace share of the whole farm, so it only calculates the harvester’s share.

10

That calculation in really only for the GUI. It SHOULD (if the devs do it) in the future take into account all of your harvesters and the plots on them. It’s not really related to the issue OP is experiencing.

1 Like
11

I have been running a multi-harvester setup since a month as well.
I have 1434 plots at 142TiB. I have 0 wins till now but all my response times are in good shape!
I am thinking of moving everything to 1 machine and see if it works any better!

1 Like
12

I don‘t see your harvester reporting back tbh or are there more lines like „farming info from peer“?

13

I am using a similar model and I am occasionally winning blocks so I can tell you that it works.

However, the consistency is far from ideal. I have about twice the amount of plots you have; a bit over 2 weeks ago I was at an expected time to win of 3 days and it took 9 days to win a block. After that one, I’ve been constantly at a time to win below 4 days and once again it’s been 9 days since my last block and I have yet to win another one.

Naturally, all this with zero downtime, no log issues etc.

I guess I’m just unlucky but nonetheless I’ll probably move all my plots to hpool after the next win. If I had my plots there during this time I would’ve made some 10 XCH instead of 2 XCH.

In other words, farming solo kind of sucks even if your total capacity exceeds 1PB. I’d say at the moment it’s viable if you’re at 4-5PB+.

2 Likes
14

[UPDATE] I sorted out the issues and its all good now on the ubuntu full chia stack.

So I consolidated ALL my plots on an ubuntu machine and started the full chia stack on it, copied the wallet and blockchain sqlite files to this machine and started.

(venv) ubuntu@ip-172-31-31-55:~/.chia/mainnet/wallet/db$ chia farm summary
Farming status: Farming
Total chia farmed: 0.0
User transaction fees: 0.0
Block rewards: 0.0
Last height farmed: 0
Plot count: 1434
Total size of plots: 142.784 TiB
Estimated network space: 16402.947 PiB
Expected time to win: 3 weeks and 2 days
Note: log into your key using 'chia wallet show' to see rewards for each key

It has been almost 2 weeks, lets see when do I win! :smiley:

2 Likes
15

I see these 6 to 7 of these messages per minute per harvester…

2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    <- farming_info from peer 6fe1fedf31362701690c5340fa9b43ee7b77fe3686beabf05013b75c362c4b45 10.233.44.109
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    <- farming_info from peer 36f9f3b8ac444ad02bf6c9e92c58637569b04ff4433b3845f262355e8accd94e 10.233.44.106
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    <- farming_info from peer 596efe8ecc1fe7a76a70885f950ea1b2dfe7117cdcec70eb0459d856bc59f680 10.233.44.108
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    Time taken to process farming_info from 6fe1fedf31362701690c5340fa9b43ee7b77fe3686beabf05013b75c362c4b45 is 0.0 seconds
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    Time taken to process farming_info from 36f9f3b8ac444ad02bf6c9e92c58637569b04ff4433b3845f262355e8accd94e is 0.0 seconds
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    Time taken to process farming_info from 596efe8ecc1fe7a76a70885f950ea1b2dfe7117cdcec70eb0459d856bc59f680 is 0.0 seconds
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    <- farming_info from peer 8f2677a3fb41692e83734cf1d7a43c50fa1ff644124402c97d9f0f786929fb59 10.233.44.105
2021-06-01T17:48:57.716 farmer farmer_server              : DEBUG    Time taken to process farming_info from 8f2677a3fb41692e83734cf1d7a43c50fa1ff644124402c97d9f0f786929fb59 is 0.0 seconds
2021-06-01T17:48:57.795 farmer farmer_server              : DEBUG    <- farming_info from peer 43b3c9657a95f9f05916ebd740b6afdcdffefe633ea0719ca490db9e3ae7f3ef 10.233.44.110
2021-06-01T17:48:57.795 farmer farmer_server              : DEBUG    Time taken to process farming_info from 43b3c9657a95f9f05916ebd740b6afdcdffefe633ea0719ca490db9e3ae7f3ef is 0.0 seconds
16

0.0 seconds is kind of weird to see tbh. I got like around 1 second there.

17

Getting the following error when SCP the database file from windows node to ubuntu node!
The transfer mode was binary.
sqlite3.DatabaseError: database disk image is malformed

I see that chia on ubuntu does not remove the following files when chia is shutdown completely. But these are removed on windows when the chia GUI is shut down.

blockchain_v1_mainnet.sqlite-shm
blockchain_v1_mainnet.sqlite-wal
peer_table_node.sqlite-shm
peer_table_node.sqlite-wal

Should I manually delete them before restarting chia stack?

(venv) ubuntu@ip-172-31-31-55:~/.chia/mainnet/db$ file blockchain_v1_mainnet.sqlite
blockchain_v1_mainnet.sqlite: SQLite 3.x database, last written using SQLite version 3031001

[UPDATE]
I deleted the files mentioned above and restarted the chia stack, it successfully digested the blockchain!

18

Can you share the math or model that makes you believe the probability of having won nothing for the past 25 days with the number of plots you have is very low? IMHO the chances of you having won anything would have been pretty low.

19

Are you asking me this question or someone else? I had 2400 plots 20 days ago, now I have 4500+ plots, not XCH… I’m just using the chia calculator. I’m not demanding XCH, I just want to make sure that I didn’t break something moving to the harvester model.

1 Like
20

Based on all you’ve written above it seems like you are doing nothing wrong. If logs are coming back good, and since Chia themselves have a “Farming on multiple machines” guide, this seems fine.

The real issue is here is Chia needs to make it more obvious if things are wrong.

1 Like