Troubleshooting Harvester disconnecting from Farmer

Chia Farming & Harvesting
1

Hi all,
I have 2 Windows 10 machines. 192.168.1.10 is my full node. 192.168.1.9 is my harvester.
To set them up I have used the following guide Farming on Many Machines | Chia Documentation.

I’m however not seeing the harvester showing up on my full node. I have set the debug log to DEBUG and have the following debug info.

2023-09-15T09:42:33.745 farmer farmer_server              : DEBUG    -> handshake to peer 192.168.1.9 bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a
2023-09-15T09:42:33.745 daemon __main__                   : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "add_connection", "data": {"success": true}, "destination": "metrics", "origin": "chia_farmer", "request_id": "0b8f9ab4fe808f30d6e24b4b1052754cdbda3786eee22f4fe0949faa8131dc75"}', extra='')
2023-09-15T09:42:33.745 farmer farmer_server              : DEBUG    -> harvester_handshake to peer 192.168.1.9 bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a
2023-09-15T09:42:33.761 daemon __main__                   : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "get_connections", "data": {"connections": [{"bytes_read": 18646, "bytes_written": 11686, "creation_time": 1694763110.7040858, "last_message_time": 1694763748.0116956, "local_port": 8447, "node_id": "0xf1906f47ffe733e0f4a0f18e5334192a91a80da8893332af233d11a0733e51d5", "peer_host": "127.0.0.1", "peer_port": 52665, "peer_server_port": 8448, "type": 2}, {"bytes_read": 8428, "bytes_written": 64, "creation_time": 1694763125.2506537, "last_message_time": 1694763747.980441, "local_port": 8447, "node_id": "0x646c402cc8e79b0963452a6449c1bc796d420da006941bf68b8af9f30563fe31", "peer_host": "127.0.0.1", "peer_port": 8444, "peer_server_port": 8444, "type": 1}, {"bytes_read": 64, "bytes_written": 64, "creation_time": 1694763753.7459009, "last_message_time": 1694763753.7459009, "local_port": 8447, "node_id": "0xbcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a", "peer_host": "192.168.1.9", "peer_port": 50052, "peer_server_port": 8448, "type": 2}], "success": true}, "destination": "wallet_ui", "origin": "chia_farmer", "request_id": "a4c8dc07b2f4029f2bf93f12de248222b91e9188de297ae0d5f1cb678e6bb165"}', extra='')
2023-09-15T09:42:33.980 farmer farmer_server              : DEBUG    <- plot_sync_start from peer bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a 192.168.1.9
2023-09-15T09:42:33.980 farmer chia.plot_sync.receiver    : DEBUG    _process: node_id bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a, message_type: ProtocolMessageTypes.plot_sync_start, message: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1694763754, 'timestamp': 1694763754}, initial True, last_sync_id 0, plot_file_count 152, harvesting_mode 1
2023-09-15T09:42:33.980 farmer chia.plot_sync.receiver    : INFO     reset: node_id bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a, current_sync: [state 0, sync_id 0, next_message_id 0, plots_processed 0, plots_total 0, delta [valid +0/-0, invalid +0/-0, keys missing: +0/-0, duplicates: +0/-0], time_done None]
2023-09-15T09:42:33.980 farmer farmer_server              : DEBUG    -> plot_sync_response to peer 192.168.1.9 bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a
2023-09-15T09:42:33.980 farmer farmer_server              : DEBUG    Time taken to process plot_sync_start from bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a is 0.0 seconds
2023-09-15T09:42:34.058 full_node full_node_server        : DEBUG    <- new_transaction from peer 9f814d87b8068d3028c7d51610ca7e335df48fffa0022342eb50e89c411d2d17 188.32.70.168
2023-09-15T09:42:34.058 full_node full_node_server        : DEBUG    -> request_transaction to peer 188.32.70.168 9f814d87b8068d3028c7d51610ca7e335df48fffa0022342eb50e89c411d2d17
2023-09-15T09:42:34.058 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 9f814d87b8068d3028c7d51610ca7e335df48fffa0022342eb50e89c411d2d17 is 0.0 seconds
2023-09-15T09:42:34.089 full_node full_node_server        : DEBUG    <- new_transaction from peer fabe4453be3c635bd925257316e2abe534085bec05ed9d94071d9d1286a32c78 46.55.162.39
2023-09-15T09:42:34.089 full_node full_node_server        : DEBUG    Time taken to process new_transaction from fabe4453be3c635bd925257316e2abe534085bec05ed9d94071d9d1286a32c78 is 0.0 seconds
2023-09-15T09:42:34.120 full_node full_node_server        : DEBUG    <- respond_transaction from peer 9f814d87b8068d3028c7d51610ca7e335df48fffa0022342eb50e89c411d2d17 188.32.70.168
2023-09-15T09:42:34.120 full_node chia.full_node.full_node: DEBUG    Processing transaction: 2d3bedb34df8e53716e4f11db2895d78107cb9a015f20097573ab1fe22ec1e7c
2023-09-15T09:42:34.120 full_node full_node_server        : DEBUG    Time taken to process respond_transaction from 9f814d87b8068d3028c7d51610ca7e335df48fffa0022342eb50e89c411d2d17 is 0.0 seconds
2023-09-15T09:42:34.152 full_node chia.full_node.mempool_manager: DEBUG    pre_validate_spendbundle took 0.0312 seconds for 2d3bedb34df8e53716e4f11db2895d78107cb9a015f20097573ab1fe22ec1e7c
2023-09-15T09:42:34.152 full_node chia.full_node.mempool_manager: DEBUG    Cost: 71708325
2023-09-15T09:42:34.167 full_node chia.full_node.mempool_manager: DEBUG    add_spendbundle 2d3bedb34df8e53716e4f11db2895d78107cb9a015f20097573ab1fe22ec1e7c took 0.02 seconds. Cost: 71708325 (0.652% of max block cost)
2023-09-15T09:42:34.167 full_node chia.full_node.full_node: DEBUG    Added transaction to mempool: 2d3bedb34df8e53716e4f11db2895d78107cb9a015f20097573ab1fe22ec1e7c mempool size: 399220810 normalized 79.844162
2023-09-15T09:42:34.167 full_node full_node_server        : DEBUG    -> new_transaction to peer 71.38.19.247 743b7b1cf9b3c4631e82eb34f2826ce8dd006e153fb549995643f92a58e35c5f
2023-09-15T09:42:34.167 full_node full_node_server        : DEBUG    -> new_transaction to peer 46.55.162.39 fabe4453be3c635bd925257316e2abe534085bec05ed9d94071d9d1286a32c78
2023-09-15T09:42:34.167 full_node full_node_server        : DEBUG    -> new_transaction to peer 92.255.225.145 8c7a6e2eafd710d4878d67e6ef954a2de272dcf62ca05503c0b3cdda15403c04
2023-09-15T09:42:34.167 full_node full_node_server        : DEBUG    -> new_transaction to peer 3.94.227.189 e5703102eb1db6fb2dfd6523662e220a02a4e0a083eacf8060b7731d99116037
2023-09-15T09:42:34.167 full_node full_node_server        : DEBUG    -> new_transaction to peer 63.250.54.166 1280d6c8db369cbcfb00027f4b7878d560c2ac7916201e8a9bec6a9a28e5edc0
2023-09-15T09:42:34.245 full_node full_node_server        : DEBUG    <- new_transaction from peer e5703102eb1db6fb2dfd6523662e220a02a4e0a083eacf8060b7731d99116037 3.94.227.189
2023-09-15T09:42:34.245 full_node full_node_server        : DEBUG    Time taken to process new_transaction from e5703102eb1db6fb2dfd6523662e220a02a4e0a083eacf8060b7731d99116037 is 0.0 seconds
2023-09-15T09:42:34.324 full_node full_node_server        : DEBUG    <- new_transaction from peer 743b7b1cf9b3c4631e82eb34f2826ce8dd006e153fb549995643f92a58e35c5f 71.38.19.247
2023-09-15T09:42:34.324 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 743b7b1cf9b3c4631e82eb34f2826ce8dd006e153fb549995643f92a58e35c5f is 0.0 seconds
2023-09-15T09:42:34.433 full_node chia.full_node.weight_proof: DEBUG    check db for sub epoch 145
2023-09-15T09:42:34.542 daemon __main__                   : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"command":"running_services","data":{},"origin":"wallet_ui","destination":"daemon","ack":false,"request_id":"68264aee0520ce4b89db82db29097e91a7a38da3e9931181782043b7788cab03"}', extra='')
2023-09-15T09:42:35.089 farmer farmer_server              : INFO     Connection closed: 192.168.1.9, node id: bcb9a8f405c72b7ccb2b0494251ebfd2f73fcf6183b5af65487acd1b4726d52a
2023-09-15T09:42:35.089 farmer chia.farmer.farmer         : INFO     peer disconnected PeerInfo(_ip=IPv4Address('192.168.1.9'), _port=8448)
2023-09-15T09:42:35.089 farmer farmer_server              : DEBUG    Inbound_handler task cancelled

So what I am seeing is my harvester connecting to the farmer but then also immediately disconnecting again. I am not sure why this is. Does anybody have an idea how to solve this?

2

What version chia are you running? In your harvester (192.168.1.9) you edited the config.yaml file to look like this
harvester:
chia_ssl_ca:
crt: config/ssl/ca/chia_ca.crt
key: config/ssl/ca/chia_ca.key
farmer_peer:
host: 192.168.1.10
port: 8447
.
.
.
And on the Harvester you run from the prompt
cd C:\Program Files\Chia\resources\app.asar.unpacked\daemon>
.\chia start -r harvester

3

One week of fiddling in and still havent figured it out. Chia version is 2.0.1 on all machines.

Config.yaml on 192.168.1.9 is as follows:

harvester:
  chia_ssl_ca:
    crt: config/ssl/ca/chia_ca.crt
    key: config/ssl/ca/chia_ca.key
  decompressor_thread_count: 0
  decompressor_timeout: 20
  disable_cpu_affinity: false
  enforce_gpu_index: false
  farmer_peer:
    host: 192.168.1.10
    port: 8447
  gpu_index: 0
  logging: *id001
  max_compression_level_allowed: 7
  network_overrides: *id002
  num_threads: 30
  parallel_decompressor_count: 0
  parallel_read: true
  plot_directories:
  - D:\PoolPlots
  - E:\PoolPlots
  - F:\PoolPlots
  - G:\PoolPlots
  - H:\PoolPlots
  - I:\PoolPlots
  plots_refresh_parameter:
    batch_size: 300
    batch_sleep_milliseconds: 1
    interval_seconds: 120
    retry_invalid_seconds: 1200
  port: 8448
  private_ssl_ca:
    crt: config/ssl/ca/private_ca.crt
    key: config/ssl/ca/private_ca.key
  recursive_plot_scan: false
  rpc_port: 8560
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/harvester/private_harvester.crt
    private_key: config/ssl/harvester/private_harvester.key
  start_rpc_server: true
  use_gpu_harvesting: false

Starting the harvester with:
C:\Users\Microserver N54L\AppData\Local\Programs\Chia\resources\app.asar.unpacked\daemon>chia.exe start harvester -r

When I was on 1.6.0 everything worked fine. I updated the main machine to V2 and it all went bad. To fix it I threw everything off my harvesters, including their configs, which might have helped in hindsight, to start fresh. Still no idea how to fix this.

4

check the log of the harvester, it looks like you have posted farmer’s log.

i could suggest to check if your 8447 port is open as for starters.

2023-09-26_11-42-19 - 8447 port
2023-09-26_11-42-19 - 8447 port671×871 58.8 KB

5

I have checked the harvester’s firewall and opened inbound/outbound ports on 8447.
Below is the log

2023-09-26T15:42:40.619 daemon filelock                   : DEBUG    Attempting to acquire lock 2197386096496 on C:\Users\Microserver N54L\.chia\mainnet\run\daemon.lock
2023-09-26T15:42:40.619 daemon filelock                   : DEBUG    Lock 2197386096496 acquired on C:\Users\Microserver N54L\.chia\mainnet\run\daemon.lock
2023-09-26T15:42:40.619 daemon chia.daemon.server         : INFO     chia-blockchain version: 2.0.1
2023-09-26T15:42:40.619 daemon filelock                   : DEBUG    Attempting to acquire lock 2197386481824 on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-09-26T15:42:40.619 daemon filelock                   : DEBUG    Lock 2197386481824 acquired on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-09-26T15:42:40.728 daemon filelock                   : DEBUG    Attempting to release lock 2197386481824 on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-09-26T15:42:40.728 daemon filelock                   : DEBUG    Lock 2197386481824 released on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-09-26T15:42:40.728 daemon chia.daemon.server         : INFO     Not implemented
2023-09-26T15:42:40.728 daemon chia.daemon.server         : INFO     Starting Daemon Server
2023-09-26T15:42:42.650 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "ping", "data": {}, "destination": "daemon", "origin": "client", "request_id": "c858145b081f241ae85a3f8a884d3ee30f800bc974e956fa1a5f07dd8632ac7a"}', extra='')
2023-09-26T15:42:42.650 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "is_keyring_locked", "data": {}, "destination": "daemon", "origin": "client", "request_id": "18170e98103714a9c25dc54fc1611d57c02fabb3f58b6ad6280aa19b6d9ecd72"}', extra='')
2023-09-26T15:42:42.650 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "is_running", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "client", "request_id": "13b70013b3c5a51cd9161c6b56472ef2d8b9a900c3c505717605a3c0cef27875"}', extra='')
2023-09-26T15:42:42.650 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "start_service", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "client", "request_id": "4b10d4ebcd8425680a3c3e09a824b9bf0286b0ba8c9268025a9c29c1fc6c5a6f"}', extra='')
2023-09-26T15:42:42.650 daemon chia.daemon.server         : DEBUG    Launching service ['C:\\Users\\Microserver N54L\\AppData\\Local\\Programs\\Chia\\resources\\app.asar.unpacked\\daemon/start_harvester.exe'] with CHIA_ROOT: C:\Users\Microserver N54L\.chia\mainnet
2023-09-26T15:42:42.681 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.CLOSE: 8>, data=1000, extra='')
2023-09-26T15:42:42.681 daemon chia.daemon.server         : INFO     ConnectionClosed. Closing websocket with ['Unknown']
6

this is no harvester’s log

7

where did you do that, farmer pc or harvester?

8

It is, this is what gets filled in the log file after chia start harvester -r.
I clear the log file. I start with harvester -r. After a couple of minutes of no activity I do stop all. The log above is what is in the log file at that point.

The harvester is the machine where I posted the config from above (192.168.1.9).

9

I have opened inbound/outbound TCP port 8447 on both the Harvester (192.168.1.9) and the Farmer pc (192.168.1.10).

10

yeah sorry looks like harvester log afterall :slight_smile:

so you saying logs are just not updating after the last line? well this is something beyond of guessing, maybe your software isnt in the path anymore

C:\\Users\\Microserver N54L\\AppData\\Local\\Programs\\Chia\\resources\\app.asar.unpacked\\daemon/start_harvester.exe

anyway, try to ask for help in discord’s channel of chia support

1 Like
11

It has been 3 months but I havent made any progress. Ive reinstalled the Farmer and the Harvester as to not have issues in config.yaml. I have also disabled Windows Defender on both machines for this test.

The behavior is still the same. The harvester connects and then disconnects.

2023-12-19T12:58:03.363 harvester harvester               : INFO     Starting service harvester ...
2023-12-19T12:58:03.363 harvester harvester               : INFO     chia-blockchain version: 2.1.3
2023-12-19T12:58:03.363 harvester harvester_server        : INFO     Service capabilities: [(1, '1'), (2, '1'), (3, '1')]
2023-12-19T12:58:03.378 harvester filelock                : DEBUG    Attempting to acquire lock 1626221716784 on C:\Users\Microserver N54L\.chia\mainnet\run\harvester.lock
2023-12-19T12:58:03.378 harvester filelock                : DEBUG    Lock 1626221716784 acquired on C:\Users\Microserver N54L\.chia\mainnet\run\harvester.lock
2023-12-19T12:58:03.378 harvester harvester               : INFO     Started harvester service on network_id: mainnet at port None
2023-12-19T12:58:03.394 harvester harvester               : INFO     Add resolved PeerInfo(_ip=IPv4Address('192.168.1.10'), _port=8447)
2023-12-19T12:58:03.394 harvester harvester_server        : DEBUG    Connecting: wss://192.168.1.10:8447/ws, Peer info: PeerInfo(_ip=IPv4Address('192.168.1.10'), _port=8447)
2023-12-19T12:58:03.410 harvester harvester_server        : DEBUG    -> handshake to peer 192.168.1.10 94c667aa348af08989df5d439918425379f1a4d552226c3d99c267c1bec759db
2023-12-19T12:58:03.410 harvester harvester_server        : INFO     Connected with farmer PeerInfo(_ip=IPv4Address('192.168.1.10'), _port=8447)
2023-12-19T12:58:03.425 harvester harvester_server        : DEBUG    <- harvester_handshake from peer 94c667aa348af08989df5d439918425379f1a4d552226c3d99c267c1bec759db 192.168.1.10
2023-12-19T12:58:03.425 harvester chia.plotting.cache     : DEBUG    Cache C:\Users\Microserver N54L\.chia\mainnet\cache\plot_manager_v2.dat not found
2023-12-19T12:58:03.425 harvester filelock                : DEBUG    Attempting to acquire lock 1626221651440 on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-12-19T12:58:03.425 harvester filelock                : DEBUG    Lock 1626221651440 acquired on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-12-19T12:58:03.457 harvester harvester_server        : DEBUG    Time taken to process harvester_handshake from 94c667aa348af08989df5d439918425379f1a4d552226c3d99c267c1bec759db is 0.03128504753112793 seconds
2023-12-19T12:58:03.472 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"ack": false, "command": "register_service", "data": {"service": "chia_harvester"}, "destination": "daemon", "origin": "chia_harvester", "request_id": "0c573ebd469b5a68c4d240376daeab9b1d6c64d16343192fd0caf2a1a69fccda"}', extra='')
2023-12-19T12:58:03.472 daemon chia.daemon.server         : INFO     Register service {'service': 'chia_harvester'}
2023-12-19T12:58:03.472 daemon chia.daemon.server         : INFO     registered for service chia_harvester
2023-12-19T12:58:03.472 daemon chia.daemon.server         : INFO     {'success': True}
2023-12-19T12:58:03.488 harvester chia.rpc.rpc_server     : DEBUG    Rpc call <- register_service
2023-12-19T12:58:03.535 harvester filelock                : DEBUG    Attempting to release lock 1626221651440 on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-12-19T12:58:03.550 harvester filelock                : DEBUG    Lock 1626221651440 released on C:\Users\Microserver N54L\.chia\mainnet\config\config.yaml.lock
2023-12-19T12:58:03.550 harvester chia.plotting.util      : DEBUG    get_filenames: 36 files found in D:\PoolPlots, recursive: False
2023-12-19T12:58:03.550 harvester chia.harvester.harvester: INFO     _plot_refresh_callback: event started, loaded 0, removed 0, processed 0, remaining 36, duration: 0.00 seconds, total plots: 0
2023-12-19T12:58:03.550 harvester chia.plot_sync.sender   : DEBUG    sync_start sync_id 0, next_message_id 0, messages 0: count 36, initial True
2023-12-19T12:58:03.550 harvester chia.plot_sync.sender   : DEBUG    sync_start 1702987083
2023-12-19T12:58:03.550 harvester chia.plotting.manager   : DEBUG    refresh_batch: 36 files in directories {WindowsPath('D:/PoolPlots')}
2023-12-19T12:58:03.566 harvester chia.plot_sync.sender   : DEBUG    _send_next_message send plot_sync_start: PlotSyncStart: identifier {'message_id': 0, 'sync_id': 1702987083, 'timestamp': 1702987083}, initial True, last_sync_id 0, plot_file_count 36, harvesting_mode 1
2023-12-19T12:58:03.566 harvester chia.plotting.manager   : DEBUG    process_file D:\PoolPlots\plot-k32-2021-07-23-21-03-0139de2aa9ebb36e823d2e34961b97f0682b540978860832d1e0d2a4d0f2d572.plot
2023-12-19T12:58:06.394 daemon chia.daemon.server         : DEBUG    Received message: WSMessage(type=<WSMsgType.CLOSED: 257>, data=None, extra=None)
2023-12-19T12:58:06.394 daemon chia.daemon.server         : INFO     Connection closed. Closing websocket with ['chia_harvester'].

The plot files were created on the farmer. Does anybody have an idea of what is happening in this log file?

12

As a reference for others, the issues were caused by this bug: