Chia Lost Sync Overnight and is No Longer Syncing

I have been running Chia plotters, harvesters, etc for a while. Recently, one of my new Windows plotters exhibited a behavior I am unsure how to deal with.

This new plotter is running Chia 1.2.11 on a Windows 10 Pro machine with Intel(R) Core™ i5-3570 CPU, 16GB of RAM and an SSD dedicated to the Chia DB.

When operating normally CPU, etc is all nominal with minimal usage.

HOWEVER, the computer lost sync approximately 2 days after syncing (farmed normally during the 2 days it was in sync). Then, it showed a red “not synced” on the GUI. After a few hours with no change, I restarted Chia (and then the machine) and in both cases opening it shows 0/1,325,361 as the “Status” and never increments. Furthermore, peers no longer appear to be connecting.

Any ideas what could cause this?

Please find the log below:

2021-12-24T15:44:06.869 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-15306' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T15:44:06.914 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:07.023 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:07.024 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T15:44:07.136 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:07.140 full_node chia.full_node.full_node: DEBUG    Address manager query count: 2. Query limit: 10
2021-12-24T15:44:07.245 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:07.246 full_node chia.full_node.full_node: DEBUG    Address manager query count: 3. Query limit: 10
2021-12-24T15:44:07.353 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:07.355 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '27.38.202.134', 'port': 8444}.
2021-12-24T15:44:07.356 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:07.358 full_node full_node_server        : DEBUG    Connecting: wss://27.38.202.134:8444/ws, Peer info: {'host': '27.38.202.134', 'port': 8444}
2021-12-24T15:44:07.416 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T15:44:07.432 full_node full_node_server        : INFO     Cannot connect to host 188.34.142.193:8444 ssl:<ssl.SSLContext object at 0x000001FE15BF61C0> [The semaphore timeout period has expired]
2021-12-24T15:44:07.619 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:07.729 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:07.730 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '122.233.198.154', 'port': 8444}.
2021-12-24T15:44:07.731 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:07.732 full_node full_node_server        : DEBUG    Connecting: wss://122.233.198.154:8444/ws, Peer info: {'host': '122.233.198.154', 'port': 8444}
2021-12-24T15:44:07.931 full_node full_node_server        : INFO     Cannot connect to host 94.230.9.194:8444 ssl:<ssl.SSLContext object at 0x000001FE2764B240> [The semaphore timeout period has expired]
2021-12-24T15:44:07.992 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:08.103 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 1.00e-03 seconds in tried table.
2021-12-24T15:44:08.105 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T15:44:08.211 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:08.212 full_node chia.full_node.full_node: DEBUG    Address manager query count: 2. Query limit: 10
2021-12-24T15:44:08.322 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:08.323 full_node chia.full_node.full_node: DEBUG    Address manager query count: 3. Query limit: 10
2021-12-24T15:44:08.433 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:08.434 full_node chia.full_node.full_node: DEBUG    Address manager query count: 4. Query limit: 10
2021-12-24T15:44:08.531 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T15:44:08.544 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:08.545 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '210.10.0.153', 'port': 8444}.
2021-12-24T15:44:08.545 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:08.547 full_node full_node_server        : DEBUG    Connecting: wss://210.10.0.153:8444/ws, Peer info: {'host': '210.10.0.153', 'port': 8444}
2021-12-24T15:44:08.812 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:08.889 full_node full_node_server        : INFO     Cannot connect to host 35.206.243.203:8444 ssl:<ssl.SSLContext object at 0x000001FE2808C5C0> [The semaphore timeout period has expired]
2021-12-24T15:44:08.920 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:08.921 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T15:44:09.029 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:09.030 full_node chia.full_node.full_node: DEBUG    Address manager query count: 2. Query limit: 10
2021-12-24T15:44:09.137 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:09.139 full_node chia.full_node.full_node: DEBUG    Address manager query count: 3. Query limit: 10
2021-12-24T15:44:09.248 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T15:44:09.249 full_node chia.full_node.full_node: DEBUG    Address manager query count: 4. Query limit: 10
2021-12-24T15:44:09.356 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:09.357 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '120.208.76.125', 'port': 8444}.
2021-12-24T15:44:09.358 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:09.359 full_node full_node_server        : DEBUG    Connecting: wss://120.208.76.125:8444/ws, Peer info: {'host': '120.208.76.125', 'port': 8444}
2021-12-24T15:44:09.546 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T15:44:09.623 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:09.732 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:09.733 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '109.194.162.229', 'port': 8444}.
2021-12-24T15:44:09.734 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:09.738 full_node full_node_server        : DEBUG    Connecting: wss://109.194.162.229:8444/ws, Peer info: {'host': '109.194.162.229', 'port': 8444}
2021-12-24T15:44:09.999 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:10.106 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:10.107 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '112.98.4.131', 'port': 8444}.
2021-12-24T15:44:10.108 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:10.109 full_node full_node_server        : DEBUG    Connecting: wss://112.98.4.131:8444/ws, Peer info: {'host': '112.98.4.131', 'port': 8444}
2021-12-24T15:44:10.372 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:10.480 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:10.481 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '86.82.183.56', 'port': 8444}.
2021-12-24T15:44:10.482 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:10.483 full_node full_node_server        : DEBUG    Connecting: wss://86.82.183.56:8444/ws, Peer info: {'host': '86.82.183.56', 'port': 8444}
2021-12-24T15:44:10.495 full_node full_node_server        : INFO     Cannot connect to host 113.101.122.216:8444 ssl:<ssl.SSLContext object at 0x000001FE169C9240> [The semaphore timeout period has expired]
2021-12-24T15:44:10.558 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T15:44:10.731 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T15:44:10.840 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:10.841 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '123.139.44.12', 'port': 8444}.
2021-12-24T15:44:10.842 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:10.843 full_node full_node_server        : INFO     Cannot connect to host 96.255.3.140:8444 ssl:<ssl.SSLContext object at 0x000001FE14436BC0> [The semaphore timeout period has expired]
2021-12-24T15:44:10.844 full_node full_node_server        : DEBUG    Connecting: wss://123.139.44.12:8444/ws, Peer info: {'host': '123.139.44.12', 'port': 8444}
2021-12-24T15:44:10.901 full_node full_node_server        : INFO     Cannot connect to host 122.233.198.154:8444 ssl:<ssl.SSLContext object at 0x000001FE29E668C0> [The remote computer refused the network connection]

If you have a copy of your base, kill the one you have and bring the save one online so you do not have to wait a couple of days. Also, lower your peers to 20 tops. It does make a difference when you get hosed. I was running .9 for weeks with no issues and I lost sync within 36 hours after .11 was installed. Very frustrating. Again, I really wish the dev team did a hard-core DVT before they go to beta.

Hi,
What do you mean by this exactly?
If this SSD only holds the db and the chia client itself is running from another disk, make sure the SSD is online (can you read from it, for instance copy the db files to another disk)?

Yes. The SSD that holds only the Chia DBs is fully accessible and all files are there. Furthermore, removing the existing 36GB ‘blockchain_v1_mainnet’ causes it to create a new one but it still does not sync.

Do you recognize any of the errors on the log?

I lowered it to 20 as part of my debugging earlier but did not seem to resolve it :confused:

You appear to be in a failed loop.

2021-12-24T15:44:10.840 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T15:44:10.841 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '123.139.44.12', 'port': 8444}.
2021-12-24T15:44:10.842 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T15:44:10.843 full_node full_node_server        : INFO     Cannot connect to host 96.255.3.140:8444 ssl:<ssl.SSLContext object at 0x000001FE14436BC0> [The semaphore timeout period has expired]
2021-12-24T15:44:10.844 full_node full_node_server        : DEBUG    Connecting: wss://123.139.44.12:8444/ws, Peer info: {'host': '123.139.44.12', 'port': 8444}
2021-12-24T15:44:10.901 full_node full_node_server        : INFO     Cannot connect to host 122.233.198.154:8444 ssl:<ssl.SSLContext object at 0x000001FE29E668C0> [The remote computer refused the network connection]
address_manager.select_peer took 0.00e+00 seconds
Addrman selected address: {'host': '123.139.44.12', 'port': 8444}
Cannot connect to host 122.233.198.154:8444 ssl:<ssl.SSLContext object at 0x000001FE29E668C0> [The remote computer refused the network connection]

So it is taking zero seconds to select a peer which seems wrong. The peer is selected in zero seconds and then always says it cannot connect because the proposed peer refuses connection.

I do not know what is causing this but the zero second selection is possibly the root of the problem. I’m sure you have already checked to make sure port 8444 is open.

I hope this helps lead you or the many other helpers here to find your problem and solution.

If that is purely a plotter, why do you have chia installed on it? Just download command line MadMax, and be done with it. Either use stotiks for Windows, or MadMax direct download for Linux (preferred).

If you do need a harvester on that machine (to harvest on the drive that is being plotted), then use chia command line: “chia start harvester -r” That will only run harvester on that box and will not need to sync.

Also, if you run UI on your harvesters, that is also no good. You should have just one full node at your location, and run command line harvesters only, pointing them to that one full node.

Here is the link to how to setup such thing: Farming on many machines · Chia-Network/chia-blockchain Wiki · GitHub

1 Like

This location has a single machine that is a full node + harvester. I don’t plot on this machine. I usually run Linux but for this specific location I had to run on Windows.

I ended up writing “plotter” due to force of habit. My bad.

Could you post a screenshot of your Connections from the Full Node panel? Sorry, didn’t notice that you have already mentioned that you don’t have any peers.

Actually, could you also post screenshots of your blockchain and wallet db folders.

Also, that CPU is kind of old. As Larry suggested, drop the number of peers down to 10 in config.yaml. This is to mainly overcome problems during dust storms.

Although, maybe those connection errors are just secondary errors that are masking the original problem. Could you shut down your chia, reboot, move all your logs to a subfolder, start your chia, and look for db problems? Here is maybe a relevant issue suggesting that db is busted:

Basically, chia is just a garbage code that runs while on critical errors, thus masking what is really happening.

1 Like

If for whatever reason a connection cannot be established (remote refuses such connection), then closing it in zero seconds is rather normal. So, I think that is not a problem. Actually, you would need to also find a pair of node connection and disconnection for the same IP, and check the time difference between those.

As far as port 8444 being open, that is also not needed. Either your node connects to peers (thus no need for an open port), or peers are connecting to you (port has to be open / port-forwarded).

A longer excerpt form the tail of the current log:

2021-12-24T20:03:51.548 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5627' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:51.550 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:51.551 full_node chia.full_node.full_node: DEBUG    Address manager query count: 2. Query limit: 10
2021-12-24T20:03:51.655 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:51.661 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '219.83.188.83', 'port': 8444}.
2021-12-24T20:03:51.662 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:51.664 full_node full_node_server        : DEBUG    Connecting: wss://219.83.188.83:8444/ws, Peer info: {'host': '219.83.188.83', 'port': 8444}
2021-12-24T20:03:51.934 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:52.011 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:52.043 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:52.044 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T20:03:52.139 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:52.142 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '2800:e2:5b80:949:5825:2da5:c552:415c', 'port': 8444}.
2021-12-24T20:03:52.143 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:52.145 full_node full_node_server        : DEBUG    Connecting: wss://[2800:e2:5b80:949:5825:2da5:c552:415c]:8444/ws, Peer info: {'host': '[2800:e2:5b80:949:5825:2da5:c552:415c]', 'port': 8444}
2021-12-24T20:03:52.146 full_node full_node_server        : INFO     Cannot connect to host 2800:e2:5b80:949:5825:2da5:c552:415c:8444 ssl:<ssl.SSLContext object at 0x0000021B87DEEEC0> [The network location cannot be reached. For information about network troubleshooting, see Windows Help]
2021-12-24T20:03:52.344 full_node full_node_server        : DEBUG    -> handshake to peer 219.83.188.83 04e7540270df4df4b5151f4b32025e8b98273b086f63f33d40743effbbcd4680
2021-12-24T20:03:52.405 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:52.514 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:52.515 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '119.74.139.141', 'port': 8444}.
2021-12-24T20:03:52.516 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:52.517 full_node full_node_server        : DEBUG    Connecting: wss://119.74.139.141:8444/ws, Peer info: {'host': '119.74.139.141', 'port': 8444}
2021-12-24T20:03:51.550 wallet chia.wallet.wallet_node    : INFO     self.sync_event.set()
2021-12-24T20:03:52.560 wallet chia.wallet.wallet_node    : INFO     Have collected 2 potential peaks
2021-12-24T20:03:52.560 wallet chia.wallet.wallet_node    : INFO     No peers to sync to
2021-12-24T20:03:52.562 wallet chia.wallet.wallet_node    : INFO     Loop end in sync job
2021-12-24T20:03:52.563 wallet chia.wallet.wallet_node    : INFO     Loop start in sync job
2021-12-24T20:03:52.566 wallet chia.rpc.rpc_server        : DEBUG    Rpc call <- get_sync_status
2021-12-24T20:03:52.567 wallet chia.rpc.rpc_server        : DEBUG    Rpc response -> get_sync_status
2021-12-24T20:03:52.568 wallet chia.rpc.rpc_server        : DEBUG    Rpc call <- get_sync_status
2021-12-24T20:03:52.569 wallet chia.rpc.rpc_server        : DEBUG    Rpc response -> get_sync_status
2021-12-24T20:03:53.026 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:53.543 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:53.545 wallet wallet_server              : DEBUG    Connecting: wss://127.0.0.1:8444/ws, Peer info: {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:52.561 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5638' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:53.576 full_node full_node_server        : INFO     Connection closed: 219.83.188.83, node id: 04e7540270df4df4b5151f4b32025e8b98273b086f63f33d40743effbbcd4680
2021-12-24T20:03:53.578 full_node full_node_server        : WARNING  Banning 219.83.188.83 for 10 seconds
2021-12-24T20:03:53.579 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '219.83.188.83', 'port': 8444}
2021-12-24T20:03:53.581 full_node full_node_server        : ERROR    Exception b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2', exception Stack: Traceback (most recent call last):
  File "chia\server\server.py", line 437, in start_client
  File "chia\server\server.py", line 335, in connection_added
  File "chia\full_node\full_node.py", line 619, in on_connect
  File "chia\full_node\full_node.py", line 585, in synced
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'

2021-12-24T20:03:53.583 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5641' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:53.585 full_node full_node_server        : INFO     Cannot connect to host 77.123.175.197:8444 ssl:<ssl.SSLContext object at 0x0000021B875384C0> [The remote computer refused the network connection]
2021-12-24T20:03:53.588 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:53.600 wallet wallet_server              : DEBUG    -> handshake to peer 127.0.0.1 04ad86314ae6992e6e92886e4d1dc38257dc306791da1d16961fc8a6c09d0e6c
2021-12-24T20:03:53.602 wallet wallet_server              : INFO     Connected with full_node {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:54.027 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:53.601 full_node full_node_server        : DEBUG    -> handshake to peer 127.0.0.1 a902566db59c04aa512906bbd451dc227f7e11e6b201c8b6d1dfd86df0a87571
2021-12-24T20:03:54.605 full_node chia.full_node.block_store: DEBUG    cache hit for block d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2
2021-12-24T20:03:54.608 wallet wallet_server              : DEBUG    Peer closed connection full_node 127.0.0.1:8444/8444
2021-12-24T20:03:54.610 wallet wallet_server              : DEBUG    Inbound_handler task cancelled
2021-12-24T20:03:54.611 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: 04ad86314ae6992e6e92886e4d1dc38257dc306791da1d16961fc8a6c09d0e6c
2021-12-24T20:03:55.041 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:54.608 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5648' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:55.620 full_node full_node_server        : INFO     Cannot connect to host 49.228.243.57:8444 ssl:<ssl.SSLContext object at 0x0000021B8DC70940> [The semaphore timeout period has expired]
2021-12-24T20:03:55.621 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:55.622 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T20:03:55.623 full_node full_node_server        : INFO     Connection closed: 127.0.0.1, node id: a902566db59c04aa512906bbd451dc227f7e11e6b201c8b6d1dfd86df0a87571
2021-12-24T20:03:55.626 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '127.0.0.1', 'port': 8449}
2021-12-24T20:03:55.628 full_node full_node_server        : ERROR    Exception <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>, exception Stack: Traceback (most recent call last):
  File "chia\server\server.py", line 292, in incoming_connection
  File "chia\server\server.py", line 335, in connection_added
  File "chia\full_node\full_node.py", line 631, in on_connect
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>

2021-12-24T20:03:55.629 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5651' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:55.727 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:55.728 full_node chia.full_node.full_node: DEBUG    Address manager query count: 2. Query limit: 10
2021-12-24T20:03:55.836 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:55.837 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '36.35.166.148', 'port': 8444}.
2021-12-24T20:03:55.839 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:55.842 full_node full_node_server        : DEBUG    Connecting: wss://36.35.166.148:8444/ws, Peer info: {'host': '36.35.166.148', 'port': 8444}
2021-12-24T20:03:56.056 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:56.088 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:56.196 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:56.197 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '66.240.197.187', 'port': 8444}.
2021-12-24T20:03:56.198 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:56.200 full_node full_node_server        : DEBUG    Connecting: wss://66.240.197.187:8444/ws, Peer info: {'host': '66.240.197.187', 'port': 8444}
2021-12-24T20:03:56.460 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:56.567 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:56.568 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T20:03:56.598 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:56.600 wallet wallet_server              : DEBUG    Connecting: wss://127.0.0.1:8444/ws, Peer info: {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:56.609 wallet wallet_server              : DEBUG    -> handshake to peer 127.0.0.1 04ad86314ae6992e6e92886e4d1dc38257dc306791da1d16961fc8a6c09d0e6c
2021-12-24T20:03:56.611 wallet wallet_server              : INFO     Connected with full_node {'host': '127.0.0.1', 'port': 8444}
2021-12-24T20:03:56.688 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\00\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.691 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\01\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.697 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\02\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.701 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\03\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.704 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\04\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.710 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\05\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.714 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\06\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.717 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\07\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.720 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\08\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.724 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\09\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.730 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\10\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.733 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\11\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.735 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\17\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.737 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\18\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.741 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\19\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.747 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\28\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.751 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\29\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.752 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\30\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.756 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\31\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.761 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\32\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.764 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\33\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.768 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\37\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.770 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\38\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.776 harvester chia.plotting.util      : DEBUG    Not checking subdirectory C:\Farm\39\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.778 harvester chia.plotting.util      : DEBUG    Not checking subdirectory E:\.Spotlight-V100, subdirectories not added by default
2021-12-24T20:03:56.779 harvester chia.plotting.util      : DEBUG    Not checking subdirectory E:\.fseventsd, subdirectories not added by default
2021-12-24T20:03:56.782 harvester chia.plotting.util      : DEBUG    Not checking subdirectory E:\System Volume Information, subdirectories not added by default
2021-12-24T20:03:56.782 harvester chia.plotting.util      : DEBUG    Not checking subdirectory E:\$RECYCLE.BIN, subdirectories not added by default
2021-12-24T20:03:56.799 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 774, duration: 0.00 seconds
2021-12-24T20:03:56.880 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {WindowsPath('C:/Farm/06'), WindowsPath('C:/Farm/31'), WindowsPath('C:/Farm/11'), WindowsPath('C:/Farm/01'), WindowsPath('C:/Farm/07'), WindowsPath('C:/Farm/04'), WindowsPath('C:/Farm/28'), WindowsPath('C:/Farm/29'), WindowsPath('C:/Farm/17'), WindowsPath('C:/Farm/08'), WindowsPath('C:/Farm/37'), WindowsPath('C:/Farm/33'), WindowsPath('C:/Farm/39'), WindowsPath('C:/Farm/02'), WindowsPath('C:/Farm/09'), WindowsPath('C:/Farm/10'), WindowsPath('C:/Farm/30'), WindowsPath('C:/Farm/03'), WindowsPath('C:/Farm/32'), WindowsPath('C:/Farm/38'), WindowsPath('C:/Farm/00'), WindowsPath('F:/plots'), WindowsPath('C:/Farm/05'), WindowsPath('E:/'), WindowsPath('C:/Farm/19'), WindowsPath('C:/Farm/18')}
2021-12-24T20:03:56.901 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.02 seconds
2021-12-24T20:03:56.902 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 474, duration: 0.02 seconds
2021-12-24T20:03:56.903 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2021-12-24T20:03:56.912 harvester chia.plotting.manager   : DEBUG    refresh_batch: 300 files in directories {WindowsPath('C:/Farm/06'), WindowsPath('C:/Farm/31'), WindowsPath('C:/Farm/11'), WindowsPath('C:/Farm/01'), WindowsPath('C:/Farm/07'), WindowsPath('C:/Farm/04'), WindowsPath('C:/Farm/28'), WindowsPath('C:/Farm/29'), WindowsPath('C:/Farm/17'), WindowsPath('C:/Farm/08'), WindowsPath('C:/Farm/37'), WindowsPath('C:/Farm/33'), WindowsPath('C:/Farm/39'), WindowsPath('C:/Farm/02'), WindowsPath('C:/Farm/09'), WindowsPath('C:/Farm/10'), WindowsPath('C:/Farm/30'), WindowsPath('C:/Farm/03'), WindowsPath('C:/Farm/32'), WindowsPath('C:/Farm/38'), WindowsPath('C:/Farm/00'), WindowsPath('F:/plots'), WindowsPath('C:/Farm/05'), WindowsPath('E:/'), WindowsPath('C:/Farm/19'), WindowsPath('C:/Farm/18')}
2021-12-24T20:03:56.960 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 300, remaining 0, batch_size 300, duration: 0.05 seconds
2021-12-24T20:03:56.961 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 174, duration: 0.05 seconds
2021-12-24T20:03:56.962 harvester chia.plotting.manager   : DEBUG    refresh_plots: Sleep 1 milliseconds
2021-12-24T20:03:56.974 harvester chia.plotting.manager   : DEBUG    refresh_batch: 174 files in directories {WindowsPath('C:/Farm/06'), WindowsPath('C:/Farm/31'), WindowsPath('C:/Farm/11'), WindowsPath('C:/Farm/01'), WindowsPath('C:/Farm/07'), WindowsPath('C:/Farm/04'), WindowsPath('C:/Farm/28'), WindowsPath('C:/Farm/29'), WindowsPath('C:/Farm/17'), WindowsPath('C:/Farm/08'), WindowsPath('C:/Farm/37'), WindowsPath('C:/Farm/33'), WindowsPath('C:/Farm/39'), WindowsPath('C:/Farm/02'), WindowsPath('C:/Farm/09'), WindowsPath('C:/Farm/10'), WindowsPath('C:/Farm/30'), WindowsPath('C:/Farm/03'), WindowsPath('C:/Farm/32'), WindowsPath('C:/Farm/38'), WindowsPath('C:/Farm/00'), WindowsPath('F:/plots'), WindowsPath('C:/Farm/05'), WindowsPath('E:/'), WindowsPath('C:/Farm/19'), WindowsPath('C:/Farm/18')}
2021-12-24T20:03:56.992 harvester chia.plotting.manager   : DEBUG    refresh_batch: loaded 0, removed 0, processed 174, remaining 0, batch_size 300, duration: 0.02 seconds
2021-12-24T20:03:56.997 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 174, remaining 0, duration: 0.02 seconds
2021-12-24T20:03:56.999 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 774, remaining 0, duration: 0.08 seconds
2021-12-24T20:03:57.001 harvester chia.plotting.manager   : DEBUG    _refresh_task: cached entries removed: 0
2021-12-24T20:03:57.001 harvester chia.plotting.manager   : DEBUG    _refresh_task: total_result.loaded 0, total_result.removed 0, total_duration 0.08 seconds
2021-12-24T20:03:57.068 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:57.567 wallet chia.wallet.wallet_node    : INFO     self.sync_event.set()
2021-12-24T20:03:57.568 wallet chia.wallet.wallet_node    : INFO     Have collected 2 potential peaks
2021-12-24T20:03:57.611 wallet chia.wallet.wallet_node    : INFO     Requesting blocks 0-32
2021-12-24T20:03:57.613 wallet chia.rpc.rpc_server        : DEBUG    Rpc call <- get_sync_status
2021-12-24T20:03:57.613 wallet chia.rpc.rpc_server        : DEBUG    Rpc response -> get_sync_status
2021-12-24T20:03:57.614 wallet wallet_server              : DEBUG    -> request_header_blocks to peer 127.0.0.1 04ad86314ae6992e6e92886e4d1dc38257dc306791da1d16961fc8a6c09d0e6c
2021-12-24T20:03:58.083 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:56.610 full_node full_node_server        : DEBUG    -> handshake to peer 127.0.0.1 a902566db59c04aa512906bbd451dc227f7e11e6b201c8b6d1dfd86df0a87571
2021-12-24T20:03:58.618 full_node chia.full_node.block_store: DEBUG    cache hit for block d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2
2021-12-24T20:03:58.619 wallet wallet_server              : DEBUG    Peer closed connection full_node 127.0.0.1:8444/8444
2021-12-24T20:03:58.619 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5662' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:58.621 full_node full_node_server        : INFO     Connection closed: 127.0.0.1, node id: a902566db59c04aa512906bbd451dc227f7e11e6b201c8b6d1dfd86df0a87571
2021-12-24T20:03:58.622 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '127.0.0.1', 'port': 8449}
2021-12-24T20:03:58.624 full_node full_node_server        : ERROR    Exception <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>, exception Stack: Traceback (most recent call last):
  File "chia\server\server.py", line 292, in incoming_connection
  File "chia\server\server.py", line 335, in connection_added
  File "chia\full_node\full_node.py", line 631, in on_connect
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>

2021-12-24T20:03:58.625 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:58.627 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '94.130.28.181', 'port': 8444}.
2021-12-24T20:03:58.628 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:58.630 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-5665' coro=<RpcServer._state_changed() done, defined at chia\rpc\rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia\rpc\rpc_server.py", line 51, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 61, in _state_changed
  File "chia\rpc\full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia\consensus\blockchain.py", line 157, in get_peak
  File "chia\consensus\blockchain.py", line 674, in height_to_block_record
  File "chia\consensus\blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2021-12-24T20:03:58.632 full_node full_node_server        : DEBUG    Connecting: wss://94.130.28.181:8444/ws, Peer info: {'host': '94.130.28.181', 'port': 8444}
2021-12-24T20:03:58.899 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:59.009 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:59.010 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T20:03:59.089 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-24T20:03:59.120 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:59.121 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '120.231.220.164', 'port': 8444}.
2021-12-24T20:03:59.122 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:59.125 full_node full_node_server        : DEBUG    Connecting: wss://120.231.220.164:8444/ws, Peer info: {'host': '120.231.220.164', 'port': 8444}
2021-12-24T20:03:59.372 full_node chia.full_node.full_node: DEBUG    Address manager query count: 0. Query limit: 10
2021-12-24T20:03:59.481 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in tried table.
2021-12-24T20:03:59.482 full_node chia.full_node.full_node: DEBUG    Address manager query count: 1. Query limit: 10
2021-12-24T20:03:59.497 full_node full_node_server        : INFO     Cannot connect to host 118.163.174.139:8444 ssl:<ssl.SSLContext object at 0x0000021B976AEA40> [The semaphore timeout period has expired]
2021-12-24T20:03:59.593 full_node chia.server.address_manager: DEBUG    address_manager.select_peer took 0.00e+00 seconds in new table.
2021-12-24T20:03:59.595 full_node chia.full_node.full_node: DEBUG    Addrman selected address: {'host': '176.74.217.177', 'port': 8444}.
2021-12-24T20:03:59.596 full_node chia.full_node.full_node: DEBUG    Num peers needed: 8
2021-12-24T20:03:59.597 full_node full_node_server        : DEBUG    Connecting: wss://176.74.217.177:8444/ws, Peer info: {'host': '176.74.217.177', 'port': 8444}
2021-12-24T20:03:58.620 wallet wallet_server              : DEBUG    Inbound_handler task cancelled
2021-12-24T20:03:59.625 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: 04ad86314ae6992e6e92886e4d1dc38257dc306791da1d16961fc8a6c09d0e6c
2021-12-24T20:03:59.626 wallet wallet_server              : DEBUG    Time for request request_header_blocks: {'host': '127.0.0.1', 'port': 8444} = 2.0139691829681396, None? True
2021-12-24T20:03:59.628 wallet chia.wallet.wallet_node    : ERROR    Error while trying to fetch from peer:Peer returned no response Traceback (most recent call last):
  File "chia\wallet\wallet_node.py", line 563, in batch_sync_to_peak
  File "chia\wallet\wallet_node.py", line 695, in fetch_blocks_and_validate
ValueError: Peer returned no response

I noticed that the Wallet service seems intermittent. It keeps appearing and disappearing from the connections list.


s

I have been trying some different things. In the newest attempt, I started anew with all the DB files except for the one of the mainnet since that one takes 2 days to sync here.

(I can only add 1 image per post since I am new user so I can’t add screenshots of the folders here but it looks like it usually does)

I did, however, confirm that if I start with all files from scratch (instead of keeping the blockchain_v1_mainnet.sqlite) the node starts to sync and everything connects as expected. I suspect, the issue is related to a corrupt DB which will likely force me to resync :frowning:

Yeah, I am not that good at parsing logs. I usually tend to ignore those - too much worthless data, too little results, unless there are some obvious things there.

If your node starts syncing when you remove both dbs, that would imply that blockchain db is busted. Usually, when wallet db is busted, you will have just spinners (UI needs wallet part operational). Potentially, when you have some data there, it implies blockchain db corruption.

Again, a quick test to see whether there is a db error at start would be to follow what I wrote there.

Although, maybe there is no point to struggle with this, as everything points to a broken blockchain db. One more option to get you up and running quickly (one hour ?) is to download blockchain db from www.chia-database.com. Yeah, I understand that that db may be poisoned, but on my book this problem should be addressed by chia, not by us that suffer days/weeks long syncing (e.g., legitimize that guy as it was outlined in another thread - whatever he is doing, he is offering a free download bandwidth).

1 Like

Thank you for your help! Hopefully after it syncs up the problem does not happen again :confused:

Chia doesn’t shut down cleanly, and sometimes those processes that handle either db are clinging to life up to reboot time. This is where db corruption comes into place. Always monitor all chia tasks before rebooting or restarting chia. Potentially during the dust storms when node goes out of whack, main db could be busted (looks like you could be hit by the latest one). To prevent that from happening on your box, change those peers in config.yaml.

Just download that db from that website. There is really no point to waste several days on syncing, where the next dust storm may screw you up again (looks like yesterday there was another short script test, or rather two short ones, potentially as a preparation to something more serious).

1 Like

The current Chia network is impossible to keep up even for me who has been running the farm for months, let alone who’s trying to sync from zero.

And yes, I always opened port 8444.

My main farm also runs on super fast internet.

I used to farm even on my 1-bar mobile internet just fine. But it’s out of the question now with the current dust storm situation.

My question is… do the devs actually have any ability/knowledge to fix the issue? Or it’s the flaw within the code that can’t be fixed unless they change the core to the point that requires us to replot (again)?

Oh, I already moved my farmer from my Raspberry Pi4 to my plotter (5950X + SSD). So far, it has had no effect at all.

Again, all those things are really irrelevant. You can run with port 8444 closed - no problem at all If you check the syncing speed, it is not bandwidth bound, but rather CPU / single core and db handling limited. If an average sync takes about 2 days, it implies that chia goes with full speed syncing by downloading roughly at 2Mbps - basically nothing.

During those dust storms, the bandwidth increase was rather negligible. On the other hand, the code was choking one single CPU core, plus debugging logs running with a millisecond speed were hindering blockchain db access. Both are entry level coding errors.

I guess, the question is whether there is any interest to fix it. From chia’s perspective, as long as 50% of nodes survive those dust storms (transactions are not hindered), all is fine - nodes like yours are just a collateral damage. As long as you drink the same cool aide and keep upgrading your rig, why bother doing anything.

You should still be able to run your RPi as long as: 1. you limit the number of peers to around 10, 2. your run your dbs from an SSD, potentially you create a small RAM drive and put your debug.logs there. Contrary to what people are saying, RPi 4 has plenty of CPU power, but it lacks on drive bandwidth.

Your 5950x box is just an overkill.

And, I would not dismiss the Flex farmer option.

What version of Chia are you running. They did tweet if you are not running .11 you will never catch up. Just another way to cover for jacked code, but nevertheless, see what .11 brings you. As Jacek has mentioned, move all your db’s and see what happens. This works most of the time. The DB’s get tweaked and then we have to fix it. Fun time…I say that in jest as well. I have been in your shoes and it hurts.

1 Like

I am running version 1.2.11.

After leaving the node to sync for almost 3 days it ended up where I started. With a red “Not Synced” warning that does not go away. :frowning:

I also noticed that the issue occurred at the same height it had happened previously. 1,326,776 of December 24th. Did anything special happen that day?

I did not use any of the previous DB files but instead let it sync from scratch.

An excerpt from the tail of the log file:

2021-12-28T11:09:35.631 full_node full_node_server        : DEBUG    <- new_signage_point_or_end_of_sub_slot from peer 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb 58.169.68.40
2021-12-28T11:09:35.633 full_node full_node_server        : DEBUG    Time taken to process new_signage_point_or_end_of_sub_slot from 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb is 0.0009918212890625 seconds
2021-12-28T11:09:35.634 full_node full_node_server        : DEBUG    -> request_signage_point_or_end_of_sub_slot to peer 58.169.68.40 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb
2021-12-28T11:09:35.662 full_node full_node_server        : DEBUG    <- respond_end_of_sub_slot from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:35.664 full_node full_node_server        : DEBUG    Time taken to process respond_end_of_sub_slot from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.0019996166229248047 seconds
2021-12-28T11:09:35.665 full_node full_node_server        : DEBUG    -> request_signage_point_or_end_of_sub_slot to peer 61.172.21.28 c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1
2021-12-28T11:09:35.698 full_node full_node_server        : DEBUG    <- respond_end_of_sub_slot from peer 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 76.190.16.194
2021-12-28T11:09:35.699 full_node full_node_server        : DEBUG    Time taken to process respond_end_of_sub_slot from 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 is 0.0009987354278564453 seconds
2021-12-28T11:09:35.701 full_node full_node_server        : DEBUG    -> request_signage_point_or_end_of_sub_slot to peer 76.190.16.194 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560
2021-12-28T11:09:35.770 full_node full_node_server        : DEBUG    <- new_transaction from peer 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 76.190.16.194
2021-12-28T11:09:35.772 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 is 0.002003192901611328 seconds
2021-12-28T11:09:35.910 full_node full_node_server        : DEBUG    <- respond_end_of_sub_slot from peer 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb 58.169.68.40
2021-12-28T11:09:35.912 full_node full_node_server        : DEBUG    Time taken to process respond_end_of_sub_slot from 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb is 0.0019989013671875 seconds
2021-12-28T11:09:35.913 full_node full_node_server        : DEBUG    -> request_signage_point_or_end_of_sub_slot to peer 58.169.68.40 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb
2021-12-28T11:09:36.122 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:36.615 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:36.616 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0xfb2c332ae36d6bff0737d25bfe0f886ff8f61c5370e39efb9fbf462b8d4d7a6c',
 'height': 1132895,
 'vdf_info': {'challenge': '0xb03e0c9deefaa30ac32294e2d05993c0e13b99cf7e36952fddccfe468f096152',
              'number_of_iterations': 31036296,
              'output': {'data': '0x01007166ce68253efa1f2ead08aa55fd412adcb90c4c0bb8ed5e51d8e6d1213c16ed71acdec638a10e527af7c1057f84ef47da364b6b3ea0db61f8b174332ac2dc500838b7d40203b9d8745021368ed6286710d874401d60742098af11132c50e6470100'}}}, already requested
2021-12-28T11:09:36.617 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.0020008087158203125 seconds
2021-12-28T11:09:36.618 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer cd914402d45a2aeb9491595a53895a4649b317b067557d437c376055fd6c5347 69.131.99.223
2021-12-28T11:09:36.711 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer cd914402d45a2aeb9491595a53895a4649b317b067557d437c376055fd6c5347 69.131.99.223
2021-12-28T11:09:36.791 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer cd914402d45a2aeb9491595a53895a4649b317b067557d437c376055fd6c5347 69.131.99.223
2021-12-28T11:09:36.878 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer cd914402d45a2aeb9491595a53895a4649b317b067557d437c376055fd6c5347 69.131.99.223
2021-12-28T11:09:36.927 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x56ae0b9eb1e01bd81d1257c9cfac5965a1b54c0883d86dad4c545c8bf5be5605',
 'height': 1057907,
 'vdf_info': {'challenge': '0xe1bddf472fce7f905f4fd50f827ea43bd2ecac8b8fb5534cb17af6f40c2f779d',
              'number_of_iterations': 31019862,
              'output': {'data': '0x02003691b58e7565639530f4400380e4a364eb8d33d1118bbb49468631206c50ddb0e0a87a1e1def8cd7eaf62bcdda6c7570a13970faaa51207a068bf5fbccbadd15f9c0740ef25af8b894a9caa95325d12917e698b6aa4116f184963e8e3a73c90b0100'}}}, already requested
2021-12-28T11:09:36.927 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from cd914402d45a2aeb9491595a53895a4649b317b067557d437c376055fd6c5347 is 0.04874920845031738 seconds
2021-12-28T11:09:37.130 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:38.129 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:38.144 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 69.139.48.189
2021-12-28T11:09:38.178 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 6c03b8a1d1ddb9e4123a1ffbeff2edb4554e0bb9798bd8f781faf053c294f09a 109.194.51.14
2021-12-28T11:09:38.535 full_node full_node_server        : DEBUG    <- new_transaction from peer 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c 88.206.52.134
2021-12-28T11:09:38.537 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c is 0.0019958019256591797 seconds
2021-12-28T11:09:38.602 full_node full_node_server        : DEBUG    <- new_transaction from peer eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 69.139.48.189
2021-12-28T11:09:38.604 full_node full_node_server        : DEBUG    Time taken to process new_transaction from eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 is 0.0019931793212890625 seconds
2021-12-28T11:09:38.611 full_node full_node_server        : DEBUG    <- new_transaction from peer 6c03b8a1d1ddb9e4123a1ffbeff2edb4554e0bb9798bd8f781faf053c294f09a 109.194.51.14
2021-12-28T11:09:38.612 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 6c03b8a1d1ddb9e4123a1ffbeff2edb4554e0bb9798bd8f781faf053c294f09a is 0.0009946823120117188 seconds
2021-12-28T11:09:38.664 full_node full_node_server        : DEBUG    <- new_transaction from peer 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 80.110.34.46
2021-12-28T11:09:38.665 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 is 0.0009989738464355469 seconds
2021-12-28T11:09:38.684 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 76.190.16.194
2021-12-28T11:09:38.686 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0xe3fe47c09b3b3aea7e180986184bda03cd9d233ac27115cdf9c15f2da0c174f8',
 'height': 393413,
 'vdf_info': {'challenge': '0xc9c8322e600e78e276d65442f68e8ab8580bf7e4636a74ae3abb85a4461255bb',
              'number_of_iterations': 23748608,
              'output': {'data': '0x0100bc04b4cf44125396f424d99aa1158bc62175364e5d033a7992263ad2cf5923af03461b84c57b4c4889ea4ff2ba6e41ae66acec9ca9a5075b82422a00cf559d18176b9afbad10648a9ecd5a35deb0cf42ddc2367a1f0f5825b31f863238c7d00d0401'}}}, _waiters
2021-12-28T11:09:38.687 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 is 0.0029990673065185547 seconds
2021-12-28T11:09:38.714 full_node full_node_server        : DEBUG    <- new_transaction from peer 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 45.90.3.75
2021-12-28T11:09:38.715 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 is 0.0009992122650146484 seconds
2021-12-28T11:09:38.736 full_node full_node_server        : DEBUG    <- new_transaction from peer 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 76.190.16.194
2021-12-28T11:09:38.738 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 is 0.0020003318786621094 seconds
2021-12-28T11:09:38.760 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb 58.169.68.40
2021-12-28T11:09:38.761 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x56ae0b9eb1e01bd81d1257c9cfac5965a1b54c0883d86dad4c545c8bf5be5605',
 'height': 1057907,
 'vdf_info': {'challenge': '0xe1bddf472fce7f905f4fd50f827ea43bd2ecac8b8fb5534cb17af6f40c2f779d',
              'number_of_iterations': 31019862,
              'output': {'data': '0x02003691b58e7565639530f4400380e4a364eb8d33d1118bbb49468631206c50ddb0e0a87a1e1def8cd7eaf62bcdda6c7570a13970faaa51207a068bf5fbccbadd15f9c0740ef25af8b894a9caa95325d12917e698b6aa4116f184963e8e3a73c90b0100'}}}, _waiters
2021-12-28T11:09:38.762 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb is 0.0019989013671875 seconds
2021-12-28T11:09:38.927 full_node full_node_server        : DEBUG    <- new_transaction from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:38.928 full_node full_node_server        : DEBUG    Time taken to process new_transaction from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.0009999275207519531 seconds
2021-12-28T11:09:38.982 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:38.984 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0xfee8fc09a120bf879049668581f66de1a06f2899d5bbfc3a87b4598786b838b2',
 'height': 1342953,
 'vdf_info': {'challenge': '0x83ec7d93738e113abdce4b45be99c4a78860619fff30746719717812eb301f9f',
              'number_of_iterations': 6291456,
              'output': {'data': '0x0100a4b55c7fea44842eec500fdeb1f0ffacfc8fb005cd09001e090cc1c733735442322a1e3dabb325a711aaa1a1f1ddbaf9c2e66e14e69d80a4e31598dfc9cf70009702fbd1f73e9ac67fe1cfc7926cfb1b86af105a05b7c1a616e699463152f8011106'}}}, _waiters
2021-12-28T11:09:38.985 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.0029964447021484375 seconds
2021-12-28T11:09:39.133 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:39.222 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:39.223 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0x60307ff9c55ebd9eb0d0dcf14b5b879c6ff20211b3d333b6d1ec2228d16587eb',
 'height': 1314201,
 'vdf_info': {'challenge': '0x8d6d2de506b3dba05fcc4fc924f2fd21e80eeeed7a061a1fbe2ee2bfc3adf175',
              'number_of_iterations': 23068672,
              'output': {'data': '0x01001e38eb889882daf901ad52b3adeb057a7d9918032f72263e42e84377b7b98ace4bf73cb88868007c036dd42d0d4b3ada7d567da983461557d64ba83caa68a52565053e6b556e881588b6c8c568539eab1d6f4dc97996678f53d3067c4fe0b80a0100'}}}, _waiters
2021-12-28T11:09:39.224 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.002000093460083008 seconds
2021-12-28T11:09:39.225 full_node full_node_server        : DEBUG    <- new_transaction from peer 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb 58.169.68.40
2021-12-28T11:09:39.226 full_node full_node_server        : DEBUG    Time taken to process new_transaction from 47da4ae906645392168886cc575f40d1fd2fa097e9ea17a4f1559a9d7d8645cb is 0.0010056495666503906 seconds
2021-12-28T11:09:39.488 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 6c03b8a1d1ddb9e4123a1ffbeff2edb4554e0bb9798bd8f781faf053c294f09a 109.194.51.14
2021-12-28T11:09:39.490 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x66b469a04d6aa0f4ca80bf44ef4f8b5e18991c3513b1d899ed0bc0126e1f3846',
 'height': 1318544,
 'vdf_info': {'challenge': '0xb2093ab677d3fcf2e22647b90196fc6bc396508f086625026936af54e3e7402b',
              'number_of_iterations': 95487734,
              'output': {'data': '0x0000583545469d7b86a59ed735fba54f3766a3376fe1ba9332017dfa1bb59d57ba411eb5e266f0645b0ec8941d30d8718313d3c7a34cc26111473775e27e93dbae0b49f7c38e3da4ef39721a7bd4da2a737d97b3f372290287c2eb708a30f50b12180403'}}}, _waiters
2021-12-28T11:09:39.491 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 6c03b8a1d1ddb9e4123a1ffbeff2edb4554e0bb9798bd8f781faf053c294f09a is 0.0029947757720947266 seconds
2021-12-28T11:09:39.843 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 331755adfcb7349a4c8e5015c65037298d50030093389b4f1b0643c1fc30e9db 87.133.177.186
2021-12-28T11:09:39.844 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0xd8c09d9a0a7e5e6c81797b7df0a3633897ef500e0962638597d9ca4dcd67ba3d',
 'height': 624549,
 'vdf_info': {'challenge': '0x902d6d54386916515090ca732c67adee48fad97d684d19e4d9e1bc5e2c4d7307',
              'number_of_iterations': 67043328,
              'output': {'data': '0x0200952eea0dc90a81e4d691d729f767f016742780d606a5d7fdfad6eb2af0108084a6d8178f23fe458d8e45e50174a530405d640f46a4a4409323176d5d0adcdd472a75e3d5e2dcd9fe3e333006e44bb5b03b00a826362dc48b0a9b56719f8f57010100'}}}, _waiters
2021-12-28T11:09:39.845 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 331755adfcb7349a4c8e5015c65037298d50030093389b4f1b0643c1fc30e9db is 0.0019998550415039062 seconds
2021-12-28T11:09:40.131 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 61.172.21.28
2021-12-28T11:09:40.133 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x3d63c4a9440398ae2edaac4cba0b218a971e4a641675d250a165ac10c6a89852',
 'height': 1131553,
 'vdf_info': {'challenge': '0xec942f0f3240a871cc55d02f92265fdb8d833952a5e61913bce3ee094cb8bab6',
              'number_of_iterations': 46413309,
              'output': {'data': '0x02005c92f29d837a487367c8fd05b8d902cd64715f715bfa7b789ed4ed918ca37df941e9480febfc6caf3cb193ff1df633ad0215c03ba1ee2bf97e810d3f5cfa0850bf50eff37b1580389dd59ef84b8eb0adc29fc88227b0168a0c206d284552b9610100'}}}, _waiters
2021-12-28T11:09:40.134 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from c6b9ec2043d872f100f84630c3038fd45232c764563f93f164f9eac6b47b5ec1 is 0.0029969215393066406 seconds
2021-12-28T11:09:40.139 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:40.267 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 76.190.16.194
2021-12-28T11:09:40.268 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0xfb2c332ae36d6bff0737d25bfe0f886ff8f61c5370e39efb9fbf462b8d4d7a6c',
 'height': 1132895,
 'vdf_info': {'challenge': '0xb03e0c9deefaa30ac32294e2d05993c0e13b99cf7e36952fddccfe468f096152',
              'number_of_iterations': 31036296,
              'output': {'data': '0x01007166ce68253efa1f2ead08aa55fd412adcb90c4c0bb8ed5e51d8e6d1213c16ed71acdec638a10e527af7c1057f84ef47da364b6b3ea0db61f8b174332ac2dc500838b7d40203b9d8745021368ed6286710d874401d60742098af11132c50e6470100'}}}, _waiters
2021-12-28T11:09:40.269 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 405b7ebcae3ff86fe2dd36c631badf9fe71078cf1bc8663e5014da055046b560 is 0.001997709274291992 seconds
2021-12-28T11:09:40.351 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 69.139.48.189
2021-12-28T11:09:40.352 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x9bf02ae3cc3df90d9af63f0adcc4f521d74773297f0e4f743b4305d4170d784d',
 'height': 1054259,
 'vdf_info': {'challenge': '0x599fb76c016804231186bd499e7ab08b850b5d8b7d0c8eea10d9948417e67fe5',
              'number_of_iterations': 46015257,
              'output': {'data': '0x02000920fdf1601710058903a3af3b2e0a7d2e29e6e5f6bd5c815fb16d68d18f324b7a1a3f189391de3b92734998400d18551fc2c9ab42a9399e3749acdfd15cd48152c10b41af84df02e5eb8c36fc8caa13ac60ed1544c36e26aa5032cb2e52c70a0100'}}}, _waiters
2021-12-28T11:09:40.353 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 is 0.0019981861114501953 seconds
2021-12-28T11:09:40.380 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 45.90.3.75
2021-12-28T11:09:40.381 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x80e4c309926e0e9039c9dd029b8c1c846b16f1d9511c7c20933e84457835c7e7',
 'height': 696902,
 'vdf_info': {'challenge': '0xdff78539d0f4a359750eb43330bc0d7a78552d575ad07ffea42a6e96776fbebc',
              'number_of_iterations': 128404815,
              'output': {'data': '0x0200d6af0c6851dd6af3b8c4721cef4adaf97257a22e3b7861af31e374d45a38cef479f07f436eb9c387c60a59cb6437e78771af509960e0c297447cea3ff5f52e25eb0ea1bc070767cba2f249bd78920e0b14da97b348641196066cc2c75107384c0100'}}}, _waiters
2021-12-28T11:09:40.382 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 is 0.0019979476928710938 seconds
2021-12-28T11:09:40.528 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 80.110.34.46
2021-12-28T11:09:40.530 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x9bf02ae3cc3df90d9af63f0adcc4f521d74773297f0e4f743b4305d4170d784d',
 'height': 1054259,
 'vdf_info': {'challenge': '0x599fb76c016804231186bd499e7ab08b850b5d8b7d0c8eea10d9948417e67fe5',
              'number_of_iterations': 46015257,
              'output': {'data': '0x02000920fdf1601710058903a3af3b2e0a7d2e29e6e5f6bd5c815fb16d68d18f324b7a1a3f189391de3b92734998400d18551fc2c9ab42a9399e3749acdfd15cd48152c10b41af84df02e5eb8c36fc8caa13ac60ed1544c36e26aa5032cb2e52c70a0100'}}}, _waiters
2021-12-28T11:09:40.530 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 is 0.0019991397857666016 seconds
2021-12-28T11:09:40.648 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c 88.206.52.134
2021-12-28T11:09:40.650 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x9bf02ae3cc3df90d9af63f0adcc4f521d74773297f0e4f743b4305d4170d784d',
 'height': 1054259,
 'vdf_info': {'challenge': '0x599fb76c016804231186bd499e7ab08b850b5d8b7d0c8eea10d9948417e67fe5',
              'number_of_iterations': 46015257,
              'output': {'data': '0x02000920fdf1601710058903a3af3b2e0a7d2e29e6e5f6bd5c815fb16d68d18f324b7a1a3f189391de3b92734998400d18551fc2c9ab42a9399e3749acdfd15cd48152c10b41af84df02e5eb8c36fc8caa13ac60ed1544c36e26aa5032cb2e52c70a0100'}}}, _waiters
2021-12-28T11:09:40.651 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c is 0.00299835205078125 seconds
2021-12-28T11:09:40.861 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 69.139.48.189
2021-12-28T11:09:40.862 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x41974fbd55c47770df6e327f39fd361d247590ac3ccf3cd0aaef76cafbde129b',
 'height': 1089719,
 'vdf_info': {'challenge': '0xb83764a150054be656ea1fcaa342dc1aa5d38af68a4fc374dbc5488fe8c992a3',
              'number_of_iterations': 132376644,
              'output': {'data': '0x02004f2d7997656cc30b72c76c96cc826e28116f311d3aaa7cc4cc374aefe0961f29ea199fa7425b405795e10a1497d7f235d9c327258b9e6127f90bf99197ec77008f0fa617ab5f0dc2a4a87c9d513719cc81bb5c50c556b62353326996641368009527'}}}, _waiters
2021-12-28T11:09:40.863 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from eecdc19e40bd5940bbe019d889876a41830c948ea01016dbbb44ac3ddb0e9521 is 0.001999378204345703 seconds
2021-12-28T11:09:41.123 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 80.110.34.46
2021-12-28T11:09:41.125 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0xd8c09d9a0a7e5e6c81797b7df0a3633897ef500e0962638597d9ca4dcd67ba3d',
 'height': 624549,
 'vdf_info': {'challenge': '0x902d6d54386916515090ca732c67adee48fad97d684d19e4d9e1bc5e2c4d7307',
              'number_of_iterations': 67043328,
              'output': {'data': '0x0200952eea0dc90a81e4d691d729f767f016742780d606a5d7fdfad6eb2af0108084a6d8178f23fe458d8e45e50174a530405d640f46a4a4409323176d5d0adcdd472a75e3d5e2dcd9fe3e333006e44bb5b03b00a826362dc48b0a9b56719f8f57010100'}}}, _waiters
2021-12-28T11:09:41.126 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 5ae75d8cd6fa880592f2d5921e6ece5a0e2ab3c5cacb6872f77e8918ae5a6603 is 0.00299835205078125 seconds
2021-12-28T11:09:41.145 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c 88.206.52.134
2021-12-28T11:09:41.146 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 4,
 'header_hash': '0x66b469a04d6aa0f4ca80bf44ef4f8b5e18991c3513b1d899ed0bc0126e1f3846',
 'height': 1318544,
 'vdf_info': {'challenge': '0xb2093ab677d3fcf2e22647b90196fc6bc396508f086625026936af54e3e7402b',
              'number_of_iterations': 95487734,
              'output': {'data': '0x0000583545469d7b86a59ed735fba54f3766a3376fe1ba9332017dfa1bb59d57ba411eb5e266f0645b0ec8941d30d8718313d3c7a34cc26111473775e27e93dbae0b49f7c38e3da4ef39721a7bd4da2a737d97b3f372290287c2eb708a30f50b12180403'}}}, _waiters
2021-12-28T11:09:41.147 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 26ba77c0c949b14525c69fad3d43745d7af613431ae8d9fc0cba3b54e711bd8c is 0.00299835205078125 seconds
2021-12-28T11:09:41.163 farmer chia.farmer.farmer         : DEBUG    update_cached_harvesters cache entries: 1
2021-12-28T11:09:41.192 full_node full_node_server        : DEBUG    <- new_compact_vdf from peer 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 45.90.3.75
2021-12-28T11:09:41.193 full_node chia.full_node.full_node: DEBUG    Ignoring NewCompactVDF: {'field_vdf': 3,
 'header_hash': '0x4e16dfd6b33ac4dce20664634496035d5b9db3eb5c5c6d08c66e48f7dda7083a',
 'height': 984503,
 'vdf_info': {'challenge': '0xe34d54a3511aa2835ff4d7b673f335e254211acaedb86dc1e270c6352494a111',
              'number_of_iterations': 116244480,
              'output': {'data': '0x00005dc3950563ccbfa9d8a5839539e9af1e3e8bbc6449cf080acd54b58e7cad756983f7bffc4838e574cd5aaf6c1e411ca9e40a26fc7720c73e1044c133074e6358b10ae95fbf6788294ab6ab238a0484f35caa66e2d59074a4df5b669a4c5f035c0100'}}}, _waiters
2021-12-28T11:09:41.194 full_node full_node_server        : DEBUG    Time taken to process new_compact_vdf from 619da964dcbac717420b1fe22585db08f46b4d27b63a1887e2c7d2fae98e1681 is 0.0019981861114501953 seconds