Full Node Sync stuck at 0/1443422 while connection status is connected

Full Node Sync stuck at 0/1443422 while connection status is connected. MiB Up/Down 0/0

Here is log

2022-01-19T15:04:45.718 daemon __main__                   : INFO     Starting Daemon Server
2022-01-19T15:04:45.719 daemon __main__                   : INFO     Not implemented
2022-01-19T15:04:45.726 daemon __main__                   : INFO     Waiting Daemon WebSocketServer closure
2022-01-19T15:04:46.720 daemon __main__                   : INFO     Register service {'service': 'wallet_ui'}
2022-01-19T15:04:46.721 daemon __main__                   : INFO     registered for service wallet_ui
2022-01-19T15:04:46.722 daemon __main__                   : INFO     {'success': True}
2022-01-19T15:04:46.723 daemon __main__                   : INFO     Register service {'service': 'chia_plotter'}
2022-01-19T15:04:46.724 daemon __main__                   : INFO     registered for service chia_plotter
2022-01-19T15:04:46.725 daemon __main__                   : INFO     {'success': True, 'service': 'chia_plotter', 'queue': []}
2022-01-19T15:04:48.095 harvester harvester               : INFO     Started harvester service on network_id: mainnet
2022-01-19T15:04:48.096 harvester harvester               : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8447}
2022-01-19T15:04:48.117 daemon __main__                   : INFO     Register service {'service': 'chia_harvester'}
2022-01-19T15:04:48.119 daemon __main__                   : INFO     registered for service chia_harvester
2022-01-19T15:04:48.120 daemon __main__                   : INFO     {'success': True}
2022-01-19T15:04:48.253 full_node chia.full_node.full_node: INFO     Default port field not found in config.
2022-01-19T15:04:48.267 full_node chia.full_node.full_node: INFO     Initializing blockchain from disk
2022-01-19T15:04:48.269 full_node chia.consensus.blockchain: INFO     Started 6 processes for block validation
2022-01-19T15:04:49.452 farmer farmer_server              : INFO     Started listening on port: 8447
2022-01-19T15:04:49.453 farmer farmer                     : INFO     Started farmer service on network_id: mainnet
2022-01-19T15:04:49.454 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2022-01-19T15:04:49.726 farmer chia.farmer.farmer         : INFO     Added pool: {'authentication_public_key': SOME KEYS HERE
2022-01-19T15:04:49.753 harvester harvester_server        : INFO     Connected with farmer {'host': '127.0.0.1', 'port': 8447}
2022-01-19T15:04:49.761 harvester chia.plotting.manager   : INFO     Loaded 804 bytes of cached data
2022-01-19T15:04:49.778 wallet chia.wallet.wallet_blockchain: INFO     Started 6 processes for block validation
2022-01-19T15:04:49.817 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 7, duration: 0.00 seconds
2022-01-19T15:04:49.829 harvester chia.plotting.manager   : INFO     Found plot F:\
2022-01-19T15:04:49.869 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 7, removed 0, processed 7, remaining 0, duration: 0.05 seconds
2022-01-19T15:04:49.871 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 7, removed 0, processed 7, remaining 0, duration: 0.05 seconds
2022-01-19T15:04:50.747 farmer chia.farmer.farmer         : INFO     GET /pool_info response: {'name': 'Space Pool', 'logo_url': 'https://pool.space/logo-250.png', 'minimum_difficulty': 1, 'relative_lock_height': 64, 'protocol_version': 1, 'fee': 0.01, 'description': 'Welcome to Space Pool. By creating this Plot NFT, you agree to the following terms https://pool.space/terms', 'target_puzzle_hash': '0x2f2c9ba1b2315d413a92b5f034fa03282ccba1767fd9ae7b14d942b969ed5d57', 'authentication_token_timeout': 5}
2022-01-19T15:04:49.754 daemon __main__                   : INFO     Register service {'service': 'chia_farmer'}
2022-01-19T15:04:50.762 daemon __main__                   : INFO     registered for service chia_farmer
2022-01-19T15:04:50.763 daemon __main__                   : INFO     {'success': True}
2022-01-19T15:04:49.842 wallet wallet                     : INFO     Started wallet service on network_id: mainnet
2022-01-19T15:04:50.858 wallet chia.wallet.wallet_node    : INFO     Loop start in sync job
2022-01-19T15:04:50.859 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2022-01-19T15:04:50.875 daemon __main__                   : INFO     Register service {'service': 'chia_wallet'}
2022-01-19T15:04:50.876 daemon __main__                   : INFO     registered for service chia_wallet
2022-01-19T15:04:50.877 daemon __main__                   : INFO     {'success': True}
2022-01-19T15:04:51.395 farmer chia.farmer.farmer         : INFO     GET /farmer response: {'authentication_public_key': SOME KEYS HERE
2022-01-19T15:04:51.440 farmer chia.farmer.farmer         : INFO     Added pool: {'authentication_public_key':SOME KEYS HERE
2022-01-19T15:04:51.492 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x00000279FB41CAC0> [The remote computer refused the network connection]
2022-01-19T15:04:52.074 farmer chia.farmer.farmer         : INFO     GET /pool_info response: {'name': 'Space Pool', 'logo_url': 'https://pool.space/logo-250.png', 'minimum_difficulty': 1, 'relative_lock_height': 64, 'protocol_version': 1, 'fee': 0.01, 'description': 'Welcome to Space Pool. By creating this Plot NFT, you agree to the following terms https://pool.space/terms', 'target_puzzle_hash': '0x2f2c9ba1b2315d413a92b5f034fa03282ccba1767fd9ae7b14d942b969ed5d57', 'authentication_token_timeout': 5}
2022-01-19T15:04:52.708 farmer chia.farmer.farmer         : INFO     GET /farmer response: {'authentication_public_key': SOME KEYS HERE
2022-01-19T15:04:52.902 wallet wallet_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001A9A4CE6DC0> [The remote computer refused the network connection]
2022-01-19T15:04:53.152 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://pool.poolsar.io, Cannot connect to host pool.poolsar.io:443 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'pool.poolsar.io'. (_ssl.c:1129)")]
2022-01-19T15:04:53.153 farmer chia.farmer.farmer         : WARNING  No pool specific authentication_token_timeout has been set for 3028c6285d621238b967c41cb978420552ddb63d30211d1dc8ea26cf5b590d08, check communication with the pool.
2022-01-19T15:04:53.213 farmer chia.farmer.farmer         : INFO     Added pool: {'authentication_public_key': SOME KEYS HERE
2022-01-19T15:04:54.480 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2022-01-19T15:04:55.914 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2022-01-19T15:04:56.521 farmer farmer_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x00000279FB4A5340> [The remote computer refused the network connection]
2022-01-19T15:04:57.958 wallet wallet_server              : INFO     Cannot connect to host 127.0.0.1:8444 ssl:<ssl.SSLContext object at 0x000001A9A4CE6F40> [The remote computer refused the network connection]
2022-01-19T15:05:53.123 wallet wallet_server              : INFO     Connected with introducer {'host': 'introducer.chia.net', 'port': 8444}
2022-01-19T15:05:53.527 wallet wallet_server              : INFO     Connection closed: introducer.chia.net, node id: 05eb2f6954fa1694d4f162cd69c0a11372b8dbf1660028439363717a749c8e00


2022-01-19T15:06:35.090 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '95.70.200.175', 'port': 8444}
2022-01-19T15:06:35.595 full_node chia.full_node.full_node: INFO     Starting to perform sync.
2022-01-19T15:06:35.596 full_node chia.full_node.full_node: INFO     Waiting to receive peaks from peers.
2022-01-19T15:06:42.090 full_node full_node_server        : INFO     Connected with full_node {'host': '75.142.137.186', 'port': 8444}
2022-01-19T15:06:42.144 full_node full_node_server        : INFO     Connection closed: 75.142.137.186, node id: a9edd98b4365465bd5fe74c13eb2fa15566deb5e671894260ce88253c18c74f5
2022-01-19T15:06:42.145 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '75.142.137.186', 'port': 8444}
2022-01-19T15:06:42.171 wallet wallet_server              : INFO     Connected with introducer {'host': 'introducer.chia.net', 'port': 8444}
2022-01-19T15:06:42.359 wallet chia.wallet.wallet_node    : INFO     Will not attempt to connect to other nodes, already connected to {'host': 'localhost', 'port': 8444}
2022-01-19T15:06:42.360 wallet chia.wallet.wallet_node    : INFO     Closing unnecessary connection to {'host': '116.23.83.55', 'port': 8444}.
2022-01-19T15:06:42.361 wallet chia.wallet.wallet_node    : INFO     Closing unnecessary connection to {'host': '123.253.86.74', 'port': 8444}.
2022-01-19T15:06:42.549 wallet wallet_server              : INFO     Connection closed: introducer.chia.net, node id: 5742bd481497c40adbb88d4db4ca1e7bde2aaf97c656a373e75dd1824b788763
2022-01-19T15:06:42.677 full_node chia.full_node.full_node: INFO     Collected a total of 3 peaks.
2022-01-19T15:06:42.678 full_node chia.full_node.full_node: INFO     Selected peak 1443422, 3d08d8b816a147da6151a6cc60a96674818f981907cd40ed225e8d596ff4b057
2022-01-19T15:06:42.747 wallet wallet_server              : INFO     Connection closed: 123.253.86.74, node id: aaae3da4a5cc0d1e9861f208f77465e2c8e89cc3d71b248a62a39a8474392352
2022-01-19T15:06:42.758 wallet wallet_server              : INFO     Connection closed: 116.23.83.55, node id: fab87c0a2898b6d198a86eb44f7334606450839c160f4de514983d66e4af6a00
2022-01-19T15:06:43.731 full_node chia.full_node.full_node: INFO     Total of 3 peers with peak 1443422
2022-01-19T15:06:43.732 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 46.155.154.89 up to height 1443422
2022-01-19T15:06:44.125 full_node full_node_server        : INFO     Cannot connect to host 219.83.188.20:8444 ssl:<ssl.SSLContext object at 0x0000020B87749840> [The remote computer refused the network connection]
2022-01-19T15:06:45.555 full_node full_node_server        : INFO     Cannot connect to host 2a02:1748:dd5e:6390:d559:44ba:8448:8699:8444 ssl:<ssl.SSLContext object at 0x0000020B853FF540> [The network location cannot be reached. For information about network troubleshooting, see Windows Help]
2022-01-19T15:06:46.272 full_node full_node_server        : INFO     Cannot connect to host 2a04:ee41:4:c1f2:9327:569e:cd58:bee9:8444 ssl:<ssl.SSLContext object at 0x0000020B853FFE40> [The network location cannot be reached. For information about network troubleshooting, see Windows Help]
2022-01-19T15:06:49.132 full_node full_node_server        : INFO     Cannot connect to host 91.154.121.149:8444 ssl:<ssl.SSLContext object at 0x0000020B87749EC0> [The semaphore timeout period has expired]
2022-01-19T15:06:50.669 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 7, duration: 0.00 seconds
2022-01-19T15:06:50.673 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 7, remaining 0, duration: 0.00 seconds
2022-01-19T15:06:50.674 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 7, remaining 0, duration: 0.00 seconds
2022-01-19T15:06:50.895 wallet chia.wallet.wallet_node    : INFO     Will not attempt to connect to other nodes, already connected to {'host': 'localhost', 'port': 8444}
2022-01-19T15:06:51.322 full_node full_node_server        : INFO     Connected with full_node {'host': '86.49.115.236', 'port': 8444}
2022-01-19T15:06:51.326 full_node full_node_server        : INFO     Connection closed: 86.49.115.236, node id: 05ced5f06947f8212d859eeb480412f15f302a776325ed51018951ca54e54e7a
2022-01-19T15:06:51.328 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '86.49.115.236', 'port': 8444}
2022-01-19T15:06:53.309 full_node full_node_server        : INFO     Connected with full_node {'host': '68.8.106.49', 'port': 8444}

Database corrupted possibly.
Unexpected shutdown?
What client version are you on?

2 Likes

latest v1.2.11 is installed.
Question for knowledge: By log viewing, How we can determine that it may be db corruption issue ?

I didn’t get that from viewing logs.
The db is on your pc, if it could be read it shouldn’t show 0/1443422.
From my personal exp, every time ive seen that it’s been my db corrupted.

1 Like

How do you correct a corrupted db?

1 Like

You don’t correct it, rather you delete it, and let chia sync (few days) thus generating a new one or download a recent snapshot - CHIA DATABASE DOWNLOAD - updated every 24 hours

1 Like

That worked! Interesting that the db got corrupted on my 2nd machine today while I was fixing the other one… :confused:

The most likely reason to get db corrupted is to stop chia and either reboot or restart it without checking whether all processes stopped. Unfortunately, it looks like sometimes they never stop, so it is just a pure luck to end up with either a clean or corrupted db.

If you think your reason to get it corrupted was different, maybe you can try to describe how it happened. It may help others to be aware of it, or maybe someone can figure out how to prevent it in such cases.

Exactly same problem here. On a Debian-based Raspberry PI 4 farmer.
My sequence:

  • Farming worked fine for weeks of being online
  • Suddenly farmer went offline (no specific event I could determine, e.g., power loss or so)
  • Restarts of the chia farming service or restarts of the machine didn’t help → During chia service startup the system ran out of memory within minutes (4GB RAM and 1GB SWAP).
  • I played with parameters, such as swap size (increased to 2GB), number of connections I accept (config.yaml), but no effect.
  • Only full resync / using the online DB fixed the issue.

Not asking for help. Posting this for statistical reasons. This seems to be no singular problem.

1 Like

On RPi, if your blockchain/wallet dbs are on SD card, you may want to move them to one of your HDs (those holding plots).

By the way, maybe you can also let people know that downloading that db helped you - CHIA DATABASE DOWNLOAD - updated every 24 hours

1 Like

DB is on external SSD (USB3.0), swap on the same SSD. Plots are all separate on HDDs (USB3.0).
And yes, thanks for pointing this out: new DB solved the issue instantly.

Latest client. Thanks. Yes it was db issue

Can we correct corrupted db instead of deleting it and sync again and again?

Not that I’m aware of, never seen it mentioned.

1 Like

Running the latest 1.2.11 on Linux. This same issue happened to me yesterday. Everything was running (full node, farmer, wallet), no unexpected shutdowns, the GUI was running but showed the height of 0, lots of connections, but no sync. It looks like some networking events precede the problem.

Possibly related, a few times over the last 1-2 months, I’ve come back to see syncing stopped. It shows connected, with many node connections, but no new blocks sync. A simple restart fixes. Before these losses of sync happen, there are also some suspect network traffic logged by Chia, similar issues with clients, banning, and timeouts.

The host is colocated in a datacenter and monitoring in place confirms no network connectivity issues around the time, and host CPU, memory, disk, etc are all normal.

Here are the logs from yesterday’s event:

2022-01-27T07:55:44.386 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '151.242.215.194', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:44.387 full_node full_node_server        : WARNING  Banning 151.242.215.194 for 10 seconds
2022-01-27T07:55:44.678 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '120.229.79.236', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:44.679 full_node full_node_server        : WARNING  Banning 120.229.79.236 for 10 seconds
2022-01-27T07:55:45.775 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '61.154.29.83', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:45.776 full_node full_node_server        : WARNING  Banning 61.154.29.83 for 10 seconds
2022-01-27T07:55:47.578 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '110.52.83.133', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:47.579 full_node full_node_server        : WARNING  Banning 110.52.83.133 for 10 seconds
2022-01-27T07:55:47.665 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '171.114.88.203', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:47.666 full_node full_node_server        : WARNING  Banning 171.114.88.203 for 10 seconds
2022-01-27T07:55:47.698 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '124.224.52.182', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:47.699 full_node full_node_server        : WARNING  Banning 124.224.52.182 for 10 seconds
2022-01-27T07:55:49.545 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '211.97.123.3', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:49.546 full_node full_node_server        : WARNING  Banning 211.97.123.3 for 10 seconds
2022-01-27T07:55:50.121 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '94.226.188.165', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:50.126 full_node full_node_server        : WARNING  Banning 94.226.188.165 for 10 seconds
2022-01-27T07:55:50.670 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '101.108.21.219', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:50.671 full_node full_node_server        : WARNING  Banning 101.108.21.219 for 10 seconds
2022-01-27T07:55:50.683 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '151.210.147.67', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:50.683 full_node full_node_server        : WARNING  Banning 151.210.147.67 for 10 seconds
2022-01-27T07:55:51.352 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '93.176.179.163', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:51.353 full_node full_node_server        : WARNING  Banning 93.176.179.163 for 10 seconds
2022-01-27T07:55:51.755 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '180.150.114.47', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:51.756 full_node full_node_server        : WARNING  Banning 180.150.114.47 for 10 seconds
2022-01-27T07:55:52.736 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '61.154.29.83', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:52.737 full_node full_node_server        : WARNING  Banning 61.154.29.83 for 10 seconds
2022-01-27T07:55:56.399 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '113.25.9.59', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:56.400 full_node full_node_server        : WARNING  Banning 113.25.9.59 for 10 seconds
2022-01-27T07:55:57.097 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '83.50.188.83', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:57.275 full_node full_node_server        : WARNING  Banning 83.50.188.83 for 10 seconds
2022-01-27T07:55:58.764 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '116.136.147.54', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:55:58.765 full_node full_node_server        : WARNING  Banning 116.136.147.54 for 10 seconds
2022-01-27T07:56:15.831 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '88.103.224.142', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:56:15.835 full_node full_node_server        : WARNING  Banning 88.103.224.142 for 10 seconds
2022-01-27T07:56:16.365 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '95.70.154.200', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:56:16.366 full_node full_node_server        : WARNING  Banning 95.70.154.200 for 10 seconds
2022-01-27T07:56:24.739 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '218.187.64.7', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:56:24.740 full_node full_node_server        : WARNING  Banning 218.187.64.7 for 10 seconds
2022-01-27T07:56:43.301 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '208.103.38.141', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:56:43.365 full_node full_node_server        : WARNING  Banning 208.103.38.141 for 10 seconds
2022-01-27T07:57:26.641 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '175.144.10.234', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T07:57:26.642 full_node full_node_server        : WARNING  Banning 175.144.10.234 for 10 seconds
2022-01-27T08:07:56.988 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '117.65.153.144', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T08:07:56.989 full_node full_node_server        : WARNING  Banning 117.65.153.144 for 10 seconds
2022-01-27T08:09:30.518 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '222.132.252.83', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T08:09:30.519 full_node full_node_server        : WARNING  Banning 222.132.252.83 for 10 seconds
2022-01-27T08:15:59.410 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '50.104.73.246', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T08:15:59.473 full_node full_node_server        : WARNING  Banning 50.104.73.246 for 10 seconds
2022-01-27T08:18:32.986 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '49.174.23.217', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T08:18:32.987 full_node full_node_server        : WARNING  Banning 49.174.23.217 for 10 seconds
2022-01-27T08:51:45.310 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '115.164.63.247', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T08:51:45.311 full_node full_node_server        : WARNING  Banning 115.164.63.247 for 10 seconds
2022-01-27T09:03:53.008 full_node full_node_server        : ERROR    Exception:  <class 'asyncio.exceptions.TimeoutError'>, closing connection {'host': '45.67.96.144', 'port': 8444}. Traceback (most recent call last):
  File "/home/chai/chia-blockchain/chia/server/server.py", line 576, in api_call
    response: Optional[Message] = await asyncio.wait_for(wrapped_coroutine(), timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

2022-01-27T09:03:53.034 full_node full_node_server        : WARNING  Banning 45.67.96.144 for 10 seconds
2022-01-27T10:50:22.962 full_node full_node_server        : WARNING  Banning 5.142.45.134 for 10 seconds
2022-01-27T10:50:22.963 full_node full_node_server        : WARNING  Invalid handshake with peer. Maybe the peer is running old software.
2022-01-27T11:50:07.723 full_node full_node_server        : WARNING  Banning 151.243.137.230 for 10 seconds
2022-01-27T11:50:07.724 full_node full_node_server        : WARNING  Invalid handshake with peer. Maybe the peer is running old software.
2022-01-27T16:03:16.257 full_node full_node_server        : WARNING  Cannot write to closing transport 169.0.136.77
2022-01-27T16:15:40.648 full_node full_node_server        : ERROR    Peer has been rate limited and will be disconnected: 116.30.100.161, message: new_compact_vdf
2022-01-27T16:15:40.649 full_node full_node_server        : WARNING  Banning 116.30.100.161 for 300 seconds

######
# Tried restarting full node here to recover, tons of exceptions and problems follow due to the corrupted chain file.
######

2022-01-27T20:28:51.958 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: UNIQUE constraint failed: coin_record.coin_name
2022-01-27T20:29:05.121 full_node asyncio                 : ERROR    Task was destroyed but it is pending!
task: <Task pending name='Task-539' coro=<FullNode.sync_from_fork_point.<locals>.fetch_block_batches() running at /home/chai/chia-blockchain/chia/full_node/full_node.py:893> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fecfdb6e220>()]> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:769]>

Happening to too many if us to be anything but a bug imo, hopefully gets sorted soon as its a pita.

Definitely a bug if a few network timeouts cause Chia’s entire networking stack to completely fall over on itself, break sync, and otherwise leave the Chia client totally oblivious to the fact that it’s in a failed state.

I have to say, the corresponding log entries with each sync event make me a bit worried that there’s a way to DoS a node over the Chia port. If network sync breaks over a few simple connection timeouts, that doesn’t instill much confidence in the rest of the system (shrug).

SQLite for a blockchain database, way to go!