Sync stops with "Not Synced" and shows nodes with SB Height 0

[aphex]

I had some issues with upgrading my full node to 1.2.x and so I ended up deleting my local blockchain db and starting a full resync. This has taken over a week and keeps getting stuck.

When I check chia show -s -c I see many connected nodes and usually it says “Syncing 55xxx/60xxx” and I can just wait to see the height increase over time.

Occasionally it will get stuck and show:

Current Blockchain Status: Not Synced. Peak height: 557751
      Time: Sun Jul 11 2021 18:08:16 UTC                  Height:     557751

The other thing I see when this happens is that every single connected node shows “SB Height” of 0:

557751                                                  -SB Height:        0    -Hash:  Info...
FULL_NODE 172.xxx.xxx.xxx                          8444/8444  d72225af... Jul 22 11:59:26      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...
FULL_NODE 164.xxx.xxx.xxx                           8444/8444  0bc3187e... Jul 22 11:59:21      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...
FULL_NODE 172.xxx.xxx.xxx                             47403/8444  ccf6821c... Jul 22 11:59:27      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...
FULL_NODE 59.xxx.xxx.xxx                           8444/8444  231d2173... Jul 22 11:59:26      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...
FULL_NODE 172.xxx.xxx.xxx                             47520/8444  4ccc909c... Jul 22 11:59:26      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...
FULL_NODE 81.xxx.xxx.xxx                           8444/8444  e6e50da0... Jul 22 11:59:24      0.0|0.0
                                                 -SB Height:        0    -Hash:  Info...

There are many more like this.

(when things are syncing properly these heights all show real values)

The logs look like this:

2021-07-22T15:08:00.688 full_node chia.full_node.full_node: INFO     peer disconnected None
2021-07-22T15:08:17.660 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: beb844b39fdce3ff82a4d7c5b3026f80de71cdbe7111235dcdd11be7da56229b
2021-07-22T15:08:17.882 full_node chia.full_node.full_node: INFO     peer disconnected None
2021-07-22T15:09:35.299 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: 33be50f4e437952ae0f42a89640bb883a625ffddd08da16578711d409050e046
2021-07-22T15:09:35.437 full_node chia.full_node.full_node: INFO     peer disconnected None
2021-07-22T15:10:52.659 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '172.17.0.1', 'port': 8444}.Inbound limit reached.
2021-07-22T15:10:52.919 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: 926aafb0054ed479cc30db61ffb667f1d666c7fe3f0bc260aa9ee4b5202b26c4
2021-07-22T15:10:52.920 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '172.17.0.1', 'port': 8444}
2021-07-22T15:12:33.178 full_node chia.full_node.full_node: INFO     Duplicate compact proof. Height: 156215. Header hash: 7eb3cebcd2bbff7b371b09f1f024638aac5a323b060c29cf03f09bb579a46581.
2021-07-22T15:12:58.826 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '172.17.0.1', 'port': 8444}.Inbound limit reached.
2021-07-22T15:12:59.086 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: c8623ce4e15fb58733ea5f89d6ad881de6ef3f80fd9a52f8218f70a60ab71919
2021-07-22T15:12:59.088 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '172.17.0.1', 'port': 8444}
2021-07-22T15:13:00.068 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '172.17.0.1', 'port': 8444}.Inbound limit reached.
2021-07-22T15:13:00.311 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: 0445797de3d275f06a8450261ca99901eb4fe697da129555cf9ceedbaf7e8d97
2021-07-22T15:13:00.313 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '172.17.0.1', 'port': 8444}
2021-07-22T15:13:43.605 full_node full_node_server        : INFO     Not accepting inbound connection: {'host': '172.17.0.1', 'port': 8444}.Inbound limit reached.
2021-07-22T15:13:43.860 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: fc8b820892b96340b3d3180209ab4533960417e4df2dec3781c939de14b97474
2021-07-22T15:13:43.862 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '172.17.0.1', 'port': 8444}
2021-07-22T15:13:46.093 full_node chia.full_node.full_node: INFO     Duplicate compact proof. Height: 156381. Header hash: d08069857c156c19e41d230215647f04cd0eb45d4223897c7d331363ac154b5c.
2021-07-22T15:13:58.300 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: e7a12af94bcf1d737fb794295420b5d90389ede364982c38d8617b73b2ebc322
2021-07-22T15:13:58.321 full_node chia.full_node.full_node: INFO     peer disconnected None
2021-07-22T15:14:42.137 full_node full_node_server        : INFO     Connection closed: 172.17.0.1, node id: 8605cac47b974be63a1849bbd53991dbccf1579f6ca50c06d20f4e083091e40c
2021-07-22T15:14:42.192 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '172.17.0.1', 'port': 8444}

This seems normal to me as I’m already connected to many peers, so no more can be added.

If I look at just errors, there is a few cases of this:

2021-07-22T14:16:22.779 full_node full_node_server        : ERROR    Exception Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x7f4a9542ea60> at 172.17.0.1, exception Stack: Traceback (most recent call last):
  File "/chia-blockchain/chia/server/server.py", line 232, in incoming_connection
    connection = WSChiaConnection(
  File "/chia-blockchain/chia/server/ws_connection.py", line 59, in __init__
    raise ValueError(f"Was not able to get peername from {self.ws_witer} at {self.peer_host}")
ValueError: Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x7f4a9542ea60> at 172.17.0.1

2021-07-22T14:16:29.734 full_node chia.full_node.full_node: INFO     Duplicate compact proof. Height: 140365. Header hash: fff3d609901506838da566cb63da14f1e441b3d5bca9c54ba6ebc3b4a734b5e9.
2021-07-22T14:17:43.005 full_node chia.full_node.full_node: INFO     Duplicate compact proof. Height: 157238. Header hash: 0b3637d58bdbaefbe338292c47a2fa7f686419601b70b68223e62e8dad6497a9.
2021-07-22T14:17:43.227 full_node chia.full_node.full_node: INFO     Duplicate compact proof. Height: 157238. Header hash: 0b3637d58bdbaefbe338292c47a2fa7f686419601b70b68223e62e8dad6497a9.
2021-07-22T14:18:48.330 full_node full_node_server        : ERROR    Exception Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x7f4a950738b0> at 172.17.0.1, exception Stack: Traceback (most recent call last):
  File "/chia-blockchain/chia/server/server.py", line 232, in incoming_connection
    connection = WSChiaConnection(
  File "/chia-blockchain/chia/server/ws_connection.py", line 59, in __init__
    raise ValueError(f"Was not able to get peername from {self.ws_witer} at {self.peer_host}")
ValueError: Was not able to get peername from <function WSChiaConnection.__getattr__.<locals>.invoke at 0x7f4a950738b0> at 172.17.0.1

What else should I try? This has happened 3 or 4 times during this resync process, and each time I can only continue by restarting my docker container, which adds a bit of a delay while everything starts up again.

Any ideas?

My setup:

• chia 1.2.2 running on the officia Chia docker image on a Synology NAS
• port 8444 is forwarded properly

[uChiaFarmer]

In my personal experience as someone who farms on a raspberry pi, it is much quicker to sync on a full powered desktop PC and then copy the db file(s) over to the lesser powered RPi.

Like your NAS, the Pi, without any reason or explanation, would seemingly hang during the syncing process, while my main PC would just power through it.

I have never successfully completed a full resync on the Pi, it was just simply too slow, and faster to sync on my main machine and copy the files over. Once the files are in place, the Pi has had no issue remaining fully synced.

I now sync my PC once a month as a back up in case I need to copy over the DB files again.

[aphex]

Hm, it seemed to have no issues the first time around on 1.7.x – I will consider doing this, though I do want to be careful about having multiple full nodes and making my router confused. Do you manually forward the port to the machine that is syncing?

[uChiaFarmer]

Nope, didn’t bother. Still worked. But if it doesn’t work for you, or if its slow feel free to (temporarily) change the port forward to your desktop.

You can turn off the NAS docker during this process if running 2 Full Nodes bothers you. But I havn’t had an issue with it.

Just know that the blockchain db file atm is 8.5GB, so it would make sense not to have multiple nodes downloading it at the same time lol.

Here are some other notes to help you with syncing.

Tips on staying synced to the Chia network

I have 4 computers and 1 pi4 4g, recently I move full node a lot among machines for various reasons, mostly kept synced, here is the things I will do for next smooth move: understand router this is the most important part from my experience, basically need choose between port forward and upnp, because I have multiple machines, I disable upnp and manually update port forward every times, my router is ddwrt and there’re a lot options to choose, for example one option is to clear port forward …

[aphex]

Thanks for the advice! I will try to sync with another full desktop machine.

[uChiaFarmer]

No problem. Feel free to drop another message if you’re still having issues. Good luck.

[distortionist]

Hey did you ever fix that? I have the same exact thing happening.
Syncing is taking forever, and eventually I have a bunch of connections at height 0 and my logs look like yours.

I was thinking maybe our ISP is doing it?

[aphex]

Yes, after moving my full node from the Synology over to a high-end desktop PC (what I was previously using for plotting) I’ve had zero issues staying in sync.

I still have all of the plots on the Synology (a few harvester nodes) and the full node connects to them.