Chia sync falling behind

pwagma · January 11, 2022, 2:45pm

Hello
Since last night my farmer got disconnect, It has been running without issues for months on a linux server.
I am checking and it seems that the sync is falling behind, I reboot the computer and try to connect to the main nodes but the system is still failing.
The chia version is 1.2.9
Any clues?
Thanks
These are the logs:

2022-01-11T11:39:25.945 full_node full_node_server : INFO 2022-01-11T11:39:25.961 full_node full_node_server : INFO 2022-01-11T11:39:25.961 full_node chia.full_node.full_node: INFO 2022-01-11T11:39:29.261 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:39:29.367 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:39:37.159 full_node full_node_server : INFO 2022-01-11T11:39:37.160 full_node full_node_server : INFO 2022-01-11T11:39:37.160 full_node chia.full_node.full_node: INFO 2022-01-11T11:39:45.922 full_node full_node_server : INFO 2022-01-11T11:40:37.732 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:38.410 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:50.433 full_node chia.full_node.full_node: ERROR 2022-01-11T11:40:50.433 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:50.433 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:50.440 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:40:50.519 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:40:58.022 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:58.022 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:58.827 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:58.827 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:59.134 full_node chia.full_node.full_node: INFO 2022-01-11T11:40:59.134 full_node chia.full_node.full_node: INFO 2022-01-11T11:41:19.533 harvester chia.harvester.harvester: INFO 2022-01-11T11:41:26.732 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:41:26.792 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:42:31.068 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:31.849 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:42.754 full_node chia.full_node.full_node: ERROR 2022-01-11T11:42:42.755 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:42.755 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:42.761 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:42:42.844 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:42:50.118 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:50.118 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:50.722 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:50.722 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:51.254 full_node chia.full_node.full_node: INFO 2022-01-11T11:42:51.254 full_node chia.full_node.full_node: INFO 2022-01-11T11:43:03.432 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:43:03.493 full_node chia.full_node.weight_proof: INFO 2022-01-11T11:43:19.740 harvester chia.harvester.harvester: INFO 2022-01-11T11:44:02.714 full_node chia.full_node.full_node: INFO 2022-01-11T11:44:02.909 full_node chia.full_node.full_node: INFO 2022-01-11T11:44:10.280 full_node chia.full_node.full_node: ERROR Connected with full_node {‘host’: ‘194.0.139.137’, ‘port’: 8444}
Connection closed: 194.0.139.137, node id: fd0c7d93590b6eeefe859a1a5ddf6f4c9b2fc09860b4d400bbdcfd356f3fd12a
peer disconnected {‘host’: ‘194.0.139.137’, ‘port’: 8444}
validate weight proof peak height 1407539
validating 3665 sub epochs
Connected with full_node {‘host’: ‘91.117.219.126’, ‘port’: 8444}
Connection closed: 91.117.219.126, node id: b213ad2ee7571848de474c700935198cbf10c1794df2d0693950123e991c6d1d
peer disconnected {‘host’: ‘91.117.219.126’, ‘port’: 8444}
Connected with full_node {‘host’: ‘80.229.6.72’, ‘port’: 8444}
Re-checked peers: total of 6 peers with peak 1407539
Start syncing from fork point at 1405450 up to 1407539
sync from fork point failed err: ‘bytes32’ object has no attribute ‘name’
long sync done
Updated peak to height 1406430, weight 2819039360, hh b25d14d06112b7f0bf7a71a31d3d2c68c6be832ec84223ae7dac797e7161a58b, forked at 140642>2022-01-11T11:40:50.436 full_node chia.full_node.mempool_manager: INFO Size of mempool: 0 spends, cost: 0 minimum fee to get in: 0
create weight proof peak b25d14d06112b7f0bf7a71a31d3d2c68c6be832ec84223ae7dac797e7161a58b 1406430
recent chain, start: 1405835 end: 1406430
Starting to perform sync.
Waiting to receive peaks from peers.
Collected a total of 3 peaks.
Selected peak 1407546, e92b2cdd37a920789f9ffd028118d638215411fe1c37a0e71d1d0c8f210f84f5
Total of 5 peers with peak 1407546
Requesting weight proof from peer 217.119.27.21 up to height 1407546
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 0, processed_plots 0, remaining_plots 0, duration: 0.07 seconds
validate weight proof peak height 1407546
validating 3665 sub epochs
Re-checked peers: total of 5 peers with peak 1407546
Start syncing from fork point at 1405450 up to 1407546
sync from fork point failed err: ‘bytes32’ object has no attribute ‘name’
long sync done
Updated peak to height 1406432, weight 2819044864, hh 830ca03ec431e490a2bef420ea5be894f365986801cefd98b03dfc26d2362f9e, forked at 140643>2022-01-11T11:42:42.758 full_node chia.full_node.mempool_manager: INFO Size of mempool: 0 spends, cost: 0 minimum fee to get in: 0
create weight proof peak 830ca03ec431e490a2bef420ea5be894f365986801cefd98b03dfc26d2362f9e 1406432
recent chain, start: 1405835 end: 1406432
Starting to perform sync.
Waiting to receive peaks from peers.
Collected a total of 3 peaks.
Selected peak 1407550, 083ca7373a578ee200daf8377a542b3ba5f16ab9ed0f20b853115fee11996276
Total of 8 peers with peak 1407550
Requesting weight proof from peer 51.79.16.87 up to height 1407550
validate weight proof peak height 1407550
validating 3665 sub epochs
refresh_batch: loaded_plots 0, loaded_size 0.00 TiB, removed_plots 0, processed_plots 0, remaining_plots 0, duration: 0.04 seconds
Re-checked peers: total of 8 peers with peak 1407550
Start syncing from fork point at 1405450 up to 1407550
sync from fork point failed err: ‘bytes32’ object has no attribute ‘name’

Bones · January 11, 2022, 3:35pm

No idea why it would be fine then stop.
But I see a few lines saying, have 5 peers, have 6 peers.

How many have you set it to have 80?
That’s the pre configured amount unless you changed it.

I dare say 5 or 6 is to low, certainly for my liking, I have 80.

xkredr59 · January 11, 2022, 3:39pm

Could be, but the 80 max is only when you have port 8444 forwarded in your router or it supports upnp.
Otherwise 8 is max and that should be ok.

If the disk the databases are on (db and wallet/db) has enough room (growing in the direction of 50GB these days…) the OP might consider updating the client to 1.2.11.
1.2.9 is falling behind also, not just the farmer…
But please stop all processes first, wait a bit to make sure and then backup the .db and .wallet/db directories somewhere save before updating the client.

Bones · January 11, 2022, 3:43pm

I’d agree, I’d certainly try updating client first.
If that doesn’t work I’d then try deleting unconfirmed transactions.
As a last resort I’d try deleting database entirely and using a backup of a known good one.

Jacek · January 11, 2022, 4:32pm

It may be that there was a dust storm going on, and that is why your node could not keep up:
https://graphchia.com/Mempool.php

Based on the original chia statement, v1.2.9 should be extra vulnerable to those storms and having problems to sync during those times.

As already said, go for an upgrade. If you have problems with v1.2.11, start with v1.2.10 first.

Still, I would like to know what CPU you have, and where is your blockchain db sitting on (HD, SSD).

As @xkredr59 stated, it looks like no inbound peers, as such the default 8 outbound peers is used. Also, because the node is struggling, it is dropping, and trying to get new peers, therefore it goes below 8 peers - behavior to be expected (gaps when dropping old / getting new peers)

Here are three nodes that I watched for few hours:

Jan 02 14:22:16
FULL_NODE 218.214.245.246   29.0|25.2    1,367,444   
FULL_NODE 209.83.95.225     427.0|13.8   1,294,370
FULL_NODE 64.67.105.184     18.4|5.7     1,358,793

Jan 02 17:40:53                                      up | down   height
FULL_NODE 218.214.245.246   29.9|27.0    1,368,103  0.9 | 1.8    659
FULL_NODE 209.83.95.225     431.8|14.8   1,294,559  4.8 | 1.0    189
FULL_NODE 64.67.105.184     21.5|6.7     1,358,793  3.1 | 1.0    0

Those three nodes show slightly different behavior. 218 is fully synced, and draws/sends little data while moving by a lot of transactions.

209 is behind, takes 5x more data than 218, but advances just by 30% of that good node.

64 is a stunned node. It draws also a lot of data but is not moving at all.

I also noticed one node that may have reduced number of peers, and my node was sending it about 10x more data than it was sending to other nodes.

So, the bottom line is that the protocol allows those stunned nodes to draw data without making any progress. I assume that those nodes get the data, cannot process it, drop it, and request the same data over and over. I think the same thing was going on with that node behind, as it is hard to explain why it took 5x more data that it should, but was advancing only at 30% rate. Also, if some nodes run with reduced number of peers, they compensate that with increased data rates sent to other peers.

pwagma · January 12, 2022, 12:46pm

Sorry for the delay, there has been some energy disruptions on my area.
The setup has been running for a while, I have 2 harvester and 1 farmer, the farmer is the original plotter so it should have enough specs to run chia.
The only problem may be the storage, right now I have used 160G/196G (glances).
Right now I am just running chia, stopped with the forks to see if it works.
I havent changed the config file in months…
It feels like the node starts to update the DB, and for some reason it looses connection with the peers, and starts again…
I could update the chia version, but I was waiting for the big update that it was supossed to come this week.

Nodes

Current Log

Bones · January 12, 2022, 12:54pm

What type of drive is your database located on?

Have you considered updating the client as has been suggested?

pwagma · January 12, 2022, 1:07pm

The Database is on a low cost sdd along with the OS
I will update the client with the new release coming soon. Also I am using foxy client (im pooling my ogs there). but has been working like a charm for months.

Here is how it is evolving… it syncs so slow, maybe now matching the new heights speed

Bones · January 12, 2022, 1:20pm

Your choice, I dont normally run first release of new client in case of bugs, but to each their own.

I know my ssd struggles a tad now, used to load instantly but not anymore due to chain size, but I sync fast once loaded.

Good luck.

wam · January 29, 2022, 6:22pm

Same issue here afetr a restart.
A client update fixed it