Full Node Sync stuck at 0/1,484,486 while connection status is connected

I check the debug log, and the follow error is shown.

2022-01-28T15:04:24.667 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: 63006236ca42575e958faff3226375237625008904e341d6f0572d86890065f0
2022-01-28T15:04:24.666 full_node asyncio                 : ERROR    Task exception was never retrieved
future: <Task finished name='Task-11042' coro=<RpcServer._state_changed() done, defined at chia/rpc/rpc_server.py:48> exception=KeyError(b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2')>
Traceback (most recent call last):
  File "chia/rpc/rpc_server.py", line 51, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 61, in _state_changed
  File "chia/rpc/full_node_rpc_api.py", line 102, in get_blockchain_state
  File "chia/consensus/blockchain.py", line 157, in get_peak
  File "chia/consensus/blockchain.py", line 674, in height_to_block_record
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: b'\xd7\x80\xd2,z\x87\xc9\xe0\x1d\x98\xb4\x9a\t\x10\xf6p\x1c;\x95\x01WA1k?\xda\x04.]{\x81\xd2'
2022-01-28T15:04:24.667 full_node full_node_server        : INFO     Connection closed: 127.0.0.1, node id: 43b61bcc01a90a0599b057f8cb0326ebc6bc0e54ce9bbcade04f7ea9969eecb6
2022-01-28T15:04:24.667 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '127.0.0.1', 'port': 8449}
2022-01-28T15:04:24.667 full_node full_node_server        : ERROR    Exception <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>, exception Stack: Traceback (most recent call last):
  File "chia/server/server.py", line 292, in incoming_connection
  File "chia/server/server.py", line 335, in connection_added
  File "chia/full_node/full_node.py", line 631, in on_connect
  File "chia/consensus/blockchain.py", line 670, in block_record
KeyError: <bytes32: d780d22c7a87c9e01d98b49a0910f6701c3b95015741316b3fda042e5d7b81d2>
1 Like

Same thing happened to me overnight. Also see the topic with identical title just below yours.
Most likely your db got corrupted. Why and how? It just seems to happen (too) frequently, doesn’t take much it seems.
Do you have a somewhat recent backup of the database?
And for the record, what type of full-node do you run and what version. I’m on an Intel NUC with 4 core Pentium with version 1.2.11.

I run it on virtural machine, Ubuntu 20.04.2, 8, cores AMD 7H12, version 1.2.11.
It runs right when I up it to 1.2.11. But it sync stuck suddenly.
How can I recover the db?

No way to recover that I’m aware of. You can wipe the db files and it will start syncing from scratch, but takes days due to the current size of the blockchain;-(
There is a site offering daily actualized db’s for download, if your comfortable with that. No incidents or problems with that reported yet.
Best is to have a local backup, I do this when updating Ubuntu once in a while, you’ll only have to sync from the latest backup then.

Take a look at the recent topic with similar title as yours for further info on that downloadsite please.

Did you take the size of the blockchain (46 GB and growing) into account when sizing the vdisk for your VM? I don’t know what happens when the SQL lite db runs out of space but nothing good I guess…

I’m also running a VM (since yesterday, with a db crash overnight after migrating to it). Now I’m thinking that the extra layer between db and physical disk could also be an extra source of problems with the high volumes of transactions on the db. A short disruption in the hypervisor’s disk management…
Maybe back to full metal?

I have 500G vmdisk, and 100G left.

Well, that should be enough;-)
As said, database corruption seems to happen now and then for mostly unknown reasons. Plenty of people reported such problems here.
I take it you stopped/started the chia client and also rebooted in between. If it still refuses to start syncing (seeing the height really updating in the full node screen, not just static 0/148xxxx or ‘not synced’) the only other option is to resync, from scratch or a (downloaded) backup.

Consider taking a look at our blockchain boostrap download if you’re having trouble getting synced up! It’ll drastically speed up the sync time! https://elysiumpool.com/bootstrap

Thank you so much for taking the time to respond to my rant. I took your advice and went through a fairly lengthy process of installing the torrent client and using the link to get the db downloaded (which took approximately 30 minutes). All in all it took about 90 minutes from start to finish and I am now FINALLY synced up. I can confirm that the elysium pool bootstrap works and am very happy to say a big thank you for helping me get to the finish line. Now on to the plotting!

1 Like

I have just had this issue. I went to copy a backed up db file and realised there was not enough space. I wonmder if a backup happens that means it needs twice as much space? i.e you should maybe keep 50GB free.
Just thinking out loud.

We’re very glad we were able to help get you set up quickly and efficiently!

We hope you consider joining Elysium Pool as you begin plotting! Best of luck!

i have same issue, but my drive has plenty of space, i have separate harvester machines that connect to this single farmer machine. should i delete the db and re sync back?

If its stuck at 0/ xxxxxxx then yes, db is corrupted and needs deleting.

o my it took several days to resync

You have no choice, you could use the torrent download that’s listed in this thread to speed things up if you want.

I’m not quite sure how to do that torrent thing, and I’m kind of hesitant also doing it in torrent maybe security thing, I have like 58K plus plots on all harvester and its a project that i is assign to me, I’m only paid to do it. And it has a considerable amount of chia coin already. Anyway i will just do it on same old way thanks.

Is the full-node machine fast?
Sync will always be slower than we want but can be a bit quicker with a couple of high Ghz CPU cores and database on NVME.
If the full-node is not like that maybe you can use another machine with higher specs to install chia with new keys and sync the database there. Then copy it to your full-node. The blockchain db is not keys dependent.

Db corruption happens alot, if your not wanting to use the torrent that’s up to you.
I’d advise once its synced stopping the client and backing up the db, then doing regular back ups client needs to be stopped first though.
At least that way if / when it happens again you can just sync from that point.

2 Likes

My Full-node machine is not a latest machines its only Intel(R) Xeon(R) CPU E31270 @ 3.40GHz processor with 32GB of RAM Memory and running on a 1TB SATA drive, i only use those NVME to my plotter Machines.

excatly thats what i really do.