CLI status is "Not Synced" but node always has the latest "Peak height" block

I have a machine where I did a fresh install of Chia (via pip), I want to eventually use this as a standalone separate node/farmer/wallet. I let it sync the blockchain, but chia show -s always states:

Current Blockchain Status: Not Synced. Peak height: <latest_block>

Where <latest_block> is the actual block number of the latest block, which I see from another machine I have which reports Full Node Synced, and is always newer than the block shown on https://www.chiaexplorer.com/blockchain/blocks (that website seems to be a few minutes delayed a lot of the time).

The list of block heights and block hashes are identical to my other synced node, every time I run chia show -s.

I also added a wallet key via mnemonic, and the wallet is also synced to the latest block, but not showing the correct balance or transactions.

It has been “Not Synced” like this for over a week now, even though the block is always the latest, and chia show -c shows connections to full nodes all the time:

$ chia show -c | grep -c FULL_NODE
21
  • and still after multiple clean installs (completely removing the user homedir, recreating the user/homedir, and attempts of letting it sync or copying a blockchain db from a synced machine).

Debug logs only show some “timed out” errors connecting to other nodes, e.g.:

2021-09-15T12:59:16.232 full_node full_node_server        : ERROR    Exception: Failed to fetch block 862457 from {'host': '<REDACTED>', 'port': 8444}, timed out, {'host': '<REDACTED>', 'port': 8444}. Traceback (most recent call last):
2021-09-15T12:59:16.235 full_node full_node_server        : ERROR    Exception: Failed to fetch block 862457 from {'host': '<REDACTED>', 'port': 8444}, timed out <class 'ValueError'>, closing connection {'host': '<REDACTED>', 'port': 8444}. Traceback (most recent call last):

I have another machine which I installed the same way, which I farm from, which always states “Full Node Synced”.

Both have Chia CLI version 1.2.6, installed via pip3 install chia-blockchain==1.2.6.

Both machines have the same (latest) Arch Linux packages, same Python/kernel version.

The machines are on the same LAN, but each has it’s own VPN connection (since my ISP employs DS-lite/CGNAT) which all internet traffic goes through, each has remote port 8444 forwarded (tested working from another external machine). I’ve swapped the VPN configs to check if was a VPN problem, but no difference. I’ve tried adding the LAN IP of the synced node to the unsynced node (chia show --add-connection <IP>:<PORT>) - no difference.

The unsynced machine seems to have stable hardware:

  • memtest86+ ran for at least 24 hours - no errors
  • prime95 ran blended test for 24 hours - no errors
  • Has been in use as a plotter 24/7 for months, no errors/crashes while using Chia mainline plotter or MadMax.
  • No SMART errors on two different SSDs, ZFS (for all data) reports no checksum errors.
  • Runs from a UPS - log says no power glitches except for the manual tests I did when I first got it.

Any ideas or suggestions of what I can look at?

1 Like

Update: Fixed! Well, just typing that all out made me realize something that I didn’t check - the time.

Forced an NTP sync and it changed to “Full Node Synced” within a few seconds and has stayed like it.

Evidently when I booted into Windows on it last (was that desperate), Windows synced the RTC to local time, and Linux expects UTC, so it was quite a bit out.

2 Likes

Hey, thanks for this, I had the same problem on a Windows client and I never thought to check the system time. Sure enough, it was the same problem with the same solution, just update the time.

Funnily enough, the time got out of sync in a mirror image of what happened to you. I mostly use Windows on that notebook, but had installed Ubuntu on it too and had recently rebooted from Ubuntu into Windows when the problem first happened (since Ubuntu had set the system clock to UTC and Windows had not yet updated it to local time).