Chia full_node becomes abnormal after some network issue

Today my router got some issues and I restarted it.
Though the network restored, but chia full_node became strange,
I tried restart full_node,
the log looks like

    2021-06-06T21:55:05.874 full_node chia.full_node.full_node: INFO     Initializing blockchain from disk
    2021-06-06T21:55:05.877 full_node chia.consensus.blockchain: INFO     Started 10 processes for block validation
    2021-06-06T21:56:16.386 wallet wallet_server              : INFO     Connected with full_node {'host': '36.57.226.13', 'port': 8444}
    2021-06-06T21:56:18.942 wallet wallet_server              : INFO     Connected with full_node {'host': '60.182.179.124', 'port': 8444}
    2021-06-06T21:56:22.531 wallet wallet_server              : INFO     Connected with full_node {'host': '114.233.159.5', 'port': 8444}
    2021-06-06T21:56:28.304 wallet wallet_server              : INFO     Connected with full_node {'host': '91.244.73.129', 'port': 8444}
    2021-06-06T21:56:33.945 wallet wallet_server              : INFO     Connected with full_node {'host': '124.238.8.68', 'port': 8444}
    2021-06-06T21:56:34.516 full_node chia.full_node.full_node: INFO     Blockchain initialized to peak 2a6b24206ddb89be27428960bd84d1e15642436bc972b8ffb6b45a1336703d46 height 391538, time taken: 88s
    2021-06-06T21:56:34.553 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 391538, weight 102040672, hh 2a6b24206ddb89be27428960bd84d1e15642436bc972b8ffb6b45a1336703d46, forked at 391537, rh: 2dc3aef3aa68c1f49916e948166155ddd8e9210b6a4dea62003f93b13bcb2020, total iters: 1279608194082, overflow: False, deficit: 15, difficulty: 1320, sub slot iters: 116916224, Generator size: 3092, Generator ref list size: 1
    2021-06-06T21:56:47.385 full_node chia.server.upnp        : INFO     Attempting to enable UPnP (open up port 8444)
    2021-06-06T21:56:48.454 full_node chia.server.upnp        : INFO     UPnP failed. This is not required to run chia, but it allows incoming connections from other peers.
    2021-06-06T21:56:55.130 wallet wallet_server              : INFO     Connected with full_node {'host': '118.163.36.245', 'port': 8444}
    2021-06-06T21:59:16.649 wallet wallet_server              : INFO     Connected with full_node {'host': '49.172.112.78', 'port': 8444}
    2021-06-06T22:04:14.423 wallet wallet_server              : INFO     Connected with full_node {'host': '60.168.43.25', 'port': 8444}
    2021-06-06T22:06:29.801 wallet wallet_server              : INFO     Connected with full_node {'host': '49.49.145.99', 'port': 8444}
    2021-06-06T22:06:35.905 wallet wallet_server              : INFO     Connected with full_node {'host': '114.233.159.5', 'port': 8444}
    2021-06-06T22:07:58.102 wallet wallet_server              : INFO     Connected with full_node {'host': '61.90.42.91', 'port': 8444}

full_node didn’t start to sync, no more ‘full_node’ log shows up
and 'chia show -c ’ shows

Connection error. Check if full node rpc is running at 8555
This is normal if full node is still starting up

What’s going on with my full_node?

1 Like

I guess the UpnP failure in log is not the problem, since I manually setup port-map on router,
and verified port 8444 is opened by “Open Port Check Tool - Test Port Forwarding on Your Router”.

1 Like

When you restart the full node. It has to validate your whole blockchain. For me it takes nearly 10 minutes. If you didn’t freak out and stop it, I imagine you saw that it went back to normal after a small amount of time.

I have waited about 30 minutes, and ‘iostat’ shows nearly no I/O on “$HOME/.chia” device.

I am afraid that something is not right.

It’s been 30 minutes and you still get the startup error from the show command? Did your log file rotate? Are you sure you’re looking at the newest one? What is CPU usage? Do you see full_node proc is still running?

1 Like

Yes, log file is rotating, 'wallet , farmer ’ services are emitting logs.

full_node process is present, and listens on 8444

(venv) xiaogang@xiaogang-chia1:~$ sudo netstat --tcp -nlp | grep 8444
[sudo] password for xiaogang:
tcp      129      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      2653/chia_full_node
tcp6       0      0 :::8444                 :::*                    LISTEN      2653/chia_full_node

cpu usages like:

Tasks: 338 total,   4 running, 334 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.3 us,  2.1 sy,  0.0 ni, 39.8 id, 43.3 wa,  0.0 hi,  1.5 si,  0.0 st
MiB Mem :  32097.6 total,    480.5 free,   9761.4 used,  21855.6 buff/cache
MiB Swap:   2048.0 total,   1136.2 free,    911.8 used.  21879.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 201996 xiaogang  20   0 3755780   2.0g   9196 D  69.1   6.3   4909:15 chia
 200859 xiaogang  20   0 3803284   1.7g   9144 R  38.2   5.3   4992:29 chia
 200862 xiaogang  20   0 3785816   1.6g   9160 D  31.2   5.2   4974:46 chia
 201710 xiaogang  20   0 3778336   1.7g   9092 D  29.2   5.4   4927:08 chia
 202281 xiaogang  20   0 3777316 973536   9184 R  25.2   3.0   4913:30 chia
 201425 xiaogang  20   0  304392  54764   9240 D   2.0   0.2   4987:39 chia

The 6 ‘chia’ processes on top are plotter.
Well the first line of ‘top’ didnt get pasted, but the ‘LA’ is about 8.
I have 12 (after hyperthread) cores, I think CPU usage is OK.

Eventually I found the cause.

After network issue, ‘full_node’ process entered a strange state, so I tried to stop/start it.
That is OK.

But actually the ‘full_node’ didn’t really quit event it logged something like 'got signal 15 … ',
though ‘chia stop all’ and ‘chia start farmer’ reported everything normal,
the ‘full_node’ process remains the same.

So I just ‘kill -9’ the ‘full_node’ , then ‘chia start farmer’ , ‘full_node’ become ‘Syncing’.

2 Likes

This was very helpful thank you!

1 Like