Deamon log spam - failed websockets

Support
1

Chia 1.1.5

-e git+https://github.com/Chia-Network/chia-blockchain.git@0b91005c6b448ab6b5aaf1125bdfc5e38fd8f511#egg=chia_blockchain
chiabip158==1.0
chiapos==1.0.1
chiavdf==1.0.1

Ubuntu Server Pi4 - Linux ubuntu 5.4.0-1034-raspi #37-Ubuntu SMP PREEMPT Mon Apr 12 23:14:49 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

All services running on this machine. harvesters on 2 other LAN machines, and one in Digital Ocean.

I can’t tell if this is harmless or terrible, didn’t see anyone else really reporting this. My log files are absolutely filling up with this:

2021-05-14T20:49:44.442 farmer farmer_server              : INFO     <- farming_info from peer a6dd9f634a0f243f5ffe8e5027cb0a6b7f8d509dec9a01d0ad5b7569463ba9fd 192.168.2.149
2021-05-14T20:49:44.456 daemon chia.daemon.server         : ERROR    Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/daemon/server.py", line 187, in safe_handle
    await socket.send(response)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
    await self.ensure_open()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-05-14T20:49:44.459 daemon chia.daemon.server         : ERROR    Unexpected exception trying to send to websocket: code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chia/chia-blockchain/chia/daemon/server.py", line 187, in safe_handle
    await socket.send(response)
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 555, in send
    await self.ensure_open()
  File "/home/chia/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
1 Like
2

today i just started to see error = 1006.

i just cant handle with it by restarting daemon.

I am on ubuntu 20.04 with full node + harvester and running two more harvesters in my local network.

3

it seems all machines are still farming but still not sure if a proof is found, eveything will be ok with this error.

4

Thanks, Doesn’t seem like any of the chia start options restart the daemon. I just ended up killing it and restarting the harvester. That seems to have eliminated the errors.

5

I am getting this several times a day now - not sure how long but have had so many other problems it has been buried. No sign of a problem on the GUI other than the “Last Attempted Proof” not updating - of course log file has no more eligible checks after this event

Nothing specifically logged around that region in the log files and the last eligibility check was about 16 seconds earlier . I had previously looked for INFO/WARNI/ERRORS but this does not have a status so had missed it - paging through the log files picked it up

Cleared up an restarted GUI and all ok now … until the next time

I think it is reasonable to ask why the connection is not retried after it has broken …

One more reason why i have this …
Farming status: Farming
Total chia farmed: 0.0
Plot count: 1227

2021-05-16T16:00:37.563 daemon __main__                   : INFO     Websocket exception. Closing websocket with chia_harvester code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 721, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../chia/daemon/server.py", line 172, in safe_handle
    async for message in websocket:
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/ianj/1.1.5/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
6

If the failure is like mine, the harvesting has stopped and no more eligibility checks take place - you are out of the lottery until you restart the chia stack

The GUI will likely still be running though - check “Last Attempted Proof” timestamps - if they are not updating you are out of the game

7

I raised a github issue - there are many that are websocket related but i tried to add as much info as possible as many are pretty sparse

https://github.com/Chia-Network/chia-blockchain/issues/5217