Chia Support Please!

Support
1

Hey Guys,

I have a raspberry pi farming my 14 drives of chia (1003 plots) and it was working fine for 2 months not a single outage or crash I brought it down 1 to update the software and it was running smooth,

then one day my pi died, I had a swap file and the sawp file killed the SD card, Replaced the card re-installed and did not install a swap file now chia goes down at least once a day I have logs showing the time it goes down.

The relevant part is here. Farming then Bam error, then the same error for a it until this one and after that I have to reboot the pie re mount the 14 drives find the folders set it up Is it the swap file that has made the difference ?

Any guidance on why it wont run smoothly would be appreciated.


2021-12-15T00:18:59.154 full_node chia.full_node.full_node: INFO     Added unfinished_block 8c49d148e7cae03a7567d6a761e5930314e137ec0c3558c202c45058c8b56f23, not farmed by us, SP: 61 farmer response time: 5.9252, Pool pk xch1jp6frj3ecddur7dxak3n7lq0j75ltquh2zyd44epdu0d6704y2hqyky5hf, validation time: 0.2235 seconds, pre_validation time 0.4931, cost: 1821857178, percent full: 16.562%
2021-12-15T00:19:02.250 full_node chia.full_node.full_node_store: INFO     Don't have rc hash 682aed39ee971a5ab07df1277ece6181d9a3bc4163575c5e22e3fd7a11d0fa3f. caching signage point 62.
2021-12-15T00:19:06.350 full_node chia.full_node.full_node: INFO     🌱 Updated peak to height 1280315, weight 2474377328, hh 443630ead272574ce68b3e0526f92d43b6a3c0dba10806a7678b9fd383bde4a0, forked at 1280314, rh: 682aed39ee971a5ab07df1277ece6181d9a3bc4163575c5e22e3fd7a11d0fa3f, total iters: 4878000636483, overflow: False, deficit: 0, difficulty: 2896, sub slot iters: 135266304, Generator size: 20507, Generator ref list size: 0
2021-12-15T00:19:06.710 full_node chia.full_node.full_node: INFO     ⏲️  Finished signage point 62/64: CC: 652922b89a3424e926c25d1a9f6823e3a1cc00dda399abdc151dec0b925870c8 RC: 52c3d7c2bd9fb5dd2d3040b120db76c80fbc100990ec5e3881d5912b27cf2a85 
2021-12-15T00:19:06.724 full_node chia.full_node.mempool_manager: INFO     Size of mempool: 7 spends, cost: 1839611128 minimum fee to get in: 0
2021-12-15T00:19:06.748 full_node chia.full_node.full_node: WARNING  Block validation time: 4.46 seconds, pre_validation time: 0.72 seconds, cost: 478897828, percent full: 4.354%
2021-12-15T00:19:08.113 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 41a3db6f81... Found 0 proofs. Time: 1.33605 s. Total 1003 plots
2021-12-15T00:19:09.723 wallet chia.wallet.wallet_blockchain: INFO     đź’° Updated wallet peak to height 1280315, weight 2474377328, 
2021-12-15T00:19:10.037 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 1003, duration: 0.00 seconds
2021-12-15T00:19:13.512 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 703, duration: 0.11 seconds
2021-12-15T00:19:13.568 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 403, duration: 0.05 seconds
2021-12-15T00:19:13.621 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 103, duration: 0.05 seconds
2021-12-15T00:19:13.644 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 103, remaining 0, duration: 0.02 seconds
2021-12-15T00:19:13.645 harvester chia.harvester.harvester: INFO     refresh_batch: event done, loaded 0, removed 0, processed 1003, remaining 0, duration: 0.23 seconds
2021-12-15T00:21:06.765 farmer farmer_server              : INFO     Connection closed: 127.0.0.1, node id: 68a094bec0b40c66a30da8aed95b6ded9b61915e831d2a3327e278426aa21963
2021-12-15T00:21:06.767 farmer chia.farmer.farmer         : INFO     peer disconnected {'host': '127.0.0.1', 'port': 8444}
2021-12-15T00:21:07.357 wallet wallet_server              : INFO     Connection closed: 127.0.0.1, node id: 68a094bec0b40c66a30da8aed95b6ded9b61915e831d2a3327e278426aa21963
2021-12-15T00:21:09.296 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-12-15T00:21:10.237 wallet wallet                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2021-12-15T00:21:14.206 harvester chia.harvester.harvester: INFO     refresh_batch: event started, loaded 0, removed 0, processed 0, remaining 1003, duration: 0.00 seconds
2021-12-15T00:21:14.471 harvester chia.harvester.harvester: INFO     refresh_batch: event batch_processed, loaded 0, removed 0, processed 300, remaining 703, duration: 0.07 seconds

2021-12-15T00:29:48.481 daemon chia.daemon.server         : INFO     Websocket exception. Closing websocket with chia_full_node code = 1006 (connection closed abnormally [internal]), no reason Traceback (most recent call last):
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 1047, in read_frame
    frame = await Frame.read(
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/chia-blockchain/chia/daemon/server.py", line 195, in safe_handle
    async for message in websocket:
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 439, in __aiter__
    yield await self.recv()
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/home/ubuntu/chia-blockchain/venv/lib/python3.8/site-packages/websockets/protocol.py", line 812, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason

2021-12-15T00:30:12.431 farmer farmer                     : INFO     Reconnecting to peer {'host': '127.0.0.1', 'port': 8444}
2
2 Likes
3

So after some digging I believe the issue here is the node is being killed when the ram fills up and my reduction of swap space is the reason I am seeing it now and not before.

I am going to enable a 1GB swap file for a week if its stable then I will disable the swap and add an external drive with a swap partition so it wont kill the Micros SD card.

Will respond back here when I confirm that this solves the issue.

2 Likes
4

Believe it or not, I just came here to see if anyone had this particular problem, like I do right now.

My pi has been located at my moms house for a few months and suddenly started dying after a few days.
Took it back with me and after syncing, I observed the RAM filling up.
Not sure, if there was a swap-file before though.

Well, I guess, i’ll let it die one time here to see, if thats rly the problem and then create a swap-file.

5

Be careful if you set up a swap file on the sd card it will die within a few weeks mine only lasted 2 months best to set it up on an external disk drive or a USB drive you don’t mind killing.

6

I bought a Samsung Pro Endurance, which should last a few years,
but it turned out, its too slow for the blockchain, so I fransfered that onto USB already.
Probably not gonna try, how the card performs as swap :sweat_smile:

7

You can always try flexfarmer if it’s a low ram model. How much ram does your pi have? I’m curious

8

I have 4GB of ram on the pi. I have the node running now and I re-enabled farmr (which consume ssome ram too and has a memory leak currently) and that combination was crashing me after a few hours but has been running all night no issue. Ill format a USB stick to ext4 and go install it this evening set up a permanent swap file on it (or at least till i kill the usb stick) and report back in a few days.

9

I didn’t know it was that bad if a 4gb can’t run a full node!

10

To be fair to the chia team on the Pi installation instruction they suggest adding a 1GB swap file for 4GB pi but yeah it eats a lot of ram, Ill do the extra to account for the farmr memory leak. should be back up solid after that, I was starting to get frustrated >_<

11

Mine has been running for almost 1 day now and still has 0,5GB Ram left.

Using Flexfarmer looks attractive to me currently, as pi4 is out of stock everywhere and I have a few pi3 laying around.

12

Key factor to running a node on the Pi4: It’s recommended by us that you do not run your Node database off of an SD card. It’s is ideal that you use an SSD instead. While I can’t say with certainty it will fix this exact issue, what I can say is that it will remove a major I/O bottleneck for you and free up a lot of resources for your system elsewhere, which will hopefully take some pressure off of whatever may be triggering this.

3 Likes
13

pi3 is a little overkill. No 1 or 2 lying around?

14

An ssd on a pie I think that’s a bit overkill a disk drive would do but the sd card was fine except the write limit due to page file on it page file on the usb is working fine let’s see how long it is stable

15

OK, after adding a swap-file to one of my HDDs, it seems to be stable again.
Times have always been fine.

16

Yes mine has been stable since I added the swap file changed yesterday to a swap file on a USB stick fingers crossed problem solved :grinning:

1 Like
17

Just to update no issues since the move to swap on external USB problem solves I think

2 Likes
18

Final update my issue is 100% gone.

Issue showed as chia would crash every day or so and when this happened a reboot of the pi was the only way to get it back up. The error in the logs is ConnectionClosedError: code = 1006.

Turns out the issue was due to page file not being enabled as it kills micro sd cards very quick, I instead set up a page file on an external usb drive.

So far its been about 5 days and no sign of crash I consider this quite stable, 2gb page file was all it needed.

Kind Regards.

1 Like