Raspberry pi farmer + full node seems to go in sycn and quickly out of sync usually within 1 block of height

Hi everyone,

I’m running a farmer+full node on a raspberry pi 4 (8GB) and it seems like the node stays in sync for a few minutes then gets behind, usually 1-2 blocks behind max height.

At first I thought this was the pi being too slow to verify the blocks quickly enough, but now I think its something else, otherwise why would it go in/out of sync all the time.

Any help would be greatly appreciated.

Thanks,
Eran

1 Like

Sneaky? You know nothing about how the code works, what the developers are working on, and the history of the development.

Chia grew faster than expected. They are fixing everything they can with the time and resources they have.

I am warning you: Stop using this forum to vent and create drama. If you have something useful to contribute, I welcome it. But so far, much of what you have been posting is vindictive and not useful.

2 Likes

If you have nothing else to lose then you might want to try running from the main branch which has some of the fixes that they mention for the upcoming 1.1.6 version.

Sneaky? You know nothing about how the code works, what the developers are working on, and the history of the development.

Maybe I used a wrong word, I am not a native speaker.
They told people before that they can farm on Raspberry Pi, but then network grew expotentially and no one stated clearly that farming on RPi is not a case now.

I removed my comment above, I should use other word than “sneaky” - I not good with synonyms. Long story short, use the desktop and later.RPi will be usable.

Fair enough, I understand.

When they first launched Chia, they did test RPi 4 and it did work. But the testnet was very small then. Chia exploded so RPi’s are having a hard time. I hear using an SSD instead of the microSD card to boot from helps.

Well, I have been surprised how much CPU my dedicated 1u box is using, and it’s a 4 core Kaby Lake (7th gen) Xeon… remember this box is only farming, it is doing literally nothing else…

How many physical disks you have connected to this machine?

JBOD connection through PCI interface, 45 drives.

I considered that. I need to get a more modern CPU for an older board I have (a 7th gen i5 with 4 cores) and I assume that will probably help it keep up.

I remember when I was running a bunch of different nodes for different coins how taxing it was on the system. Specifically Ethereum needed quite bit of resources with the rate of transactions going on.

Hopefully the CPU will get here soon as I can test if that was indeed the case. I think Raspberry Pis are more than enough to be farmers but that combined with a node is probably too much.

1 Like

It doesn’t surprise me that a raspberrypi 4 is having trouble keeping up.
I’m running it on a threadripper 1920x and that can’t keep up.

Seems like chia_full_node is effectively single threaded. I see 15 chia_full_node processes and only 1 always pegged at 100%CPU and losing sync.

My PI 4 with raspian64 bit is keeping sync.
But I have detailed logging off and no other activities running on it.
Obviously not plotting on it

I also just throttle d it a bit also, but even before throttling, it was almost always keeping in sync

I have had occasion sporadic out of syncs but very few

1 Like

I literally just set my new Pi 4 (Ubuntu 20.04 LTS 64-bit) up for harvesting two days ago. I was seeing similar issues with syncing after moving my full node database over to the Pi.

Part of my issue is that my network is behind carrier-grade NAT (meaning you can’t forward 8444 to your full node without a publicly-routable IP address), so that meant the number of peers I had was quite small.

This is a great article on how to get around that if this is your only issue.

However, it did seem the Pi would get fully synced, then go out of sync very quickly.

I have detailed logging on as I wanted to run chiamon, but maybe this is an issue?

What I did to get around this is set up a $12/mo Digital Ocean droplet (2GB RAM, their “premium” Intel CPU) as a full node, then configured my Pi and Mac mini (plotter) as harvesters.

I’m still not fully synced on the droplet, but this might be a good stopgap solution. EDIT: I’m now fully synced and farming – seems to be working very well.

I hope the core team is able to find appropriate performance fixes, as I know they believe a Pi 4 should be able to participate on the network.

2 Likes

I have also noticed that the reason for getting unsynced was actually because the full node was using too much system resources. As soon as the RAM or CPU reaches 100% the node gets unsynced.

So I have completely changed my setup. now I’m using a 5 years old i5 8GB ram laptop just to do farming. And even this laptop uses more than half of system resources with only the windows GUI running. nothing else works on this machine. It feels like the bigger the network gets, the more resources the program uses :confused:

1 Like

I don’t think I’d draw too many conclusions about the ability of a given setup (i.e. Raspberry Pi) to farm at this point. I believe there are things happening on the network that are causing sync problems that are not related to whatever hardware you may be running. I’ve seen many complaints for several days now of sync issues for people running various hardware/configurations. I’ve been happy that I wasn’t seeing issues until yesterday where for several hours it was in and out of sync. The one pattern I noticed was that when out of sync many of my connected peers showed no “height” value and had no data transfer up or down. I started grooming out these connections. Did it help? Maybe. Fulll node is sync’d and happy today. Connections all look healthy. Real source of problem? Not sure, but I’m convinced there is something going on with the network.

I’ve been farming on a raspi4 for at least a month now with no issues: connected to 10 peers (check this), search times (on filtered plots) is always <1sec, and I haven’t gone out of sync (see image). Granted I’m farming only 1TB on the pi, as an experiment, but nonetheless. Plus, I’m running Umbrel and Urbit on it. :slight_smile:

I gather you checked the pi’s temp, throttle, and memory (see below) and monitor the system with vmstat/iostat to see if there are any obvious anomalies?

Temperature
-----------
temp=47.7'C

Throttling
----------
throttled=0x0

Memory usage
------------
              total        used        free      shared  buff/cache   available
Mem:           7.8G        5.2G        317M        115M        2.3G        2.6G
Swap:          4.1G        1.6G        2.5G

2 Likes

Those Pi’s tend to get quite toasty if you put any sustained load on them, and many have them in those tiny cases with 0 active cooling

As I understand with what little experience with Chia I have, you can only harvest with low power computer. When it comes to farming you really need fast/powerful computer.

I am having hard time with Syncing issues. Looks like they have to work a lot to improve Syncing.

Yeah, I did check thinks and it all seems to be ok.

I copied the DB because it was just a few blocks from max height to an older desktop I had lying around and it synced without a problem and is still synced for the past 12 hours or so.

So its not UPNP (its disabled), the port is forwarded directly and its not Comcast (my cable internet provider) doing the fancy NATing issues that sometimes happen.

It’s definitely the Pi running the full node, farming and rsyncing from time to time new plots. Specifically, it couldn’t sync when it wasn’t rsyncing new plots and sometimes it did sync while doing rsyncs. So its all so strange.

I had about the same issue. My solution was to
chia start farmer-no-wallet

That way the RPi has less load to deal with. AFAIK the wallet is not needed for harvesting. In my case I still want the RPi to be my main (full) node for main farming, and for my plotter (+ harvester) to connect to it, until disks are full and physically moved to the RPi.
I also have the GUI set up on my workstation, which also is a full-node + wallet. That way I also ensure that I have at least one machine on my network which has a current copy of the blockchain, so I can sync faster (locally) if I ever need to do any maintenance on my RPi.

I am running fine with the ‘farmer-no-wallet’ setup for a few days now. CPU is also much lower than before.

2 Likes

So, assuming your raspi isn’t throttled (not a CPU issue), if you want to go semi-hardcore into it then wait until it is sync’d and in separate terminals start monitoring:

  1. run dmesg -Tw --facility=kern,user,daemon,auth,syslog --level=debug,info,notice,warn,err,crit,alert,emerg
  2. run vmstat and iostat at 1sec intervals
  3. run tcpdump
  4. finally, run chia show -s at 2sec internals until it shows it’s not in sync

Then, take a look at everything above. Ideally, you can do that without it rsync’ing plots to avoid that added variable.

My raspi is connected to my OpenWrt router over wired ethernet. Oddly, when I first started running it, the wired network connection on the pi was quite flaky. To try to figure out what was happening, I enabled its wireless interface so I can see what was happening when the wired connection flaked out. Magically, the wired ethernet problem disappeared.
Heisenbug. ¯\_(ツ)_/¯

1 Like