Chia Proofs start to slow and then goes to 0/2000 after about 15 hours?

Hey Farmers,

I’m using 1.6.0 and to save on power costs moved my Full node to a VM on my Qnap NAS AMD Ryzen V1500B Quad-Core 2.2MHz (8 threads) with 32GB of Ram. The Win 10 VM is on a 2TB SSD. It has been running 2000 odd NFT Plots in Space Pool… till now.

The symptom is I see the number of proofs and estimated size on Space Pool begin to drop. When I remote desktop, I can see the rather than the usual 4 entries a minute bigger gaps and eventually, it stops. This happens over about 12-15 hours…

There is a Dust storm on at the moment, so I guess it might be that? The Full Node & Wallet sync indicators remain green.

When I look at connections - 40+, I see a range of xxxx/8444 & 8444/8444.

The strange thing is when I reboot my ASUS router, it seems to start getting proof again, I guess it could be a red herring, but it’s annoying me… Any thoughts or suggestions are welcome!

1 Like

Also seeing this type of warning in the logs a lot!

2022-10-27T19:44:09.483 full_node full_node_server : WARNING Rate limiting ourselves. message type: new_compact_vdf, peer: 218.90.122.33

All the plots are on this NAS or do you have separate harvester(s)? Did you check if farmer reports correct plot count (CLI: chia farm summary)?

Yes, I believer rate limiting warning is due to dust storm.

Look at he Chia harvester log but make sure it is running at INFO level (default is WARNING):

  • To see you plot scan rate look for lines with “eligible for farming” this will show you # of plots that pass filter
  • To see any plot scan slowdowns look for warnings with “Looking up qualities” which will give you the path to plots that took > 5sec to scan
  • To see your proofs look for lines with “Farmed unfinished_block”

I’m monitoring my farm & harvesters 24/7 and have not noticed any issues a part from the rate limiting but that’s normal under the current condition. I also farm NFT plots solo so that may be why I’m not experiencing issues. Look at the log to confirm if you are seeing the 5-7 plot scans per min and compare proofs in log to what pool is reporting as the issue may be on pool side.

If you suspect something with the router you can setup 8444 port forward manually on the router to the internal IP of the farmer and restart the farmer. This will eliminate possible issues with UPnP which you are using if no port forwarding is manually setup.

When the slowdown occurs you can runt he above farm summary CLI command and check for “Farming status: Farming” and if it is not farming then do “chia show -s” and check if your node is out of sync.

How did you configure the VM? If not enough vCPU resources committed, your node may struggle to keep up during the dust storm. Anything else taxing the CPU in the NAS? Like running a backup job f.i.?

60% of the plots are on the NAS the VM is on. The rest are on a second NAS, with SAMBA shares to the Windows VM. I know this isn’t the quickest way but has worked in the past…

I double-checked and have Log set to INFO.

I have port forwarding set up and UPnP off. I checked the port via one of those test websites, and it says it’s open.

When you say check CLI Status, I’m using the GUI - is that the same as the GUI ‘Your Farm Overview’ where it says Farming Status… I’ll wait for it to happen but pretty sure that it still says farming and stays green.

Hey,

I configured it with 4 vCPU of the 8 available. The CPU seems to sit around 15 -30%, with 53% of the 16.6GB memory in use.

I do have it set up to run malware and virus scans, but, they were not running at the time.

Thats alot of ram being used.
I was having an issue where my ram was creeping up and causing response delays, a reboot fixed it.
Mine crept up over days / weeks though, not 15 hrs.

That really is alot of ram usage just for farming, of my 32gb, 25% is used, and thats on a windows full node with apx 7k plots.

I’m not sure the Qnap Visualisation panel is all that exact… just looked in the VM’s Window task manager, and it’s showing 2036mb of ram used on the Chia process and 33% in use overall.

1 Like

Thats much more inline with what chia should be using.

1 Like

It take it the plots on the VM’s NAS are also connected by samba-shares to the vm? Or rather the VM has samba shares to the NAS it’s running on?
Didn’t think a VM could have ‘direct’ access to NAS volumes with plots.
Maybe an easier, more resource friendly way would be to run chia node in a docker container, which can directly access plots on the NAS. Plenty of youtube tutorials how to set this up, even form QNAP itself.

Also if the other (40%) NAS is docker capable, run a chia remote harvester in docker there also.
This will solve any resource related issues if those are in effect.

Saves a Windows license also :grin:

Only thing (just realizing) it might be a hassle to get a GUI running/connected… Couldn’t find much info on it yet :thinking:

1 Like

I’m no networking geek; I just originally had all the spare disks on both NAS’s setups with simple shares. Then my old ETH miner PC just had those shares mounted over the network (in Windows). When I retired my ETH miner, I spun up the VM. It could see my network, and, therefore, all the drives from both NASs - I just mounted them.

There is, for sure, a look-up penalty with the extra latency with all that, but when there was no dust storm, and till now, it’s been fine - even found a block couple of weeks ago (yes, congrats and f-u etc :joy:).

I did look at CLI and docker stuff, all looked like a lot of work when I’ve never used Linux CLI stuff.

The second NAS isn’t powerful enough to run anything, other than basic Plex.

I got a free windows licence, so that’s not a factor…

About the router, maybe it gets overloaded by the number of incoming peers it has to do NAT for (long shot). Did you try disabling the port forwarding so you node would only connect outbound to 8 (or 10) peers?
On the other hand, if this was your setup with the other PC it should have been an issue then also…

1 Like

Disabling port forwarding does not limit number of connections, it only forces Chica farmer to use UPnP to setup port forwarding automatically. If you want to limit node clients you have to set that in farmer config file. I suggest looking at router CPU utilization and if it’s not 80% or higher then it’s likely not a problem with number of connections.

1 Like

True, if upnp is enabled on the router. Should have said that, thanks.
However if you don’t want to edit config files and router gui makes it easy to check/disable upnp it would be a quicker option. Disabling upnp in general is something to consider imo;-)

UPnP is disabled on the ASUS router. Just forwarding the port.

It’s okay at the moment, so probably not a fair test, but the router shows two CPU’s 5% / 18%, so nothing crazy. Mem at 44% 223mb/512mb. I’ll check again when it stops passing proofs. :slight_smile: