Having stales but log nothing showing anything?

getting some stales last night when I was asleep, wake up to find the log has no ERROR in it and log goes back to at least 48 hours.

what can be the cause of this?

Dust storm and/or low end hardware

1 Like

Yes, there was a miniature dust storm yesterday. Actually, today mempool size is also elevated (1/10 of what was yesterday, though).

However, saying that, I am leaning toward thinking that during those events, the pools may be the sources of those stales.

I was watching my node yesterday (when I noticed that storm) and there was nothing special. My node was running roughly at normal CPU load all the time (i5-8250U), no load on NVMe hosting blockchain db, and most (except two) connections were fully synced and exchanging data with my node. It looked like another boring day. However, my stale ratio went from below 0.1% up to 0.5%.

Looking at the top miners at my pool, my stales correlate rather closely with those other farmers (both as far as percentage as well as roughly times when they occurred). Maybe that would imply that nodes operated by pool owners are getting behind, and I would imagine that those nodes are no slouches.

1 Like

yea that make sense. I didn’t know dust storm can cause this, I went from 0% stale to 0.3% yesterday then back down to 0%

I also don’t know what the cause for stales could be on the pool side during those storms.

I checked again Flex and Space pools (top miners for stales). All top farmers for Flex have stales around 0.3% (Flex reports the last 24 hours). On the other hand, Space has some farmers that were not affected at all (reports one week).

Someone posted some time ago a page with how well various pools handled dusts storms, and if I remember right, Space was at the top (more like not affected at all), where Flex had some issues.

I am not trying to say that pools are the only source of stales. Of course, if some node cannot handle those storms well, most likely such node will be the source of those stales. That may actually be what Space pool top miners show - some had those stales, some didn’t, as such maybe those that did couldn’t keep up.

@Chris22 Is there a problem with Flex pool right now? Since yesterday (Thursday mini storm) the number of stales jumped to around 0.3% looks like for all Flex farmers. Looking at the distribution of those stales (Flex’s top miners, and myself), everyone has them roughly at the exact same time.

Thursday’s mini storm lasted several hours (around 12 or so). Right now is about 1/10th of that level (still being a bit elevated).

Dust storm affects even our hosted node slightly at times. Alex was doing a lot of programming today to counter it.

I’m not a programmer so I could be wrong but I believe because we have so many partials that if farmers have issues it’s obvious and you can see them pretty accurately while when there’s less partials sometimes the pool may not see if their having issues.

In the end the solution is a better node. Ours is now synced and now on to improving speed and reducing resource use.

Did Alex make any progress with those stales?

Could you have him check those stale partials again. It used to be that they were on the below 0.1% level. However, for about a week / two (maybe since the mem pool got a bit elevated), they are around 0.2% - 0.3%. Also, if you compare virtually any two farmers, you will see that those stales are coming at exactly same times for all farmers. Comparing that to mem pool rather doesn’t suggest that it is due to the network issues, rather everything points to Flexppol being the source of it.

It is potentially not a big deal right now, but maybe understanding and fixing it now, when those problems are there present all the time will also fix the problem when things get denser (next dust storms).