Flexpool not seeing all my plots

Jacek · November 2, 2021, 3:46am

That explains why we got results comparable to a single drive (spanned volume / jbod).

Looks good to me, and it should not cause any problems. Although, I would also schedule fsck, just to put that to sleep, whether there is nothing lurking there. Although, I would delay it.

(However, IMO, you should not use spanned volumes, as in case of any issues, everything will be lost, i.e., a lot of new plotting.)

So far, we know that your lookup times are not the best, but not at 80% over the budget level (5 secs). That kind of points to your satellite side. If it adds 2-3 secs, that would be the issue.

Could you ping flex host that is close to you (e.g., ping xch-us-west.flexpool.io)

olivierLA75 · November 2, 2021, 3:49am

I’m not familiar to the ping thing, could you tell me how to do?

Jacek · November 2, 2021, 3:52am

That’s it. If you are not in US, you will need to change “xch-us-west” to something that is closer to you.

Here are those hosts:
xch-de.flexpool.io
xch-us-east.flexpool.io
xch-us-west.flexpool.io
xch-sg.flexpool.io

Actually, you can go to this page below, and it will run those tests for you::

I have about 30 ms according to Flex’s website, but only 13 ms based on raw ping. Either one is good. I have les than 200 to both Europe and Singapore (to give you a reference).

olivierLA75 · November 2, 2021, 3:56am

I’m in LA

olivierLA75 · November 2, 2021, 3:59am

olivierLA75 · November 2, 2021, 4:00am

average is around 28 ms but there was one with 106 ms

Jacek · November 2, 2021, 4:01am

Yup, Wilshire shines as usual.

Just to clarify. Those timings are from your office, using your satellite? Is that really a satellite or just a dish pointed at Wilshire direction (or Verizon, etc.)?

One off from time to time is nothing to worry. Also, we are looking for 2-3 seconds, not really 100 milliseconds.

olivierLA75 · November 2, 2021, 4:04am

it’s this company called Wiline, it’s satellite internet for businesses, it’s not cheap I think we pay $1800 a month but we’re having some issues with it, service is up and down sometimes. We’re supposed to have our own At&T fiber installed soon but they are taking forever

Chris22 · November 2, 2021, 4:05am

Consider Flexfarmer as we corrected a lot of the errors that cause stale partials. If even flexfarmer doesn’t solve it then you got serious issues.

olivierLA75 · November 2, 2021, 4:06am

I was actually thinking of trying it

olivierLA75 · November 2, 2021, 4:07am

now the pool thinks I have 6Tb of plots when I have 500TB +…

Jacek · November 2, 2021, 4:07am

Take a look at lookup times up there. Those are not the best lookups. So, this is the first problem. However, even from 1-2 sec lookups to get 80% stale partials that means something needs to be wrong, and to me maybe not on the box.

Jacek · November 2, 2021, 4:16am

Would be good to see that chart as well. Not seeing that, it kind of points to two issues. First is some heavy, potentially network issue, as dropping your space means that you are not responding to those challenges, where the second one is that you are getting your results late, when everything else seems to be OK.

By the way, I would keep that ping alive 24/7, so maybe that could show some network hiccups.

To me, your setup sounds fine (so far), although we know that it is not, and we cannot pinpoint it. Switching to flex is good option, but doesn’t really answer what is amiss. So, I would still rather bang on it for some time.

Don’t you think that you may have some internal router problems, where that sucker thinks that your box is an issue?

Before giving up, I would still disconnect those volumes, and run chia for some time, to get results for each individual volume.

Also, maybe this is a good time to start parsing your logs? Although, I am not that good at that. I mean, the obvious thing is to cat your logs, and grep for ERRORS, maybe up that to WARNINGS, and see what that will give us.

I will need a break - need to get some food.

Also, maybe tomorrow, you can ask your net admin to run some monitoring tools (pings) to get some daily charts. Something like Nagios. Just running manual pings is good, but doesn’t really show the trends, if one misses it.

olivierLA75 · November 2, 2021, 4:29am

Will look into this and thank you so much for trying to help. Have a good evening!

Chris22 · November 2, 2021, 3:47pm

Thats Flexfarmer? Flexfarmer supports reported TB so I’m assuming it isn’t. With that many stales there are serious issues with your setup which FF may solve but no promises.

olivierLA75 · November 3, 2021, 2:12am

Well, I think I need to give some credit to Chris and Flexpool now, switched to Flexfamer an hour ago and here are the results on the right, still don’t understand why this happened though

Jacek · November 3, 2021, 2:49am

I think we know what happened. Chia’s code is just crap. As I mentioned already, everything that we touched was sound. We were trying to split hairs, to find something that was not there. What a bummer. Not that long ago (few days) @Aspy68 stated that Chia’s priority is not us farmers, but rather banking. We did our part, we elevated XCH price well above what they estimated will be ($20). Now, looks like we are not needed anymore, the demo code should be good for us.

It would be really, really helpful, if you could take the exact same screenshot in an hour or so (to have more Flex data), and put a red mark roughly around the time you switched to flex, and posted it here.

I think that this is the best example how bad the official full node is. So many people are wasting their time trying to fine tune stuff, but this one takes the cake.

I would also encourage you to go to github Issues page, and post that picture there, also stating that you wasted countless hours chasing hardware, networking issues, and it was leading nowhere. A simple switch to a properly written node basically put all their (Chia) FUD about underpowered nodes to rest. Here is that github page:

Really, to me, this is one of the most important threads for really long time.

As you said, kudos to to @Chris22 and Flex. This is what code written by professional devs does.

olivierLA75 · November 3, 2021, 2:51am

Ok I will be posting screenshots often

Jacek · November 3, 2021, 2:53am

We know now that there were no issues whatsoever. His setup was as sound as it could be. Your code didn’t fix anything, rather your code works! Again, kudos!!!

olivierLA75 · November 3, 2021, 2:53am

this is the same worker (I renamed it on Flexfarmer), this is just insane! 84% stale vs 0.04% stale!