Flex pool - does it have a bad luck, or something just happened to it

Jacek · April 30, 2022, 3:35am

It looks like Flex luck is going down to drain in the past couple of days. Looking at their block page, I have two questions.

The first is of course what is happening right now. The last block took 4:44 (four hours, 55 minutes) instead of average / expected 20 minutes. That is 15x behind the schedule (likelihood of that is basically zilch). The previous is not that far behind - 2:41 (8x behind), few previous blocks are in 1 or 2 hours range (3x or more). On the other hand, there is not that many as expected or better to compensate for those delays (for the past two/three days). Comparing that to Space pool, their number of blocks found each day is fairly consistent and there are no such drops in the past few days, rather to the contrary, today they are above average. Sure, Space is 10x bigger, but this is not waiting for 1 block over 4 months, but rather instead of getting around 70+ blocks per day, it is less than 30. Also, comparing that with OpenChia pool, they also don’t have such decline in the past few days, and they are 1/10th of Flex, so should have more fluctuations.

The second question is about consistency over longer periods. Space is maybe not so much flat, but rather with small variance (less than 5% on average, with only 1 outlier over 10% over the past month. On the other hand, Flex has 20-30% jumps all over the last month. Sure, OpenChia fluctuates a bit more, but their pool is only around 50 PB, so on average 8 blocks or so per day, so just shifting one block around midnight shows as a decent jump.

And I don’t think this is related to Flex’s bigger farms having problems. I went through top 10 list, and there are no drops in farm sizes, nor increased number of stales in their collected points, all are basically flat as it can possibly be.

Anyone else is seeing their earnings dropped for the past few days?

huntingground · April 30, 2022, 5:46am

https://www.xchcentral.com/chia-pools

ludodivx · April 30, 2022, 5:57am

I asked them on twitter if they were aware of any issues and they answered they thought it was pure back luck but were still loking into it.

Let’s see…

Jacek · April 30, 2022, 5:59am

Actually, I have exchanged few emails with them about their stats. Most likely, they will need to look at what they have there right now, as there were some obvious issues. So, I would not yet put all my money on those results yet (I am waiting for an update from them).

Jacek · April 30, 2022, 6:38am

Yeah, bad luck is an easy way out.

If you take a look at the last three results, those were 8.75x, 15.36x, 5.5x behind. To put it in a perspective, if you had a farm that should be getting 1 win per month, you would just get 3 blocks if almost 30 months. Do you think this calls for a bad luck, and if so, wouldn’t you be doing some checks while the first block was behind (already a couple of days in this case)? Those were just 3 last blocks. The whole day is like that, what makes it even worse.

findchia_support · April 30, 2022, 8:57am

I think the problem might be in a so-called “dust storm”, it’s affecting us a little too.

Ronski · April 30, 2022, 9:42am

My estimated monthly earnings are around 1.75 XCH a month with 211TiB of plots, it fluctuates up and down, but has been around that since beginning of April when I finished plotting. That estimated amount is in line with what calculators predict once you add the 0.25.

My luck at winning blocks has been awful, I went six months roughly between wins when it was estimated to be two months, I did win twice in quick succession in March though, but nothing since.

I’m using the standard Chia GUI to pool with an ETW of 1 month, so am now a couple of weeks or so overdue a win.

Jacek · April 30, 2022, 3:20pm

It looks like they have identified the problem and fixed it. Looking at the current results, since they fixed it all wins are coming with basically less that 0.2x delay. That means that the wins were coming at normal rate, but were not processed basically for those couple of days, and are turning up right now. So far, there is about 100 blocks like that in the last 8 hours, where the average daily is below 70 per day. IIRC, they had the same problem during one of the previous high-intensity storms.

So, the bottom line is that it looks like they run their business blindly (not having any active monitoring), and are only reacting when people bring problems up. Still trying to weasel out of it by blaming the luck.

By the way, this ongoing dust storm stopped on Apr 26th, so before this happened.

UPDATE
After about 10 hours of processing block won in the past couple of days (one block every 2-3 minutes, where the average for Flex is ~20 mins), looks like it is back to normal.

So, I would have two questions. Whether the problem was solved, or just a box got rebooted and we can expect the same in the future. The second is whether an active monitoring will be implemented, so we don’t need to wait a couple of days to get Flex to wake up.

Digital · May 1, 2022, 10:55am

Flexpool have some of the most competent developers in the crypto space, they do not run their business blindly. The discrepancy was noticed and investigated, their pooling reward claim system lagged for some farmers and once the issue was identified these rewards were promptly claimed and issued.

Jacek · May 1, 2022, 8:21pm

Sure, we can wave our hands and make some unsubstantiated claims, or we can look at the data and draw some conclusions based on that.

Flex processing blocks:

Flex’s reaction to being notified about the issue

Space Pool luck

If you take a look at the first chart (Flex’s block processing), you will notice that on Apr 30, there were 3x more blocks processed than on average. I was monitoring it, and it started at around 3AM Pacific, and run till about 1PM that day (about 10 hours). As that is 3x more than average, it implies that those blocks accumulated for over than 2 previous days (as those days still had some blocks coming). This implies that the problem started on Apr 27 or so. So, here goes your claim that “they do not run their business blindly. The discrepancy was noticed” Flex started to look at the issue when @ludodivx notified them via twitter (as that tweet was made several long hours before Flex reacted).

If you take a look at Flex’s tweet, you may notice that the first reaction was that “It’s likely just poor luck” and further digging in by trying to use some other / irrelevant event to justify that. Not really stating “Give me some time, I will get back to you as soon as I have more info.”

Your another statement " their pooling reward claim system lagged for some farmers" also doesn’t hold much water. There was no “for some farmers,” as those were stalled blocks, and no rewards were given for them to any farmers.

As Flex didn’t process basically 2 days of rewards, I stand by my claim that they do not have any monitoring of what is going on, but catastrophic failures. They are the second biggest pool, and could check Poisson distribution for roughly how long one block can be delayed, and should be able to react to such problems in basically 2-6 hours, not ~70, and only when pinged.

As far as your ‘Flexpool have some of the most competent developers in the crypto space, they do not run their business blindly.’ statement, ‘having the most competent developers’ is rather orthogonal to how business is being run. Also, if you check some of my previous comments (from other threads), you may notice that I gave them a bit of credit for the work they did on FF, so I am aware what they can do, and this whole thing is not about that.

Lastly, I have also attached a screenshot from Space Pool. You may notice that their average luck is around 102 and the difference between the best and worst luck around 13. On the other hand, if Flex’s page is right, their luck is at 110%, and in April, there is plenty of ~50% up differences in luck / blocks processed. I am not really sure, whether this can be attributed just to a smaller number of processed blocks (smaller pool), or it is also influenced by Flex not really handling the current dust storm properly (what was already brought up, but brushed off by Flex). Someone posted some time ago various pools performance during dust storms (before this one started in Feb), and Flex was rather in the lower end of those comparisons.