Better Harvester client for XCH

Hi Support,

Possible to come out more simple harvester client for Full node chia. I unable to running a full node with > 30k plots.

My scenarios like this.
Solo Plot : 7k plots
NFT Plot : 26k plots

I got 4 machines (PC) running 1 full node & 3 harvester. My biggest issue was chia client was unable to handle > 3k plots … the speed too slow

I think its your pc not the client.
I have 1 pc, its apx 10 yr old, fx8350 8 core 8 threads, handles full node and > 7 k plots.

1 Like

i using i7 Gen10 with 64 gb ram to farm. Issue was I still unable bring the to 5 sec in every time.
other 3 machine was 5950x not more than 10k plots is working fine

the issue i have now master node got more than 10k plots … its will submit late e.g 6 - 7 second.

I seriously doubt that would stop a win, but completely understand not wanting it to happen.

Id have thought a 5950x would handle all plots fine, but it seems im wrong.

How are they all connected, internal drives, jbod or what?

1 Like

Chia harvester is very efficient and using 10gen i5/i7 or 5950X for just harvesting is massive over kill. I have multiple harvesters with few thousand plots each running just with an ARM SoC that has only 2G of RAM on each harvester and have no problem staying <5sec. Your issues are likely on IO latency accessing the plots and not CPU power or amount of RAM. Review the logs for warning messages indicating scans >5sec to see if they come from any specific drives. If you are having slowdown across the board then it is likely due to drives going to standby which will case >10sec delays and a lot of ware on your HDDs. You can control drive power savings mode with tools like openSeaChest.

2 Likes

Are you storing your plots on the C:\ ? And is your C: a SSD?

1 Like

You are not wrong, but you already knew that. As dctech says, “…massive overkill”.

This pure and simple a lookup issue related to connection, drives, sleep, but not CPU.

Sleepwalker, what size are your drives? What connection USB2,3, sata, sas? Any USB hubs in there? Do you know which drive(s) are slow, or is it any and all?

2 Likes

@Magicwalker is your issue the 5+ second lookup times being reported in your debug log?

I had the same issue, when I gave my harvester too many plots to handle.

I was still winning XCH close to my estimated time for winning. But I was uncomfortable with seeing those “risk of losing rewards” messages in my debug log. And those messages were all over the place, making it difficult to see if some other important message showed up in the debug log.

I was planning to re-plot from K32 to higher K values. Upon doing so, my plot count dropped, and the 5+ second messages were no more.

I do not know what the number is… but a harvester starts to have trouble when handling too many plots.

My system that had that issue is a 5950x running Windows 10 Home.

1 Like

I have found that a harvester can handle a very large amount of plots … if they are all on the same device. If the harvester has to close a connection to a device and open a connection to another device to continue it’s query, that is when things go badly. For example, if you have a group of 1000 plots on an attached storage array and then another 1000 on a NAS device. If you put those 2000 plots on one harvester, there will be a problem. But if all 2000 were on one device or the other, no problem. I eventually just setup a harvester for each device. And those harvesters can all run on the same system. You just can’t group the plots on one harvester. See the thread below from back when I did all this testing. Hope this helps.

1 Like

I have 2.3PB from 5 remote harvesters over vpn. No issues. If you are doing network shares you will quickly run into problems. I also run 8 forks on all my harvesters and full node.

I noticed on an older 8 core cpu that i ran into problems running full node, since going to 16 cores and an i9 it handles everything fine. There is 100% a correlation between plots and forks and the cpu you need.

Also try changing your logging level, you will kill your ssd pretty fast from all the logs.

1 Like

I use an i3 gen 10 on ubuntu with 16Gb and 256 M.2 and no issue with farming close to 1PB and its uses only less the 20% of the processor.
I recommend using Linux for farming because is more reliable, with no freezing…

2 Likes

yes, 5+ second lookup time very trouble some. After i split more machine (harvester) bring down the lookup time between 0.XXXX - 1 sec. The rate of winning block was every day. My number of plots i have was 33,000 plots. Now i got more machine to run each machine not more 3k plots

all my plots is K32. Its seems lookup time very impacting winning block rate. My solo plot 7,069 now look very effective at least now alway belong ETW .

system no freezing, just checking the logs lookup rate is not good its impact the wining rate.

Does it, iirc ppl have got a reward with 25s response, if it wasnt 25, it was def 20.

all hdd drive was physical connected to PC. Only issue was lookup PC having problem when i got 10,400 plots in single PC.

Just hope that having better harvester client to splitting the look up … e.g. Now the limitation of chia client was unable run multi instance in same pc and its very heavy harvester. Just need to simple harvester client like execution exe or .sh shell script to run a harvester (light weight)
Like

  1. ./run_harvester.sh config_first_3000_plots.yaml
  2. ./run_harvester.sh config_second_3000_plots.yaml
  3. ./run_harvester config_third_3000_plots.yaml

The harvester is not heavy at all when ran alone. And you can run multiple instances using VM’s. It isn’t the load of the harvester causing the issue. I tested this and my current setup proves this. I agree, there should be a way to run multiple instances of a harvester like you mention. But that is not going to happen any time soon. I recommend being the “wise man” here and take advantage of the work others have already done. Spend an hour or so loading and configuring and your problem will be solved. Then you can move on and enjoy a smooth running farm. Good luck!

Flexfarmer is one solution to this for your NFT plots. You can run multiple instances of it if you have too many plots for one to handle. Our largest farmer has 10.2 PB running on FF. It supports docker and almost all hardware including 32bit.
It will also help solve I/O latency issues and reduce your CPU use to minimal. We also allow you to set your difficulty (for all users not just FF).
Example

CPU+RAM use (this will increase somewhat as plots increase)


Farming 1PB of Plots, Startup takes 1 Minute

That is a proprietary solution for one pool. Not an option in my opinion. Nice product, but I’m not a fan of the limitation.

1 Like

That why i like flexfarmer … i running multi instance in same pc without any issue. So far only flexfarmer able to solve my plots look up time.

Now i very happy … mostly 1 - 2 days I will be winning 0.25. It help me win 0.25 and i also happy sharing 1.75 with flexfarmer pool miner. If anyone in big miner in flex. May be try out open multi flexfarmer client it will reduce the look up time and increase winning rate.

2 Likes