Network topology, am I doing it wrong?

Chia Farming & Harvesting
1

Hi,

Let me start with saying I’m not tech savvy at all, and I’ve used windows/osx since forever.

So I’ve been pondering the networking topology needed to for an efficient chia plotter-harvester-farmer-full node network.

I have 1 plotter pc currently (an i7 7700) which can churn around 10 plots a day, still optimizing for better results.

But I have several old i3 and a macbook pro 2017 13 inch that I want to use for all this since I don’t want to invest heavily into a plotter machine, I’m thinking that I’d better use the capital to get farming HDDs instead.

What I want to do is
Use 4 Plotters that I’m happy enough if it churns 40 plots/day since my investment would mostly be the SSDs.
1x i7 PC
2x i3 PC
1x Macbook Pro

And then run another older machine with an i3 3230, 6GB DDR3 RAM for the Full Node Farmer and Harvester all in one, if I use PCIe to SATA adapter, I can add up to 8 SATA HDD inside?

Planning to give this machine a 400w 80 gold PSU to plug in more HDD arrays via USB 3 later

I do have another 3 i3 2120 lying around with 4GB RAM if I need to use it for a separate harvester.

Picture attached (forgive the impeccable drawing skills and even worse handwriting)

IMG_6892
IMG_68922085×2290 1020 KB

Flow is :
4 Plotters putting temps on Smaller SSD
Robocopy it through the network to the Drives on the Main Machine, 1Gbps would mean that 1 plot being takes 15mins to copy, with network congestion that would mean my 40 plots would take 10-13 hours to copy since hopefully having their own network would reduce the network load.

Questions :

  1. is this network topology doable?
  2. I plan to use Ubuntu server for the full node machine, I have no linux experience as I’ve mentioned in another thread, will this be a problem?
  3. Will the filesystem present a problem when copying files with robocopy from windows to NFS mounted with linux? I think I need to use Samba to do that, is that correct?
  4. Do I need a separate harvester? or is the main machine powerful enough to do all the full node-farming-harvesting as well as keep tons of HDD there?

Thank you in advance, people.

2 Likes
2

1 GbE interface is far to enough since you have 4 plotter there, try add a 2.5gbe usb network card a least, and an 2.5gbe switcher.

3

Hi, thanks for the reply.

in total they’re only (hopefully) churn 40 plots a day, will that still be too much for the 1 GbE interface?

My backup plan is to use SFP, but that would definitely wont include the macbook as the SFP to USB converter costs a fortune.

If that happens I might run a harvester on the macbook and not just the plotter, and then move the USB HDD manually to the main machine harvester, I do hope this is possible…

4

Now I’m wondering, does copying files over the network eat CPU and RAM resources?

5

With your plots staggered then yes it would work just fine since there are 96 15 minute time slots in a day :slight_smile:
However if I were you I would put your farming disks into your plotters until they are full (except the mbp of course) and them physically move them. Just share them and mount from the full node machine so you can still farm as they are created.

2 Likes
6

Thanks for the idea, yes with the plots staggered I do believe that moving the 40 plots in a 24h timeframe through only a 1GbE dedicated network would be possible.

I am a lazy bum and wanted everything to be done automatically so I won’t have to do the manual labor of checking whether drives are full and would like to avoid shutting down PCs so that they can run almost forever to do the plotting at least until the main machine drives are full.

Yes I’m considering the physical movement as well since I’m familiar with the concept of a plotter and harvester only machine talking to the full node.

But I’m still unsure of how to do it since I haven’t implemented them yet.

What I understood is windows… but I’m eager to learn linux since I’m building new plotter+harvesters anyway.

I’ve watched these guides both from the wiki and youtube and think that I should be able to do it on windows, while on ubuntu perhaps it needed more learning curve
is this what I’m supposed to do? basically close the GUI don’t enter the private key and only use the CLI to plot and harvest as per the guide?

Wait, before that is my i3 3240 3.4Ghz with 6GB DDR 3 RAM enough to run the full node main machine? I mean it’s old but I think with 2 cores it’s still more powerful than a raspberry pi 4? I do plan to run Ubuntu Server on the main machine… but looking at how already complicated things are for my it looks like I’ll be running windows machines for now :sweat_smile:

1 Like
7

I did this previously with 10gbe network, but finally 2.5gbe just working fine for me, and maybe 1 GbE is still fine if you have just 40 plots a day.

1 Like
8

that is reassuring, thanks, does the copy process put a burden on the proc/ram by any means?

9

I don’t understand why people don’t answer the questions about the harvesters LOL I’m still so confused about that too…

I have the exact same topology as you however, this has a single point of failure if your full node goes down you are fucked… so Im trying to figure out how to configure multiple full nodes pointing to a NAS

1 Like
10

thanks for the reply.

Yes I was waiting for someone to answer that part too, but this forum gets a lot of traffic yet little replies, I think because of the nature of the forum to be helpful and friendly, so no /r/chia trolls here

perhaps I should include memes, pop culture references and use snide remarks to the replies to add more controversy and get more traffic to the post?

that said, the question you’re asking has just dawned to me, I want all this to be semi-automatic so basically something that I will go to when maybe something doesn’t work or adding more disks when they’re full.

Is there a technical solution to the problem of single full node failure means the end of our farming story?

I need to go maybe I will write a more detailed concern after :laughing:

11

[Chia Farmers-only need apply! - YouTube](Farmers only install explained)

I go through a “controversial” setup, however explain clearly the reasoning based on the actual documentation.

IMG_9999
IMG_99994032×3024 3.89 MB

1 Like
12

Wow I just saw this post, thank you, I’ll get into the video soon enough

1 Like
13

I just saw your video.

This is really an interesting way of seeing the network architecture, although this doesn’t answer your initial question on what if the full node goes down, or was it meant as an answer?

And, having multiple farmers seems like it would relay the one challenge from the full nodes several times to the harvesters, is this a correct way of imagining it?

I’m still questioning the benefits though, will it only be syncing speed?

Or will one full node that’s connected to multiple farmers challenging the same harvester multiple time gets more chance to win a block?

If that’s true then perhaps I’ll turn all my old android phones into chia farmers just for that :rofl: install a super lightweight linux and then run dozens of farmers 24/7 to the same harvester+fullnode

14

Signing of with “Chia Later” :rofl: that one cracked me up…

On a serious note… how to start a farmer only without GUI on windows?

15

Chia start farmer ?

I guess that would be where to start, or maybe write -h there for more help

I do love the Chia later signoff

16

Here is my current understanding, please critique:
On the premise that farming - finding a proof of space for a block challenge - is NOT a race against competing (external other) farmers but only has to be completed within the given 30 sec time-window (practically in under 5 sec) , multiple internal farmers would only have a benefit if the capacity of the main farmer is not able to keep up with the challenges as they come from the full node. This should be a matter of looking at the lookup times in the INFO/debug.log. Even if this were the case, it appears unclear how (in a predictable / deterministic way) each farmer would pick / get the challenges that the other(s) missed while being busy.

Instead, in the above topology with multiple farmers, it appears that multiple farmers likely will all process the challenge for the same block and task the single harvester to check all the plots for proofs - in other words 4 farmers ask 1 harvester to lookup for the same challenge 4 times?

The question that bugs me is, the reason why not to run multiple full nodes and on the internal network. With upnp disabled on all but one, their cross-chatter should be silenced, and all nodes should be independent from each generate block challenges - it is the full node that syncs the blockchain with other full nodes, and each node will try to add new blocks, hence issue challenges to connected farmers for proof. Ideally the more nodes one would control the more independent attempts to create/ win a new block should be processed.

I would sencerely wish for more details on the protocol from the Chia team to address our collective speculation.

17

I posted the PS commend in the description of the video

START "" powershell -NoLogo -WindowStyle Hidden "C:\Users\derek\AppData\Local\chia-blockchain\app-1.1.6\resources\app.asar.unpacked\daemon\.\chia start farmer-only"

I’m going to be doing a follow up hopefully today on this as well some other interesting testing I’ve done regarding affinity weighs and optimization per thread

1 Like
19

Ive done the followup video and the theory doesn’t stand up

Farmer only follow-up