Lost my internet connection for 2 hours, and now syncing is dragging

[seymour.krelborn]

Greetings,

This morning, I lost my internet connection for approximately 2 hours.

I am now connected, and Chia is syncing. But it is slow, to the point where it is not keeping pace with new transactions.

After my internet outage, when I re-established syncing, I was behind by 384 units (you know, where the GUI reads “1,167,900/1,168/490” for the syncing status).

So although my syncing is climbing, the fully synced value is climbing away faster than I am climbing.

I have connections to over 50 peers that are fully synced, plus a handful of other peers that are far behind (see a photo of my screen, which shows one screen’s worth of connections).

Should I simply wait, assuming that the Chia tool will eventually get me synced up?

Or should I trash some of my connections? If yes, how do I choose the bad ones that should be trashed?

Or are there other, better options to get me synced?

Thank you.

[Aspy68]

Does not sound normal to me.

I’ve seen much discussion recently about too many peers being an issue causing syncing to slow.

Lsherring finally overcame his syncing issue after Jacek helped him reduce his peers to 20.

@Lsherring

Jacek is a Chia Stud! After four hours of troubleshooting, my rig is back to normal. There was never a network issue, we proved that. The resolution was to reduce the # of peer connections from 80 to 20. We saw instant impact for the better on memory and I got over a gig of download. The twenty peers eventually went to 10 as I was getting more than a gig of data. I highly suggest changing the peer count while synching. What took two weeks was done in a 36 hours.

I have no idea if this is your problem but it’s not a bad place to start.

[Jacek]

Which version are you running? What is your hardware?

Syncing speed depends on how fast your blockchain db can be handled. There are three factors to be considered: 1. underlying media (HD, SSD, NVMe), 2. number of peers, 3. amount of RAM (your cache).

Out of those three, the fastest one to try is #2. I would reduce the number of peers to 10 for time being, and see whether that will help. At the same time, if you are not on NVMe, I would start preparing to upgrade your box. Switching to NVMe is a long term solution, where reducing number of peers is basically doing a job that Chia should do for all of us (dynamically handle choking points).

All your peers look good (so you are not fully hosed), that at least mean that you are not fully choked. There is no point to “Or should I trash some of my connections? If yes, how do I choose the bad ones that should be trashed?” unless you are looking for a full time job competing with what protocol does 24/7.

Also, please read that thread that @Aspy68 suggested, as it has more info.

Although, I would also consider looking into Flex farmer. That is a superior product (IMO), but it gets some bad rap.

What we need to also understand is that as blockchain db grows every day; therefore, the problem is not going away, but every day will become more and more serious. Dust storm pushed network traffic a bit, but what you are experiencing is proving that the network is on the verge of just not being able to handle the traffic as is, with the current state of chia code.

[seymour.krelborn]

@Aspy68 @Jacek

I am running Chia 1.2.9
CPU: AMD Ryzen 9 5950x, fluctuating between 40% - 70% usage (due to plotting)
64GB memory, at approximately ½ in use.

OS and Chia app, including all block-chain, wallet, etc, are all on an NVMe drive, and the drive nearly never exceeds 1% utilization, and it has 379 GB of free space.

I run nothing other than Chia on the system.

I have two other NVMe drives, both of which are dedicated for use as temp space for plotting.

I do not think that my system has any “syncing” related bottlenecks.

[seymour.krelborn]

@Aspy68 @Jacek
I would like to try reducing the number of my peers (unless you talk me out of it).

Out of the ones that are fully synced, how do I choose which ones should be terminated?

Also, is there anything in my config.yaml file that needs to reflect my changing the number of peers?

Thank you.

[Lsherring]

Also give the peers time to connect and remap up. It usually takes about an hour to get up to speed. Patience…

[Aspy68]

@seymour.krelborn

Also, is there anything in my config.yaml file that needs to reflect my changing the number of peers?

I believe it is:
target_peer_count : xx

[Jacek]

@seymour.krelborn

I am running Chia 1.2.9

Well that would imply that you are the first one confirming what Chia stated, that v1.2.9 is just garbage, and may not sync back. Still available for download.

1. Stop plotting until you sync (if you are able to sync)
2. Drop those peers to 10, if still too much, go down to 6 or so (your wallet, farmer, and 3-4 peers)

See, whether that will speed up syncing. If not, you will need to reinstall chia (either v1.2.10/11 or v1.2.7/6). I am still on v1.2.6, and had ISP outage yesterday for over an hour. No problem with syncing up back to normal.

@seymour.krelborn

I do not think that my system has any “syncing” related bottlenecks.

Those syncing bottlenecks are just manifestation of bad code. (If you cannot sync with your setup, we don’t need Dust Storm II.) Therefore, I suggested Flex.

@seymour.krelborn

Out of the ones that are fully synced, how do I choose which ones should be terminated?

All those peers are good, and you should not worry about those. Just follow that config.yaml change that Aspy suggested, and it will take care of it with you being fully synced.

[Aspy68]

@seymour.krelborn

Out of the ones that are fully synced, how do I choose which ones should be terminated?

@Jacek

All those peers are good, and you should not worry about those. Just follow that config.yaml change that Aspy suggested, and it will take care of it with you being fully synced.

Terminating/Deleting peers in the GUI just makes things worse.

Jacek actually understands it, but simply for me, too many peers get in each others way. Once limited to a less troublesome number of peers, the torrent part of the Chia GUI takes care of it’s business best when left alone.

[Jacek]

@Aspy68

Terminating/Deleting peers in the GUI just makes things worse.

Maybe it doesn’t make it worst, just gives one 24/7 job that produces zero results.

@Aspy68

but simply for me, too many peers get in each others way

All those peers compete for blockchain db access, basically slowing db down to crawl. Also, chia is not handling those requests properly, so those peers keep repeating the same requests over an over. It is really bad code.

[Aspy68]

And, every time you delete a peer it just has to find and connect with a new one using the same torrent methodology. Just a small waste of clicks, CPU cycles, and bandwidth.

[seymour.krelborn]

@Aspy68

Terminating/Deleting peers in the GUI just makes things worse.

I already disconnected, via the GUI, every peer that was behind me – thinking that my resources need to focus only on catching up.

I also reduced the number of peers to 35, because after disconnecting peers, I was left with 35 fully synced peers.

I will now go back in to my config.yaml file, and reduce the target_peer_count from 35 to 8 (or maybe 10), as @Jacek recommended.

I will also not kick off any more plotting on this syncing machine (#1), to see if that helps (but I have another plotter (#2) chugging away, that machine #1 harvests (in other words: machine #1, my syncing machine, harvests both itself and machine #2 – both machines plot)).

On machine #1, I have plotting in progress. I will not start any more plotting on machine #1.
But machine #2 I will keep plotting.

Sound right?

[Aspy68]

But take the details from Jacek, lolz! I think 10 is your number.

[Jacek]

@seymour.krelborn

thinking that my resources need to focus only on catching up.

That is possibly the biggest misconception. Chia does not parallelize syncing downloads. It just gets data from one peer at a time. So, in theory, you just need one willing peer to sync up. To give those peers that you will be leeching from a brake, it is good to have 3-4 at minimum. Also, one of those peers may be leeching from you, but again, the problem is not really with one peer leeching from you, but all those peers generating db traffic.

@seymour.krelborn

I also reduced the number of peers to 35

Go down to 10.

@seymour.krelborn

Sound right?

So, what is your main problem right now? Is it plotting or syncing? Pick one and stick with it till your setup is sound again!

Just follow what both @Lsherring and @Aspy68 said - patience, and focus on one thing only.

[seymour.krelborn]

@Jacek
I updated my config.yaml “target_peer_count” value. It is now: 10

My only issue is syncing (although I have an old issue with not all of my plots being counted).
So I will let my current plotting complete (I have k33 and k34 plots in progress).

Between my lower peer count, and eventually no plotting, I will hopefully sync up.

But there has to be something wrong with the code. No one should have to stop plotting to remain synced.

Should I upgrade to the latest Chia version?
Should I fall back to a previous Chia version?

[Jacek]

@seymour.krelborn

Between my lower peer count, and eventually no plotting, I will hopefully sync up.

Your plotting is not a persistent problem, but rather a small factor for time being - while you are trying to catch up. Again, based on Chia statement, you may be SOL right now. So, what you are trying to do is just smooth things out while trying to catch up.

With the gap you have right now, if the process works, you should be up to speed in no time (say a couple of hours or so).

I would give it an hour or so to get things sorted out (with peers), and after that start checking whether you are closing the gap. Give it another hour or two, and if you are not closing the gap, maybe you need to abort, and reinstall.

@seymour.krelborn

But there has to be something wrong with the code

Not sure whether you are still thinking that it is not

Another part that is contributing to this problem is db choice. Chia is using sqlite that is not really meant to be run as high performance / high scale db. That db is also not that much multi-threading aware what makes it even worst.

@seymour.krelborn

Should I upgrade to the latest Chia version?
Should I fall back to a previous Chia version?

I would let it sync first, or have it prove to us that syncing is not possible. Once we know the outcome, changing version potentially will be the next step. As mentioned, I am on v1.2.6 (v1.2.7 should be also good), and have more or less zero issues (still suffered during dust storm). I see both v1.2.10/11 as not worthy to upgrade, as those were rushed ones. Although, folks run those as well.

One more thing to consider is that v1.2.10 added that password, so it may be more challenging to drop down to v1.2.7 if needed.

Again, I would add to that list Flex farmer, as that is rather a superior code comparing to what chia produces.

By the way, could you post a screenshot of your connections that you have right now.

[seymour.krelborn]

I’ll get the screen-shot. It takes a bit of time, because on my Chia systems, I do nothing other than Chia processing (to keep the systems clean). I do not browse the web on those systems. I am writing in this forum from a non-Chia, general-use computer.

So I have to grab a screen-shot, transfer the .png file to this computer (where I am typing), and upload it to this forum.

i'll be back

[seymour.krelborn]

@Jacek

By the way, could you post a screenshot of your connections that you have right now.

I was originally behind by 384 syncing units.
I am now behind by over 1,000.

And although I set the config.yaml’s “target_peer_count” to 10, and I disconnected all but 10 peers, Chia keeps adding more, anyway.

In the screen-shot, there are another two connections that were below the capture.

[Jacek]

And you get this after you dropped the connection count to 10? Did you restart chia?

You should have there just 10 connections (or less, as P2P is sorting things out for you).

That value is checked only at chia startup.

[seymour.krelborn]

@Jacek

And you get this after you dropped the connection count to 10? Did you restart chia?

You should have there just 10 connections (or less, as P2P is sorting things out for you).

I took two actions:

1. Updated the “target_peer_count” in config.yaml (and saved my edit)
2. Used the GUI to disconnect the extra connections.

Please note that “target_peer_count” appears twice in my config.yaml file.
I did not put the second occurrence there. It’s value is set to: 5

Should I restart the GUI?
I did not want to take any steps without guidance, due to the risk of possibly further complicating matters.