That to me means that this is not a real error (e.g., shit happened), but rather the system cannot handle something beyond that point. Basically, the only task that his box is doing is syncing (i.e., no plots, just clean start). In that case, the offending factors would be either the ISP side or local. Assuming that it would be the ISP, we should still see some data trickling, and we don’t see it. So, we are left with the system. On the system level, you have two main processes involved. The one is handling getting data from all those peers, the other is channeling that data through the db. Again, assuming that the peer handling process is at fault, we basically don’t have much traffic there, as chia is handling only one peer at a time, but doing it in a round robin cycle. So, we can assume that this process has data to feed to the db handling process. That puts us at the db not being handled right, and basically screwing synchronization with that peer handling process. At least, this is how I see it working. Those two suggestions (NVMe, lower peer count) have slightly different implications. The first one (NVMe), lets chia handle db faster. The second one is lowering issues related to synchronization between those two processes (but it also indirectly lowers db reads/writes as for every peer, chia needs to go through blockchain db).
Actually, considering those two suggestions (NVMe and peers), assuming that it would be just the first problem (slow drive holding blockchain db), that would also manifest in that drive being choked most of the time. I don’t recall anyone stating that (and that was also not the case on my box). That further points to the synchronization code between that peer handling process and db updating process. Reducing peers is removing some stress exactly there.
If that would be the case, chia would not be able to connect to those peers. We see it connecting, and we see it working up to that ~500 level rather fine. So, that would not be my choice of what could go wrong.
Again, his box runs fine to that ~500 level, so up to that point everything is sound. It implies that there are no other interferences, as the syncing process is rather linear as far as received data. The only thing that is non-linear is blockchain db access.
That you for that, but I am also not an expert here. We are all like those blind people touching an elephant.
One hope for me is that Flex will have a full node potentially soon, and to me that would be the way to go, as I see no hope that chia will address those issues.