Syncing too slowly, full sync taking too long

@elysiumpool Could you stop spamming all threads with your service.

You offer no help, no understanding of the problems, just write some garbage and peddle your site.

2 Likes

So yes, unfortunately, this is normal. It can take that long (8-12 hours) to sync the blockchain the first time.

It’s been more like 8-12 days moving onto weeks. Both wallet and Farming Status.
I don’t know what else to do. Running version 1.2.11 on Debian. Deleted the files in the mainnet/wallet/db and then entered the 24 word and passphrase numerous times and nothing seems to work.
Watched lots of videos and tried their repairs, install the latest version of the software, which I have.
Port 8444 is open.

Any pointers.

It’s a couple of days that’s normal currently with the growing blockchain, but 8 or more is really bad.
If you delete the db files it will start from zero again of course but that’s not what you mean by 8-12 days I suppose!?!

What disk is your db directory on? HDD, SSD, NVME?
Syncing is very intensive for the processor, what is your CPU?

If all fails (and it’s not for lack of you having tried already!) you can download an actual blockchain db from a few providers these days.

https://elysiumpool.com/bootstrap/

for instance has this service.

Of course your wallet would still need to sync from zero and even that takes time, depending on your CPU power mostly but can be many hours (over 24 even).

The drives are HDD’s and the processor is a Ryzen™ 5 3600 Processor, 3.6GHz w/ 35MB Cache, 32 GB RAM, on an Asus TUF GAMING X570-PLUS.

Take a look at this thread - My sync remains 0 for last two days - as it is a bit related.

You need to provide more info / screenshots for anyone to be able to understand what is going on on your side and provide help.

1 Like



The pool shot shows a transaction that won’t complete and I think this is the problem.

1 Like

I would not worry about pool / wallet, as long as you are not fully synced.

Whether you have port 8444 open or not, it really doesn’t matter, as it is irrelevant for syncing.

Does it always show in the Full Node section as “Not Synced?” That implies that for whatever reason your node is not making syncing progress at all, so waiting will not help (will not fix itself). Although, on your Blocks section there are blocks that were processed on Feb 22. So, what happened that day? It kind of hints that your db may be corrupted (your box got rebooted while chia was modifying db?).

A screenshot of your Connections would help. If all Up/Down are at 0, that confirms that your node has stopped syncing.

Take a look at that other thread and follow instructions there to grab ERRORs from your debug.log file, as those most likely will point to a db corruption (whether blockchain or wallet).

Actually, as the wallet db is a secondary one, you can try to delete that wallet db (.chia/mainnet/wallet/db/*) and see whether it would help. If not, I think that all signs point to blockchain db corruption.

Your CPU would suggest that you should be able to sync in 2-3 days or so. Although, the media that holds your dbs (HD, SSD, NVMe?) could further slow it down (say 4 days).

Actually, do you still have room on your c: drive (assuming that your dbs are sitting on it)? Is that a HD or SSD?

[quote]=“Jacek, post:20, topic:1316”
Actually, as the wallet db is a secondary one, you can try to delete that wallet db (.chia/mainnet/wallet/db/*) and see whether it would help. If not, I think that all signs point to blockchain db corruption.

Your CPU would suggest that you should be able to sync in 2-3 days or so. Although, the media that holds your dbs (HD, SSD, NVMe?) could further slow it down (say 4 days).

Actually, do you still have room on your c: drive (assuming that your dbs are sitting on it)? Is that a HD or SSD?[/quote]
The drives are HDD. I have about 27 GB in the /home directory. I have deleted the wallet db numerous time from ./chia/mainnet/wallet/db

I’ll go over the other post

That is not the best setup. Once you get synced, if you cannot / don’t want to add NVMe/SSD, I would suggest that you split your dbs and logs across two / three different HDs (at least to separate blockchain db from both wallet and logs),

My blockchain db is around 71GB, and wallet over 10GB. For the past two/three weeks, the mem pool is elevated, and it looks like blockchain db grows faster than normal (was below 60 maybe a couple of weeks ago). So, you may be forced to move at least your blockchain db soon due to hot having enough room in /home dir

Sorry, I missed your original post where you mentioned it. I guess, all signs point to a corrupted blockchain db.

Maybe the fastest thing to do would be:

  1. Download blockchain db from one of those two places (should take ~30 mins or so)
  2. Shut down chia, potentially reboot, if not sure if it was fully stopped
  3. Move your original blockchain db to another drive
  4. Move that downloaded db to your .chia/mainnet/db folder
  5. Start chia

Keep your fingers crossed and let’s hope that chia will be almost synced, and will show syncing state (processing new blocks, peers with moving Down states). This last part of catching up is really slow, as that main full node process is basically hosed, and even though you have plenty of extra cores to crunch those blocks, they are starving, as that main process is really SOL and is not giving them enough blocks to crunch.

If it will be working, you may want to watch whether it is syncing all the time (processing blocks all the time). If it will be falling behind, and then catching up, you would need to trim the number of peers (free, and immediate result), and start thinking what to do with your blockchain db (where to put it to make it faster).

Kind of two things that are going against you right now is that elevated mem pool (more blocks to process), and really big blockchain db (slow access).

1 Like

OK, I downloaded as yuou stated and then moved the databases onto a different drive (250 GB free) and changed the locations in the config.yaml file. The wallet syncs very fast, but the Full Node was taking days and never did sync. I stopped the process and upgrade to 1.13.1 hoping this may help.

The wallet appears to be v2, while the Full Node database appears to be v1.

Started the sign on again at 1330 hours 22-03-18.

Sorry, but it looks like what I wrote may not be complete, or at least I don’t understand what you did. I asked you to:

  1. Download a new db
  2. Move your original one somewhere else (outside of your c:…/mainnet/db folder)
  3. Move the new db to c:/…/mainnet/db

This would let chia to pick up the new db from your original location, where your old one would be moved to some backup place. My understanding is that once you moved your original db to another place, you also made config.yaml modification to point to it. If that is the case, then the new db was not used. Although, it looks like this is already post mortem. Although, this may affected your config.yaml db locations.

I am not sure, but I think that during the upgrade, v1.3.1 couldn’t find blockchain db, and started from scratch. Although, it also looks like it started from scratch only for the blockchain db and defaulted to v2 db. On the other hand, it found your wallet db and upgraded it to v2. At least, this is what I can understand from what you wrote.

Maybe you could:

  1. paste from your config.yaml entries that specify locations of those dbs - something similar to:
full_node:
  database_path: d:/chia/blockchain/blockchain_v1_CHALLENGE.sqlite
...
wallet:
  database_path: d:/chiab/wallet/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
  1. Get two screenshots of file explorer, each pointing to folders where those two dbs should be (in the above example that would be d:/chia/blockchain and d:/chiab/wallet).

That should clarify what your db situation is right now.

Sorry, but it looks like what I wrote may not be complete, or at least I don’t understand what you did. I asked you to:

1. Download a new db
2. Move your original one somewhere else (outside of your c:…/mainnet/db folder)
3. Move the new db to c:/…/mainnet/db

This would let chia to pick up the new db from your original location, where your old one would be moved to some backup place. My understanding is that once you moved your original db to another place, you also made config.yaml modification to point to it. If that is the case, then the new db was not used. Although, it looks like this is already post mortem. Although, this may affected your config.yaml db locations.

I dowbloaded from https://elysiumpool.com/bootstrap/, then I copied it and the old db off of the main /home drive to a secondary drive with more room.

wallet:
  database_path: /media/scott/DS500Seagate/.chia/mainnet/db/wallet/db/blockchain_wallet_v2_mainnet_xxxxxxxx.sqlite
full_node:
  database_path: /media/scott/DS500Seagate/.chia/mainnet/db/blockchain_v1_mainnet.sqlite



Great!

sqlite should have 3 files, the way they are used by chia (main sqlite, index - shm, and write-ahead wall). More or less, all three files need to be current. You can look at wal as a tiny caching file for the main one. Once it reaches some threshold, it dumps those transactions to the main file (say every minute or so, so the date on the main file needs to be updated). The snapshot from your blockchain folder shows that the main file was not updated since Mar 07, where both shm and wal were touched today. One possibility is that the main file is being locked for use since that day (thus, the screenshot doesn’t really show what is going on), the other is that the blockchain db is corrupted and the process that dumps wal data is not biting. I would go with the second guess.

Your wallet db looks OK, though. However, I don’t know whether it is updating from your main node, or rather is looking for some peers (as we have db version mismatch).

Take a look at that post - How should I upgrade Chia v1 DB with boot drive almost out of space? - #4 by elysiumpool

@elysiumpool outlined steps to get it going (download v2 blockchain file and modify your config.yaml to be v2). Hopefully, that will get you going.

By the way, in my config.yaml, both dbs have “CHALLENGE_KEY” instead of the fingerprint. Maybe you can try to modify that in your config.yaml first (chia restart will be needed). You can copy/paste those lines and prefix the original with “#” char (no quotes, but keep the indentation same). (I mean, I would still download the v2 db, as keeping v1 doesn’t make sense anymore.)

i checked a couple of harversts and my farmers config.yaml and they all have the “challenge” statement not the numbers.

1 Like

Maybe this is not a problem for one wallet installations, as the code is substituting that ‘challenge’ for a current fingerprint, or takes it verbatim if that ‘challenge’ is not there. However, it may be that the code will barf, if the current fingerprint doesn’t match the one in config.yaml, as it cannot make such substitution. Or maybe this is where it defaults to an assumed naming format This may explain why even though v1.3 update didn’t modify wallet entry in config.yaml, but was still able to use the upgraded one (or maybe quickly siphoned a new one off the full node).

On the other hand, this is not a problem for the blockchain, as that is just one db for all wallets (nothing to do with fingerprints).

I updated chia to v1.3.1 about 30 mins ago on my plotter (i9-10900 / Win). One thing that I have missed in the above screenshots is that wallet db is right now in MB (v1 was around 10GB). So, finally someone realized that it should contain just records relevant to transactions, not 20% junk records from the main blockchain. This is potentially the biggest improvement / change in this version.

When I started this update, that node was 26 days behind (~30 GB worth of of blockchain data). The wallet updated rather fast, but the blockchain is struggling as before. I was hoping that this version will have some improvements in the main start_full_node process (dispatching one). However, it doesn’t look like there are any improvements. That one process is choking on one core, and the additional 8 start_fulll_node processes (that crunch blocks) are starving for data. Currently, the speed is about 1Mbps / 125KBps - what is basically exactly the same as before - rather pathetic. Although, it is still working on v1 db. (Once it syncs up, I will kill the db, and start over with v2, to see whether it will start moving faster.)

Also, I was monitoring both blockchain and wallet folders. The blockchain continues with v1 database. However, the wallet started right away with v2 db migration. It connected to 4 peers plus the local node. It didn’t pull any data from the local node - everything came from those external peers (nonsense). Looks like, it didn’t bother to even open my v1 wallet db (time stamp is still from Feb). There is no change in config.yaml with respect to v2 wallet db, so the db name is a default one that is hardcoded (why would you do that), and the location is derived from the v1 db path (I moved my dbs to d: drive, and the new v2 wallet is on that drive - c: drive has no db traces - good).

So, I guess, this answers the puzzle about whats/hows about wallet db with v1.3.x upgrade. The speed of updating that db is greatly improved, as there is no junk stored in that db anymore. The size is smaller by order of 3 magnitudes (just think what garbage it had to be before in that db to get such gains).

@Powderjockey looks like your wallet db may be in a good shape, as it was updated independently. Also, if it is broken, it looks like wallet rebuild will take minutes, so this is not that big deal to kill it to test things out. However, for some reason your v1 blockchain db is screwed. Just download that v2 db from Elysium pool, as otherwise you will be looking at 2-4 days of syncing.

1 Like

They spoke in Keybase that after the chia db upgrade, the config.yaml node db gets changed to v2 and the wallet uses the v2 db (hard coded in) but still display v1 for downgrade ability is what was said. You can change it to v2 if you know your not going to downgrade.

1 Like

@Jacek Thanks for the post. I placed the new v2 db into the drive and changed the config.yaml as required. Running 1.13.1 and the Full Node just hit Not Synced. Peak Height is 1,722,490. Wallet is synced at 1,722,534. My Plots were then showing as Not Available.

Short time later. the Plots show Farming, Full Node is Synced and the Wallet is as well. The Peak Height is 1,722,545.

Thanks for your help,
Scott

1 Like

Glad to see you sorted!