Above is the only reference I can find to somebody actively backing up their blockchain database every day. I’m also doing this now after experiencing a corrupt database and recovering from another node that I happened to have: Node stopped syncing - #31 by enderTown
Why am I concerned? Because the blockchain database is closing in on 3gb. It takes a LONG time to sync from scratch - this isn’t just download 3gb, your node has to download and crytographically verify each block in order after it has consensus on that block from several peers. Has anybody sync’d from scratch recently? My guess is that it will take well over 24hours. If the drive hosting your blockchain database crashes, you’ll be out of the running for winning blocks until you are caught back up!
My questions to the experts are:
Is there an upper limit to the blockchain database? Does it “turn over” at some point? At what size? Am I correct about the reasons for the slow synching or can this be improved by a faster connection, faster processor, more peers, etc? What is the best way to back up this database file? Does sqlite have built-in backup options for a in-use database file? Or should we just shut down the client, backup, then start back up again? Maybe running two nodes locally IS a better setup for local blockchain redundancy? Just be sure to turn UPNP off on one of them? Or will this decrease the quality of the network/netspace?
Well that definitely rules faster connections out - I assume connection speed is not an issue at your DC!
Syncing feels like it probably won’t ever get faster by design. I suppose the whole point is for each node to independently verify the chain block by block, including consensus, and that is just gonna take time. Unlike the plot files, I think this blockchain database file does deserve some disaster recovery planning and strategy. Syncing from yesterday’s backup is always going to be way faster than starting over!
This DC we are connected at 100 megabit so max 10mb/sec throughput. I am trickling some plots on there over the weekend via Google Drive to fill some “holes” in the drives from bad plots, kind of literally weeding my garden lol
Run a periodic job to scatter it across all your farming disks and/or all your remote harvesters - backup job done. In the event of a rebuild, you don’t have to sync from scratch.
Just make sure you have ample space on those farming disks. I got 51g usable space on my farming disks so it’s “still” covered.
If you do have remote harvesters, you can set them up as standby DR full nodes by again periodically copying the blockchain db to .chia/mainnet/db. You only start harvester processes in remotes so it doesn’t matter if you have another copy on the target location.
It’s all about setting things up to failure and how to recover quickly in the event of a disaster.
I backup my DB and WALLET folders once a week, cause if there’s ever a time I need to start from scratch it’s those two folders that take the longest to ‘catch up’
If the database is modified by the node process during copy then the backup can get corrupted. So the node should be shut down before backing up. However, this obviously stops farming which isn’t ideal and makes automation more complex.
You can run a separate node (that isn’t farming) solely for the purpose of backups but that’s not ideal either.
The ideal way is to use the sqlite backup API which can perform backups safely on live databases.
This is in a script that is launched by Duplicati before it makes a deduplicated backup to another machine. This significantly reduces the amount of space required and you can restore from any previous backup point (e.g. hourly, daily, weekly).
The SQLite backup command can take 30 minutes or so to complete because it restarts every time the source database is being written to, which is exactly how it should work to avoid corruption. The node writes to the database very frequently but there are some idle moments which allows the backup to proceed in small chunks.
This enables regular, versioned, reliable and automated backups without needing to shut down the node or operate a separate node. Can even backup to the cloud if you like.
I have two full nodes. A backup of each other. So I don’t have to download from scratch if one gets corrupted. Yeah I have run into this kind of problem many times and I don’t want to wait for days to sync.
I can just copy the db from the non-corrupted node into the other node. Works pretty well. I don’t have to wait for hours/days to fully sync.
NEVER trust a database you find online somewhere. ALWAYS sync at least 1 system and then you can copy that database and wallet to another system.
Keeping a copy of the DB is good idea I never thought about (because I have multiple synced systems). Just remember that you will still have to sync after the wallet and DB are copied over because the height will inevitably go up before you can get the system up and going again.
What’s up guys? I see all this talk about backing up wallet and blockchain DB’s, but EVERY time I’ve ever tried to use one of my backups to sync, I ended up syncing the entire chain again!? Not sure where I went wrong. Can someone please give me a couple of steps to correctly use on of these backups? I basically just copied all of the files in my wallet db /db and tried to sync on the new machine. Chia blockchain started syncing from block #1 every time I did this.
Did you shut down the full node process, which means, if you use GUI shutting it down and then ensuring that it closes all the chia processes running in the background? If not you copied a live database and it would most likely have been corrupt. I closed it down, copied it and started it back up and it synced in a few minutes when moving to a new farmer machine.
I will just add this to my list of maintenance to do on my farmer/s around once every 1-2 weeks, which includes windows updates, driver updates, virus scans, plots-check, pruning any bad plots, reboots etc… and now backing up the blockchain files after closing the node, to some of the empty space on plot drives, which is an excellent idea from @spyghost.
I don’t use an official backup solution for this. What you need to do ideally is reboot your computer. Then save/make a copy of the .chia folder or the wallet and db folders without any Chia process running. I made the mistake of trying to copy the wallet and db folders on my farmer while farming and then when I tried to use them to start a new plotter, the whole Chia app (GUI) got messed up and refused to get back into sync.
Maybe that was my issue! I can’t remember. Might have had the farmer running when I was trying copy the DB file. I’ll just make a backup of the entire folder! Good idea! Thank you!
Is there an upper limit to the blockchain database? Does it “turn over” at some point? At what size?
