Share your Tales of Chia Plot Loss Woe and Agony! šŸ˜±

Weā€™ve all done itā€¦ lost plots through stupid moves on our part.

Share your tale of Chia woe! How many plots did you lose? Let us commiserate with you and collectively cry about those lost plots which coulda been winners! :trophy:

Hereā€™s mine: I had a dump of about 55 plots on a 10tb drive from an 8c/16t machine. I thought, oh, Iā€™ll be smart and use robocopy /MIR to copy the files over to this blank directoryā€¦ wellā€¦ turns outā€¦ MIRROR really means MIRROR as in ā€œmake the two directories look the sameā€, and I specified the target destination as the mirror target AND IT WAS EMPTY.

Per the robocopy documentation:

/mir Mirrors a directory tree (equivalent to /e plus /purge). Using this option with the /e option and a destination directory, overwrites the destination directory security settings.

And what is /purge?

/purge Deletes destination files and directories that no longer exist in the source. Using this option with the /e option and a destination directory, allows the destination directory security settings to not be overwritten.

So I basically pushed a button and instead of copying, deleted 55 plots, due to my own idiocy. :sob: Man was I pissed off. :face_with_symbols_over_mouth: But I learned: never use that mirror command unless you really know what youā€™re doing

13 Likes

One things I always do to my plots under a linux system is mark them as immutable via sudo chattr +i.

3 Likes

Posting my shameful story of woe under duress :wink:

Last Friday we had a power outage and I failed to check the systems had correctly come up. Only realised last night that the docker service wasnā€™t running and I hadnā€™t been harvesting for 5 days :sob::sob::sob::sob::sob:

I should have an alert on this.

4 Likes

Sometimes you shuck an Easystore, and sometimes an Easystore shucks you. Hear yee the tale of my harrowing ordeal, so yee might learn from my stupidity.

I lost a full 10 TB disk to the stupidest set of actions. My first mistake was trusting articles about how exfat is the cross-platform file system, and how it is modern and most importantly ā€“ did I mention that ā€“ cross platform. I I needed it to be because my plan was to plot into usb drives and them aggregate the someplace else.

After my best machine at the time (a fast windows laptop) filled up that drive, I wanted to move it to a central location, especially because full nodes were going in and out (which is why there should only be a few dedicated ones). I was feeling anxious because half of my full nodes were down, my plot output rate sucked, and the network grew fast, and my friends were earning sweet, sweet chia hand over fist while I was lost in space.

I already figured out why the partition could be read by windows, but not anything else. Apparently windows exfat is special or feels that it is special, and especially special when you format a shucked drive in a usb cradle. This results in an extra partition that is invisible to both Windows and Mac stock disk partition tools, but visible to some third party software, and also gparted. Deleting that partition did not do anything to help mac visibility. I figured that I needed to move the big data partition.

Why didnā€™t you just copy everything onto antother drive? Oh, itā€™s because I wanted to get the plots gambling for me as soon as possible. I paid $50 or so for EaseUS Partition Master ā€“ twice, because I uninstalled it on one machine, but then needed it on another and the license is tied down, etc. Anyhow, I decided to move the partion with the disk in the usb cradle. The move crashed midway, and my beautiful plots are now sitting in there all alone. I tried to look for them with gparted and with the stupid EaseUS, and with fsck. Maybe someone who knows what they are doing could recover it, but I picked up myself from the floor and just went on doing more farming.

2021-04-20_16-35-49

Here are the lessons that Iā€™ve learned:

  1. Donā€™t use usb drives when possible: attach to sata
  2. Donā€™t use crappy partitioning software and hope itā€™ll work
  3. Exfat is harder to use across oses than NTFS ā€“ everything can read and write NTFS, even Macs with a special driver
  4. Learn about how partitions and file formats work
  5. Donā€™t pay attention to what everyone else is learning
2 Likes

I donā€™t even feel bad now that Iā€™ve seen others have lost 50-100 plots. I plot on HDDs and use the same directory for -t and -d on some disks while also using those disks for -d when plotting to disks that arenā€™t big enough to store any plots.

I got used to cleaning up .tmp files from failed plots using rm -f /mnt/disk/*.tmp. One day I noticed I had some extra .tmp files without realizing they were .plot.tmp and deleted 8 of them. Iā€™m 99% sure they copied correctly, but werenā€™t renamed into .plot for some reason.

Now I use find with a -not filter to exclude .plot.tmp files.

3 Likes

Yeah I was going to say, this weekend I had a weird situation where it seemed like I just double clicked too much and lost 4 active plots at once because the terminal crashed.

BUT thatā€™s not 50-100 at once :slight_smile:

My wife was going to labor last week, so, before going to the hospital I scheduled 10 plots in parallel, each one queuing 19 plots. After three days (baby is great btw) and I got home, I noted only 5 plots were completed, caused by a crash in the system which made it reboot. I still donā€™t know the reason it failed.

After that I switched to linux, my plotting time went from 30k to 20k, and I am also building a plotting management tool to avoid this kind of situation in the future.

1 Like

oh man, there is a chiadog tool that maybe can help you next time

1 Like

ME TOO. So frustrating. I have a mix of Ubuntu, MacOS and Windows so I thought - great - here we go. Same thing as you. 30 plots bit the dust as soon as I tried to write to an Ubuntu exFat partitioned disk that I wanted to move to a Mac. WTF? The whole partition just melted down.

I have decided to have a master harvester for each os and I copy files over to the master once they are done plotting across a local ethernet network. NTF for windows boxes, ext4 for Ubuntu and APF for Macs. No problems since then. knock on wood.

2 Likes

Just this afternoon, I came back from the office and found some new drives arrived.
Went to put them in and a bit too late realised I have pulled the wrong tray out which already had a drive in it. 16TB, ~140 plots. Iā€™ve put it back in but NAS reported crashed volume and a corrupt partition. A couple of restarts and some dicking around later it has fortunately recovered. :relieved:

1 Like

Ha. Congrats on the recovery. That must have been a rush.

I just realized that when I swapped my -d disk (because it was full) this afternoon I forgot to mount the new one, so all my plots that were just lining up to copy just now ground to a halt when the OS disk filled up. I had 6 near complete plots get wasted. Itā€™s a small mistake, but a dumb one. Lol.

At least I noticed before I went to bed and they can re-plot while I sleep :slight_smile:

1 Like

I havenā€™t lost anything yet, but Iā€™m still pouring one out for all the plots I could be making more if I could try out performance tweaks without flushing the pipeline, if I had more hardware, if resuming was implemented so I wouldnā€™t have to start early on my desktop which I canā€™t leave on overnightā€¦ :sweat_smile:

Just knocked over a brand new FULL 12TB External Drive. DEAD!!
No more external drives for me.
FML

6 Likes

A moment of silence for those 12tb of plotsā€¦ been there brotherā€¦ RIP :coffin:

2 Likes

I decided to go with ā€œstart drinking now.ā€

Hopefully one of them hit in the past few days while filling it up so its short life was not completely pointless.

When 1.0.4 came out, the consensus was still that no bitfield was faster than bitfield. In the release notes I saw that required temp space dropped to 256GB. I setup all my plotters with a long queue because I wouldnā€™t be home for a few days. Come back, and to my surprise, all the plotters failed. Learned the hard way that no bitfield was still 356GB temp space.

2 Likes

Yeah thereā€™s a lot of ā€œgood adviceā€ from earlier betas that doesnā€™t apply any moreā€¦ I hope they clean up the docs! Bitfield option should have been removed completely IMO, just like other options before it that became obsoleteā€¦

1 Like

Lost some 3.7 TB of plots with a raid going awry. Not to mention a full day trying to fix it. JBOD for me from now on lol.

2 Likes

Yeah no doubt. KISS. The simpler the better. I have learned this lesson the hard way myself, so I feel you.

1 Like

So far not bad, but dumb:

I was doing like 3 parallel plots on my desktop that were almost done. I was setting up a NAS and unplugged my desktop without thinking.

1 Like