I have set up some scripts to shutdown my farm and power off when the temperature in my farm room gets too hot. I can just force-shutdown windows where the Full Node is running and also shutdown harvesters of course, but I want to ensure there is no database corruption in the process.
I have used WMI commands to close applications in the past, but not tried it yet with Chia Full Node.
Of course I can trial it, but don’t want to do so on my active Full Node as this will obviously kill farming. Just wondered if anyone has encountered this requirement before I start tests.
Of course, it hasn’t gotten warm enough yet to trigger my scripts, but it surely will soon.
[Edit] I meant to add, I’ve browsed the full node api but don’t see anything to control the application/service.
Also, check the main CLI doc - CLI Overview | Chia Documentation (look for “Other commands (not all are fully documented)” - “stop” is listed there). Basically, use “-h” to get more info for a given command.
Just in case, check in both db and wallet folders that there are no *wal, *shm db files except *sqlite, as that means that db was cleanly shut down.
Not sure if there is a way to use chia CLI to kill GUI. By the way, the GUI part has the first letter capitalized (Chia.exe), where daemon is lower case (chia.exe). Maybe you could poke the GUI from cmd line to see whether it responds to anything.
Also, maybe there is some additional flag for “chia stop all” that will bring the GUI down (don’t have chia running to test). Something like “chia stop all -f” (yeah, it should be “chia stop all --really-all” ).
Microsoft has a Power Automate tool, where one of its options is to record your actions, click-by-click.
So you can have it record yourself gracefully shutting down the GUI.
I just do not know if there is a way to have that recorded flow triggered via your script. But there is probably a way to kick off a saved Power Automate flow from the command line.
Here is a video of the host recording opening an app, performing some operations within his app, and then closing his app (in the first half of his video).
Later, he gets into using Power Automate with Excel, which is not relevant to what you are seeking.
Yes, I’ve used the Microsoft Power Automate tools in the past, and they can work well. I’ve also used my own automation scripts based on the AutoIt3 engine, as I have mentioned here in the past. They don’t work well in some circumstances however, particularly if a machine is headless and left without a ‘desktop’ 99.99% of the time, so a proper scripted method is sometimes required.
I did a cursory check on the sqlite temporary files and they do disappear following completion of ‘chia stop all’. So I am happy with killing the GUI with the taskkill command after this.
It appears Windows (from the command line) cannot/does not differentiate between chia.exe and Chia.exe, it’s not typically case-sensitive like *nix systems.
Anyway, for now, I am happy with the solution. Will just await to see if it ever actually gets warm enough to execute the scripts.
By the way, I am not executing the shutdown scripts based on ambient temperature in the farming room, but based on average hard disk temperature across all disks. If this average reaches a threshold, or if any one disk temperature reaches another (higher) threshold, then down the farm goes (cleanly) followed (after a time) by complete power removal.
I’ve set my temperature thresholds from long term observations of disk temperatures, so I know what is ‘normal’ versus what I would consider too warm.
Yes, Win is case agnostic as far as file names, so you either need to be in the current folder or explicitly specify path to the binary you want to run. I am not sure if that is a disadvantage (comparing to (*nix), rather a different way that also has some benefits.
Actually, I have Dell sc200 JBODs that by default are quite loud. I put together some crude scripts that check HD temps and adjust individual fan speeds (per JBOD). This way, most of the time my max HD temps are fluctuating around 39C (temp that I set as default) or lower during the cooler days. I was thinking about using a low cost ESP32 to do the job but am not there yet (those JBODs have a serial port for fan speed control - kind of sucks).
Also, what I did is I put 200mm “case” fans in front of those JBODs (manual speed control, so this would be a good start for an ESP32 controller). By default, all those JBODs have small diameter fans; therefore, are running at high speed. On the other hand, those 200mm fans run at low speed, but that is already good enough to drop HD temps by few C.
Lastly, if you have a mix HD brands you may want to check which HDs are running consistently high, and swap those with those that run cooler. In my case Toshiba HDs run cooler than WD / Seagate (they draw 1-2W less), so I put those in the middle of those JBODs (those are 3x4 JBODs and columns 2/3 usually run hotter).
Coincidentally, 39C is my average shutdown temperature. When this average is reached, obviously some disks are hotter (likely in the mid 40’s C) and there may be some that are cooler (but it won’t be by much…).
Most of my farm is now external USB drives (having lost a lot of my original farm size due to relocated datacenter-class storage), although I do have a single NetApp DS4246 still. Again coincidentally, I use 200mm case fans sat on top of the USB drives - one fan per 2x drives on average, and power them from 5V rather than 12V, which makes them silent but still keeps drive temperatures significantly below what they were without them. All the USB drives are sitting on stacked wire-mesh trays, so airflow is good across them all.
Finally, I have a single 18" floor fan which forces a stream of air across and through the multiple wire-mesh trays. This is only switched on when average disk temperatures reach 37C, and off again when 36C or lower. The fan only stays on for typically 5 minutes every hour, even on warmer days, so the power usage is negligible.
I am using power supply / speed (Voltage) controller from this thing - https://www.amazon.com/gp/product/B088WB9D5T and connecting to that this splitter - Amazon.com. Also, added one more standard fan-connector splitter from Noctua. This way, I can drive 4x 200mm fans in front of my JBODs. Yes, this power supply will not be able to drive those fans anywhere close to 100%, still works great for low speeds / power (there are more powerful power supply similar to this one, but this little one is a great to just start testing with it). Just bumping up the speed a tiny bit drops temps by a couple of C.
That is why I was thinking about using an ESP32 controller instead of that power supply. This way, I could get both HDs and ambient temps, and auto-regulate fan speeds. Without that ESP32 controller for those front fans, for hotter days I just manually bump up the speed for those 200mm fans (one knob takes care of all of them). On the other hand, those other scripts take care of the JBOD fan speeds (minimizing speed, while trying to not exceed 39C max for any HD). With that setup, those JBODs run anywhere between 6-30% of fan speed. Just out of the box, I think that the default speeds for those JBODs is around 50-60% what is quite loud (even without HDs, what sucks).
Thought I’d update this. The scripts executed in anger for the first time yesterday. All worked as expected, so my farm is now down with power removed.
I will start the farm up again when the temperature falls, whenever that will be. I may even look to automate this too. Perhaps when there is a run of forecast temperatures below x or something similar as I don’t want the farm starting up and shutdown down again on a daily basis.
Currently my farm is in the UK. We had approx. 78 F yesterday forecast 80+ F nearly all week.
I have already added code to retrieve the 5 day forecast from the UK Met Office and power the farm back on if the average forecasted maximum daily temperatures of three close surrounding locations across the 5 days does not exceed 20 C and any of the location’s maximum forecasted temperatures across the 5 days does not exceed 22C.
These parameters may need some tuning. The philosophy is to not power up the farm if it’s going to be too warm which will just result in it powering down again after a few hours.
Another way to control heat (power consumption) would be to manipulate config.yaml - plot folders. Once you remove a HD from being farmed, it should go to standby and/or sleep mode (around 1-2W, depending on brand / model). If you have Seagate drives, you could use SeaChest PowerControl util to speed up going to standby mode (specify timeout before going to standy; don’t recall what are the defaults but bringing it down to 1-5 mins should be good enough (challenges are every 9.375 secs, so 1 min should be OK)).
This way, you could still have your farm running with a couple of drives that have better airflow (or no HDs at all).
Also, instead of using a forecast, maybe you could use an API to get the current temp, and act on that accordingly. Looks like they have a nice API - https://open-meteo.com/
With that scrip, you could have your farm just playing with individual HDs based on the current temp. For instance, yesterday we had 96 F during the day, but it dropped to ~60 F during the night. My JBOD’s fans went from 10% to around 30-35% (quite noisy, but temps still at 39C; at night they drop to about 35C when JBOD’s fans are running at 10%).
Yes, I did consider this but the main issue is that the farming room is relatively poorly insulated. It is OK in cooler temperatures as the farm provides background heat which is useful, but in summer when the outdoor ambient temperature climes beyond 22-23 C, there is a noticeable build-up of heat in the room, not helped by the solar radiation that hits building causing heat-soak which then keeps the room temperature artificially elevated in warm sunny weather. As an added issue, if there is no-one present to ventilate the room (as is the case most of the time), it makes matters worse.
So, I shutdown the farm based on actual measured hdd temperatures (as per original posts above) and now restart the farm based on forecasted highs (rather than actual room temperature as this can remain high even when the weather cools down). [Edit] I update the forecasted highs every hour, so I always know when the cooler days are coming, so I can restart the farm in advance.
Not ideal overall, but better than burning energy to heat an already hot room!
).
