You got this right, you just didn’t add in everything that impacts it, Proof of Space AND Time, PoST…
while (Timelords) “Time” isn’t rewarded with blocks, it acts as a metronome keeping a steady pace on the network (VDF - Verifiable Delay Function), if time speeds up, more signage points come, if time slows down, less signage points come, and so difficulty adjusts to this as more or less blocks are made. Keep the same pace of time, but add or remove space, the same thing happens.
This is part of the secret sauce allowing us to store our proof of work and turn it into space (plots) which consume far less energy over time, so we don’t have to compute everything on the fly like Bitcoin, Litecoin, etc., in fact once SSDs take over storage in the future, we’ll use 10-100x less energy, so we can increase security with more space while consuming even less energy!
Pretty wild stuff when you sit back and compare the future of Chia to other coins.
P.S. Here are a couple links I’ve been posting on reddit for those who want to dive into things more.