Mock all you IO and then use a test framework that test combinations of failure (disks failing, network partitioning, time, etc). DST gives you a deterministic way of replicating bugs in all kinds of edge cases.
Model checkers are usefull for checking that something is possible in theory, DST allows you to test your implementation (this idea came from TigerBeetle). Joran and Dominik Tornow highlights how Paxos is proved to be correct in theory, but every implementation is riddled with bugs.
I think it would be useful to haev programming languages with Algebraic Effect Types so that you already have mocked out all IO at the language level. If you don't have this your forced to replace the entire universe with your own mocked versions, which is not possible for third party libraries (so you end up having to mock those as well). However in a language with Algebraic Effect Types you've already mocked out IO at a language level, so it would be perfect to slap a DST testing library on top.
I think it would be better to take a programming language like Effekt and implement DST in it (given that it already have Algebraic Effect Types). However maybe we could take Rust and use something like effing-mad and implement DST on top of it?
"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson - how FoundationDB did it.