Ok we crashed again, and got back up in 15 minutes. There was a 15-minute revert to the save done at 12:00 shard time.
I have found the cause of the problem... For now I have just removed the item that caused the crash, but I am yet to fix the code so that this crash can't be triggered in the future (assuming there are other items with data that would be able to trigger this bug).
Someone was just bragging in chat about an hour earlier, about a possible bug in RunUO code that could crash this server... Maybe related.
Found the item that caused it, and see who did the final double-click and gump button click that caused it, though I will have to investigate further before I can come to any conclusion if this was intentional or accidental. Since it's a long-term player here, it would make more sense that this was just an accident/coincidence. If someone wanted to do harm, they'd probably use a VPN and a fresh account, so that they can't be traced.
During this I also learned how to find the cause of a StackOverflowException in a crashed application... So far, when this happened, it was usually some completely fresh, within hours of a restart with an update, and so I knew right away where the bug probably is. Today, it could have been anything really, and it did turn out to be some code in the very old parts of RunUO.
Along with the crash-inspection, I also checked the log files of the players who were active in that last minute before the crash, and what exact actions they did, and as I reproduced those actions I managed to crash it again, this time knowing exactly what crashes it. I guess, to avoid another crash, I could do that on a staging server, but it's just easier and more reliable to work directly on the same system where the crash occured, because the setup might be a bit different on a clone of the server, and the same action wouldn't cause the bug to go off.

- 2024-11-19 it aint content but its honest work.jpg (79.73 KiB) Viewed 2732 times