Excelsior

Zazriel wrote: Tue Nov 19, 2024 7:26 am Anyone able to log in?

Yep. We're all at TC and +C is handing out free ancient titan hammers.

GreenBar wrote: Tue Nov 19, 2024 7:27 am

Zazriel wrote: Tue Nov 19, 2024 7:26 am Anyone able to log in?

Yep. We're all at TC and +C is handing out free ancient titan hammers.

Well dang, might just be me and my internet again, psssttt grab one for me!

Zazriel wrote: Tue Nov 19, 2024 7:32 am

GreenBar wrote: Tue Nov 19, 2024 7:27 am

Zazriel wrote: Tue Nov 19, 2024 7:26 am Anyone able to log in?

Yep. We're all at TC and +C is handing out free ancient titan hammers.

Well dang, might just be me and my internet again, psssttt grab one for me!

And were'e back! Darn, those ancient hammers aren't here anymore. Time to redo the last 1/2 hours activities.

Ok we crashed again, and got back up in 15 minutes. There was a 15-minute revert to the save done at 12:00 shard time.

I have found the cause of the problem... For now I have just removed the item that caused the crash, but I am yet to fix the code so that this crash can't be triggered in the future (assuming there are other items with data that would be able to trigger this bug).
Someone was just bragging in chat about an hour earlier, about a possible bug in RunUO code that could crash this server... Maybe related.

Found the item that caused it, and see who did the final double-click and gump button click that caused it, though I will have to investigate further before I can come to any conclusion if this was intentional or accidental. Since it's a long-term player here, it would make more sense that this was just an accident/coincidence. If someone wanted to do harm, they'd probably use a VPN and a fresh account, so that they can't be traced.

During this I also learned how to find the cause of a StackOverflowException in a crashed application... So far, when this happened, it was usually some completely fresh, within hours of a restart with an update, and so I knew right away where the bug probably is. Today, it could have been anything really, and it did turn out to be some code in the very old parts of RunUO.
Along with the crash-inspection, I also checked the log files of the players who were active in that last minute before the crash, and what exact actions they did, and as I reproduced those actions I managed to crash it again, this time knowing exactly what crashes it. I guess, to avoid another crash, I could do that on a staging server, but it's just easier and more reliable to work directly on the same system where the crash occured, because the setup might be a bit different on a clone of the server, and the same action wouldn't cause the bug to go off.

We had two disconnects yesterday, which didn't seem like a big deal, probably just a random network glitch. Another disconnect today, so now there's more chance that there's an issue that needs addressing. I'm aware of the issue and monitoring it, for now it seems just like a network issue in the datacenter, which might not repeat often, or just go away without any intervention.

Server is moving. Going down in a few minutes now at 15:30 shard time, expected downtime is 30-60 minutes, could be longer.
Keep in mind that even if you are able connect, the server might go down again in just a few minutes... there's a bunch of internal services & databases, and I need to make sure that the server is connected to everything as it should, and a quick server restart with an edited configuration is the simplest way to do that.

I'll let you know when it all seems stable & operational.

OK we are moved, and basically stable. Still offline are the Portal & Topsite voting, which should return in an hour or so.

There were some lag spikes yesterday around 19:00 shard time, I'd say that's just random occurrence, not related to the server move (and the new setup or any forgotten configuration). I'm keeping an eye out though.