Staff Death Posted June 23, 2022 Staff Report Posted June 23, 2022 Wanted to give a quick update on the ongoing stability issues, what's the cause, and what we're doing to address the issues, as well as make it right to our players. This is completely unacceptable and is not what we've worked so hard for. Cause of instability We've identified the cause of the recent stability issues to be hardware related. We were able to finally confirm this today and have ordered new RAM to address these issues for good. We currently have no ETA but it'll be within the week. Faulty hardware has been replaced and stability has been restored! Thanks again for your patience and understanding What we're doing about it In the meantime, we've moved US Survival over to a spare dedicated machine to spread the load and avoid further crashes until the new hardware arrives. To clarify, this machine isn't a fix for the ongoing stability fix, this will be fixed within the week once we swap out RAM. This is strictly just preventing future issues. Additionally, we've ordered a brand new machine to replace our current US infrastructure with the latest and greatest hardware money can buy. We already do this once a year but we've decided to do it early as we cannot tolerate any more issues. We expect this machine to be built and racked in 4-6 weeks. Is it safe to play now? After moving US Survival over to a spare dedicated machine, we believe the load on our main machine will never exceed the potential for crashes and therefore stability should no longer be an issue. Once we replace the bad hardware within the next week we will be able to move survival back and everything should be back to normal. This of course is just a technical assumption and we can't be sure until we run the servers for a couple of days. In the event stability is not addressed, please consider taking a break and come back when we're back to normal in a week. Compensation We cannot express how deeply sorry we are for this month's awful stability. So instead, we're offering refunds for any purchases made, excluding memberships, this wipe for those who have decided to stop playing this wipe due to the ongoing issues. You can submit a refund request over at https://rustez.com/help Additionally, we will be hosting exciting events in the last week of June for players to hang out, have fun, and win exciting rewards for the next wipe. You'll find these events posted in Discord! 9 7
CHIEF BUD Posted June 25, 2022 Report Posted June 25, 2022 Thank you for the update. You can expect to see me and my team around more once this goes live. - Chief 1 1
Staff Death Posted June 25, 2022 Author Staff Report Posted June 25, 2022 Quick update on this, we've got the remaining US servers stable as best as we can until the new hardware arrives. Servers are saving every 5 minutes instead of every 20 minutes to prevent rollbacks, and I've adjusted a few settings to keep memory utilization as low as possible. Crashes shouldn't be that common, but if they occur, we've taken precautions to minimize their impact. We'll keep working non-stop until the issues are addressed for good. 3 2
Staff Death Posted June 28, 2022 Author Staff Report Posted June 28, 2022 In addition to replacing hardware, we've ordered brand new server machines to replace our entire fleet. This includes US, EU and our AU servers. All of our servers will be rocking the fastest server hardware available to greatly improve server performance and stability. Specs: CPU: Intel Core i9-12900KS 16C / 24T 3.9 GHz / 5.7 GHz Memory: 128 GB DDR5 5600mhz Storage: 2x 1 TB NVMe ETA on the new machines is about 3 months. Again, the stability issues on our US servers will be fixed within the week. This is simply covering all bases and future proofing our infrastructure for years to come. 2 2
Staff Death Posted July 2, 2022 Author Staff Report Posted July 2, 2022 Hardware has been replaced! We apologize again for the recent issues. 3 1
Recommended Posts