UO Gateway Status

Red Squirrel

Red Squirrel

Member
Just a heads up that the server that hosts UO Gateway, along with many other services I run got hacked and trashed. It was an old distro that was well overdue to be upgraded but while migrating to a whole new server and newer distro, it got hacked again. Clearly some kind of sophisticated targeted attack. The new server did not even have any sites on it yet as I was in middle of setting it up. There was not any services on it and it was an up to date distro, and it still got hacked. They are using some kind of injection attack in Apache that allows to run remote code as root.

Most of the stuff I run online is old and not really maintained much anymore and I decided to just shut everything down instead of dealing with this stress. Time to move on to different hobbies. One of my sites had been online for at least 20 years, shutting that down too. Pains me to do it, but if I keep letting this stress me, the hacker won. If I use this as an opportunity to move on in life and find a new passion, then I won. Will consider it a new chapter in life. Recently bought off grid land, time to focus on that dream.

At this point I don't have plans for what to do with UOG, I feel that I don't want it to die completely but for now that is probably what I will do until I can let time pass and either try to rehost it again, or maybe eventually give it to the UO community.

Lastly and most importantly for you, if by chance you had an account on the UOG forum or UOG itself, I suggest you change that password on any site that you may have reused it. I cannot tell how much data the hacker actually took but to play things safe I am assuming it's everything. The bandwidth graph on my server provider's control panel does not indicate any large amount of data being transferred though.

I've always shook my head when a corporation gets hacked and leaks data, and here I am. But in my defence, I don't have a multi billion dollar budget for IT either. ;)
 
I was wondering why I wasn't able to vote on it anymore. Thanks for the update and for running it for as long as you did!
 
I did the same thing I liked playing UO for a long time and building a server was a learning experience. But gave that up to get back into music, guitar and playing the drums. Time for the real world, the internet is not what it used to be. Sad but true.
 
So I've been pondering about this, and I want to look at handing UOG to someone in the UO community that can be well trusted to continue running it well. I still want to keep the domain for now though, but I would renew it for the max time it lets me, and point it to new owner's DNS. I'd make sure to be contactable in case there is any domain related issues such as switching to another host, and provide initial support to get it going.

I'm not sure what is the best way to go about doing this, and if it's something maybe mods here can help with, but basically I've already had several people ask if they could have it, and I'm sure there will be multiple here interested too, so I just want to figure out the fairest way to pick someone trustworthy. Maybe via a vote or something. This way in the end, it does not just die off. I'm not that involved with UO anymore and I shut down my shard so figured I can pass on the torch.

Maybe if a mod/admin wants to help coordinate something that would be great, I'm not sure what is the best approach.

Whoever does end up getting it would get the site "as is" along with the poller, source code and the latest backup of the database, with cleared passwords. (ex: shared owners would be required to do a reset)

I'd be available at first with any questions etc and to help set it up. Some of the code is rough around the edges and requires some fixing and improvements but it does work. Ideally the person that gets it should know php, mysql and C++. The biggest challenge might be getting the C++ mysql library to work, as it's very finicky when trying to load it on a new server. But other than that, once it's setup it's not too involved and it mostly runs itself. I had a watchdog script that would relaunch it if it crashed, since it does have some very rare crash issue that I never took the proper time to debug.

Whoever gets it would of course be free to change it if they want, the code would essentially be theirs at that point. Only thing I ask is that you keep it clear on the site that it's under new ownership, that can just be a small line at the bottom of the page or something "under new ownership" or something. At least for a year or so. I just think it's courteous to make sure shard owners know. There is also a small historical page that talks about when it was acquired etc so can just update that. These are all details we could discuss further and agree on what to do at the time of transaction. I mean ultimately if new owner just wants to do something completely new, then that's fine too. Basically I'll just point the domain to your site.

But yeah, just leaving this open to see what everyone thinks, and can go from here to figure someone out that is the best candidate. I don't actually post here all that much and don't really know anyone too well (I do recognize the odd name but that's about it) so I don't feel right being the one to decide.
 
I know I'm not a widely known member of the community but have been in the UO scene long enough to see many shards, portals, launchers and the like come and go, I've had my main domain for many years (my old domain is going on 20 years now) and again although it's not widely known or used I have continued to keep it up as there are files I've hosted so they aren't lost to history.

I would love to put my name in the hat so to speak however it's planned to work it out and would definitely plan to keep it going for as long as history will allow me to.

Anyways that's all I have to say for now..
 
Voxpire said:
Do you think you can redirect the domain to shardportal.com while everything gets sorted out? :)
Click to expand...


That could not hurt, I can do that.

In fact... I am wondering if maybe just linking to an existing site is the easiest approach, at least for now. Gives me more time to ponder this entire situation before I officially decide to give it up. I can always relaunch way later down the line and hope whatever exploit the hacker is using is patched by then. Part of me is sad for giving up all my sites/services, and another part of me sees it as a way to move on to new things, so still need to process everything.
 
I'd be interested in reviving UOG! I have experience with running various Linux game and web servers for Ultima Online as well as other games. I have been around since the RUNUO days and have even written a few scripts.
I can also program in C++ (albeit a bit rusty), C#, PHP as well as MYSQL.

It would be awesome to see UOGateway up and running again!
 
Thanks to those of you who have offered to take it over. I am still letting this brew for a bit, as deep down I kinda want to try hosting it again, so I may in fact do that once I can figure out how the hacker got in the 2nd box. I am also tempted to just try again, but tweak security settings more (ex: actually learn to use Selinux and configure it properly) and then hope for the best. I will keep my email and lot of other stuff hosted on the current shared hosting plan I have now, just to minimize attack surface. It could very well be they got in through a dovecot or postfix vulnerability of sorts. I keep assuming apache but when I think about it, the weird stuff in the logs is in the mail logs too. It's some kind of remote code execution vulnerability it seems.

There is also a very good possibility it's already patched, it could have been a 0 day.

I am also working with a security expert to secure UOG itself. There was a few flaws that were found and there may be more. They were fairly minor and don't think they were a cause of the hack, but still something I will need to look into to be safe.
 
I see you're planning to relaunch - awesome and wishing you good luck with it :)

Btw I dont know if you'll try multiple sites at once, or just uogateway for now, but I would recommend hosting somewhere so you can isolate each of the sites. A cpanel-based service can be quite cheap (but usually the low promotional price only applies if you pay for 12 or 24 months in advance), but perhaps it's worth it so you can focus on the actual functionality, not the infrastructure.
 
Yeah that is the plan, I am thinking of doing only uogateway for now. I won't even do DNS or mail, I will leave that all on the shared host.

Then I can slowly reintroduce the other sites, ideally after doing full code audit. Will also downsize as lot of my sites I can just keep shut down as they were dead anyway, including my shard.
 
Is it possible you were hit by this it coincides with the time you went down:

www.huntress.com

Critical RCE Vulnerability: log4j - CVE-2021-44228

Our team is currently investigating CVE-2021-44228, a critical vulnerability that’s affecting a Java logging package.
www.huntress.com www.huntress.com

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.
 
tr1age said:
Is it possible you were hit by this it coincides with the time you went down:

www.huntress.com

Critical RCE Vulnerability: log4j - CVE-2021-44228

Our team is currently investigating CVE-2021-44228, a critical vulnerability that’s affecting a Java logging package.
www.huntress.com www.huntress.com

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.
Click to expand...

Funny when the server got hacked I thought "I bet there will be a big vulnerability found soon" and sure enough this came out. Only thing this seems to only affect java applications, but I do wonder if somehow some other applications impliment the same code. Apache itself, dovecot, postfix, maybe openssh? Those were basically the only services running on the 2nd server when it got hacked. They were mostly default and there were no sites running yet. So I'm still very baffled at how they got in.


john burns said:
Hey Red Squirrel, if someone wanted to start something like this, how, where, and what software would be needed?

Just curious.
Click to expand...

Basically a fairly standard Linux setup, and then whatever programming language you feel more comfortable in, for the poller. In the case of mine the site is written in php and poller is written in C++. They both talk to the same database so that is how the data gets updated between both. Ex: when you submit a shard, the poller will see it, or when poller polls a shard, the site will pull the new data etc.


As for road map to get it back up and running I am hoping to get it done by end of month, though no promises. I hate that I don't know how the hacker got in as there is no way to know that it won't just happen again, but I will try a different distro, one that does not use systemD (which is a pita to deal with anyway) and hope for the best. I read briefly that SystemD has over 1M lines of code and tons of vulnerabilities so that makes for a fairly big attack surface. So could very well be that's how they got in. I feel it has to do with logging or sending some specially crafted string to a service, because when the hack happens, all system logs will show giberish in them. So it seems to be something that affects the system at a bigger level.
 
Figured I'd post an update. Ran into some set backs setting up the server and had to restart setting it up but now I'm in a good position to get the site going again soon. The good news is once all this is setup, any future server failure should not be this long to rebuild from. I moved towards doing a VM instead of hosting on the bare metal server. So once everything is setup I will backup the OS disk of the VM locally. If I ever need to redeploy at least I don't have to fully reconfigure everything again. Could not do that before as without physical access to the server I cannot do an image backup. So doing a VM is kind of a way around that. Unless of course the entire host craps out, but at least it's quick enough to setup another VM host and restore the VM image. I've always had good data and config backups, but restores still involve lot of reconfiguring etc as config files are often not drop in, especially when going to another distro.

Going from CentOS 6 to newer distro poses lot of challenges too as tons of stuff has changed including php syntax, apache config file syntax and many other things. So I have to modify a lot of my code as I go, but I'm getting there. Before I go live I will probably want to also update my local dev environment so I'm not making code changes in production like I'm doing right now. But I'll deal with that after and just take note of what file I changed so I can pull it back to dev environment.

The forum may remain down, since I think it too requires lot of code changes to work and I already had a totally separate project on the go to create a new forum so I rather just focus on getting that up and running as my next big project. UOG will get it's own section on that forum.

I'm aiming for end of month to get it back up unless I run into a snag.
 
And it's back!

There's lot of back end stuff to tweak and I also need to upgrade my local dev environment to match since lot of the code had to change to work on the newer distro and I will want to reflect that in dev to make further updates sync properly. But for now the list and existing shards are working. I reset all the stats as well. I still need to get the email portion of the server to work so that registering or resetting a password works, I will do that in next few days. Just wanted to get the site up at least.

Still monitoring and keeping an eye on things to make sure things keep working but I'm confident it should be fine.

The forum will stay down for now since it would require tons of code changes to work on new server, and I plan to launch a new general discussion forum site soon so it will be best to just combine everything into that forum.
 
You must log in or register to reply here.

Active Shards

Donations

Total amount
$0.00
Goal
$1,000.00
View
Back