Wiki Downtime

Hello all,

The main server was down today (or rather super slooooow). I had some time to actually dig into the reason this time around. Thanks to a list of processes that Paul was able to grab I was able to find the culprit.

So around 1:23 last night 4.79.204.36 ip spidered our system. There isn’t anything wrong with this necessarily but it uses a rather aggressive scanning…basically just blasts us with as many GET requests as possible. Originally it looked like Passenger (the module that handles our rails apps) kicked off MajorLazer (Or lazer management) app which then spun off a bunch of apache services. I added limits to Passenger and updated all the gems on MajorLazer but it seems the apache processes actually came from the wiki.

I’ve added mod_security rules to rate limit per IP to hopefully cut down on allowing someone to thrash the site. Also since the wiki and a lot of the extensions are outdated I’m going to take some time to update these. The wiki maybe down for a period as I perform updates.

Craig

Are you going to do the updates to the wiki today? Just curious because I’ve been editing it a bit in the last hour and don’t want to lose any work.

Thanks for finding the problem!

Yes, probably in the next 15 minutes. I just took a backup of the SQL db about 5 minutes ago… should I do it again?

Yeah, please take another backup. I’ll work on something else in the meantime. Sorry about that!

No worries. Took another backup. The main wiki should be backup soon, extensions will probably take longer. I’m glad you mentioned you were working on it. I’ll let you know the moment I’m done.

OK, wiki is backup and updated. Let me know if you see any problems. I’m still working on updated extensions but it should be functional now.

Just finished updating all of the extensions. Good thing is that with the new version of mediawiki you can use Composer as an extension manager…bad thing is that it doesn’t really work. Extensions are just now moving that direction and when it worked it was great but I found that I had to manual use git repos and overwrite some of the install from composer still. No idea why but I figure I would mention it in case somebody wants to use it after me.

I also disabled the SemanticMaps. Mediawiki:Maps is working and has been updated to 3.0 but SemanticMaps was tossing an error. I don’t know if we have used any of that functionality but it is currently disabled in the SemanticBundleSettings.php. Flickr extension was also updated and I saw it was throwing a lot of errors before. Hopefully the new version will fix those issues.

Test the wiki and let me know if anything doesn’t appear to be working correctly.

Craig

Looks like the new version of Flickr needs an API key. http://www.mediawiki.org/wiki/Extension:Flickr This wasn’t in our settings file before. If someone has the API key send it to me and I’ll add it to the settings file.

Nevermind I found it. Sorry to spam the list.

I was poking around the wiki and it suddenly went to a blank page, any clue what’s up?

Wow this new extension system is a PITA. Not sure how to fix it yet but it seems to be an issue with SemanticMediaWiki…

Ok I believe I have it fixed. We were getting hit by this bug https://bugzilla.wikimedia.org/show_bug.cgi?id=43787

To solve it I downloaded and built the git version of SemanticBundle, I also remove composer’s installations of Maps and Validator. Even though SemanticBundle should be pulling from the same git repo there seemed to be a difference. Maps is also installed directly via git now. I hit random pages a bunch and it seems to be working…Although I did that last time too, so we’ll see.