The outage of 2009-07-01/2009-07-02

Discuss recent changes, make suggestions, etc.

The outage of 2009-07-01/2009-07-02

Postby crfriend » Fri Jul 03, 2009 1:10 am

The astute -- and even the not-so-astute -- will surely have noticed that we've been off the air for a bit. This was due to the detection of a compromise of our site that made it a haven for spammers and what looks like spamming search-engines. Analysis of the full nature of the issues is underway, but we seem to have culled the immediate threat and are now back on the air.

Our hosting provider noticed the problem, likely through an automated scan of our site and the files contained therein, and moved the primary file hierarchy out of view from the front-line web-servers that all of us "talk to" to access SkirtCafe. This was a professional action on their part, and they made sure not to destroy or damage any of the content and history of SkirtCafe -- and it was that non-destructive action that allowed us to restore things to the point where they were immediately prior to the off-lining event.

We have, to the best of our knowledge, removed the problem, and we will be carefully monitoring the situation over the next few weeks; if our provider offlines us again, we shall work with them to remove the problem and take measures to ensure that any security hole is plugged to the best level possible.

Please bear with us for what may possibly be a few rocky days ahead as we tease out the full root-cause of the event and establish countermeasures to keep it from happening again.
User avatar
crfriend
Master Barista
 
Posts: 8689
Joined: Fri Nov 19, 2004 9:52 pm
Location: New England (U.S.)

Re: The outage of 2009-07-01/2009-07-02

Postby ChrisM » Fri Jul 03, 2009 3:23 am

Thanks for the update Carl....and for the education I gleaned just by reading it!

<smile>

Chris
ChrisM
Member Extraordinaire
 
Posts: 468
Joined: Thu Mar 18, 2004 12:49 am
Location: Vancouver, British Columbia, Canada

Re: The outage of 2009-07-01/2009-07-02

Postby Milfmog » Fri Jul 03, 2009 10:35 am

Carl,

Can you tell us where the compromised security originated? Is it due to a hole in phpBB, the operating system on the servers or in a setting specific to this forum? I ask because others might need to do something about their forums.

Thanks,


Ian.
Do not argue with idiots; they will drag you down to their level and beat you with experience.
Cogito ergo sum - Descartes
Cogito cogito ergo cogito sum - Ambrose Bierce
User avatar
Milfmog
Moderator
 
Posts: 2229
Joined: Tue Jul 18, 2006 7:30 pm
Location: Buckinghamshire, UK

Re: The outage of 2009-07-01/2009-07-02

Postby stefan » Fri Jul 03, 2009 11:02 am

I am also interested in some more descriptions. I know there are a lot of forums out there based on phpBB and if that itself was the attack point then more forums might need patching / updates to avoid this type of exploits.

Thanks,

Stefan
User avatar
stefan
Active Member
 
Posts: 88
Joined: Thu Jun 04, 2009 3:20 am
Location: Stockholm, Sweden

Re: The outage of 2009-07-01/2009-07-02

Postby crfriend » Fri Jul 03, 2009 12:14 pm

Analysis of the event remains underway, but we suffered a file-system injection that allowed computer crackers to maintain a separate "pagespace" for the purpose of link-spamming. There are hints that the scripts are Eastern European or Russian in origin as the character set used for some comments is not in the Latin character set, but the cadencing and pitching "looks" Russian and the glyphs in use on my VT-100 emulator look like what gets presented when confronted with the Cyrillic alphabet. At this point in time, I have not identified the specific exploit, nor concrete methods of stopping it in the future.

We had two separate compromises -- both of the same ilk -- that look like they date back into May of this year. We detected neither of them because without hitting a specific URL (the "index.php" file in a separate cracker-created subdirectory) -- whch we have no links to -- we never saw it.

For those who care what the red-flag filenames are, they're "shablom.html", "admdoor", and "add.php" with the foreign-language (to me at least) commentary in the latter. There were also 400-odd sequentially-numbered *.php files in the cracker-created directory and a ZIP archive of the whole bunch.

From what it looks like at this tentative stage is that a custom-created ZIP archive was created for SkirtCafe (pretty easy to automate, really), uploaded via the phpbb software or directly via the web-server, and then exploded. Once the files were un-archived, the exploit was fully in place and ready for various nefarious uses. I so hate script-kiddies and crackers.
User avatar
crfriend
Master Barista
 
Posts: 8689
Joined: Fri Nov 19, 2004 9:52 pm
Location: New England (U.S.)

Re: The outage of 2009-07-01/2009-07-02

Postby stefan » Fri Jul 03, 2009 12:19 pm

Yea. I wish these kids could use their talents on doing something productive. I hate all this destructive work. Thanks for the information.
Do you know if this in any way could have affected our own computers directly or indirectly? My antivirus software has not warned me about anything, it has been quite so I guess not.

Thanks for the report.

/Stefan
User avatar
stefan
Active Member
 
Posts: 88
Joined: Thu Jun 04, 2009 3:20 am
Location: Stockholm, Sweden

Re: The outage of 2009-07-01/2009-07-02

Postby crfriend » Fri Jul 03, 2009 2:54 pm

stefan wrote:Do you know if this in any way could have affected our own computers directly or indirectly? My antivirus software has not warned me about anything, it has been quite so I guess not.

For it to have done any malice to your system you would have had to accessed any of the malicious pages. As these were not linked to from anywhere else on SkirtCafe, unless you were actively poking around you would not have stumbled upon them. Specifically, unless you were poking at the old /phpbb2 directory hierarchy or the "images/thumbs" hierarchy you never accessed that particular "content".

Mostly, the accesses were confined to search engines and people directed here by those search engines; the files that we were inadvertently hosting contained mainly links to other sites. Now, those pages may be harmful, but the content of the stuff that was here looks benign from a purely infectious standpoint.

With luck, I'll be speaking to Bob sometime over the weekend and we can develop plans for rapid detection of any possible future compromises. Part of this may require upgrading the phpbb software, and if that's the case there will likely be another outage whilst the migration is performed.
User avatar
crfriend
Master Barista
 
Posts: 8689
Joined: Fri Nov 19, 2004 9:52 pm
Location: New England (U.S.)

Re: The outage of 2009-07-01/2009-07-02

Postby stefan » Fri Jul 03, 2009 3:20 pm

Thanks for the clarification. I doubt any of us poke around the the directories. We use the forums nothing else.
User avatar
stefan
Active Member
 
Posts: 88
Joined: Thu Jun 04, 2009 3:20 am
Location: Stockholm, Sweden

Re: The outage of 2009-07-01/2009-07-02

Postby Milfmog » Fri Jul 03, 2009 3:48 pm

Thanks Carl.

Have fun,


Ian.
Do not argue with idiots; they will drag you down to their level and beat you with experience.
Cogito ergo sum - Descartes
Cogito cogito ergo cogito sum - Ambrose Bierce
User avatar
Milfmog
Moderator
 
Posts: 2229
Joined: Tue Jul 18, 2006 7:30 pm
Location: Buckinghamshire, UK

Re: The outage of 2009-07-01/2009-07-02

Postby crfriend » Fri Jul 03, 2009 3:53 pm

Milfmog wrote:Thanks Carl.

You're quite welcome. I'm just doing my job.

Spacer text to create a blank line - if you are reading this you are even sadder than I was for writing it

Well, I guess that makes me one sorry SOB. In any event, I've been called vastly worse!

It's a long weekend here in the States, and I'm logged into work to mop up at least two other unrelated computer messes. What's one more?
Retrocomputing -- It's not just a job, it's an adventure!
User avatar
crfriend
Master Barista
 
Posts: 8689
Joined: Fri Nov 19, 2004 9:52 pm
Location: New England (U.S.)

Re: The outage of 2009-07-01/2009-07-02

Postby Since1982 » Sat Jul 04, 2009 6:39 am

I, for one am glad you all figured out and fixed your problem, I don't know what I'd do if that hit S4M other than beg Carl for help. He's SUCH a treasure to any site owner. My site certainly wouldn't have survived this long without him. 8)
I had to remove this signature as it was being used on Twitter. This is my OPINION, you NEEDN'T AGREE.

Story of Life, Perspire, Expire, Funeral Pyre!
I've been skirted part time since 1972 and full time since 2005. http://skirts4men.myfreeforum.org/
User avatar
Since1982
Member Extraordinaire
 
Posts: 3449
Joined: Mon Oct 18, 2004 2:13 pm
Location: My BUTT is Living in the USA, and sitting on the tip of the Sky Needle, Ow Ow Ow!!. Get the POINT?

Re: The outage of 2009-07-01/2009-07-02

Postby sapphire » Sat Jul 04, 2009 2:15 pm

I like him too :D
Moderation is for monks. To enjoy life, take big bites.
-------Lazarus Long
User avatar
sapphire
Member Extraordinaire
 
Posts: 1308
Joined: Thu Aug 16, 2007 5:42 pm
Location: New England

Re: The outage of 2009-07-01/2009-07-02

Postby crfriend » Sat Jul 04, 2009 7:55 pm

Oh, stop. You're making me blush.
Retrocomputing -- It's not just a job, it's an adventure!
User avatar
crfriend
Master Barista
 
Posts: 8689
Joined: Fri Nov 19, 2004 9:52 pm
Location: New England (U.S.)

Re: The outage of 2009-07-01/2009-07-02

Postby Bob » Mon Jul 06, 2009 3:36 am

Thank you Carl, for an amazing job on this issue. While Carl was fixing up SkirtCafe and getting it back on-line without delay, I didn't even have Internet. I was calling Verizon to get them to come out and fix my DSL, which would stop working every time it rained.
Bob
Barista Emeritus
 
Posts: 587
Joined: Tue Oct 21, 2003 9:31 pm
Location: New England

Re: The outage of 2009-07-01/2009-07-02

Postby Since1982 » Mon Jul 06, 2009 4:59 am

Unless you're in love with Verizon DSL, check into the comparable pricing with HughesNet Broadband. They are the Online arm of DirecTV and all their feeds come from satellites. I used to have ATT DSL and thought it was the fastest computer speed on the planet until I tried HughesNet. Plus the price is about ¾ the price of ATT and maybe Verizon. Just a thought, check it out. :alien: :hide: :alien:
I had to remove this signature as it was being used on Twitter. This is my OPINION, you NEEDN'T AGREE.

Story of Life, Perspire, Expire, Funeral Pyre!
I've been skirted part time since 1972 and full time since 2005. http://skirts4men.myfreeforum.org/
User avatar
Since1982
Member Extraordinaire
 
Posts: 3449
Joined: Mon Oct 18, 2004 2:13 pm
Location: My BUTT is Living in the USA, and sitting on the tip of the Sky Needle, Ow Ow Ow!!. Get the POINT?

Next

Return to Changes at the Cafe

Who is online

Users browsing this forum: No registered users and 1 guest