Page 1 of 2

The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 1:10 am
by crfriend
The astute -- and even the not-so-astute -- will surely have noticed that we've been off the air for a bit. This was due to the detection of a compromise of our site that made it a haven for spammers and what looks like spamming search-engines. Analysis of the full nature of the issues is underway, but we seem to have culled the immediate threat and are now back on the air.

Our hosting provider noticed the problem, likely through an automated scan of our site and the files contained therein, and moved the primary file hierarchy out of view from the front-line web-servers that all of us "talk to" to access SkirtCafe. This was a professional action on their part, and they made sure not to destroy or damage any of the content and history of SkirtCafe -- and it was that non-destructive action that allowed us to restore things to the point where they were immediately prior to the off-lining event.

We have, to the best of our knowledge, removed the problem, and we will be carefully monitoring the situation over the next few weeks; if our provider offlines us again, we shall work with them to remove the problem and take measures to ensure that any security hole is plugged to the best level possible.

Please bear with us for what may possibly be a few rocky days ahead as we tease out the full root-cause of the event and establish countermeasures to keep it from happening again.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 3:23 am
by ChrisM
Thanks for the update Carl....and for the education I gleaned just by reading it!

<smile>

Chris

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 10:35 am
by Milfmog
Carl,

Can you tell us where the compromised security originated? Is it due to a hole in phpBB, the operating system on the servers or in a setting specific to this forum? I ask because others might need to do something about their forums.

Thanks,


Ian.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 11:02 am
by stefan
I am also interested in some more descriptions. I know there are a lot of forums out there based on phpBB and if that itself was the attack point then more forums might need patching / updates to avoid this type of exploits.

Thanks,

Stefan

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 12:14 pm
by crfriend
Analysis of the event remains underway, but we suffered a file-system injection that allowed computer crackers to maintain a separate "pagespace" for the purpose of link-spamming. There are hints that the scripts are Eastern European or Russian in origin as the character set used for some comments is not in the Latin character set, but the cadencing and pitching "looks" Russian and the glyphs in use on my VT-100 emulator look like what gets presented when confronted with the Cyrillic alphabet. At this point in time, I have not identified the specific exploit, nor concrete methods of stopping it in the future.

We had two separate compromises -- both of the same ilk -- that look like they date back into May of this year. We detected neither of them because without hitting a specific URL (the "index.php" file in a separate cracker-created subdirectory) -- whch we have no links to -- we never saw it.

For those who care what the red-flag filenames are, they're "shablom.html", "admdoor", and "add.php" with the foreign-language (to me at least) commentary in the latter. There were also 400-odd sequentially-numbered *.php files in the cracker-created directory and a ZIP archive of the whole bunch.

From what it looks like at this tentative stage is that a custom-created ZIP archive was created for SkirtCafe (pretty easy to automate, really), uploaded via the phpbb software or directly via the web-server, and then exploded. Once the files were un-archived, the exploit was fully in place and ready for various nefarious uses. I so hate script-kiddies and crackers.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 12:19 pm
by stefan
Yea. I wish these kids could use their talents on doing something productive. I hate all this destructive work. Thanks for the information.
Do you know if this in any way could have affected our own computers directly or indirectly? My antivirus software has not warned me about anything, it has been quite so I guess not.

Thanks for the report.

/Stefan

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 2:54 pm
by crfriend
stefan wrote:Do you know if this in any way could have affected our own computers directly or indirectly? My antivirus software has not warned me about anything, it has been quite so I guess not.

For it to have done any malice to your system you would have had to accessed any of the malicious pages. As these were not linked to from anywhere else on SkirtCafe, unless you were actively poking around you would not have stumbled upon them. Specifically, unless you were poking at the old /phpbb2 directory hierarchy or the "images/thumbs" hierarchy you never accessed that particular "content".

Mostly, the accesses were confined to search engines and people directed here by those search engines; the files that we were inadvertently hosting contained mainly links to other sites. Now, those pages may be harmful, but the content of the stuff that was here looks benign from a purely infectious standpoint.

With luck, I'll be speaking to Bob sometime over the weekend and we can develop plans for rapid detection of any possible future compromises. Part of this may require upgrading the phpbb software, and if that's the case there will likely be another outage whilst the migration is performed.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 3:20 pm
by stefan
Thanks for the clarification. I doubt any of us poke around the the directories. We use the forums nothing else.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 3:48 pm
by Milfmog
Thanks Carl.

Have fun,


Ian.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Fri Jul 03, 2009 3:53 pm
by crfriend
Milfmog wrote:Thanks Carl.

You're quite welcome. I'm just doing my job.

Spacer text to create a blank line - if you are reading this you are even sadder than I was for writing it

Well, I guess that makes me one sorry SOB. In any event, I've been called vastly worse!

It's a long weekend here in the States, and I'm logged into work to mop up at least two other unrelated computer messes. What's one more?

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Sat Jul 04, 2009 6:39 am
by Since1982
I, for one am glad you all figured out and fixed your problem, I don't know what I'd do if that hit S4M other than beg Carl for help. He's SUCH a treasure to any site owner. My site certainly wouldn't have survived this long without him. 8)

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Sat Jul 04, 2009 2:15 pm
by sapphire
I like him too :D

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Sat Jul 04, 2009 7:55 pm
by crfriend
Oh, stop. You're making me blush.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Mon Jul 06, 2009 3:36 am
by Bob
Thank you Carl, for an amazing job on this issue. While Carl was fixing up SkirtCafe and getting it back on-line without delay, I didn't even have Internet. I was calling Verizon to get them to come out and fix my DSL, which would stop working every time it rained.

Re: The outage of 2009-07-01/2009-07-02

PostPosted: Mon Jul 06, 2009 4:59 am
by Since1982
Unless you're in love with Verizon DSL, check into the comparable pricing with HughesNet Broadband. They are the Online arm of DirecTV and all their feeds come from satellites. I used to have ATT DSL and thought it was the fastest computer speed on the planet until I tried HughesNet. Plus the price is about ¾ the price of ATT and maybe Verizon. Just a thought, check it out. :alien: :hide: :alien: