The Neverending Fight Against On-line Junk
- crfriend
- Master Barista
- Posts: 14486
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
The Neverending Fight Against On-line Junk
One of the tactics we've developed over the past couple of years to keep the level of robot-posted junk advertising out of the forum is a way to detect whether an individual (or, vastly more likely, a virus-hijacked computer known as a "zombie") has posted junk to other forums and automatically deactivate the newly-registered account and temporarily blacklist the IP address in question. This tactic, while not one hundred percent has the benefit that there are no modifications to the phpBB software that drives the system here.
We normally get a dozen or so registrations per day, and given the niche nature of this forum, it's (unfortunately) fairly rare we have new human participants registering. The automated techniques find and get rid of most of the robot-registrations leaving the moderation staff to deal with the occasional straggler. Added onto this is the requirement that newly-registered folks need to have a certain number of posts approved by the moderators before becoming full members of the community; this means that sometimes a robot will register and before being found out and disabled can leave dozens of messages in the moderation queue.
Every so often I'll purge all the disabled account from the database, mainly to save space and improve performance; I did this just yesterday where I deleted all the disabled accounts from April 1 to December 31, 2013 -- all 6,067 of them. It took a while.
Here's the 10,000-foot view -- it's a graph of the number of accounts that were either automatically disabled by the machinery or that the moderators disabled over time: Of note are the spikes in the data; those represent assorted malfunctions of the machinery, some of which were do to "pilot error" (read, "me doing something stupid") or upstream problems with the data-source I use (StopForumSpam.com). The average number of account-disablings per day counting the spikes was 22.1; with the spikes removed, 15.6. From the beginning of August, 2013 the numbers average 9.7 per day; that smoothing corresponds to when I reinstated a blanket ban on China by blacklisting every net-block I could find. This is the reason for the ban. Yes, it's ham-fisted, but it works, and there has never been a registrant from China who has contributed to the forum -- it's all been junk.
Note that this operation did NOT affect valid registrations or accounts that are active, even if they're idle.
We normally get a dozen or so registrations per day, and given the niche nature of this forum, it's (unfortunately) fairly rare we have new human participants registering. The automated techniques find and get rid of most of the robot-registrations leaving the moderation staff to deal with the occasional straggler. Added onto this is the requirement that newly-registered folks need to have a certain number of posts approved by the moderators before becoming full members of the community; this means that sometimes a robot will register and before being found out and disabled can leave dozens of messages in the moderation queue.
Every so often I'll purge all the disabled account from the database, mainly to save space and improve performance; I did this just yesterday where I deleted all the disabled accounts from April 1 to December 31, 2013 -- all 6,067 of them. It took a while.
Here's the 10,000-foot view -- it's a graph of the number of accounts that were either automatically disabled by the machinery or that the moderators disabled over time: Of note are the spikes in the data; those represent assorted malfunctions of the machinery, some of which were do to "pilot error" (read, "me doing something stupid") or upstream problems with the data-source I use (StopForumSpam.com). The average number of account-disablings per day counting the spikes was 22.1; with the spikes removed, 15.6. From the beginning of August, 2013 the numbers average 9.7 per day; that smoothing corresponds to when I reinstated a blanket ban on China by blacklisting every net-block I could find. This is the reason for the ban. Yes, it's ham-fisted, but it works, and there has never been a registrant from China who has contributed to the forum -- it's all been junk.
Note that this operation did NOT affect valid registrations or accounts that are active, even if they're idle.
You do not have the required permissions to view the files attached to this post.
Retrocomputing -- It's not just a job, it's an adventure!
-
- Member Extraordinaire
- Posts: 2921
- Joined: Sat Jan 14, 2006 3:51 pm
- Location: Scottish West Coast
Re: The Neverending Fight Against On-line Junk
Carl, you are doing a fantastic job! I am glad the forum has you in control!
I am the God of Hellfire! and I bring you truffles!
-
- Member Extraordinaire
- Posts: 247
- Joined: Wed Oct 17, 2012 11:54 pm
- Location: Rockland County, New York, USA
Re: The Neverending Fight Against On-line Junk
Carl- I didn't fully understand all of the technical things that you mentioned, but I'm grateful for your obsession to maintain the forum.
Re: The Neverending Fight Against On-line Junk
Thanks Carl, keep up the good work.
Re: The Neverending Fight Against On-line Junk
Carl,
Nice to see you back in the pink and not worrying about groceries!
I didn't know about block bans but would like to block all traffic from everywhere except the US and Canada....could you point me to a resource I could read so I could figure out how to do this?
Thanks!
Nice to see you back in the pink and not worrying about groceries!
I didn't know about block bans but would like to block all traffic from everywhere except the US and Canada....could you point me to a resource I could read so I could figure out how to do this?
Thanks!
-John
______________________
You see, ya can't please everyone, so ya got to please yourself (Rick Nelson "Garden Party")
______________________
You see, ya can't please everyone, so ya got to please yourself (Rick Nelson "Garden Party")
-
- Member Extraordinaire
- Posts: 908
- Joined: Sun Dec 09, 2012 7:05 pm
Re: The Neverending Fight Against On-line Junk
Why on earth would you want to do that?JRMILLER wrote:Carl,
I didn't know about block bans but would like to block all traffic from everywhere except the US and Canada....could you point me to a resource I could read so I could figure out how to do this?
Thanks!
-
- Member Extraordinaire
- Posts: 309
- Joined: Wed Oct 22, 2003 1:49 pm
- Location: Metrowest Suburbs of Boston
Re: The Neverending Fight Against On-line Junk
In a word:
BRAVO!
BRAVO!
- crfriend
- Master Barista
- Posts: 14486
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: The Neverending Fight Against On-line Junk
Thanks. It's nice to be back to normal.JRMILLER wrote:Nice to see you back in the pink and not worrying about groceries!
Maps of IP address space are available in various spots on the Intar-Tubes, but unless you're running a forum with a very limited geographical scope such a broad blacklist is probably not what you're looking for. It's just too broad a block.I didn't know about block bans but would like to block all traffic from everywhere except the US and Canada....could you point me to a resource I could read so I could figure out how to do this?
I can see binning places like China and Pakistan outright as nothing really hits these shores from there that isn't trouble of one sort or other, but why nix Europe?
Generally speaking, the path of least effort is the most useful one in these regards. For instance, when I see a suspicious registration, I look over at StopForumSpam to see if the IP address or e-mail address of the registrant is known for forum-spamming; I also look using "whois" to find out where the IP address is. If it's in China, the entire netblock immediately goes into the local blacklist (I should probably be using .htaccess for this, but the one in phpBB seems good enough), permanently; I figure I've probably got about 90% of China so configured, but new netblocks frequently get added and need to be addressed on an ongoing basis (Hey, ICANN, they've got the Great Firewall. Give 'em a single class-C and let 'em NAT the entire shebang until they clean up their act.). Most everything else I simply let the system deal with.
Retrocomputing -- It's not just a job, it's an adventure!
Re: The Neverending Fight Against On-line Junk
Carl, I have run various forums in the past so I know the amount of behind the scenes work it really is. Thank you for taking the time and trouble!
Ordained Deacon and Ruling Elder, Associate Reformed Presbyterian Church.
- crfriend
- Master Barista
- Posts: 14486
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: The Neverending Fight Against On-line Junk
I am a big fan of automation, and the more things I can write that'll enable the forum to "look after and defend itself" the better. That said, I do remain "hands-on", so that needn't be a worry, but I find that if I have to keep doing mechanical things repeatedly then it's time to dust off my programming and system chops.BobM wrote:Carl, I have run various forums in the past so I know the amount of behind the scenes work it really is. Thank you for taking the time and trouble!
Retrocomputing -- It's not just a job, it's an adventure!
- crfriend
- Master Barista
- Posts: 14486
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: The Neverending Fight Against On-line Junk
As the community here knows, one must be logged in in order to post, but what the community has not seen is that over the past week or so we've been deluged with bogus registrations and spam. This is invisible to the community because we require that the first few posts from new registrants be vetted by the moderation staff. Some folks are put off by this, but on the whole the moderation team are a pretty relaxed lot and if something doesn't blatantly contravene the social mores of the community we approve it and let it go.
Your administrator, on the other hand, has grown increasingly tired of spending 30 or 45 minutes every morning deleting adverts posted by "new registrants" and reporting them, one by one, to his favourite blacklist provider.
It first came to a head about two weeks ago when I finally got sick of the level of abuse coming from China, and summarily blocked access -- by the web-server process proper -- to the script that allows one to register (If you're not registered, you cannot post -- ergo, no spam). Today, I put in the entire range of netblocks that are owned by "DataShack, LC" and "Peg Tech" into the same categories as there has been precisely nothing but a barrage of hostile traffic therefrom. The former just seems to be a rogue server-hosting provider; the latter looks to be entirely a Chinese-owned US-based proxy-farm.
If the admins (or bots) at either of the above are reading, Clean up your act! Your "service" is a public nuisance.
Your administrator, on the other hand, has grown increasingly tired of spending 30 or 45 minutes every morning deleting adverts posted by "new registrants" and reporting them, one by one, to his favourite blacklist provider.
It first came to a head about two weeks ago when I finally got sick of the level of abuse coming from China, and summarily blocked access -- by the web-server process proper -- to the script that allows one to register (If you're not registered, you cannot post -- ergo, no spam). Today, I put in the entire range of netblocks that are owned by "DataShack, LC" and "Peg Tech" into the same categories as there has been precisely nothing but a barrage of hostile traffic therefrom. The former just seems to be a rogue server-hosting provider; the latter looks to be entirely a Chinese-owned US-based proxy-farm.
If the admins (or bots) at either of the above are reading, Clean up your act! Your "service" is a public nuisance.
Retrocomputing -- It's not just a job, it's an adventure!
Re: The Neverending Fight Against On-line Junk
They are a nuisance. Thank you for all your hard work in keeping this place running so smoothly and elegantly. It may be awkward, but if I read aright what you have done doesn't quite entirely lock out intelligent real people from those locations - should any exist they can email to request an account be generated, and then log in by normal means.
human@world# ask_question --recursive "By what legitimate authority?"
- crfriend
- Master Barista
- Posts: 14486
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: The Neverending Fight Against On-line Junk
True enough, but they'd have to ask really politely and cogently -- something I have to suspect is highly unlikely. The ones originating within the proxy-blocks would get told to get a real ISP. Proxies are pernicious.Tor wrote:[... I]f I read aright what you have done doesn't quite entirely lock out intelligent real people from those locations - should any exist they can email to request an account be generated, and then log in by normal means.
Personally, given the way that China operates, I think that IANA should have allocated the entire country a single Class-C (256 IP addresses) netblock and told them to stuff off until they get a sane view on things. If they want to run the "Great Firewall" then let them run it responsibly and keep the miscreants from running wild; else, let 'em NAT everything through a /24 and make it easy to firewall them.
I'm starting to regard Russia and Ukraine in the same vein. Both are marginally European countries, but both are a lot of trouble. I have seen nothing of value from China, and a whole shed-load of trouble therefrom. I do not like being ham-fisted about stuff like this, but a scalpel-approach just doesn't work given the number of addresses in play.
Retrocomputing -- It's not just a job, it's an adventure!
- melsav
- Member Extraordinaire
- Posts: 431
- Joined: Sat Feb 15, 2014 11:39 am
- Location: Johannesburg South Africa
Re: The Neverending Fight Against On-line Junk
Crfreind. It looks like you have a lot of hard work and trouble to deal with everyday. But I for one appreciate the hard work that you do to keep this forum what it is. Many many thanks ![Very Happy :D](./images/smilies/icon_biggrin.gif)
![Very Happy :D](./images/smilies/icon_biggrin.gif)
Re: The Neverending Fight Against On-line Junk
I Second that.melsav wrote:Crfreind. It looks like you have a lot of hard work and trouble to deal with everyday. But I for one appreciate the hard work that you do to keep this forum what it is. Many many thanks