Starting to go viral?
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
2022-09-30: 31,465 requests from 1,171 unique IP addresses. This would seem to indicate more intent reading by folks even if the number of unique IPs is falling.
My suspicion is that we'll be "back to normal" traffic by midweek next. Nice to get a shot in the arm, though!
My suspicion is that we'll be "back to normal" traffic by midweek next. Nice to get a shot in the arm, though!
Retrocomputing -- It's not just a job, it's an adventure!
Re: Starting to go viral?
It really is a nice shot in the arm. I’m sure the computer experts on this forum have some idea how many of the unique IPs would be bots? I don’t know what a zombie is except from movies.
-
- Member Extraordinaire
- Posts: 1733
- Joined: Fri Oct 24, 2003 7:03 am
- Location: West Midlands, England, UK
Re: Starting to go viral?
Right, everyone: best behaviour for a week!
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
There's no easy way of determining that number as a request from a bot looks very much the same as a request from a person.
A "zombie" in this instance is someone's personal computer that's been hijacked and is being used for nefarious purposes without the victim's knowledge.
Numbers for August 31: 26,030 requests from 722 unique IP addresses. I think things are getting back to normal, even if "normal" is more of a hunch than a concretely known value.
Retrocomputing -- It's not just a job, it's an adventure!
Re: Starting to go viral?
Guess I wonder how many bots typically show up to a site daily. Even 722 seems like a lot of interest to me. Even if half are bots, there are over 300 people checking in on this forum. I find that impressive for what right now is a niche area of the internet.
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
The bald truth is that without a highly detailed analysis process (which would likely yield incomplete results) one simply cannot know. One could, perhaps, identify the ones that identify themselves, but that leaves the ones that don't. It also doesn't catch the various zombies that come from random hijacked PCs in people's homes. It also would not catch the large sections of the globe that are blackholed from (certain0 access by virtue of excessive spam (China and Pakistan, mainly, with several tame proxies in the United States). So, it is possible? Yes, anything is technically possible. It is worth the expenditure in time and effort for a likely incomplete result? I posit, "No."
The data from 2022-09-01: 28,547 hits from 663 unique IP addresses.
Methodology:
Code: Select all
[carl@runabout ~]$ wc -l access.log.2022-09-01
28547 access.log.2022-09-01
[carl@runabout ~]$ awk '{print $1}' access.log.2022-09-01 | sort -u | wc -l
663
[carl@runabout ~]$
Retrocomputing -- It's not just a job, it's an adventure!
Re: Starting to go viral?
Yeah, seems like a waste of time. But still 663 is still a lot of hits even if half are bots/zombies.
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
Here's the only way to identify them via the logs -- by their User-Agent identification (and only if they're being honest, and tell me how many of the ones below (a subset of the entries that actually contain the string "bot", 32 out of 258) are being completely honest:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/104.0.5112.101 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Neevabot/1.0; +https://neeva.com/neevabot)
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0
serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)
I see quite a few in there that the forum software doesn't know about. As you can see, it's very easy to drown in this quagmire.
Retrocomputing -- It's not just a job, it's an adventure!
Re: Starting to go viral?
Wow, is that Greek? Seems like we should just assume at least half of the unique IPs are real people. Still a lot of folkscrfriend wrote: ↑Fri Sep 02, 2022 2:03 pmHere's the only way to identify them via the logs -- by their User-Agent identification (and only if they're being honest, and tell me how many of the ones below (a subset of the entries that actually contain the string "bot", 32 out of 258) are being completely honest:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/104.0.5112.101 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Neevabot/1.0; +https://neeva.com/neevabot)
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0
serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)
I see quite a few in there that the forum software doesn't know about. As you can see, it's very easy to drown in this quagmire.
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
Not Greek but Geek. Now you see what deep-layer admins get to deal with. As I mentioned, that was but a tiny subset (32 of 258 on that one day alone), and each of those would need to be analysed for "usefulness" and validity. Most of the ones in that subset lie outright or are, at best, misleading. At least one ought to be identified by netblock and blackholed. The last one alone amounted to 3300+ hits which adds to the bandwidth draw.
This is a highly fluid environment, and changes each and every day; therefore it'd be a full-time job to identify, classify, and decide what to do with each one discovered. Then there's the problem that the things are mortal and the dead ones need to be removed. This, hopefully begins to bring the scope of the problem into view. And still does not address the zombie problem, which in itself is enormous.
Retrocomputing -- It's not just a job, it's an adventure!
- crfriend
- Master Barista
- Posts: 14433
- Joined: Fri Nov 19, 2004 9:52 pm
- Location: New England (U.S.)
- Contact:
Re: Starting to go viral?
The decline in distinct IP addresses accessing Skirt Cafe continues -- as expected -- to drop. The numbers from 2022-09-02 indicate 430 distinct IP addresses and 27,238 requests.
I wish I'd sampled it before the CNN article published.
I wish I'd sampled it before the CNN article published.
Retrocomputing -- It's not just a job, it's an adventure!
Re: Starting to go viral?
From a peak of 3200 IP addresses, this was a great article. Shows a lot of people took interest enough to google it themselves as I didn’t see a direct link to the forum. That’s distinct interest. Like a snowball rolling downhill, this was awesome but only a start
- mishawakaskirt
- Member Extraordinaire
- Posts: 720
- Joined: Wed Sep 16, 2015 9:59 pm
- Location: INDIANA USA
- Contact:
Re: Starting to go viral?
All these bots, what do they actually do? Search for?? That they hang around here?
All forums and other websites?
What are the ways of the bots oh tech savvy members,?
I do hope the exposure continues, keep the snowball rolling.
Share these articles with others, friends, family, spouse es
. Also on social media. Etc
All forums and other websites?
What are the ways of the bots oh tech savvy members,?
I do hope the exposure continues, keep the snowball rolling.
Share these articles with others, friends, family, spouse es
. Also on social media. Etc
Mishawakaskirt @2wayskirt on Twitter
Avoid the middle man, wear a kilt or skirt.
Avoid the middle man, wear a kilt or skirt.
-
- Member Extraordinaire
- Posts: 2649
- Joined: Mon Dec 16, 2019 4:40 am
- Location: Southeast Michigan
Re: Starting to go viral?
Some of the bots index the public areas of the site for search engine listings.mishawakaskirt wrote: ↑Mon Sep 05, 2022 3:17 am All these bots, what do they actually do? Search for?? That they hang around here?
All forums and other websites?
What are the ways of the bots oh tech savvy members,?
Some of the bots trawl for email addresses, names, etc... for advertising purposes.
The list can go on quite a bit. This site has a short listing of different types:
https://soft-surge.com/what-are-bots/
Also see:
https://www.cloudflare.com/learning/bots/what-is-a-bot/
WikiPedia has a writeup as well:
https://en.wikipedia.org/wiki/Internet_bot
Re: Starting to go viral?
A bot is what’s inside your skirt, at the back.