AI Bots and website scraping at DN.ca (8.Viewing)

ADMIN · 2025-11-28T07:59:45-0500

DUE TO AI ROBOTS WE ARE NOW SEPARATING GUESTS FROM BOTS

Bots have never been in our online user list, we have always ignored them and they do not contribute to site stats. However due to AI scrapping we are seeing hundreds of hourly robot crawlers visit the site setting us over our server limits and forcing us to upgrade our server capacity.

To more closely monitor this activity we are now tracking the bots from TikTok, Google, Microsoft, Amazon etc and have found they are single handedly scrapping our content for AI results and using up almost as much if not more than members in resources. This has driven our costs exponentially and all attempts to block the bots have failed. There are thousands of IP addresses which change at random so blocking has thus far failed. Additionally we have had to upgrade web stats to single out robots from humans.

You will now see how many robots are online at DN.ca but all known robots do not increase usage stats.

We will update this topic as things change moving forward.

MapleDots · 2025-11-28T08:05:39-0500

I read there was a class action lawsuit somewhere because AI is scrapping data without consent. Our costs are insane, I'm getting overage charges everywhere because the bots are sending our usage through the roof.

These bots (other than Google) have no permission to scrape our site for content so I'm looking forward to any legal action against them. Our server went crazy the last few days with AI scrapping content hundreds of times per minute.

The HUSHED section is safe and receives no scraping and we are considering going to a members only version where guests can no longer view unless they become a member. That would not bid well for search results and opens up a whole new can of worms so we are currently in a monitoring state and will make decisions moving forward. We have upgraded all our systems yesterday to handle the extra load but a members only version might be the only way if bot activity continues to grow as rapidly as it has.

MapleDots · 2025-11-28T08:11:05-0500

DN.ca on chatGPT

Here is a previous topic where I was talking about the ChatGPT bods but the Amazon and TikTok bots are crazy as well.

Amazon is the underpinnings (servers) for a lot of AI websites so they are probably all identified as Amazon or AWS.

silentg · 2025-11-28T09:10:17-0500

Yeah they're crawling websites for more data and don't want to pay up.

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity Response to Cloudflare | Hacker News

Why AI Crawlers Are Slowing Down Your Site | InMotion Hosting
AI Crawlers Are Reportedly Draining Site Resources & Skewing Analytics

DomainRecap · 2025-11-28T12:53:54-0500

I truly believe that unless it's stopped, AI is eventually going to kill the internet.

And that's not hyperbole, as it's absolutely nuts out there, with AI bots taking up an ever-increasing slice of total online traffic and bandwidth.

MapleDots · 2025-11-28T13:36:53-0500

Text Box:

43.156.110.253
Robot: Sogou Web Spider
Viewing thread Afternic Fast Transfer - Danger 3 minutes ago
47.128.125.106
Robot: Bytespider
Viewing latest content 3 minutes ago
47.128.113.51
Robot: Bytespider
Viewing latest content 3 minutes ago
66.249.66.11
Robot: Googlebot
Viewing thread Rick Sold AiReviews.com For $1,340,250 USD (LTO) 3 minutes ago
66.249.66.194
Robot: Googlebot
Viewing thread Rick Sold AiReviews.com For $1,340,250 USD (LTO) 3 minutes ago
49.51.166.228
Robot: ThinkBot
Viewing unknown page 3 minutes ago Viewing an error
57.141.4.24
Robot: Meta-ExternalAgent
Viewing unknown page 3 minutes ago Viewing an error
74.7.228.54
Robot: OpenAI Searchbot
Viewing unknown page 3 minutes ago Viewing an error
124.156.194.80
Robot: Sogou Web Spider
Viewing thread Government of Anguilla and Identity Digital Announce Partnership 3 minutes ago
47.128.62.237
Robot: Bytespider
Viewing latest content 3 minutes ago
47.128.37.237
Robot: Bytespider
Viewing latest content 3 minutes ago
66.249.66.38
Robot: Googlebot
Viewing thread OldFashioned.ca 4 minutes ago
66.249.66.167
Robot: Googlebot
Viewing thread Show off your Christmas flavoured domains 4 minutes ago
88.99.80.227
Robot: UptimeRobot
Viewing thread canuckdomains 4 minutes ago
43.159.39.29
Robot: Sogou Web Spider
Viewing forum list 4 minutes ago Viewing an error
47.128.20.47
Robot: Bytespider
Viewing latest content 4 minutes ago
47.128.127.51
Robot: Bytespider
Viewing latest content 4 minutes ago
57.141.4.58
Robot: Meta-ExternalAgent
Viewing thread Oliver.com Sold For 5 Figures 5 minutes ago
66.249.66.74
Robot: Googlebot
Viewing thread GoDaddy and NameSilo gained; Team Internet Group and Tucows faced headwinds 5 minutes ago
47.128.113.156
Robot: Bytespider
Viewing latest content 5 minutes ago
47.128.122.4
Robot: Bytespider
Viewing latest content 5 minutes ago
43.131.23.154
Robot: ThinkBot
Viewing unknown page 5 minutes ago Viewing an error
47.128.61.157
Robot: Bytespider
Viewing latest content 6 minutes ago
47.128.41.59
Robot: Bytespider
Viewing latest content 6 minutes ago
119.28.89.249
Robot: ThinkBot
Viewing unknown page 6 minutes ago Viewing an error
17.22.237.12
Robot: Applebot
Viewing thread TBR Musings - August 4, 2021 6 minutes ago
43.154.250.181
Robot: ThinkBot
Viewing unknown page 6 minutes ago Viewing an error
52.167.144.137
Robot: Bingbot
Viewing thread What domain was worthless in 2000 but priceless by 2025? 6 minutes ago
57.141.4.43
Robot: Meta-ExternalAgent
Viewing thread CMI.ca - Sold $16,000 CAD 6 minutes ago
47.128.35.184
Robot: Bytespider
Viewing latest content 7 minutes ago
47.128.111.249
Robot: Bytespider
Viewing latest content 7 minutes ago
43.156.247.77
Robot: Sogou Web Spider
Viewing thread My favourite Twitter post today 7 minutes ago
116.179.32.227
Robot: Baidu Spider
Viewing thread Is the CIRA Still Sending out .CA Registration Emails? 7 minutes ago
150.109.46.88
Robot: ThinkBot
Viewing unknown page 7 minutes ago Viewing an error
47.128.126.168
Robot: Bytespider
Viewing latest content 7 minutes ago
47.128.33.62
Robot: Bytespider
Viewing latest content 7 minutes ago
68.221.67.171
Robot: ChatGPT-User
Viewing thread Afternic’s 2024 TLD Rundown: .com Reigns, .ai Surges 8 minutes ago
223.109.207.89
Robot: Sogou Web Spider
Viewing thread Domain Meets & Events 8 minutes ago
43.157.20.63
Robot: ThinkBot
Viewing unknown page 8 minutes ago Viewing an error
47.128.63.211
Robot: Bytespider
Viewing latest content 8 minutes ago
47.128.120.75
Robot: Bytespider
Viewing latest content 8 minutes ago
43.134.91.37
Robot: Sogou Web Spider
Viewing thread AutoMed.ai Sold For $25,000 USD At Atom.com 9 minutes ago
43.157.174.69
Robot: ThinkBot
Viewing unknown page 9 minutes ago Viewing an error
47.128.37.17
Robot: Bytespider
Viewing latest content 9 minutes ago
47.128.42.12
Robot: Bytespider
Viewing latest content 9 minutes ago
78.46.215.1
Robot: UptimeRobot
Viewing thread canuckdomains 9 minutes ago
8.29.198.27
Robot: Feedly Fetcher
Viewing forum list 9 minutes ago
40.77.167.59
Robot: Bingbot
Viewing thread Andrew Rosener & MediaOptions Pioneer New Frontiers in Digital Asset Investment 10 minutes ago
47.128.118.56
Robot: Bytespider
Viewing latest content 10 minutes ago
47.128.61.130
Robot: Bytespider
Viewing latest content 10 minutes ago
223.109.211.141
Robot: Sogou Web Spider
Viewing thread Most expensive domain ever sold 10 minutes ago
213.180.203.195
Robot: Yandex Bot
Viewing unknown page 10 minutes ago Viewing an error
43.163.104.54
Robot: ThinkBot
Viewing thread canuckdomains 10 minutes ago
95.217.59.55
Robot: Semantic Visions
Viewing thread dnfs 10 minutes ago
47.128.125.114
Robot: Bytespider
Viewing latest content 11 minutes ago
47.128.99.166
Robot: Bytespider
Viewing latest content 11 minutes ago
43.131.26.226
Robot: ThinkBot
Viewing unknown page 11 minutes ago Viewing an error
57.141.4.39
Robot: Meta-ExternalAgent
Viewing unknown page 11 minutes ago Viewing an error
199.16.157.183
Robot: Twitterbot
Viewing thread DomainEasy Launches an Ambitious Platform 11 minutes ago
220.181.108.81
Robot: Baidu Spider
Viewing thread Opponent.ca 11 minutes ago Viewing an error
47.128.109.34
Robot: Bytespider
Viewing latest content 12 minutes ago
47.128.30.128
Robot: Bytespider
Viewing latest content 12 minutes ago
216.73.216.162
Robot: ClaudeBot
Viewing thread More gloom predicted for .com 12 minutes ago
57.141.4.53
Robot: Meta-ExternalAgent
Viewing thread Drew.ai Sold For $25,000 USD At Sedo By Mark Ghoriafi 12 minutes ago
47.128.52.75
Robot: Bytespider
Viewing latest content 12 minutes ago
47.128.98.231
Robot: Bytespider
Viewing latest content 12 minutes ago
40.77.167.58
Robot: Bingbot
Viewing thread HealRight.com Sold For $210,000 USD 13 minutes ago
43.135.115.233
Robot: ThinkBot
Viewing unknown page 13 minutes ago Viewing an error
170.106.37.134
Robot: ThinkBot
Viewing unknown page 13 minutes ago Viewing an error
47.128.44.117
Robot: Bytespider
Viewing latest content 13 minutes ago
47.128.52.94
Robot: Bytespider
Viewing latest content 13 minutes ago
17.246.23.101
Robot: Applebot
Viewing thread CIRA is raising prices 14 minutes ago
47.128.99.109
Robot: Bytespider
Viewing latest content 14 minutes ago
47.128.51.123
Robot: Bytespider
Viewing latest content 14 minutes ago
52.167.144.147
Robot: Bingbot
Viewing unknown page 14 minutes ago Viewing an error
168.119.123.75
Robot: UptimeRobot
Viewing thread canuckdomains 14 minutes ago
43.130.141.193
Robot: ThinkBot
Viewing unknown page 15 minutes ago Viewing an error
47.128.47.66
Robot: Bytespider
Viewing latest content 15 minutes ago
47.128.40.228
Robot: Bytespider
Viewing latest content 15 minutes ago
57.141.4.51
Robot: Meta-ExternalAgent
Viewing member profile Prashanna 15 minutes ago
43.135.211.148
Robot: ThinkBot
Viewing unknown page 16 minutes ago Viewing an error
47.128.35.215
Robot: Bytespider
Viewing latest content 16 minutes ago
47.128.50.111
Robot: Bytespider
Viewing latest content 16 minutes ago
116.179.32.233
Robot: Baidu Spider
Viewing thread MyID TBR Issue 16 minutes ago
57.141.4.1
Robot: Meta-ExternalAgent
Viewing unknown page 16 minutes ago Viewing an error
8.29.198.26
Robot: Feedly Fetcher
Viewing forum list 16 minutes ago
47.128.114.4
Robot: Bytespider
Viewing latest content 16 minutes ago
47.128.96.230
Robot: Bytespider
Viewing latest content 16 minutes ago
66.249.66.12
Robot: Googlebot
Viewing thread Merging DN.ca with NamePros? 17 minutes ago
57.141.4.40
Robot: Meta-ExternalAgent
Viewing unknown page 17 minutes ago Viewing an error
66.249.66.37
Robot: Googlebot
Viewing thread If GPT-6 can build websites, which domains will skyrocket in value? 17 minutes ago
47.128.44.104
Robot: Bytespider
Viewing latest content 17 minutes ago
47.128.99.30
Robot: Bytespider
Viewing latest content 17 minutes ago
57.141.4.38
Robot: Meta-ExternalAgent
Viewing unknown page 17 minutes ago Viewing an error
54.39.177.92
Robot: Linkfluence
Viewing unknown page 18 minutes ago Viewing an error
47.128.18.180
Robot: Bytespider
Viewing latest content 18 minutes ago
47.128.35.68
Robot: Bytespider
Viewing latest content 18 minutes ago
23.98.142.191
Robot: ChatGPT-User
Viewing thread Get Verified 18 minutes ago
43.164.196.57
Robot: ThinkBot
Viewing unknown page 18 minutes ago Viewing an error
57.141.4.77
Robot: Meta-ExternalAgent
Viewing thread Bellucci.ca sold for $8,888 USD 19 minutes ago
173.252.87.3
Robot: Facebook External Hit
Viewing unknown page 19 minutes ago Viewing an error
47.128.40.34
Robot: Bytespider
Viewing latest content 19 minutes ago
47.128.22.172
Robot: Bytespider
Viewing latest content 19 minutes ago
66.249.66.165
Robot: Googlebot
Viewing latest content 19 minutes ago
43.157.175.122
Robot: ThinkBot
Viewing unknown page 19 minutes ago Viewing an error
49.13.134.145
Robot: UptimeRobot
Viewing thread canuckdomains 19 minutes ago
57.141.4.35
Robot: Meta-ExternalAgent
Viewing unknown page 19 minutes ago Viewing an error
43.130.154.56
Robot: ThinkBot
Viewing unknown page 19 minutes ago Viewing an error
66.249.66.193
Robot: Googlebot
Viewing thread Diamond.com Sold By Saw.com Brokerage 20 minutes ago
47.128.51.53
Robot: Bytespider
Viewing latest content 20 minutes ago
47.128.115.152
Robot: Bytespider
Viewing latest content 20 minutes ago
47.128.122.247
Robot: Bytespider
Viewing latest content 20 minutes ago
47.128.50.46
Robot: Bytespider
Viewing latest content 20 minutes ago
220.181.108.158
Robot: Baidu Spider
Viewing thread Geo domains in Europe 21 minutes ago
43.153.192.98
Robot: ThinkBot
Viewing unknown page 21 minutes ago Viewing an error
43.131.45.213
Robot: ThinkBot
Viewing unknown page 21 minutes ago Viewing an error
47.128.112.135
Robot: Bytespider
Viewing latest content 21 minutes ago
47.128.41.167
Robot: Bytespider
Viewing latest content 21 minutes ago
57.141.4.12
Robot: Meta-ExternalAgent
Viewing thread Market Guidelines 22 minutes ago
57.141.4.23
Robot: Meta-ExternalAgent
Viewing unknown page 22 minutes ago Viewing an error
47.128.96.78
Robot: Bytespider
Viewing latest content 22 minutes ago
47.128.21.55
Robot: Bytespider
Viewing latest content 22 minutes ago
43.155.162.41
Robot: ThinkBot
Viewing unknown page 22 minutes ago Viewing an error
47.128.41.85
Robot: Bytespider
Viewing latest content 23 minutes ago
47.128.58.166
Robot: Bytespider
Viewing latest content 23 minutes ago
66.249.66.195
Robot: Googlebot
Viewing thread Why Are There So Many Domain Conferences Happening Right Now? 23 minutes ago
43.131.32.36
Robot: ThinkBot
Viewing unknown page 23 minutes ago Viewing an error
47.128.34.229
Robot: Bytespider
Viewing latest content 24 minutes ago
47.128.41.211
Robot: Bytespider
Viewing latest content 24 minutes ago
40.77.167.75
Robot: Bingbot
Viewing thread Goons.ca is for sale 24 minutes ago
157.90.155.240
Robot: UptimeRobot
Viewing thread canuckdomains 24 minutes ago
47.128.41.43
Robot: Bytespider
Viewing latest content 24 minutes ago
47.128.26.23
Robot: Bytespider
Viewing latest content 24 minutes ago
34.244.181.137
Robot: Snap URL Preview Service
Viewing thread nafti 25 minutes ago
220.181.108.167
Robot: Baidu Spider
Viewing thread Welcome, NameBio 25 minutes ago
47.128.41.81
Robot: Bytespider
Viewing latest content 25 minutes ago
47.128.61.88
Robot: Bytespider
Viewing latest content 25 minutes ago
43.154.127.188
Robot: ThinkBot
Viewing unknown page 26 minutes ago Viewing an error
47.128.51.138
Robot: Bytespider
Viewing latest content 26 minutes ago
47.128.126.171
Robot: Bytespider
Viewing latest content 26 minutes ago
43.157.191.20
Robot: ThinkBot
Viewing unknown page 27 minutes ago Viewing an error
43.156.63.87
Robot: Sogou Web Spider
Viewing member profile marouane25 27 minutes ago
57.141.4.72
Robot: Meta-ExternalAgent
Viewing unknown page 27 minutes ago Viewing an error
47.128.114.246
Robot: Bytespider
Viewing latest content 27 minutes ago
47.128.115.64
Robot: Bytespider
Viewing latest content 27 minutes ago
144.76.22.77
Robot: Trendiction-Bot
Viewing unknown page 27 minutes ago Viewing an error
54.144.185.255
Robot: Amazonbot
Viewing thread Seller hiked the price from $38,000 to $380,000 after discovering Pay.com.au was the buyer 27 minutes ago
3.22.209.42
Robot: Python Requests Library
Viewing unknown page 27 minutes ago Viewing an error
34.236.41.241
Robot: Amazonbot
Viewing member profile gebtillet 27 minutes ago
18.214.124.6
Robot: Amazonbot
Viewing thread TBR Musings - September 29, 2021 28 minutes ago
3.221.156.96
Robot: Amazonbot
Viewing latest content 28 minutes ago
98.82.63.147
Robot: Amazonbot
Viewing latest content 28 minutes ago
47.128.39.72
Robot: Bytespider
Viewing latest content 28 minutes ago
47.128.33.42
Robot: Bytespider
Viewing latest content 28 minutes ago
3.93.253.174
Robot: Amazonbot
Viewing latest content 28 minutes ago
101.32.52.164
Robot: ThinkBot
Viewing thread canuckdomains 28 minutes ago
43.157.62.101
Robot: ThinkBot
Viewing unknown page 28 minutes ago Viewing an error
34.206.212.24
Robot: Amazonbot
Viewing latest content 28 minutes ago
98.83.177.42
Robot: Amazonbot
Viewing latest content 28 minutes ago
57.141.4.6
Robot: Meta-ExternalAgent
Viewing thread Squarespace buys Google Domains for $180 million 28 minutes ago
47.128.54.63
Robot: Bytespider
Viewing latest content 28 minutes ago
47.128.96.54
Robot: Bytespider
Viewing latest content 28 minutes ago
40.77.167.71
Robot: Bingbot
Viewing thread MCM.ai Sold For $50,000 USD 29 minutes ago
57.141.4.78
Robot: Meta-ExternalAgent
Viewing unknown page 29 minutes ago Viewing an error
47.128.58.182
Robot: Bytespider
Viewing latest content 29 minutes ago
47.128.26.63
Robot: Bytespider
Viewing latest content 29 minutes ago
18.192.166.72
Robot: UptimeRobot
Viewing thread canuckdomains 29 minutes ago
116.179.32.92
Robot: Baidu Spider
Viewing unknown page 30 minutes ago Viewing an error
47.128.32.73
Robot: Bytespider
Viewing latest content 30 minutes ago
47.128.125.55
Robot: Bytespider
Viewing latest content 30 minutes ago
8.29.198.25
Robot: Feedly Fetcher
Viewing forum list 31 minutes ago
40.77.167.33
Robot: Bingbot
Viewing thread Adsense account buyer 31 minutes ago
47.128.111.25
Robot: Bytespider
Viewing latest content 31 minutes ago
47.128.17.194
Robot: Bytespider
Viewing latest content 31 minutes ago
47.128.40.63
Robot: Bytespider
Viewing latest content 32 minutes ago
47.128.36.96
Robot: Bytespider
Viewing latest content 32 minutes ago
100.28.57.133
Robot: Amazonbot
Viewing unknown page 32 minutes ago Viewing an error
43.166.224.244
Robot: ThinkBot
Viewing unknown page 32 minutes ago Viewing an error
57.141.4.20
Robot: Meta-ExternalAgent
Viewing thread Is CIRA whois down?? 32 minutes ago
43.135.36.201
Robot: ThinkBot
Viewing unknown page 32 minutes ago Viewing an error
47.128.25.172
Robot: Bytespider
Viewing latest content 33 minutes ago
47.128.52.26
Robot: Bytespider
Viewing latest content 33 minutes ago
173.252.82.113
Robot: Facebook External Hit
Viewing unknown page 33 minutes ago Viewing an error
47.128.118.14
Robot: Bytespider
Viewing latest content 33 minutes ago
47.128.115.103
Robot: Bytespider
Viewing latest content 33 minutes ago
43.154.140.188
Robot: ThinkBot
Viewing unknown page 33 minutes ago Viewing an error
47.128.44.142
Robot: Bytespider
Viewing latest content 34 minutes ago
47.128.17.165
Robot: Bytespider
Viewing latest content 34 minutes ago
220.181.108.174
Robot: Baidu Spider
Viewing thread micahdomains 34 minutes ago
43.157.181.189
Robot: ThinkBot
Viewing thread canuckdomains 35 minutes ago
17.22.237.49
Robot: Applebot
Viewing thread WHOIS is dead 35 minutes ago
47.128.46.247
Robot: Bytespider
Viewing latest content 35 minutes ago
47.128.20.104
Robot: Bytespider
Viewing latest content 35 minutes ago
57.141.4.7
Robot: Meta-ExternalAgent
Viewing member profile Bertrell 35 minutes ago
57.141.4.3
Robot: Meta-ExternalAgent
Viewing thread 65,000 Dead Spawning Salmon in BC Canada 35 minutes ago
43.157.22.109
Robot: ThinkBot
Viewing unknown page 35 minutes ago Viewing an error
119.28.140.106
Robot: ThinkBot
Viewing thread canuckdomains 35 minutes ago
66.249.66.76
Robot: Generic Bot
Viewing thread X2.ca 36 minutes ago
47.128.54.151
Robot: Bytespider
Viewing latest content 36 minutes ago
47.128.55.174
Robot: Bytespider
Viewing latest content 36 minutes ago
47.128.31.97
Robot: Bytespider
Viewing latest content 37 minutes ago
47.128.116.231
Robot: Bytespider
Viewing latest content 37 minutes ago
20.194.1.2
Robot: ChatGPT-User
Viewing thread The dilemma listing on DAN.com when using landers. 37 minutes ago
47.128.99.23
Robot: Bytespider
Viewing latest content 37 minutes ago
47.128.18.156
Robot: Bytespider
Viewing latest content 37 minutes ago
52.167.144.229
Robot: Bingbot
Viewing thread M4 and M7 dot ca 38 minutes ago Viewing an error
74.7.242.141
Robot: OpenAI Searchbot
Viewing thread Cybersquatting dispute decision reversed on appeal 38 minutes ago
47.128.126.162
Robot: Bytespider
Viewing latest content 38 minutes ago
47.128.45.232
Robot: Bytespider
Viewing latest content 38 minutes ago
66.249.66.75
Robot: Googlebot
Viewing thread Edible Arrangements takes on a whole new meaning 38 minutes ago
20.215.214.21
Robot: ChatGPT-User
Viewing thread Veramed.com sold for $167,500 by Mike Mann 39 minutes ago
140.179.74.207
Robot: Headless Chrome
Viewing thread nafti 39 minutes ago
116.179.32.12
Robot: Baidu Spider
Viewing member profile Spitfire 39 minutes ago
47.128.118.133
Robot: Bytespider
Viewing latest content 39 minutes ago
47.128.111.203
Robot: Bytespider
Viewing latest content 39 minutes ago
43.164.197.209
Robot: ThinkBot
Viewing unknown page 39 minutes ago Viewing an error
47.128.17.18
Robot: Bytespider
Viewing latest content 40 minutes ago
47.128.126.216
Robot: Bytespider
Viewing latest content 40 minutes ago
20.194.157.191
Robot: ChatGPT-User
Viewing thread Sedo and self verification 40 minutes ago
104.210.140.140
Robot: OpenAI Searchbot
Viewing thread canuckdomains 40 minutes ago
47.128.27.209
Robot: Bytespider
Viewing latest content 41 minutes ago
47.128.57.31
Robot: Bytespider
Viewing latest content 41 minutes ago
57.141.4.71
Robot: Meta-ExternalAgent
Viewing unknown page 41 minutes ago Viewing an error
3.79.134.69
Robot: UptimeRobot
Viewing thread canuckdomains 41 minutes ago
47.128.37.24
Robot: Bytespider
Viewing latest content 41 minutes ago
47.128.31.158
Robot: Bytespider
Viewing latest content 41 minutes ago
173.252.127.21
Robot: Facebook External Hit
Viewing thread nafti 42 minutes ago
47.128.127.249
Robot: Bytespider
Viewing latest content 42 minutes ago
47.128.97.31
Robot: Bytespider
Viewing latest content 42 minutes ago
17.246.19.70
Robot: Applebot
Viewing thread Hurting.ca 42 minutes ago
43.156.232.190
Robot: ThinkBot
Viewing unknown page 42 minutes ago Viewing an error
47.128.62.58
Robot: Bytespider
Viewing latest content 43 minutes ago
47.128.59.44
Robot: Bytespider
Viewing latest content 43 minutes ago
66.249.66.39
Robot: Googlebot
Viewing thread Welcome, abdo 44 minutes ago
47.128.122.92
Robot: Bytespider
Viewing latest content 44 minutes ago
47.128.56.221
Robot: Bytespider
Viewing latest content 44 minutes ago
57.141.4.70
Robot: Meta-ExternalAgent
Viewing unknown page 44 minutes ago Viewing an error
47.128.21.64
Robot: Bytespider
Viewing latest content 45 minutes ago
47.128.29.144
Robot: Bytespider
Viewing latest content 45 minutes ago
52.167.144.216
Robot: Bingbot
Viewing unknown page 45 minutes ago Viewing an error
52.187.246.134
Robot: ChatGPT-User
Viewing category Market 45 minutes ago
43.166.244.66
Robot: ThinkBot
Viewing unknown page 45 minutes ago Viewing an error
52.187.246.135
Robot: ChatGPT-User
Viewing forum list 45 minutes ago
47.128.59.130
Robot: Bytespider
Viewing latest content 46 minutes ago
47.128.27.133
Robot: Bytespider
Viewing latest content 46 minutes ago
43.130.101.151
Robot: ThinkBot
Viewing unknown page 46 minutes ago Viewing an error
47.128.114.12
Robot: Bytespider
Viewing latest content 46 minutes ago
47.128.113.34
Robot: Bytespider
Viewing latest content 46 minutes ago
47.128.113.95
Robot: Bytespider
Viewing latest content 47 minutes ago
47.128.16.253
Robot: Bytespider
Viewing latest content 47 minutes ago
18.232.36.1
Robot: Amazonbot
Viewing latest content 48 minutes ago
3.216.227.216
Robot: Amazonbot
Viewing thread Velocity.io Sold For $100,000 USD 48 minutes ago
47.128.48.233
Robot: Bytespider
Viewing latest content 48 minutes ago
47.128.63.159
Robot: Bytespider
Viewing latest content 48 minutes ago
116.179.32.76
Robot: Baidu Spider
Viewing unknown page 48 minutes ago Viewing an error
47.128.51.119
Robot: Bytespider
Viewing latest content 49 minutes ago
47.128.49.23
Robot: Bytespider
Viewing latest content 49 minutes ago
38.250.157.106
Robot: Mobile AdSense
Viewing unknown page 49 minutes ago Viewing an error
43.166.242.189
Robot: ThinkBot
Viewing unknown page 50 minutes ago Viewing an error
47.128.32.92
Robot: Bytespider
Viewing latest content 50 minutes ago
47.128.25.164
Robot: Bytespider
Viewing latest content 50 minutes ago
43.153.62.161
Robot: ThinkBot
Viewing unknown page 50 minutes ago Viewing an error
47.128.28.11
Robot: Bytespider
Viewing latest content 50 minutes ago
47.128.43.201
Robot: Bytespider
Viewing latest content 50 minutes ago
87.250.224.55
Robot: Yandex Bot
Viewing unknown page 51 minutes ago Viewing an error
223.109.207.53
Robot: Sogou Web Spider
Viewing thread AutoCoverage .ca and 5 other .CAs $99 each 51 minutes ago
168.119.96.239
Robot: UptimeRobot
Viewing thread canuckdomains 51 minutes ago
47.128.45.170
Robot: Bytespider
Viewing latest content 51 minutes ago
47.128.29.109
Robot: Bytespider
Viewing latest content 51 minutes ago
101.33.81.73
Robot: ThinkBot
Viewing unknown page 52 minutes ago Viewing an error
47.128.98.146
Robot: Bytespider
Viewing latest content 52 minutes ago
47.128.123.185
Robot: Bytespider
Viewing latest content 52 minutes ago
57.141.4.31
Robot: Meta-ExternalAgent
Viewing thread Spaceship Domain Marketplace 52 minutes ago
116.179.32.135
Robot: Baidu Spider
Viewing thread BlockchainBarcelona.com 52 minutes ago
47.128.17.92
Robot: Bytespider
Viewing latest content 53 minutes ago
47.128.56.188
Robot: Bytespider
Viewing latest content 53 minutes ago
52.200.93.170
Robot: Amazonbot
Viewing latest content 53 minutes ago
57.141.4.22
Robot: Meta-ExternalAgent
Viewing unknown page 53 minutes ago Viewing an error
18.215.49.176
Robot: Amazonbot
Viewing latest content 53 minutes ago
54.83.240.58
Robot: Amazonbot
Viewing latest content 53 minutes ago
52.54.95.127
Robot: Amazonbot
Viewing latest content 53 minutes ago
34.193.2.57
Robot: Amazonbot
Viewing latest content 54 minutes ago
47.128.125.103
Robot: Bytespider
Viewing latest content 54 minutes ago
47.128.22.118
Robot: Bytespider
Viewing latest content 54 minutes ago
52.0.105.244
Robot: Amazonbot
Viewing latest content 54 minutes ago
43.134.186.61
Robot: ThinkBot
Viewing unknown page 54 minutes ago Viewing an error
47.128.120.11
Robot: Bytespider
Viewing latest content 54 minutes ago
47.128.18.250
Robot: Bytespider
Viewing latest content 54 minutes ago
47.128.43.47
Robot: Bytespider
Viewing latest content 55 minutes ago
47.128.114.65
Robot: Bytespider
Viewing latest content 55 minutes ago
47.128.29.96
Robot: Bytespider
Viewing latest content 56 minutes ago
47.128.27.190
Robot: Bytespider
Viewing latest content 56 minutes ago
167.235.143.113
Robot: UptimeRobot
Viewing thread canuckdomains 56 minutes ago
47.128.41.76
Robot: Bytespider
Viewing latest content 57 minutes ago
47.128.110.134
Robot: Bytespider
Viewing latest content 57 minutes ago
220.181.108.114
Robot: Baidu Spider
Viewing member profile realone 57 minutes ago
40.77.167.55
Robot: Bingbot
Viewing thread Company loses UDRP against domain it could have bought for $1,995 57 minutes ago
5.255.231.195
Robot: Yandex Bot
Viewing unknown page 58 minutes ago Viewing an error
47.128.96.6
Robot: Bytespider
Viewing latest content 58 minutes ago
47.128.127.212
Robot: Bytespider
Viewing latest content 58 minutes ago
57.141.4.21
Robot: Meta-ExternalAgent
Viewing unknown page 58 minutes ago Viewing an error
47.128.32.221
Robot: Bytespider
Viewing latest content 59 minutes ago
47.128.96.245
Robot: Bytespider
Viewing latest content 59 minutes ago
57.141.4.27
Robot: Meta-ExternalAgent
Viewing unknown page 59 minutes ago Viewing an error
43.130.72.40
Robot: ThinkBot
Viewing unknown page 59 minutes ago Viewing an error
40.77.167.47
Robot: Bingbot
Viewing thread Super.xyz Sold For $287,607 USD At Atom.com 59 minutes ago
47.128.114.76
Robot: Bytespider
Viewing latest content 59 minutes ago
47.128.45.20
Robot: Bytespider
Viewing latest content 59 minutes ago
40.77.167.74
Robot: Bingbot
Viewing thread Foxx.ca - Nice Site - Nice use of double XX Today at 12:35 PM
43.166.128.86
Robot: ThinkBot
Viewing thread canuckdomains Today at 12:35 PM
47.128.118.28
Robot: Bytespider
Viewing latest content Today at 12:34 PM
47.128.117.201
Robot: Bytespider
Viewing latest content Today at 12:34 PM
47.128.120.232
Robot: Bytespider
Viewing latest content Today at 12:34 PM
47.128.32.232
Robot: Bytespider
Viewing latest content Today at 12:34 PM
78.47.98.55
Robot: UptimeRobot
Viewing thread canuckdomains Today at 12:33 PM
57.141.4.52
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:33 PM Viewing an error
220.181.108.146
Robot: Baidu Spider
Viewing member profile Rebel Today at 12:33 PM
43.159.143.190
Robot: ThinkBot
Viewing member profile Aquarius Today at 12:33 PM
43.130.26.3
Robot: ThinkBot
Viewing unknown page Today at 12:33 PM Viewing an error
47.128.45.160
Robot: Bytespider
Viewing latest content Today at 12:33 PM
47.128.114.18
Robot: Bytespider
Viewing latest content Today at 12:33 PM
66.249.66.166
Robot: Googlebot
Viewing thread The TLD Extension Showdown Today at 12:33 PM
74.7.229.224
Robot: OpenAI Searchbot
Viewing thread Royal Bank of Canada launches RBCx, with the .com Today at 12:33 PM
47.128.122.40
Robot: Bytespider
Viewing latest content Today at 12:32 PM
47.128.38.66
Robot: Bytespider
Viewing latest content Today at 12:32 PM
95.217.84.249
Robot: Semantic Visions
Viewing thread dnfs Today at 12:31 PM
47.128.127.36
Robot: Bytespider
Viewing latest content Today at 12:31 PM
47.128.56.79
Robot: Bytespider
Viewing latest content Today at 12:31 PM
47.128.21.75
Robot: Bytespider
Viewing latest content Today at 12:30 PM
47.128.21.174
Robot: Bytespider
Viewing latest content Today at 12:30 PM
57.141.4.15
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:30 PM Viewing an error
47.128.50.33
Robot: Bytespider
Viewing latest content Today at 12:30 PM
47.128.125.239
Robot: Bytespider
Viewing latest content Today at 12:30 PM
134.122.135.50
Robot: Googlebot
Viewing unknown page Today at 12:29 PM Viewing an error
47.128.45.47
Robot: Bytespider
Viewing latest content Today at 12:29 PM
47.128.25.109
Robot: Bytespider
Viewing latest content Today at 12:29 PM
52.167.144.160
Robot: Bingbot
Viewing thread Two letter .ca in the wild Today at 12:29 PM
116.179.32.14
Robot: Baidu Spider
Viewing thread Big domain name drops of 2020 Today at 12:28 PM
57.141.4.14
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:28 PM Viewing an error
49.13.164.148
Robot: UptimeRobot
Viewing thread canuckdomains Today at 12:28 PM
47.128.113.44
Robot: Bytespider
Viewing latest content Today at 12:28 PM
47.128.119.189
Robot: Bytespider
Viewing latest content Today at 12:28 PM
49.51.141.76
Robot: ThinkBot
Viewing thread FROSTHAVEN.CA — Premium Brandable .CA Domain For Sale Today at 12:27 PM
47.128.30.199
Robot: Bytespider
Viewing latest content Today at 12:27 PM
47.128.36.235
Robot: Bytespider
Viewing latest content Today at 12:27 PM
47.128.39.84
Robot: Bytespider
Viewing latest content Today at 12:26 PM
47.128.34.8
Robot: Bytespider
Viewing latest content Today at 12:26 PM
47.128.29.82
Robot: Bytespider
Viewing latest content Today at 12:26 PM
47.128.59.34
Robot: Bytespider
Viewing latest content Today at 12:26 PM
17.241.75.213
Robot: Applebot
Viewing thread I advise everyone to use Escrow only for large amounts of money Today at 12:25 PM
47.128.17.111
Robot: Bytespider
Viewing latest content Today at 12:25 PM
47.128.116.87
Robot: Bytespider
Viewing latest content Today at 12:25 PM
57.141.4.50
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:25 PM Viewing an error
52.230.164.185
Robot: ChatGPT-User
Viewing thread Canadians are pivoting online in record numbers with .CA domains Today at 12:24 PM
47.128.120.246
Robot: Bytespider
Viewing latest content Today at 12:24 PM
47.128.16.82
Robot: Bytespider
Viewing latest content Today at 12:24 PM
220.181.108.157
Robot: Baidu Spider
Viewing thread What a load of bull crap Today at 12:24 PM
189.78.105.139
Robot: Mobile AdSense
Viewing unknown page Today at 12:23 PM Viewing an error
47.128.111.172
Robot: Bytespider
Viewing latest content Today at 12:23 PM
47.128.98.4
Robot: Bytespider
Viewing latest content Today at 12:23 PM
17.241.219.139
Robot: Applebot
Viewing thread Sorry, but you're going to have to pay me to take this domain name Today at 12:23 PM
43.155.26.193
Robot: ThinkBot
Viewing unknown page Today at 12:23 PM Viewing an error
223.109.211.187
Robot: Sogou Web Spider
Viewing thread Practice your posting skills Today at 12:23 PM
47.128.45.90
Robot: Bytespider
Viewing latest content Today at 12:22 PM
47.128.52.9
Robot: Bytespider
Viewing latest content Today at 12:22 PM
52.167.144.231
Robot: Bingbot
Viewing member profile artem Today at 12:22 PM
47.128.42.128
Robot: Bytespider
Viewing latest content Today at 12:22 PM
47.128.57.127
Robot: Bytespider
Viewing latest content Today at 12:22 PM
47.128.55.60
Robot: Bytespider
Viewing latest content Today at 12:21 PM
47.128.47.192
Robot: Bytespider
Viewing latest content Today at 12:21 PM
52.167.144.162
Robot: Bingbot
Viewing unknown page Today at 12:21 PM Viewing an error
47.128.62.164
Robot: Bytespider
Viewing latest content Today at 12:20 PM
47.128.48.0
Robot: Bytespider
Viewing latest content Today at 12:20 PM
47.128.54.169
Robot: Bytespider
Viewing latest content Today at 12:19 PM
47.128.62.144
Robot: Bytespider
Viewing latest content Today at 12:19 PM
43.165.189.110
Robot: ThinkBot
Viewing unknown page Today at 12:18 PM Viewing an error
47.128.17.200
Robot: Bytespider
Viewing latest content Today at 12:18 PM
47.128.96.66
Robot: Bytespider
Viewing latest content Today at 12:18 PM
57.141.4.56
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:18 PM Viewing an error
49.13.24.81
Robot: UptimeRobot
Viewing thread canuckdomains Today at 12:18 PM
47.128.42.5
Robot: Bytespider
Viewing latest content Today at 12:18 PM
47.128.44.12
Robot: Bytespider
Viewing latest content Today at 12:18 PM
57.141.4.54
Robot: Meta-ExternalAgent
Viewing unknown page Today at 12:17 PM Viewing an error
17.22.253.84
Robot: Applebot
Viewing member profile supraturbo Today at 12:17 PM
47.128.45.120
Robot: Bytespider
Viewing latest content Today at 12:17 PM
47.128.36.53
Robot: Bytespider
Viewing latest content Today at 12:17 PM
173.252.127.0
Robot: Facebook External Hit
Viewing thread nafti Today at 12:17 PM
43.133.66.51
Robot: ThinkBot
Viewing unknown page Today at 12:16 PM Viewing an error
47.128.119.73
Robot: Bytespider
Viewing latest content Today at 12:16 PM
47.128.35.203
Robot: Bytespider
Viewing latest content Today at 12:16 PM
47.128.51.242
Robot: Bytespider
Viewing latest content Today at 12:15 PM
47.128.120.48
Robot: Bytespider
Viewing latest content Today at 12:15 PM
47.128.19.219
Robot: Bytespider
Viewing latest content Today at 12:14 PM
47.128.125.42
Robot: Bytespider
Viewing latest content Today at 12:14 PM
95.216.226.163
Robot: Semantic Visions
Viewing forum Sold Domains Today at 12:14 PM
47.128.113.142
Robot: Bytespider
Viewing latest content Today at 12:13 PM
47.128.20.182
Robot: Bytespider
Viewing latest content Today at 12:13 PM
43.157.43.147
Robot: ThinkBot
Viewing unknown page Today at 12:13 PM Viewing an error
23.178.112.210
Robot: Let's Encrypt validation server
Viewing unknown page Today at 12:13 PM Viewing an error
23.178.112.211
Robot: Let's Encrypt validation server
Viewing unknown page Today at 12:13 PM Viewing an error
47.128.96.74
Robot: Bytespider
Viewing latest content Today at 12:13 PM
47.128.21.162
Robot: Bytespider
Viewing latest content Today at 12:13 PM
116.179.32.86
Robot: Baidu Spider
Viewing thread Welcome, Toff Today at 12:12 PM
47.128.97.249
Robot: Bytespider
Viewing latest content Today at 12:12 PM
47.128.124.159
Robot: Bytespider
Viewing latest content Today at 12:12 PM
34.226.89.140
Robot: Amazonbot
Viewing thread I lost 1,000,000 subscribers Today at 12:11 PM
52.44.229.124
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
50.19.102.70
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
18.232.12.157
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
44.193.102.198
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
23.23.214.190
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
47.128.122.23
Robot: Bytespider
Viewing latest content Today at 12:11 PM
47.128.47.198
Robot: Bytespider
Viewing latest content Today at 12:11 PM
52.0.63.151
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
54.84.102.81
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
54.163.136.244
Robot: Amazonbot
Viewing latest content Today at 12:11 PM
213.180.203.77
Robot: Yandex Bot
Viewing unknown page Today at 12:10 PM Viewing an error
47.128.21.133
Robot: Bytespider
Viewing latest content Today at 12:10 PM
47.128.40.48
Robot: Bytespider
Viewing latest content Today at 12:10 PM

Some of the bots on right now

Viewing error is a login, contact, hushed, etc page where you have to be logged in.

Byte Spyder is TikTok

rlm · 2025-11-28T14:55:47-0500

I had the same problems early this year, so using cloudflare's free service has helped a lot for me. they now have an option to allow them to block AI scrapers, but before that I created this php code that gets included in the config file and run on every request, and it helped a lot too. It blocks traffic on different criteria such as specific user agents, specific abusive ip blocks I didn't want, and most importantly blocks on country codes.

While this doesn't stop everything, it stopped a majority of the worst abuse. You can add/remove from the preg matches to your preference.

Text Box:

if (preg_match("/cloudflare/i",$_SERVER['HTTP_CDN_LOOP'])) {

  # BLOCK UNWANTED BOTS
  $BAD = "DataForSeoBot|meta-externalagent|HeadlessChrome";
  if (preg_match("/($BAD)/i",$_SERVER['HTTP_USER_AGENT'])) { block(); }

  # BLOCK ALIBABA/ALIEXPRESS BOTS
  if (preg_match("/^47\.(74|75|76|77|78|79|80|81|82|83|84|85|86|87)\./",$_SERVER['HTTP_CF_CONNECTING_IP'])) { block(); }

  #  BLOCK SPECIFIC COUNTRIES
  # IN BD VN IQ PK TR SA  (others to consider blocking...)
  if (preg_match("/^(BR|SG|RU|IQ)$/",$_SERVER['HTTP_CF_IPCOUNTRY'])) { block(); }

  # OR, BLOCK ANYTHING BUT GOOD COUNTRIES
  if (!preg_match("/^(US|CA|GB)$/",$_SERVER['HTTP_CF_IPCOUNTRY'])) { block(); }

}

function block () {

  # -------------------------------------------------------------
  #  WHITELIST GOOGLE
  # IF THEY SAY THEYRE FROM  OR REFERRED BY GOOGLE, LET IT PASS
  # -------------------------------------------------------------
  if (preg_match("/google/i",$_SERVER['HTTP_USER_AGENT']) ) { return; }
  if (preg_match("/google/i",$_SERVER['HTTP_REFERER'])    ) { return; }

  header('HTTP/1.0 403 Forbidden', true, 403);
  exit;
}

MapleDots · 2025-11-28T15:04:55-0500

Thanks for that, I tried cloudfare and our up times were terrible, I can't believe how many times cloudflare actually goes down. We are using separate dns servers and I have the ability to edit or add code so I will analyze what you have provided to see if it can be put to use.

rlm · 2025-11-28T15:19:55-0500

The most effective part for me was using cloudflare's country code data and blocking the blatant bad countries that have no useful value, and eventually the list was getting so long I said screw it, just allow specified good countries. Of course its important to ensure that the data is reliable - I had to assume it was from cloudflare. Blocking IP ranges, as in entire "B" blocks was also useful. Large abusive companies can be filtered out this way too. I also had additional code in there to log the suspicious data first, which I later used to make permanent blocks based on IP or whatever.

You can add this code:

Text Box:

     $file = $_SERVER['DOCUMENT_ROOT'].'/abusers.log';
     $line = sprintf("%-40s %-2s %s\n",$_SERVER['HTTP_CF_CONNECTING_IP'],$_SERVER['HTTP_CF_IPCOUNTRY'],$_SERVER['HTTP_USER_AGENT']);
     file_put_contents($file, $line, FILE_APPEND | LOCK_EX);

That'll log the IP, CountryCode and User Agent which you can then review to see what the most abusive ones are. I then just do something like:

awk '{print $2}' abusers.log | sort | uniq -c | sort -n

That example will count up the duplicates in the second column (country code), showing you just how many hits are coming from useless countries, which you can then block.

rlm · 2025-11-28T15:21:23-0500

obviously none of this is ideal, but the abuse by scraping bots is unprecedented and off the charts.

MapleDots · 2025-11-28T17:40:30-0500

rlm said:
Blocking IP ranges, as in entire "B" blocks was also useful.

I tried blocking IP ranges but the IP addresses change as soon as I block one another appears completely different and from a different.

rlm said:
The most effective part for me was using cloudflare's country code data and blocking the blatant bad countries

I have country blocking software and so far have blocked India, Bangladesh, Russia, and Pakistan due to spam and dangerous software injection.

But I always get some member that are legitimate, for instance I had to unban China for a recent member that likes to post.

Even places like Dynadot that use cloudflare makes things so cumbersome for login. Porkbun also recently dumped cloudflare and I understand why. It was a disaster for me, sometimes the site would hand 1-2 minutes on login and verify for this action, verify for this action, it was a royal pain.

I just ponied up for extra resources to handle the scraping load and made sure I was able to identify the bots. There was always a basic way of doing that but because all my resources went over the limits I had to get more involved and there seems to be no end in sight. At this point everyone is paying for AI and in return the AI companies are costing me money by eating away at our resources.

They really need to contribute if they are going to keep using that much of our resources, there are always close to 500 bots on the site now. Fortunately they don't contribute to our stats because we ignore them but on my Umami Webstats I can see them and it's pretty scary.

rlm · 2025-11-28T18:29:46-0500

nothing is ideal, and sometimes small sacrifices have to be made for the greater good. And yes, cloudflare can be annoying, but getting rid of them is like throwing out the baby with the bathwater. They're using their scale to fight the AI scale, so if you want to go it alone, it's going to be a never ending time-suck for you.

I also use a honeypot, immediately using iptables to block IP addresses that hit common (but non-existent) pages looking for vulnerabilities. You can also log IP addresses, dynamically counting/blocking/unblocking them.

Have you tried using iptables and the hashlimit module? This might be the best solution.

AI Bots and website scraping at DN.ca (8.Viewing)

ADMIN

DN.ca

MapleDots

MapleDots.ca

MapleDots

MapleDots.ca

silentg

DomainRetail.com

DomainRecap

Member

MapleDots

MapleDots.ca

rlm

Bonfire.ca

MapleDots

MapleDots.ca

rlm

Bonfire.ca

rlm

Bonfire.ca

MapleDots

MapleDots.ca

rlm

Bonfire.ca

Sponsors who contribute to keep dn.ca free for everyone.

Sponsors who contribute to keep dn.ca free.

Members who recently read this topic: 10