Rocksolid Light

Welcome to Rocksolid Light

mail  files  register  newsreader  groups  login

Message-ID:  

"We can't schedule an orgy, it might be construed as fighting" -- Stanley Sutton


interests / Offtopic / Re: Re I cant believe it the baidu spider is scanning def3 on tor

SubjectAuthor
* I can't believe it, the baidu spider is scanning def3 on tortrw
`* I can't believe it, the baidu spider is scanning def3 on torAnonUser
 `* I can't believe it, the baidu spider is scanning def3 on torAnonUser
  `* I can't believe it, the baidu spider is scanning def3 on torAnonUser
   `* Re I cant believe it the baidu spider is scanning def3 on tortrw
    `* Re I cant believe it the baidu spider is scanning def3 on torAnonUser
     +* Re I cant believe it the baidu spider is scanning def3 on torAnonUser
     |`- Re I cant believe it the baidu spider is scanning def3 on torGuest
     `- Re I cant believe it the baidu spider is scanning def3 on tortrw

1
I can't believe it, the baidu spider is scanning def3 on tor

<qodh8v$dkc$1@i2pn2.org>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=112&group=rocksolid.shared.offtopic#112

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!.POSTED!not-for-mail
From: trw@i2pmail.org (trw)
Newsgroups: rocksolid.shared.offtopic
Subject: I can't believe it, the baidu spider is scanning def3 on tor
Date: Fri, 18 Oct 2019 19:18:35 -0400
Organization: Dancing elephants
Lines: 3
Message-ID: <qodh8v$dkc$1@i2pn2.org>
Reply-To: trw <trw@i2pmail.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 18 Oct 2019 23:18:56 -0000 (UTC)
Injection-Info: i2pn2.org; posting-account="def.i2p";
logging-data="13964"; mail-complaints-to="usenet@i2pn2.org"
User-Agent: FUDforum 3.0.7
X-FUDforum: 6666cd76f96956469e7be39d750cc7d9 <77569>
 by: trw - Fri, 18 Oct 2019 23:18 UTC

Seems like a strange thing, and maybe it is a fake: baidu shows no hit when searching for the name or the onion address.
Or maybe they just harvest the info, and censor it right after...

trw
Posted on def3

Re: I can't believe it, the baidu spider is scanning def3 on tor

<ea7ac6c84bc58f535bcd51f5f9d9e505$1@news.novabbs.com>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=113&group=rocksolid.shared.offtopic#113

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!rocksolid2!.POSTED.localhost!not-for-mail
From: AnonUser@rslight.i2p (AnonUser)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: I can't believe it, the baidu spider is scanning def3 on tor
Date: Fri, 18 Oct 2019 23:34:00 -0000 (UTC)
Organization: Rocksolid Light
Message-ID: <ea7ac6c84bc58f535bcd51f5f9d9e505$1@news.novabbs.com>
References: <qodh8v$dkc$1@i2pn2.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 18 Oct 2019 23:34:00 -0000 (UTC)
Injection-Info: novabbs.com; posting-account="retrobbs1"; posting-host="localhost:127.0.0.1";
logging-data="30846"; mail-complaints-to="usenet@novabbs.com"
User-Agent: rslight (http://news.novabbs.com)
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on novabbs.com
X-Rslight-Site: $2y$10$U6rZtKAl./VW9KRa4KAA9enwAxXYynr0qhs/HWmabaWm49X/ktnIu
 by: AnonUser - Fri, 18 Oct 2019 23:34 UTC

trw wrote:

> Seems like a strange thing, and maybe it is a fake: baidu shows no hit when
searching for the name or the onion address.
> Or maybe they just harvest the info, and censor it right after...

I see baidu search in my logs also. I assume it's someone running the
spider themself (or spoofing the user-agent). May not be the site. Often
they disable delays between page requests and ignore robots.txt, which
most real sites won't do.

--
Posted on Rocksolid Light

Re: I can't believe it, the baidu spider is scanning def3 on tor

<9746930af4a5b77ceceed35784021788$1@retrobbs.i2p>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=114&group=rocksolid.shared.offtopic#114

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.rocksolid3!not-for-mail
From: anonuser@retrobbs.rocksolidbbs.com.remove-kn3-this (AnonUser)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: I can't believe it, the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 19:15:30 +0000
Organization: RetroBBS
Message-ID: <9746930af4a5b77ceceed35784021788$1@retrobbs.i2p>
References: <ea7ac6c84bc58f535bcd51f5f9d9e505$1@news.novabbs.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.com; posting-account="rslight.i2p"; posting-host="rocksolid3:192.241.178.238";
logging-data="27264"; mail-complaints-to="usenet@novabbs.com"
User-Agent: rslight (http://news.novabbs.com)
To: AnonUser
X-Comment-To: AnonUser
In-Reply-To: <ea7ac6c84bc58f535bcd51f5f9d9e505$1@news.novabbs.com>
X-FTN-PID: Synchronet 3.17a-Linux Dec 29 2018 GCC 6.3.0
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on rocksolidbbs.com
X-Rslight-Site: $2y$10$1.k567n35I6b.QWkmjn/M.x1NHHBTBUFrz8nYSiqZlNH5598nm5KO
X-Gateway: retrobbs.rocksolidbbs.com [Synchronet 3.17a-Linux NewsLink 1.110]
 by: AnonUser - Sat, 19 Oct 2019 19:15 UTC

To: AnonUser
this seems most likely.

people think nothing of it if googlebot or any of the major search engine
bots is crawling your site without a delay, or even ignoring robots.txt or
simply don't want any attention.

source: i've written crawlers in the past and spoofed the user agent
because of those reasons.
--
Posted on RetroBBS

Re: I can't believe it, the baidu spider is scanning def3 on tor

<11a1968b180a7a511b542f97c8c7a9d8$1@retrobbs.i2p>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=115&group=rocksolid.shared.offtopic#115

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.rocksolid3!not-for-mail
From: anonuser@retrobbs.rocksolidbbs.com.remove-9vm-this (AnonUser)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: I can't believe it, the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 19:17:59 +0000
Organization: RetroBBS
Message-ID: <11a1968b180a7a511b542f97c8c7a9d8$1@retrobbs.i2p>
References: <9746930af4a5b77ceceed35784021788$1@retrobbs.i2p>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.com; posting-account="rslight.i2p"; posting-host="rocksolid3:192.241.178.238";
logging-data="28862"; mail-complaints-to="usenet@novabbs.com"
User-Agent: rslight (http://news.novabbs.com)
To: AnonUser
X-Comment-To: AnonUser
In-Reply-To: <9746930af4a5b77ceceed35784021788$1@retrobbs.i2p>
X-FTN-PID: Synchronet 3.17a-Linux Dec 29 2018 GCC 6.3.0
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on rocksolidbbs.com
X-Rslight-Site: $2y$10$rjXsVUSDH9PC3WhZ5WWvaeYkj6oqmlsKGZ7yBrengxsjHkA0nCrea
X-Gateway: retrobbs.rocksolidbbs.com [Synchronet 3.17a-Linux NewsLink 1.110]
 by: AnonUser - Sat, 19 Oct 2019 19:17 UTC

To: AnonUser
googlebot and the other well known bots usually also got preferential
treatment.
--
Posted on RetroBBS

Re: Re I cant believe it the baidu spider is scanning def3 on tor

<qofqgp$fe2$1@i2pn2.org>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=116&group=rocksolid.shared.offtopic#116

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!.POSTED!not-for-mail
From: trw@i2pmail.org (trw)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: Re I cant believe it the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 16:08:38 -0400
Organization: Dancing elephants
Lines: 6
Message-ID: <qofqgp$fe2$1@i2pn2.org>
References: <11a1968b180a7a511b542f97c8c7a9d8$1@retrobbs.i2p>
Reply-To: trw <trw@i2pmail.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Sat, 19 Oct 2019 20:08:58 -0000 (UTC)
Injection-Info: i2pn2.org; posting-account="def.i2p";
logging-data="15810"; mail-complaints-to="usenet@i2pn2.org"
User-Agent: FUDforum 3.0.7
X-FUDforum: 6666cd76f96956469e7be39d750cc7d9 <79297>
 by: trw - Sat, 19 Oct 2019 20:08 UTC

>source: i've written crawlers in the past and spoofed the user agent because of those reasons.

were those for clearnet or for darknets (or both) ? i don't mind bots as long as they do not consume too many resources...

cheers

trw
Posted on def3

Re: Re I cant believe it the baidu spider is scanning def3 on tor

<5cd2c23bb04cf218ce44d57e40117e31$1@retrobbs.i2p>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=117&group=rocksolid.shared.offtopic#117

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.rocksolid3!not-for-mail
From: anonuser@retrobbs.rocksolidbbs.com.remove-3rb-this (AnonUser)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: Re I cant believe it the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 22:10:06 +0000
Organization: RetroBBS
Message-ID: <5cd2c23bb04cf218ce44d57e40117e31$1@retrobbs.i2p>
References: <qofqgp$fe2$1@i2pn2.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.com; posting-account="rslight.i2p"; posting-host="rocksolid3:192.241.178.238";
logging-data="25486"; mail-complaints-to="usenet@novabbs.com"
User-Agent: rslight (http://news.novabbs.com)
To: trw
X-Comment-To: trw
In-Reply-To: <qofqgp$fe2$1@i2pn2.org>
X-FTN-PID: Synchronet 3.17a-Linux Dec 29 2018 GCC 6.3.0
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on rocksolidbbs.com
X-Rslight-Site: $2y$10$8zE76nzMHX0GAW3whHdH2.WZ63NHaLzdODzX9z2Km5PR1K6VNz5Ee
X-Gateway: retrobbs.rocksolidbbs.com [Synchronet 3.17a-Linux NewsLink 1.110]
 by: AnonUser - Sat, 19 Oct 2019 22:10 UTC

To: trw
mainly clearnet. i did also crawl the darknet without a delay back then,
simply because the network was really slow.

years ago the darknets were even slower than they are now. one could even
say they are "fast" nowadays.

you can always rate limit your eepsite if bots become an issue, it
generally isn't worth it for any crawler to re-create tunnels very often,
as that is computationally expensive and also takes a bit to warm up. i
also do not think most people would bother with that.
--
Posted on RetroBBS

Re: Re I cant believe it the baidu spider is scanning def3 on tor

<eb067b108b84f148a81f0537ad44cca3$1@retrobbs.i2p>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=118&group=rocksolid.shared.offtopic#118

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!i2pn.org!rocksolid2!.POSTED.rocksolid3!not-for-mail
From: anonuser@retrobbs.rocksolidbbs.com.remove-ob-this (AnonUser)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: Re I cant believe it the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 22:13:53 +0000
Organization: RetroBBS
Message-ID: <eb067b108b84f148a81f0537ad44cca3$1@retrobbs.i2p>
References: <5cd2c23bb04cf218ce44d57e40117e31$1@retrobbs.i2p>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.com; posting-account="rslight.i2p"; posting-host="rocksolid3:192.241.178.238";
logging-data="27123"; mail-complaints-to="usenet@novabbs.com"
User-Agent: rslight (http://news.novabbs.com)
To: AnonUser
X-Comment-To: AnonUser
In-Reply-To: <5cd2c23bb04cf218ce44d57e40117e31$1@retrobbs.i2p>
X-FTN-PID: Synchronet 3.17a-Linux Dec 29 2018 GCC 6.3.0
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on rocksolidbbs.com
X-Rslight-Site: $2y$10$jRI0orRrTzd39e1yCZFK7ezOdnbfVFmh4B.nxgXuBw.TkBXccN.ji
X-Gateway: retrobbs.rocksolidbbs.com [Synchronet 3.17a-Linux NewsLink 1.110]
 by: AnonUser - Sat, 19 Oct 2019 22:13 UTC

To: AnonUser
it just came to my mind that back then there were clearnet websites which
provided access to darknet websites, so clearnet search engines did index
the darknets for a while.

i do not know if any such website is still active, though i would
seriously doubt it because of the possible legal issues and constant
dmca/takedown requests.
--
Posted on RetroBBS

Re: Re I cant believe it the baidu spider is scanning def3 on tor

<qog2g4$qo2$1@i2pn2.org>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=119&group=rocksolid.shared.offtopic#119

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!.POSTED!not-for-mail
From: guest@retrobbs.rocksolidbbs.com (Guest)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: Re I cant believe it the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 18:24:50 -0400
Organization: Dancing elephants
Lines: 2
Message-ID: <qog2g4$qo2$1@i2pn2.org>
References: <eb067b108b84f148a81f0537ad44cca3$1@retrobbs.i2p>
Reply-To: Guest <guest@retrobbs.rocksolidbbs.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Sat, 19 Oct 2019 22:25:09 -0000 (UTC)
Injection-Info: i2pn2.org; posting-account="def.i2p";
logging-data="27394"; mail-complaints-to="usenet@i2pn2.org"
User-Agent: FUDforum 3.0.7
X-FUDforum: 6666cd76f96956469e7be39d750cc7d9 <79615>
 by: Guest - Sat, 19 Oct 2019 22:24 UTC

>i do not know if any such website is still active,

oh yes, there are many of them that are active. best to be avoided, imo, both as a client and as a server.
Posted on def3

Re: Re I cant believe it the baidu spider is scanning def3 on tor

<qog2t5$rft$1@i2pn2.org>

 copy mid

https://news.novabbs.org/interests/article-flat.php?id=120&group=rocksolid.shared.offtopic#120

 copy link   Newsgroups: rocksolid.shared.offtopic
Path: i2pn2.org!.POSTED!not-for-mail
From: trw@i2pmail.org (trw)
Newsgroups: rocksolid.shared.offtopic
Subject: Re: Re I cant believe it the baidu spider is scanning def3 on tor
Date: Sat, 19 Oct 2019 18:31:47 -0400
Organization: Dancing elephants
Lines: 11
Message-ID: <qog2t5$rft$1@i2pn2.org>
References: <5cd2c23bb04cf218ce44d57e40117e31$1@retrobbs.i2p>
Reply-To: trw <trw@i2pmail.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Sat, 19 Oct 2019 22:32:06 -0000 (UTC)
Injection-Info: i2pn2.org; posting-account="def.i2p";
logging-data="28157"; mail-complaints-to="usenet@i2pn2.org"
User-Agent: FUDforum 3.0.7
X-FUDforum: 6666cd76f96956469e7be39d750cc7d9 <79639>
 by: trw - Sat, 19 Oct 2019 22:31 UTC

>you can always rate limit your eepsite if bots become an issue

on i2p everything is dandy, and the java package allows finetuning of these service settings to a very high extent.
it is the tor side that is usually making trouble, and that one has no such measures (unless you add your own code to the service or you play around with iptables).
anyway, bots and spiders are just something that any service has to deal with somehow, it is kind a stress test sometimes, so it helps to find weak spots in the server.

i did some crawling of both tor and i2p some years ago, only by using wget in a script. this was semi successful...

cheers

trw
Posted on def3

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor