Julieta Shem <jshem@yaxenu.org> wrote

> I would like to locate any message by message-id, no matter in
> which group it was posted. Is that a dream?

You didn't mention whether or not you already know the message id.
If you already know it, what's wrong with the Howard Knight site?
<http://al.howardknight.net/>

If you don't already know the message id, I'm not sure of any lookup that
will give it to you - as I don't use the narkives & I think dejagoogle
stopped archiving the headers a few years ago (as I recall).
<https://news.admin.net-abuse.usenet.narkive.com>
<https://groups.google.com/g/news.admin.net-abuse.usenet>

How are you going to find the message id if you don't already have the
article's headers in your possession (e.g., from your news server)?

On 07.12.23 19:15, metaed@newjersey.metaed.com wrote:
> On Google Groups, there's a "public-ntp-discuss" group.

Time servers are different ball game.
Probably you mean NNTP.

--
"Gutta cavat lapidem." (Ovid)

On Dec 8, 2023 at 11:43:08 PM CST, "Grant Taylor" <gtaylor@tnetconsulting.net>
wrote:

> On 12/8/23 21:47, Julieta Shem wrote:
>> That's already something of value, but I don't even know how to do that
>> anymore. I used to. By seeing how the interface changes from time to
>> time, I gave up on Google Groups even as an archive. I still think we
>> need a all-time USENET serious archive. I would like to locate any
>> message by message-id, no matter in which group it was posted. Is that
>> a dream?
>
> I think that finding a collection of all the messages is going to be the
> biggest hurtle.
>
> I had a collection going back to late 2018 and it was 27 million
> articles. Had because I've since removed articles from Google Groups.
>
> So if it's 27 million articles ~5 years, I don't want to fathom what it
> would be going back to the '80s.
>
> I strongly suspect much of that data is gone.

I have a little over 20 years of many Usenet hierarchies, 336,216,091
messages, about 1.5TB uncompressed. Using ZFS file system with compression and
CNFS buffers gets it down to ~450GB. It is far from "complete" but I'm working
on it.

Stalled on getting the rest imported from archive.org due storage, but it is
on the roadmap (waiting on some some high endurance SSDs).

Between the Uztoo archive, others on archive.org, and what is still available
on commercial providers, I believe we have the vast majority of Usenet's
history available, but there are bound to be gaps or missing articles
throughout any archive.

On 09/12/2023 14:09, Jesse Rehmer wrote:
>
> I have a little over 20 years of many Usenet hierarchies, 336,216,091
> messages, about 1.5TB uncompressed. Using ZFS file system with compression and
> CNFS buffers gets it down to ~450GB. It is far from "complete" but I'm working
> on it.
>
> Stalled on getting the rest imported from archive.org due storage, but it is
> on the roadmap (waiting on some some high endurance SSDs).
>
> Between the Uztoo archive, others on archive.org, and what is still available
> on commercial providers, I believe we have the vast majority of Usenet's
> history available, but there are bound to be gaps or missing articles
> throughout any archive.

I'm trying to do the same thing, I'll contact you privately to
coordinate efforts, I currently have 24TB of storage waiting to be
filled (I'm currently at 2TB, and the script is continuing to import
from various sources, all the way back to 2000 )

If you have stuff, we can discuss what the best route is.
The hierarchy that is currently giving me the most problems is alt.* as
it is the most "sewer" of all the others.

Sincerely

--
Ivo Gandolfo

On 12/9/23 09:17, Ivo Gandolfo wrote:
> I'm trying to do the same thing,

I should have a snapshot or an older copy of my new spool from earlier
this year that will have all articles my server received from late '18
through early '23 if someone wants a copy of it.

--
Grant. . . .

On Fri, 8 Dec 2023 23:43:08 -0600,
Grant Taylor <gtaylor@tnetconsulting.net> wrote:
> On 12/8/23 21:47, Julieta Shem wrote:
> > That's already something of value, but I don't even know how
> > to do that anymore. I used to. By seeing how the interface
> > changes from time to time, I gave up on Google Groups even as
> > an archive. I still think we need a all-time USENET serious
> > archive. I would like to locate any message by message-id, no
> > matter in which group it was posted. Is that a dream?
>
> I think that finding a collection of all the messages is going
> to be the biggest hurtle.

Ouch! Hope you can clear the hurdle without injury. ;)

--
Ted Heise <theise@panix.com> West Lafayette, IN, USA

On Dec 9, 2023 at 9:17:04 AM CST, "Ivo Gandolfo" <usenet@bofh.team> wrote:

> On 09/12/2023 14:09, Jesse Rehmer wrote:
>>
>> I have a little over 20 years of many Usenet hierarchies, 336,216,091
>> messages, about 1.5TB uncompressed. Using ZFS file system with compression and
>> CNFS buffers gets it down to ~450GB. It is far from "complete" but I'm working
>> on it.
>>
>> Stalled on getting the rest imported from archive.org due storage, but it is
>> on the roadmap (waiting on some some high endurance SSDs).
>>
>> Between the Uztoo archive, others on archive.org, and what is still available
>> on commercial providers, I believe we have the vast majority of Usenet's
>> history available, but there are bound to be gaps or missing articles
>> throughout any archive.
>
> I'm trying to do the same thing, I'll contact you privately to
> coordinate efforts, I currently have 24TB of storage waiting to be
> filled (I'm currently at 2TB, and the script is continuing to import
> from various sources, all the way back to 2000 )
>
> If you have stuff, we can discuss what the best route is.
> The hierarchy that is currently giving me the most problems is alt.* as
> it is the most "sewer" of all the others.
>
>
> Sincerely

Yep, hit me up. Most of what I have I sucked from other sources, and yes,
alt.* is ridiculous. I leave pyClean on for most operations but effectively
disable a lot of the filters so it is basically just the EMP body filter I
rely on. The commercial sources are filled with misplaced binaries, so if you
don't filter at least some it's going to a useless sewer.

Ivo Gandolfo <usenet@bofh.team> wrote:
>On 09/12/2023 14:09, Jesse Rehmer wrote:

>>I have a little over 20 years of many Usenet hierarchies, 336,216,091
>>messages, about 1.5TB uncompressed. Using ZFS file system with
>>compression and CNFS buffers gets it down to ~450GB. It is far from
>>"complete" but I'm working on it.

>>Stalled on getting the rest imported from archive.org due storage, but it is
>>on the roadmap (waiting on some some high endurance SSDs).

>>Between the Uztoo archive, others on archive.org, and what is still
>>available on commercial providers, I believe we have the vast majority
>>of Usenet's history available, but there are bound to be gaps or missing
>>articles throughout any archive.

>I'm trying to do the same thing, I'll contact you privately to
>coordinate efforts, I currently have 24TB of storage waiting to be
>filled (I'm currently at 2TB, and the script is continuing to import
>from various sources, all the way back to 2000 )

>If you have stuff, we can discuss what the best route is.
>The hierarchy that is currently giving me the most problems is alt.* as
>it is the most "sewer" of all the others.

This shouldn't be available on line where Google can index it. It makes
it painful to search for Usenet articles as so much other stuff from the
Web comes up. Similarly, if I'm searching the Web, normally I don't want
Usenet articles coming up.

Plenty of Usenet shouldn't be archived. Too many newsgroups are nothing
but reposts of articles from the Web whose true author had not posted
them to Usenet. That's plagarism.

Also I think some of the longest-running flame wars -- going back three
decades as someone pointed out the other day -- are cron jobs.

There never has been a lot on Usenet that shouldn't be ephemeral.

On Dec 9, 2023 at 12:00:08 PM CST, ""Adam H. Kerman"" <ahk@chinet.com> wrote:

> Ivo Gandolfo <usenet@bofh.team> wrote:
>> On 09/12/2023 14:09, Jesse Rehmer wrote:
>
>>> I have a little over 20 years of many Usenet hierarchies, 336,216,091
>>> messages, about 1.5TB uncompressed. Using ZFS file system with
>>> compression and CNFS buffers gets it down to ~450GB. It is far from
>>> "complete" but I'm working on it.
>
>>> Stalled on getting the rest imported from archive.org due storage, but it is
>>> on the roadmap (waiting on some some high endurance SSDs).
>
>>> Between the Uztoo archive, others on archive.org, and what is still
>>> available on commercial providers, I believe we have the vast majority
>>> of Usenet's history available, but there are bound to be gaps or missing
>>> articles throughout any archive.
>
>> I'm trying to do the same thing, I'll contact you privately to
>> coordinate efforts, I currently have 24TB of storage waiting to be
>> filled (I'm currently at 2TB, and the script is continuing to import
>> from various sources, all the way back to 2000 )
>
>> If you have stuff, we can discuss what the best route is.
>> The hierarchy that is currently giving me the most problems is alt.* as
>> it is the most "sewer" of all the others.
>
> This shouldn't be available on line where Google can index it. It makes
> it painful to search for Usenet articles as so much other stuff from the
> Web comes up. Similarly, if I'm searching the Web, normally I don't want
> Usenet articles coming up.
>
> Plenty of Usenet shouldn't be archived. Too many newsgroups are nothing
> but reposts of articles from the Web whose true author had not posted
> them to Usenet. That's plagarism.
>
> Also I think some of the longest-running flame wars -- going back three
> decades as someone pointed out the other day -- are cron jobs.
>
> There never has been a lot on Usenet that shouldn't be ephemeral.

If Google wanted to index NNTP servers, they could, but they don't, and I
believe Ivo and I are talking about NNTP.

I don't share the same views regarding archival. Minus binaries, pretty much
everything on Usenet is accessible via the web, whether anyone likes it or
not. Besides Google Groups there are several, mostly not well known, web front
ends to NNTP servers that are indexable. Usenet's content isn't that
interesting anymore and I've never had a Google search pull up site containing
a Usenet article unless I was specifically looking for one.

On Sat, 09 Dec 2023 00:47:37 -0300, Julieta Shem wrote:

> I would like to locate any
> message by message-id, no matter in which group it was posted. Is that
> a dream?

In addition to the Howard Knight site...

Put the following two together and you can do this for messages posted
within the last 20 years (text posts) or 15 years (binary posts).

1) Some news servers have over 20 years of text retention and over
15 years of binary retention.
2) Some NNTP newsreader client will let you download a message by
message-id, no matter in which group it was posted.

--
Kind regards
Ralph Fox
🦊

The self-edge makes shew of the cloth.

Le 09/12/2023 19:00, Adam H. Kerman a écrit :
>
> [...]
>
> Plenty of Usenet shouldn't be archived. [...]
>
> There never has been a lot on Usenet that shouldn't be ephemeral.

Agreed.

--
Olivier Miakinen

Le 09/12/2023 19:57, Ralph Fox a écrit :
>
> 1) Some news servers have over 20 years of text retention and over
> 15 years of binary retention.

That is way too much! The nntp server that I use has one month of text
retention, which is consistent with the notion of « news » (not « olds »)

--
Olivier Miakinen

Olivier Miakinen <om+news@miakinen.net> wrote:
> Le 09/12/2023 19:57, Ralph Fox a écrit :
> >
> > 1) Some news servers have over 20 years of text retention and over
> > 15 years of binary retention.
>
> That is way too much! The nntp server that I use has one month of text
> retention, which is consistent with the notion of « news » (not « olds »)

The context - Julieta Shem's questions - was archives. For archives, a
month of retention is useless.

FYI, because there are no good - searchable - Usenet archives, I keep
my own archive, for my selected/subscribed groups and largely filtered
on trolls, spam, kooks, etc.. My retention is nearly 20 years.

Le 09/12/2023 20:28, Frank Slootweg a écrit :
> Olivier Miakinen <om+news@miakinen.net> wrote:
>> Le 09/12/2023 19:57, Ralph Fox a écrit :
>> >
>> > 1) Some news servers have over 20 years of text retention and over
>> > 15 years of binary retention.
>>
>> That is way too much! The nntp server that I use has one month of text
>> retention, which is consistent with the notion of « news » (not « olds »)
>
> The context - Julieta Shem's questions - was archives. For archives, a
> month of retention is useless.

Yes, it is ok for archives, accessible via the web. But some news servers
have up to one year accessible by NNTP, or even more. And IMHO *that* is
way too much (though I know that my opinion is not widely shared).

> FYI, because there are no good - searchable - Usenet archives, I keep
> my own archive, for my selected/subscribed groups and largely filtered
> on trolls, spam, kooks, etc.. My retention is nearly 20 years.

Ok for that.

--
Olivier Miakinen

On 12/9/23 13:40, Olivier Miakinen wrote:
> Yes, it is ok for archives, accessible via the web. But some news servers
> have up to one year accessible by NNTP, or even more. And IMHO *that* is
> way too much (though I know that my opinion is not widely shared).

If it's okay for an archive to have ~20 years of retention and it's okay
for an archive to be accessible via the web (HTTP(S)), why is it not
also okay for the archive to be accessible via NNTP(S).

Remember, NNTP(S) is the (current) native news protocol where as HTTP(S)
and others are not nearly as native.

>> FYI, because there are no good - searchable - Usenet archives, I keep
>> my own archive, for my selected/subscribed groups and largely filtered
>> on trolls, spam, kooks, etc.. My retention is nearly 20 years.
>
> Ok for that.

Why does it matter if there are any subscribers or not?

News operators can choose to have as much or as little disk space
backing their spool / archive as they want to have. It's their choice.

--
Grant. . . .

Le 09/12/2023 20:49, Grant Taylor a écrit :
>
>> Yes, it is ok for archives, accessible via the web. But some news servers
>> have up to one year accessible by NNTP, or even more. And IMHO *that* is
>> way too much (though I know that my opinion is not widely shared).
>
> If it's okay for an archive to have ~20 years of retention and it's okay
> for an archive to be accessible via the web (HTTP(S)), why is it not
> also okay for the archive to be accessible via NNTP(S).

Maybe I have a bad newsreader or I don't know how to use it ?

The fact is, when I have to look for a given topic, and I register to
the corresponding newsgroup, it takes me a lot of time to read the
whole newsgroup when there are several months (or years) of retention
on the server.

--
Olivier Miakinen

On 12/9/23 14:09, Olivier Miakinen wrote:
> Maybe I have a bad newsreader or I don't know how to use it ?

Eh ... I'm not convinced that there are any -- what I would consider to
be -- good news readers left. All that I've tried have left something
to be desired.

> The fact is, when I have to look for a given topic, and I register to
> the corresponding newsgroup, it takes me a lot of time to read the
> whole newsgroup when there are several months (or years) of retention
> on the server.

Oh, definitely.

When I have Thunderbird (re)download a newsgroup for the first time it
will often pop up a window asking how many messages to download and what
to do with the rest of them. I usually have it download the last 500
and mark all older messages as read.

But I consider this to be a client side implementation problem and has
very little influence on server side retention.

After all, multiple years of a newsgroup that gets fewer than 1 message
a week can be less of a load on a news reader than a month of a
newsgroup that gets more than one message an hour. It's a numbers game
and how well news readers work with massive numbers of messages.

Or the news server can retain all the messages it wants to devote disk
space to and allow the client to decide how many to download / display. ;-)

Grant. . . .

On Sat, 9 Dec 2023 21:09:36 +0100, Olivier Miakinen wrote:

> Maybe I have a bad newsreader or I don't know how to use it ?
>
> The fact is, when I have to look for a given topic, and I register to
> the corresponding newsgroup, it takes me a lot of time to read the
> whole newsgroup when there are several months (or years) of retention
> on the server.

> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
> Firefox/52.0 SeaMonkey/2.49.4

Set this in SeaMonkey:

Edit >> Mail & Newsgroup Account Settings >> (select newsgroup account) >> Server Settings
[x] Ask me before downloading more than [ 100 ] messages

Pick a number to fit how far back you want messages.

When there are many headers to download, SeaMonkey will ask you like this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ QUOTE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Download Headers

news.admin.rhubarb.yadda
There are 185969 new message headers to download for
this newsgroup

( ) Download all headers
(*) Download [ 100 ] headers
[ ] Mark remaining headers as read

[ Download ] [ Cancel ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ QUOTE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Choose the option to download only a limited number of headers.
Pick a number which fits how far back you want messages.

Put a check-mark in "Mark remaining headers as read". This is
so that the group's count of unread messages will not include
the older messages not downloaded.

--
Kind regards
Ralph Fox
🦊

A broken sack will hold no corn.

Ivo Gandolfo <usenet@bofh.team> writes:

> On 09/12/2023 14:09, Jesse Rehmer wrote:
>> I have a little over 20 years of many Usenet hierarchies, 336,216,091
>> messages, about 1.5TB uncompressed. Using ZFS file system with
>> compression and CNFS buffers gets it down to ~450GB. It is far from
>> "complete" but I'm working on it. Stalled on getting the rest
>> imported from archive.org due storage, but it is on the roadmap
>> (waiting on some some high endurance SSDs). Between the Uztoo
>> archive, others on archive.org, and what is still available on
>> commercial providers, I believe we have the vast majority of Usenet's
>> history available, but there are bound to be gaps or missing articles
>> throughout any archive.
>
> I'm trying to do the same thing, I'll contact you privately to
> coordinate efforts, I currently have 24TB of storage waiting to be
> filled (I'm currently at 2TB, and the script is continuing to import
> from various sources, all the way back to 2000 )
>
> If you have stuff, we can discuss what the best route is.
> The hierarchy that is currently giving me the most problems is alt.*
> as it is the most "sewer" of all the others.

You two have my respect. That's a pretty serious project. The USENET
carries principles such as decentralization and censorship resistance.
And likely way more important than that is the principles of
self-organization that might be the only real promise of any social
organization.

Trimming the destinataries to news.admin.net-abuse.usenet only.

Wally J <walterjones@invalid.nospam> writes:

> Julieta Shem <jshem@yaxenu.org> wrote
>
>> I would like to locate any message by message-id, no matter in
>> which group it was posted. Is that a dream?
>
> You didn't mention whether or not you already know the message id.
> If you already know it, what's wrong with the Howard Knight site?
> <http://al.howardknight.net/>

I had never heard of this website. It's asking me for a username and
password, which I don't have. It also presenting a certificate issued
to a different hostname.

> If you don't already know the message id, I'm not sure of any lookup that
> will give it to you - as I don't use the narkives & I think dejagoogle
> stopped archiving the headers a few years ago (as I recall).
> <https://news.admin.net-abuse.usenet.narkive.com>
> <https://groups.google.com/g/news.admin.net-abuse.usenet>

I do know the message-id. I think it's nice if posts and papers could
reference a USENET article by the message-id and with a click of the
mouse people could look it up on the web or gopher, gemini or something.
Having to mention the group (along) seems undesired.

> How are you going to find the message id if you don't already have the
> article's headers in your possession (e.g., from your news server)?

Someone sends me an e-mail saying --- ``see <message-id> for how X can
be accomplished'' --- and some archive-service could display the message
given the <message-if>. The USENET could be what electronic archives
for scientific papers still can't do. (You read a paper, find a
reference and must still manually go to questionably legal services to
try to find a copy.)

Thanks for the pointers.

On Sun, 10 Dec 2023 02:29:07 -0300
Julieta Shem <jshem@yaxenu.org> wrote:
>
> Trimming the destinataries to news.admin.net-abuse.usenet only.
>
> Wally J <walterjones@invalid.nospam> writes:
>
> > Julieta Shem <jshem@yaxenu.org> wrote
> >
> >> I would like to locate any message by message-id, no matter in
> >> which group it was posted. Is that a dream?
> >
> > You didn't mention whether or not you already know the message id.
> > If you already know it, what's wrong with the Howard Knight site?
> > <http://al.howardknight.net/>
>
> I had never heard of this website. It's asking me for a username and
> password, which I don't have. It also presenting a certificate issued
> to a different hostname.

The username and password stuff was news to me and seems to be the case if
you go to https://al.howardknight.net .But if you go to
http://al.howardknight.net/ , there's none of that stuff.

On Sat, 9 Dec 2023 20:40:28 +0100
Olivier Miakinen <om+news@miakinen.net> wrote:
> Le 09/12/2023 20:28, Frank Slootweg a écrit :
> > Olivier Miakinen <om+news@miakinen.net> wrote:
> >> Le 09/12/2023 19:57, Ralph Fox a écrit :
> >> >
> >> > 1) Some news servers have over 20 years of text retention and over
> >> > 15 years of binary retention.
> >>
> >> That is way too much! The nntp server that I use has one month of text
> >> retention, which is consistent with the notion of « news » (not « olds »)

"news" is just a name. The groups I mostly read have invaluable (mostly
technical) information which should be retained for as long as possible ,
certainly decades.

> > The context - Julieta Shem's questions - was archives. For archives, a
> > month of retention is useless.
>
> Yes, it is ok for archives, accessible via the web. But some news servers
> have up to one year accessible by NNTP, or even more. And IMHO *that* is
> way too much (though I know that my opinion is not widely shared).

Count me among those who do not share your opinion. As far as I'm concerned ,
the longer retention there is , the better.

On Sun, 10 Dec 2023 02:17:44 -0300, Julieta Shem <jshem@yaxenu.org> wrote:
>Ivo Gandolfo <usenet@bofh.team> writes:
>> On 09/12/2023 14:09, Jesse Rehmer wrote:
>>> I have a little over 20 years of many Usenet hierarchies, 336,216,091
>>> messages, about 1.5TB uncompressed. Using ZFS file system with
>>> compression and CNFS buffers gets it down to ~450GB. It is far from
>>> "complete" but I'm working on it. Stalled on getting the rest
>>> imported from archive.org due storage, but it is on the roadmap
>>> (waiting on some some high endurance SSDs). Between the Uztoo
>>> archive, others on archive.org, and what is still available on
>>> commercial providers, I believe we have the vast majority of Usenet's
>>> history available, but there are bound to be gaps or missing articles
>>> throughout any archive.
>>
>> I'm trying to do the same thing, I'll contact you privately to
>> coordinate efforts, I currently have 24TB of storage waiting to be
>> filled (I'm currently at 2TB, and the script is continuing to import
>> from various sources, all the way back to 2000 )
>> If you have stuff, we can discuss what the best route is.
>> The hierarchy that is currently giving me the most problems is alt.*
>> as it is the most "sewer" of all the others.
>
>You two have my respect. That's a pretty serious project. The USENET
>carries principles such as decentralization and censorship resistance.
>And likely way more important than that is the principles of
>self-organization that might be the only real promise of any social
>organization.

plus ones (one plus its own shadow) . . . unmoderated is free speech

Spiros Bousbouras <spibou@gmail.com> writes:

> On Sun, 10 Dec 2023 02:29:07 -0300
> Julieta Shem <jshem@yaxenu.org> wrote:
>>
>> Trimming the destinataries to news.admin.net-abuse.usenet only.
>>
>> Wally J <walterjones@invalid.nospam> writes:
>>
>> > Julieta Shem <jshem@yaxenu.org> wrote
>> >
>> >> I would like to locate any message by message-id, no matter in
>> >> which group it was posted. Is that a dream?
>> >
>> > You didn't mention whether or not you already know the message id.
>> > If you already know it, what's wrong with the Howard Knight site?
>> > <http://al.howardknight.net/ " rel="nofollow" target="_blank">http://al.howardknight.net/>
>>
>> I had never heard of this website. It's asking me for a username and
>> password, which I don't have. It also presenting a certificate issued
>> to a different hostname.
>
> The username and password stuff was news to me and seems to be the case if
> you go to https://al.howardknight.net .But if you go to
> http://al.howardknight.net/ , there's none of that stuff.

Thanks! That's a pretty useful service. Special thanks to Howard
Knight. (It would be nice if we could get a raw copy of the article.)

Frank Slootweg <this@ddress.is.invalid> wrote:
> FYI, because there are no good - searchable - Usenet archives, I keep
>my own archive, for my selected/subscribed groups and largely filtered
>on trolls, spam, kooks, etc.. My retention is nearly 20 years.

And from my perspective, 20 years only goes back to 2003 when Usenet was
already on the way down. I'd like to see maybe 1983 to 2010 and I don't
much care about anything after then.

Guys like Larry Lippman and Bill Vermillion left a lot of really useful
posts. They are now deceased and all that information is lost.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."