Spiros Bousbouras <spibou@gmail.com> writes:

> On Sat, 9 Dec 2023 20:40:28 +0100

[...]

>> Le 09/12/2023 20:28, Frank Slootweg a écrit :

[...]

>> Yes, it is ok for archives, accessible via the web. But some news servers
>> have up to one year accessible by NNTP, or even more. And IMHO *that* is
>> way too much (though I know that my opinion is not widely shared).
>
> Count me among those who do not share your opinion. As far as I'm concerned ,
> the longer retention there is , the better.

Sure, but, assuming we have a ``complete'' archive somewhere, a sysadmin
with few resources could run a thin server and these thin users could
have a smart client that whenever we would like to fetch a nonexistent
article we would ask the archive.

Say I'm reading a post and I'd like to take a look at its parent, which
has been expired. My client asks Howard Knight, fetches a copy and I'm
good. So an archive with an API for fetching raw articles is a pretty
nice service.

kludge@panix.com (Scott Dorsey) writes:

> Frank Slootweg <this@ddress.is.invalid> wrote:
>> FYI, because there are no good - searchable - Usenet archives, I keep
>>my own archive, for my selected/subscribed groups and largely filtered
>>on trolls, spam, kooks, etc.. My retention is nearly 20 years.
>
> And from my perspective, 20 years only goes back to 2003 when Usenet was
> already on the way down. I'd like to see maybe 1983 to 2010 and I don't
> much care about anything after then.
>
> Guys like Larry Lippman and Bill Vermillion left a lot of really useful
> posts. They are now deceased and all that information is lost.

I care about that too. There's history, too. Say we're talking about
the birth of the GNU system. Why can't we cite the message-id of the of
the post by Richard Stallman? What's sad is that keeping a service
online on the Internet is never an easy job --- there's attacks and
whatnot. I wish I could put a well set-up service online and turn my
back on it forever. But that's a dream I could never realize.

On 12/10/23 10:37, Julieta Shem wrote:
> Sure, but, assuming we have a ``complete'' archive somewhere, a sysadmin
> with few resources could run a thin server and these thin users could
> have a smart client that whenever we would like to fetch a nonexistent
> article we would ask the archive.

I question the veracity of "few resources".

I'm seeing 5+ million articles per year in my short (5+ year) archive.
-- Those numbers might not stay the same, but they are a starting point
for this discussion.

If we say ~5 million articles per year going back 30+ years, that's 150
million messages. That's not just a "few resources" to make accessible
in relatively short order.

> Say I'm reading a post and I'd like to take a look at its parent, which
> has been expired. My client asks Howard Knight, fetches a copy and I'm
> good. So an archive with an API for fetching raw articles is a pretty
> nice service.

We already have an API for fetching articles from a server. Network
News Transfer Protocol does perfectly adequate job at it. NNTP is also
already very well supported in most news clients. So why not use it?

I think what you are after is an NNTP server that has an extremely deep
(30+ year) history.

There may be some room for added complexity in the servers so that they
can look up an old article somewhere else outside of their corpus. But
that's a different problem for a different day.

I believe that an NNTP server meant for archive access and doesn't allow
posting could provide what you're after /if/ such a corpus could be put
together and such a system could be hosted somewhere (and likely
supported by it's user base).

--
Grant. . . .

On 12/10/23 10:51, Julieta Shem wrote:
> I care about that too. There's history, too. Say we're talking about
> the birth of the GNU system. Why can't we cite the message-id of the of
> the post by Richard Stallman?

I believe you can.

What is different about citing an article from yesterday vs an article
from 25 years ago /other/ /than/ lack of the latter on the news server
you're using?

I believe that current methodologies can provide what is being asked for
if the data was available and people were willing to commit the
resources for it.

> What's sad is that keeping a service
> online on the Internet is never an easy job --- there's attacks and
> whatnot. I wish I could put a well set-up service online and turn my
> back on it forever. But that's a dream I could never realize.

It's not trivial. But it's not impossible either.

--
Grant. . . .

Grant Taylor <gtaylor@tnetconsulting.net> wrote:
>On 12/10/23 10:51, Julieta Shem wrote:
>> I care about that too. There's history, too. Say we're talking about
>> the birth of the GNU system. Why can't we cite the message-id of the of
>> the post by Richard Stallman?
>
>I believe you can.
>
>What is different about citing an article from yesterday vs an article
>from 25 years ago /other/ /than/ lack of the latter on the news server
>you're using?

The lack of access to the latter from any source at all. I mean, I can
cite the article (if I could find it), but there is no point in citing
something that nobody can read.

>I believe that current methodologies can provide what is being asked for
>if the data was available and people were willing to commit the
>resources for it.

Dejanews provided exactly what was asked for, and so did Google for a
while, until they "improved" things to the point where it was completely
broken.

Committing the resources is an issue, but not the big one. The real problem
is availability of the data. When dejanews was created an enormous amount of
effort went into finding pre-dejanews material and passing it off to dejanews.
At this point, most of that likely no longer exists except in google's
database. And there's no point in it being in there if people can't find it.
--scott

--
"C'est un Nagra. C'est suisse, et tres, tres precis."

On 12/10/23 11:23, Scott Dorsey wrote:
> The lack of access to the latter from any source at all.

ACK

> I mean, I can
> cite the article (if I could find it), but there is no point in citing
> something that nobody can read.

I think there is some value in saying something and saying where it came
from. It offers a modicum of veracity and enables the reader to fact
check. What the reader does and does not have access to is not your
primary problem. Though it does behoove you to cite sources that you
know the reader has access to.

> Dejanews provided exactly what was asked for, and so did Google for a
> while, until they "improved" things to the point where it was completely
> broken.

ACK

> Committing the resources is an issue, but not the big one.

I think the necessary resources are going to be a bigger issue than many
other people.

INN's traditional spool directories are going to tax just about any file
system when you try to put hundreds of thousands of files in a single
directory. Some newsgroups will have significantly more articles than
others.

Sure, there are ways to store articles so that they aren't all in one
directory. But that's now a news server change. It can happen, but
it's not as simple as a configuration change. It will likely be a code
and a configuration change.

There are other message stores, but the ones that I'm aware of tend to
by cyclical in nature and of a fixed size which is antithetical to the
archive forever goal.

This is probably a case for a custom NNTP server that is really a
gateway (of sorts) to some sort of object store that is distributed and
designed to scale to millions of objects in a container (newsgroup).

Whatever is done needs to be flexible and have the ability to be
reconfigured as things grow. It should also have a little bit of
redundancy as the more systems that are added to it, the more fragile it
will become.

> The real problem is availability of the data.

I agree that's probably the /primary/ problem and supersedes the storage
in such as I believe that computer science / systems people can overcome
the problem mentioned above. -- I'm not as confident that we can
recover a full archive of Usenet any more. After all, according to
Wikipedia, we're talking about 43+ years of data. Much of that data was
deemed ephemeral by most people. Much of that data was difficult to
collect 20+ years ago. The intervening 20 years won't have helped the
matter.

I maintain that storage and accessibility to the data is the /secondary/
largest problem.

> When dejanews was created an enormous amount of
> effort went into finding pre-dejanews material and passing it off to dejanews.
> At this point, most of that likely no longer exists except in google's
> database. And there's no point in it being in there if people can't find it.

Eh ... given Google's propensity to not get rid of things, I sort of
suspect that the old DeJa News archive is probably still exists
somewhere. It may actually be directly behind Google Groups Usenet
gateway and the gateway itself may be munging articles as they are
presented. Google really is antithetical to destroying data. Getting
to that data, that's an entirely different issue.

--
Grant. . . .

Scott Dorsey <kludge@panix.com> wrote:
> Frank Slootweg <this@ddress.is.invalid> wrote:
> > FYI, because there are no good - searchable - Usenet archives, I keep
> >my own archive, for my selected/subscribed groups and largely filtered
> >on trolls, spam, kooks, etc.. My retention is nearly 20 years.
>
> And from my perspective, 20 years only goes back to 2003 when Usenet was
> already on the way down. I'd like to see maybe 1983 to 2010 and I don't
> much care about anything after then.

Yes, that would be nice. I started on Usenet at about that time, or
perhaps a year later or earlier. 'We' tought that Deja/Google would keep
it for us. How wrong/gullible we were!

At the time I tried, my oldest article I could find in Google Groups
was of February 24, 1989 (of course just '89' at that time! :-)). It's
still there. (To preserve my privacy and those of others, I won't give
the URL.)

So maybe you can still find something of what you're looking for.

> Guys like Larry Lippman and Bill Vermillion left a lot of really useful
> posts. They are now deceased and all that information is lost.

Perhaps someone on the admin or/and newsreader groups has some idea
about how to get (acces to) this kind of material. IIRC, there have been
recent discussions about old archives and how to get/preserve them.

Brian Gregory <void-invalid-dead-dontuse@email.invalid> wrote
> I'm using news.individual.net and I don't see anything from Google
> Groups here.

Hi Brian,
Your admin appears to be working on the problem as shown here.
<https://groups.google.com/g/news.admin.peering/c/AgrNUeZuAkw/m/f9PSZkv4AAAJ>

Here's what he said, verbatim.
My apologies to the Individual.net news server admins.

==< cut here >==
We follow the discussion here and are aware of the Google spam problem. We
also have some anti-spam measures for our reader servers. But it is really
easy to find our contact address (news@individual.net) on
https://news.individual.net/

In the past, issuing a UDP� has often not been particularly successful.

Heiko (for Newsmaster-Team of individual.net)
https://en.wikipedia.org/wiki/Usenet_Death_Penalty

Julieta Shem <jshem@yaxenu.org> wrote

>> You didn't mention whether or not you already know the message id.
>> If you already know it, what's wrong with the Howard Knight site?
>> <http://al.howardknight.net/>
>
> I had never heard of this website. It's asking me for a username and
> password, which I don't have. It also presenting a certificate issued
> to a different hostname.

I'm sorry about that.
I care that you get your answer that you need.
But I didn't respond at first as I have nothing more to help you with.

Nobody else has responded to this, unfortunately.
So let's home someone else can help you explain what is happening.
Especially as it purports to do exactly what it is you asked for.

To be clear, I can't explain your results as when I go to howardknight.net,
it only asks me for the message id. But I'm just a long-time user.

I'm not an admin in any way.

All we can do to help is ask the others here why Howardknight.net would ask
for a password from you - when it has never asked for a password from me.

Wally J wrote:

> Julieta Shem wrote
>
>>> You didn't mention whether or not you already know the message id.
>>> If you already know it, what's wrong with the Howard Knight site?
>>> <http://al.howardknight.net/>
>>
>> I had never heard of this website. It's asking me for a username and
>> password, which I don't have. It also presenting a certificate issued
>> to a different hostname.
>
> I'm sorry about that.
> I care that you get your answer that you need.
> But I didn't respond at first as I have nothing more to help you with.
>
> Nobody else has responded to this, unfortunately.
> So let's home someone else can help you explain what is happening.
> Especially as it purports to do exactly what it is you asked for.
>
> To be clear, I can't explain your results as when I go to howardknight.net,
> it only asks me for the message id. But I'm just a long-time user.
>
> I'm not an admin in any way.
>
> All we can do to help is ask the others here why Howardknight.net would ask
> for a password from you - when it has never asked for a password from me.

To expand the URLs slightly as an explanation ...

<http://al.howardknight.net:80/>
is different from

<https://al.howardknight.net:443/>
which redirects to
<https://al.howardknight.net:443/login_up.php>

The former is where the lookup service is running (no need for
encryption) the latter is where howard administers the
shared?/virtual?/colo? host that it runs on, that does need to be
encrypted to keep the server safe.

Grant Taylor <gtaylor@tnetconsulting.net> writes:

> On 12/10/23 10:37, Julieta Shem wrote:
>> Sure, but, assuming we have a ``complete'' archive somewhere, a sysadmin
>> with few resources could run a thin server and these thin users could
>> have a smart client that whenever we would like to fetch a nonexistent
>> article we would ask the archive.
>
> I question the veracity of "few resources".
>
> I'm seeing 5+ million articles per year in my short (5+ year)
> archive. -- Those numbers might not stay the same, but they are a
> starting point for this discussion.
>
> If we say ~5 million articles per year going back 30+ years, that's
> 150 million messages. That's not just a "few resources" to make
> accessible in relatively short order.

I meant a sysadmin with little money.

>> Say I'm reading a post and I'd like to take a look at its parent, which
>> has been expired. My client asks Howard Knight, fetches a copy and I'm
>> good. So an archive with an API for fetching raw articles is a pretty
>> nice service.
>
> We already have an API for fetching articles from a server. Network
> News Transfer Protocol does perfectly adequate job at it. NNTP is
> also already very well supported in most news clients. So why not use
> it?

I totally agree. (Whatever works.)

> I think what you are after is an NNTP server that has an extremely
> deep (30+ year) history.

Yes, I'd like that. I think it's something we can all use.

> There may be some room for added complexity in the servers so that
> they can look up an old article somewhere else outside of their
> corpus. But that's a different problem for a different day.
>
> I believe that an NNTP server meant for archive access and doesn't
> allow posting could provide what you're after /if/ such a corpus could
> be put together and such a system could be hosted somewhere (and
> likely supported by it's user base).

That makes sense --- an NNTP made to be used as as reference, optimized
only to provide a message per message-id.

We should also have a solution for keeping addresses for a very long
time. In the same way we have organized ourselves to keep the USENET
rolling, we should also organize ourselves to make things last. But
that's seems less easy. When there is money involved, people set up
institutions that keep things going. We should have things like that.
Could a sum of money be allocated, providing interest with which we can
keep basic things like a hostname so that

nntp://arxiv.use.net/<message@id>

always fetches the article with <message@id>? I think that's the chance
we have of running things properly in the world. The representative
idea is likely at the end of its time now, finally. Electronic
communication and the electronic printing press might give us the means
to run stuff ourselves now. (An interesting keyword --- sortition.)

Grant Taylor <gtaylor@tnetconsulting.net> writes:

> On 12/10/23 11:23, Scott Dorsey wrote:
>> The lack of access to the latter from any source at all.
>
> ACK
>
>> I mean, I can
>> cite the article (if I could find it), but there is no point in citing
>> something that nobody can read.
>
> I think there is some value in saying something and saying where it
> came from. It offers a modicum of veracity and enables the reader to
> fact check. What the reader does and does not have access to is not
> your primary problem. Though it does behoove you to cite sources that
> you know the reader has access to.

Sure. I cite these messages when I need to. But people often wonder
--- how can I get it? Sometimes there is an answer such as some
website, but such websites scramble over the years. The USENET deserves
the highest standard in archiving.

> INN's traditional spool directories are going to tax just about any
> file system when you try to put hundreds of thousands of files in a
> single directory. Some newsgroups will have significantly more
> articles than others.
>
> Sure, there are ways to store articles so that they aren't all in one
> directory. But that's now a news server change. It can happen, but
> it's not as simple as a configuration change. It will likely be a
> code and a configuration change.
>
> There are other message stores, but the ones that I'm aware of tend to
> by cyclical in nature and of a fixed size which is antithetical to the
> archive forever goal.
>
> This is probably a case for a custom NNTP server that is really a
> gateway (of sorts) to some sort of object store that is distributed
> and designed to scale to millions of objects in a container
> (newsgroup).
>
> Whatever is done needs to be flexible and have the ability to be
> reconfigured as things grow. It should also have a little bit of
> redundancy as the more systems that are added to it, the more fragile
> it will become.

ACK. When you said NNTP, I said --- okay ---, but I would think that a
whole new system should be set-up. I would think that archive.org would
be interested in getting a proposal for something like that. They seem
pretty serious about archiving the Internet.

>> The real problem is availability of the data.
>
> I agree that's probably the /primary/ problem and supersedes the
> storage in such as I believe that computer science / systems people
> can overcome the problem mentioned above. -- I'm not as confident
> that we can recover a full archive of Usenet any more. After all,
> according to Wikipedia, we're talking about 43+ years of data. Much
> of that data was deemed ephemeral by most people. Much of that data
> was difficult to collect 20+ years ago. The intervening 20 years
> won't have helped the matter.

Sadly, this could be true, but I feel optimistic about it. If we could
get people's attention, I believe some people would show up saying ---
oh, hey, I got a lot of data here and there and the whole thing would
appear.

>> When dejanews was created an enormous amount of effort went into
>> finding pre-dejanews material and passing it off to dejanews. At
>> this point, most of that likely no longer exists except in google's
>> database. And there's no point in it being in there if people can't
>> find it.
>
> Eh ... given Google's propensity to not get rid of things, I sort of
> suspect that the old DeJa News archive is probably still exists
> somewhere. It may actually be directly behind Google Groups Usenet
> gateway and the gateway itself may be munging articles as they are
> presented. Google really is antithetical to destroying data. Getting
> to that data, that's an entirely different issue.

I totally bet on the same.

Wally J <walterjones@invalid.nospam> writes:

> Julieta Shem <jshem@yaxenu.org> wrote
>
>>> You didn't mention whether or not you already know the message id.
>>> If you already know it, what's wrong with the Howard Knight site?
>>> <http://al.howardknight.net/>
>>
>> I had never heard of this website. It's asking me for a username and
>> password, which I don't have. It also presenting a certificate issued
>> to a different hostname.
>
> I'm sorry about that.
> I care that you get your answer that you need.
> But I didn't respond at first as I have nothing more to help you with.
>
> Nobody else has responded to this, unfortunately.
> So let's home someone else can help you explain what is happening.
> Especially as it purports to do exactly what it is you asked for.
>
> To be clear, I can't explain your results as when I go to howardknight.net,
> it only asks me for the message id. But I'm just a long-time user.
>
> I'm not an admin in any way.
>
> All we can do to help is ask the others here why Howardknight.net would ask
> for a password from you - when it has never asked for a password from me.

Someone helped me with it --- it was an address difference. And, by the
way, I have no current need of anything, but it's nice to know what to
do when the need arise. Thank you so much for your attention!

Grant Taylor <gtaylor@tnetconsulting.net> wrote:
>Eh ... given Google's propensity to not get rid of things, I sort of
>suspect that the old DeJa News archive is probably still exists
>somewhere. It may actually be directly behind Google Groups Usenet
>gateway and the gateway itself may be munging articles as they are
>presented. Google really is antithetical to destroying data. Getting
>to that data, that's an entirely different issue.

I agree. It's in there, but we can't get it out, and that is the
most frustrating thing possible. We gave it to them, but we can't
it it back.
--scott

--
"C'est un Nagra. C'est suisse, et tres, tres precis."

On Dec 10, 2023 at 12:13:06 PM CST, "Grant Taylor"
<gtaylor@tnetconsulting.net> wrote:

> I think the necessary resources are going to be a bigger issue than many
> other people.
>
> INN's traditional spool directories are going to tax just about any file
> system when you try to put hundreds of thousands of files in a single
> directory. Some newsgroups will have significantly more articles than
> others.

I have 1.6TB of tradspool on NVMe and no performance issues. Some groups have
over 4 million articles.

> Sure, there are ways to store articles so that they aren't all in one
> directory. But that's now a news server change. It can happen, but
> it's not as simple as a configuration change. It will likely be a code
> and a configuration change.
>
> There are other message stores, but the ones that I'm aware of tend to
> by cyclical in nature and of a fixed size which is antithetical to the
> archive forever goal.

You can set CNFS buffers to be written to sequentially, and as long as you're
paying attention you can add new buffers without ever losing anything and
wrapping to the beginning of the metabuffer. I've experimented with 100GB and
1TB buffer files and performance doesn't seem to matter the CNFS buffer size.

On a ZFS filesystem with CNFS buffers I get roughly 2.86X compression so that
1.6TB of tradspool shrinks down to ~450GB of CNFS buffers.

> This is probably a case for a custom NNTP server that is really a
> gateway (of sorts) to some sort of object store that is distributed and
> designed to scale to millions of objects in a container (newsgroup).
>
> Whatever is done needs to be flexible and have the ability to be
> reconfigured as things grow. It should also have a little bit of
> redundancy as the more systems that are added to it, the more fragile it
> will become.

If one wants to dabble in distributed NNTP architecture, Diablo and its
accompanying dreaderd are the way to go. Or NNTPSwitch if you can get it to
compile on anything.

>
>> The real problem is availability of the data.
>
> I agree that's probably the /primary/ problem and supersedes the storage
> in such as I believe that computer science / systems people can overcome
> the problem mentioned above. -- I'm not as confident that we can
> recover a full archive of Usenet any more. After all, according to
> Wikipedia, we're talking about 43+ years of data. Much of that data was
> deemed ephemeral by most people. Much of that data was difficult to
> collect 20+ years ago. The intervening 20 years won't have helped the
> matter.

I believe we have every bit of Usenet (may not *every single article* but
close) with all of the archives on archive.org, and what is still available on
commercial Usenet spools (most go back ~20 years for text retention).

It's a matter of putting it all back together on a usable NNTP spool that's
not as easy, but not undoable. There are a few of us working on projects of
our own with the same goal.

On 12/10/23 15:01, Julieta Shem wrote:
> I meant a sysadmin with little money.

I think it's actually going to take a fair bit more than a little money.

> That makes sense --- an NNTP made to be used as as reference, optimized
> only to provide a message per message-id.

NNTP itself wouldn't be changed at all. Or at least it shouldn't need
to be. Changing NNTP would fundamentally alter how clients interact and
what clients would be able to interact.

Or did you mean a server daemon / software that speaks standard NNTP?

> We should also have a solution for keeping addresses for a very long
> time.

Please elaborate on what you mean by "addresses" in this context.

> In the same way we have organized ourselves to keep the USENET
> rolling, we should also organize ourselves to make things last.

Making things last is quite a bit more difficult than many like.
Hardware fails. Software gets corrupted. Disks fill up. Systems get
compromised.

Things are going to require care, feeding, burping, and changing.

> But that's seems less easy. When there is money involved, people set
> up institutions that keep things going. We should have things like that.

That's done for significant sums of money. Trust funds probably work
for five or six digits. Institutions are formed for more digits.

> Could a sum of money be allocated, providing interest with which we can
> keep basic things like a hostname so that
>
> nntp://arxiv.use.net/<message@id>
>
> always fetches the article with <message@id>?

Sadly, I don't think that's going to work as well as you might hope.

That's a singular host name, which implies a singular point of
communications. Which in and of itself sort of implies a single point
of failure.

Sure, there are ways to make some of that non-singular, but the
complexity goes way up and things start being duplicated.

> I think that's the chance
> we have of running things properly in the world.

"properly" starts to get problematic to define the more people you add
to the mix and the longer you want things to last.

> The representative
> idea is likely at the end of its time now, finally. Electronic
> communication and the electronic printing press might give us the means
> to run stuff ourselves now. (An interesting keyword --- sortition.)

I don't know.

There is also the problem that such an archive isn't static. It is
currently growing at thousands, if not tens of thousands of messages a day.

--
Grant. . . .

On 12/10/23 18:28, Julieta Shem wrote:
> Sure. I cite these messages when I need to. But people often wonder
> --- how can I get it? Sometimes there is an answer such as some
> website, but such websites scramble over the years.

Agreed.

> The USENET deserves the highest standard in archiving.

Unfortunately most people don't agree.

I reluctantly don't agree by the very fact that I'm purging messages
from Google Groups from my private server.

I think that there is a LOT of Usenet that isn't worth archiving for
very long. In fact, many articles become increasingly less valuable
with each successively larger unit of time.

> ACK. When you said NNTP, I said --- okay ---, but I would think that a
> whole new system should be set-up.

Please clarify what you mean by "whole new system". Are you referring
to a new server that speaks NNTP to make articles available?

Or are you referring to something more radical?

> I would think that archive.org would
> be interested in getting a proposal for something like that. They seem
> pretty serious about archiving the Internet.

Maybe.

There are static Usenet archives on archive.org already. But they are
both incomplete and /static/. They also aren't readily usable the way
that a news server that speaks NNTP is.

> Sadly, this could be true, but I feel optimistic about it. If we could
> get people's attention, I believe some people would show up saying ---
> oh, hey, I got a lot of data here and there and the whole thing would
> appear.

I am not that optimistic.

I suspect each digit in the percentage is going to be an order of
magnitude more difficult to get than the previous digit.

I suspect that we can easily get 9%.

I don't know if we could get to 90% without support from people like
Google providing the DeJa News archive (unmodified).

Getting the next digit, 99.9% is going to be like pulling hens teeth.

Then what about 99.99%?

> I totally bet on the same.

ACK

--
Grant. . . .

On 12/10/23 19:04, Scott Dorsey wrote:
> I agree. It's in there, but we can't get it out, and that is the
> most frustrating thing possible. We gave it to them, but we can't
> it it back.

Having a better idea than many how Google lawyercats behave, I suspect
they would say "you didn't give it to us" and / or "we bought the
company and the data they had". They likely feel zero obligation to
provide it.

Then there's the "you gave X number of articles to the free global
Usenet out of the Y total articles therein" where X is significantly
smaller than Y and as such you get an infinitesimally small part back,
if that.

--
Grant. . . .

On 12/10/23 19:08, Jesse Rehmer wrote:
> I have 1.6TB of tradspool on NVMe and no performance issues. Some groups have
> over 4 million articles.

That sounds like a very nice setup.

I suspect that most people wouldn't want to spend the money /today/
especially for Usenet, and 40 year old articles therein.

But there are a number of people who would.

But then it does require some money to acquire the equipment, host it,
keep it online, etc. etc. etc.

Can it be done? Yes.

Is it static? Absolutely not.

> You can set CNFS buffers to be written to sequentially, and as long as you're
> paying attention you can add new buffers without ever losing anything and
> wrapping to the beginning of the metabuffer. I've experimented with 100GB and
> 1TB buffer files and performance doesn't seem to matter the CNFS buffer size.

Good to know.

But it sounds like you need to monitor it and make sure that your CNFS
buffers don't fill / wrap.

I'm sure there are ways to streamline this. But it's antithetical to
static / simple which is the domain most archives operate in.

> On a ZFS filesystem with CNFS buffers I get roughly 2.86X compression so that
> 1.6TB of tradspool shrinks down to ~450GB of CNFS buffers.

ACK

I'm glad to know that others are using ZFS too.

I'm not at all surprised to learn about the 2.86x compression.

> If one wants to dabble in distributed NNTP architecture, Diablo and its
> accompanying dreaderd are the way to go. Or NNTPSwitch if you can get it to
> compile on anything.

Does Diablo + dreader use it's own protocol? Or is it standard NNTP?

If it is it's own protocol, could someone write a gateway to be an NNTP
server that is a dreader client on the back end?

There is also the fact that with distributed systems, there need to be
the multiple systems to distribute the load over. -- Something that's
definitely possible, but it strongly speaks to something that needs
constant care and feeding.

> I believe we have every bit of Usenet (may not *every single article* but
> close) with all of the archives on archive.org, and what is still available on
> commercial Usenet spools (most go back ~20 years for text retention).

I would be very pleasantly surprised to learn if that is indeed the case.

I would not bet a fancy coffee on it. Maybe I'm more cynical than most.

> It's a matter of putting it all back together on a usable NNTP spool that's
> not as easy, but not undoable.

Agreed.

As I indicated in another post, I think the biggest issue will be
acquiring the corpus of articles.

> There are a few of us working on projects of our own with the same goal.

Cool!

Let me know if I can help in any way.

--
Grant. . . .

In article <ul64k8$s6g$3@tncsrv09.home.tnetconsulting.net>,
Grant Taylor <gtaylor@tnetconsulting.net> wrote:
>On 12/10/23 19:04, Scott Dorsey wrote:
>> I agree. It's in there, but we can't get it out, and that is the
>> most frustrating thing possible. We gave it to them, but we can't
>> it it back.
>
>Having a better idea than many how Google lawyercats behave, I suspect
>they would say "you didn't give it to us" and / or "we bought the
>company and the data they had". They likely feel zero obligation to
>provide it.
>
>Then there's the "you gave X number of articles to the free global
>Usenet out of the Y total articles therein" where X is significantly
>smaller than Y and as such you get an infinitesimally small part back,
>if that.

Yes, precisely.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

In news.admin.net-abuse.usenet Joerg Lorenz <hugybear@gmx.net> wrote:
> Probably you mean NNTP.

The point was to share a successful outcome, using Bughunters to report a spam
attack on a Google Group having ties to a Google development team. What worked
for that Group might not work for another, but a person could try it. My
suggestion was that spam overrunning comp.mobile.android could be posted on
Bughunters to the attention of the Android development team. The Android team
might be invested enough and in a position to escalate it internally and get it
fixed, just as the Time team did.

(Nothing to do with NTP and NNTP having similar spelling.)

Cheers! Edward

Scott Dorsey <kludge@panix.com> wrote

>>Then there's the "you gave X number of articles to the free global
>>Usenet out of the Y total articles therein" where X is significantly
>>smaller than Y and as such you get an infinitesimally small part back,
>>if that.
>
> Yes, precisely.

BTW, on the Android newsgroup some of us are discussing WHY they're doing
all this spam, where not every newsgroup is being spammed, it seems.

These are:
<https://groups.google.com/g/alt.internet.wireless>
<https://groups.google.com/g/alt.comp.microsoft.windows>
<https://groups.google.com/g/uk.telecom.mobile>
<https://groups.google.com/g/rec.photo.digital>
<https://groups.google.com/g/alt.home.repair>
<https://groups.google.com/g/comp.mobile.android>
etc.

These are not:
<http://alt.comp.os.windows-10.narkive.com>
<http://alt.comp.os.windows-11.narkive.com>
<https://groups.google.com/g/comp.mobile.ipad>
<https://groups.google.com/g/misc.phone.mobile.iphone>
etc.

I haven't looked extensively but they don't seem to be spamming groups
(such as the Windows 10 and 11 newsgroups most people post to) which aren't
auto-archived - but that could also be because the Google-to-Usenet portal
might not work for groups that aren't part of the DejaNews archives.

Dunno what they're doing for real, but it's only some newsgroups.
Not all.

On 12/12/23 19:38, Wally J wrote:
> BTW, on the Android newsgroup some of us are discussing WHY they're doing
> all this spam, where not every newsgroup is being spammed, it seems.

I'm of the opinion that it's a bunch of different spam campaigns, likely
by almost as many spammers.

> I haven't looked extensively but they don't seem to be spamming groups
> (such as the Windows 10 and 11 newsgroups most people post to) which aren't
> auto-archived - but that could also be because the Google-to-Usenet portal
> might not work for groups that aren't part of the DejaNews archives.

^DejaNews^Usenet

I know that the newsgroups for Thunderbird / Firefox support and some of
the newer versions of Windows don't have Usenet newsgroups inside of
Google Groups.

In traditional news parlance, Google doesn't carry said newsgroups /
they aren't in Google's active newsgroups file. As such there is
nothing inside of Google Groups that the spammers can post to using the
Google Groups Usenet gateway.

> Dunno what they're doing for real, but it's only some newsgroups.
> Not all.

I don't think I've ever seen a single spam campaign hit all of the
newsgroups that I subscribe to, much less all of the thousands in my
server's active file. All of them are one or few groups and there is no
rime or reason that I'm aware of in their selection of groups.

Grant. . . .

Grant Taylor <gtaylor@tnetconsulting.net> wrote

>> BTW, on the Android newsgroup some of us are discussing WHY they're doing
>> all this spam, where not every newsgroup is being spammed, it seems.
>
> I'm of the opinion that it's a bunch of different spam campaigns, likely
> by almost as many spammers.
>
>> I haven't looked extensively but they don't seem to be spamming groups
>> (such as the Windows 10 and 11 newsgroups most people post to) which aren't
>> auto-archived - but that could also be because the Google-to-Usenet portal
>> might not work for groups that aren't part of the DejaNews archives.
>
> ^DejaNews^Usenet

I wish we had a name for it but it's like when we talk about unwanted spam.
No matter what name we use, everyone knows what we're talking about.

I call it dejagoogle sometimes.
a. It's a search engine to me
b. And it's a cite reference (better'n a message-ID is anyway)

> I know that the newsgroups for Thunderbird / Firefox support and some of
> the newer versions of Windows don't have Usenet newsgroups inside of
> Google Groups.

Yup. I tried valiantly, as did you, I believe, to get Google to add them to
the auto-archives so that others can run searches to find answers before
they post a question. I failed.
<https://groups.google.com/g/alt.comp.software.thunderbird> 404 not found
<https://groups.google.com/g/alt.comp.software.firefox> Content unavailable
NOTE: Interesting I got two different errors for those two URIs just now.

We might check for the Google spam on the narkives though.
<http://alt.comp.software.thunderbird.narkive.com> (unsafe)
<http://alt.comp.software.firefox.narkive.com> (unsafe)
But my browser setup won't let me.
> In traditional news parlance, Google doesn't carry said newsgroups /
> they aren't in Google's active newsgroups file. As such there is
> nothing inside of Google Groups that the spammers can post to using the
> Google Groups Usenet gateway.

Yeah. That must be why the Gspammers aren't spamming these Windows ngs.
<http://alt.comp.os.windows-10.narkive.com>
<http://alt.comp.os.windows-11.narkive.com>

>> Dunno what they're doing for real, but it's only some newsgroups.
>> Not all.
>
> I don't think I've ever seen a single spam campaign hit all of the
> newsgroups that I subscribe to, much less all of the thousands in my
> server's active file. All of them are one or few groups and there is no
> rime or reason that I'm aware of in their selection of groups.

I've noticed a lot of the spam is in funky characters, which I find odd,
but many also have URLs so they could be phishing attacks for all I know.

The M0VIE spam is particularly repetitive - I suspect they want to get into
search engine results - but I don't know that for a fact. I'm just guessing
as a lot of my tutorials posted to Usenet end up in the first page of hits
from a "normal" www.google.com search - so Usenet _does_ show up there too.

BTW, regarding the message from Individual.net, I was heartened they care.
Anybody have any new datapoints from Giganews & Highwinds admins yet?

Wally J <walterjones@invalid.nospam> wrote:
>Scott Dorsey <kludge@panix.com> wrote
>I haven't looked extensively but they don't seem to be spamming groups
>(such as the Windows 10 and 11 newsgroups most people post to) which aren't
>auto-archived - but that could also be because the Google-to-Usenet portal
>might not work for groups that aren't part of the DejaNews archives.
>
>Dunno what they're doing for real, but it's only some newsgroups.
>Not all.

And it is not only Usenet. I saw the spam happening on pandoc-users, a
googlegroup about the pandoc software which is not gated to Usenet. A
user complained on-list, I tried to reply, got a bounce ("You are not
allowed to write to this list"). I finally shrugged it off and tried
ot unsubscribe via the Mail address that is in the headers, got a
bounce ("You are not allowed to write to this list"). Tried to
unsubscribe via the URL from the header, got redirected to an error
page saying that the group is being blocked due to malicious content.

I eneded up putting the mail address (thankfully, an exclusive mail
address just for this group) to ":fail: Depeer Google Groups Now".

Google doesn't even care about its own groups any more.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834