I noticed that with a peer (GRC) that ignores custom Message-ID that a
duplicate message is created in the newsgroup.

Steps:

1. Post a message via the local server. The message posts with the
Message-ID generated by rslight.

2. The remote INN server accepts the message on sync but strips the
Message-ID and replaces it with its own Message-ID.

This should be reproducible by posting to grc.test.

If the remote server strictly generates its own message-id in place of
the provided message-id, rslight is not aware of this. So rslight then
stores its original message in the spool with its own Message-ID, then
pulls the message from remote containing the remote-generated
Message-ID. This results in the same message appearing twice with two
Message-ID values.

Is there any configuration option to get rslight to accept the
Message-ID from the remote, or at least discard one of the duplicate
messages and refrain from re-copying the duplicate to and from the
remote?

--
@firefly@neon.nightbulb.net | https://neon.nightbulb.net/firefly
Grand Garter General of the Imperial Baggy Jeans Mafia

SugarBug wrote:

> I noticed that with a peer (GRC) that ignores custom Message-ID that a
> duplicate message is created in the newsgroup.

> Steps:

> 1. Post a message via the local server. The message posts with the
> Message-ID generated by rslight.

> 2. The remote INN server accepts the message on sync but strips the
> Message-ID and replaces it with its own Message-ID.

> This should be reproducible by posting to grc.test.

> If the remote server strictly generates its own message-id in place of
> the provided message-id, rslight is not aware of this. So rslight then
> stores its original message in the spool with its own Message-ID, then
> pulls the message from remote containing the remote-generated
> Message-ID. This results in the same message appearing twice with two
> Message-ID values.

> Is there any configuration option to get rslight to accept the
> Message-ID from the remote, or at least discard one of the duplicate
> messages and refrain from re-copying the duplicate to and from the
> remote?

Interesting. So the remote server is ignoring the supplied MID and creating it's own.

Does the remote server send a reply to state this? If it does, rslight could then modify it's MID. If not, we'll need to code it to NOT generate a MID for a certain <section>. If it DOES send a reply, that would be better. Can you see if it does that?

--
Retro Guy

Retro Guy wrote:

> SugarBug wrote:

>> I noticed that with a peer (GRC) that ignores custom Message-ID that a
>> duplicate message is created in the newsgroup.

>> Steps:

>> 1. Post a message via the local server. The message posts with the
>> Message-ID generated by rslight.

>> 2. The remote INN server accepts the message on sync but strips the
>> Message-ID and replaces it with its own Message-ID.

>> This should be reproducible by posting to grc.test.

>> If the remote server strictly generates its own message-id in place of
>> the provided message-id, rslight is not aware of this. So rslight then
>> stores its original message in the spool with its own Message-ID, then
>> pulls the message from remote containing the remote-generated
>> Message-ID. This results in the same message appearing twice with two
>> Message-ID values.

>> Is there any configuration option to get rslight to accept the
>> Message-ID from the remote, or at least discard one of the duplicate
>> messages and refrain from re-copying the duplicate to and from the
>> remote?

> Interesting. So the remote server is ignoring the supplied MID and creating it's own.

> Does the remote server send a reply to state this? If it does, rslight could then modify it's MID. If not, we'll need to code it to NOT generate a MID for a certain <section>. If it DOES send a reply, that would be better. Can you see if it does that?

Actually, since the message is sent to the remote server AFTER posting locally, we'll need to create a config option by <section>. The config option would require that the post is NOT posted locally at all, but ONLY after retrieving it back from the remote server (since we won't know the MID at the time of posting)

--
Retro Guy

Am 17.01.2024 um 11:56:33 Uhr schrieb SugarBug:

> 2. The remote INN server accepts the message on sync but strips the
> Message-ID and replaces it with its own Message-ID.

That is the conceptual problem.
As message-ID is intended to be an identifier of a message and must not
be changed because then the message can't be identified anymore.

Tell the operator of that server not to change the Message ID.

On Wed, 17 Jan 2024 18:24:23 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> Retro Guy wrote:
>
> > SugarBug wrote:
>
> >> I noticed that with a peer (GRC) that ignores custom Message-ID
> >> that a duplicate message is created in the newsgroup.
>
> >> Steps:
>
> >> 1. Post a message via the local server. The message posts with the
> >> Message-ID generated by rslight.
>
> >> 2. The remote INN server accepts the message on sync but strips the
> >> Message-ID and replaces it with its own Message-ID.
>
> >> This should be reproducible by posting to grc.test.
>
> >> If the remote server strictly generates its own message-id in
> >> place of the provided message-id, rslight is not aware of this. So
> >> rslight then stores its original message in the spool with its own
> >> Message-ID, then pulls the message from remote containing the
> >> remote-generated Message-ID. This results in the same message
> >> appearing twice with two Message-ID values.
>
> >> Is there any configuration option to get rslight to accept the
> >> Message-ID from the remote, or at least discard one of the
> >> duplicate messages and refrain from re-copying the duplicate to
> >> and from the remote?
>
> > Interesting. So the remote server is ignoring the supplied MID and
> > creating it's own.
>
> > Does the remote server send a reply to state this? If it does,
> > rslight could then modify it's MID. If not, we'll need to code it
> > to NOT generate a MID for a certain <section>. If it DOES send a
> > reply, that would be better. Can you see if it does that?
>
> Actually, since the message is sent to the remote server AFTER
> posting locally, we'll need to create a config option by <section>.
> The config option would require that the post is NOT posted locally
> at all, but ONLY after retrieving it back from the remote server
> (since we won't know the MID at the time of posting)

Please hold off on making any code changes. I will need to examine it in
very fine detail to be sure of exactly what is happening.

--
CRYP7010G3R | Read between the signals. | Read between the signs.
<6ef76f5d853272d23e209476783618ed$1@sybershock.com>
3883@sugar.bug | sybershock.com | alt.sources.crypto

On Thu, 18 Jan 2024 05:44:24 -0600, SugarBug wrote:

> On Wed, 17 Jan 2024 18:24:23 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>> Retro Guy wrote:
>>
>>> SugarBug wrote:
>>
>>>> I noticed that with a peer (GRC) that ignores custom Message-ID
>>>> that a duplicate message is created in the newsgroup.
>>
>>>> Steps:
>>
>>>> 1. Post a message via the local server. The message posts with the
>>>> Message-ID generated by rslight.
>>
>>>> 2. The remote INN server accepts the message on sync but strips the
>>>> Message-ID and replaces it with its own Message-ID.
>>
>>>> This should be reproducible by posting to grc.test.
>>
>>>> If the remote server strictly generates its own message-id in
>>>> place of the provided message-id, rslight is not aware of this. So
>>>> rslight then stores its original message in the spool with its own
>>>> Message-ID, then pulls the message from remote containing the
>>>> remote-generated Message-ID. This results in the same message
>>>> appearing twice with two Message-ID values.
>>
>>>> Is there any configuration option to get rslight to accept the
>>>> Message-ID from the remote, or at least discard one of the
>>>> duplicate messages and refrain from re-copying the duplicate to
>>>> and from the remote?
>>
>>> Interesting. So the remote server is ignoring the supplied MID and
>>> creating it's own.
>>
>>> Does the remote server send a reply to state this? If it does,
>>> rslight could then modify it's MID. If not, we'll need to code it
>>> to NOT generate a MID for a certain <section>. If it DOES send a
>>> reply, that would be better. Can you see if it does that?
>>
>> Actually, since the message is sent to the remote server AFTER
>> posting locally, we'll need to create a config option by <section>.
>> The config option would require that the post is NOT posted locally
>> at all, but ONLY after retrieving it back from the remote server
>> (since we won't know the MID at the time of posting)
>
> Please hold off on making any code changes. I will need to examine it in
> very fine detail to be sure of exactly what is happening.

Got it! I'll wait to hear back.

I did take a peek at the code, and it shouldn't be too difficult to just
NOT insert on post, just wait until it returns from the remote server.

I guess I'll just finish my Sprouts order for today instead :)

On Wed, 17 Jan 2024 21:10:44 +0100, Marco Moock wrote:

> Am 17.01.2024 um 11:56:33 Uhr schrieb SugarBug:
>
>> 2. The remote INN server accepts the message on sync but strips the
>> Message-ID and replaces it with its own Message-ID.
>
> That is the conceptual problem.
> As message-ID is intended to be an identifier of a message and must not
> be changed because then the message can't be identified anymore.
>
> Tell the operator of that server not to change the Message ID.

I agree that it's not a good thing to do. At least it should reject the
message with a reply that says something useful.

But since rslight is really a READER, it can't complain too much. If I
wrote the server that did this I'd say, "We're just trying to avoid
spoofing other servers in mid." I would though produce a proper nntp
response so the sender could see what happened in their logs.

On Wed, 17 Jan 2024 18:21:51 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> SugarBug wrote:
>
> > I noticed that with a peer (GRC) that ignores custom Message-ID
> > that a duplicate message is created in the newsgroup.
>
> > Steps:
>
> > 1. Post a message via the local server. The message posts with the
> > Message-ID generated by rslight.
>
> > 2. The remote INN server accepts the message on sync but strips the
> > Message-ID and replaces it with its own Message-ID.
>
> > This should be reproducible by posting to grc.test.
>
> > If the remote server strictly generates its own message-id in place
> > of the provided message-id, rslight is not aware of this. So
> > rslight then stores its original message in the spool with its own
> > Message-ID, then pulls the message from remote containing the
> > remote-generated Message-ID. This results in the same message
> > appearing twice with two Message-ID values.
>
> > Is there any configuration option to get rslight to accept the
> > Message-ID from the remote, or at least discard one of the duplicate
> > messages and refrain from re-copying the duplicate to and from the
> > remote?
>
> Interesting. So the remote server is ignoring the supplied MID and
> creating it's own.
>
> Does the remote server send a reply to state this? If it does,
> rslight could then modify it's MID. If not, we'll need to code it to
> NOT generate a MID for a certain <section>. If it DOES send a reply,
> that would be better. Can you see if it does that?

Rslight logs say, "Removed: 1" ==>

spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
<uob54a$cqf$1@GRC> Removed: 1

spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article received
<uob4gi$cjq$1@GRC> Removed: 1

Two of the test messages generated duplicates. Two did not:

https://rslight.sybershock.com/grc/thread.php?group=grc.test

Look for subjects containing, "Testing Duplicate Message Bug".

Here is more logging from my rslight:

nntp.log:Jan 18 12:19:27 grc.test:28 240 Article received OK (posted)
nntp.log:Jan 18 12:28:20 grc.test:30 240 Article received OK (posted)
nntp.log.1:Jan 18 12:02:30 grc.test:21 240 Article received OK (posted)
nntp.log.1:Jan 18 12:02:47 grc.test:22 240 Article received OK (posted)
nntp.log.1:Jan 18 12:03:16 grc.test:23 240 Article received OK (posted)
spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
<uob54a$cqf$1@GRC> Removed: 1
spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article received
<uob4gi$cjq$1@GRC> Removed: 1

--
Night Bulb | nightbulb.net | alt.light.bulb

On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:

> On Wed, 17 Jan 2024 18:21:51 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>> SugarBug wrote:
>>
>>> I noticed that with a peer (GRC) that ignores custom Message-ID
>>> that a duplicate message is created in the newsgroup.
>>
>>> Steps:
>>
>>> 1. Post a message via the local server. The message posts with the
>>> Message-ID generated by rslight.
>>
>>> 2. The remote INN server accepts the message on sync but strips the
>>> Message-ID and replaces it with its own Message-ID.
>>
>>> This should be reproducible by posting to grc.test.
>>
>>> If the remote server strictly generates its own message-id in place
>>> of the provided message-id, rslight is not aware of this. So
>>> rslight then stores its original message in the spool with its own
>>> Message-ID, then pulls the message from remote containing the
>>> remote-generated Message-ID. This results in the same message
>>> appearing twice with two Message-ID values.
>>
>>> Is there any configuration option to get rslight to accept the
>>> Message-ID from the remote, or at least discard one of the duplicate
>>> messages and refrain from re-copying the duplicate to and from the
>>> remote?
>>
>> Interesting. So the remote server is ignoring the supplied MID and
>> creating it's own.
>>
>> Does the remote server send a reply to state this? If it does,
>> rslight could then modify it's MID. If not, we'll need to code it to
>> NOT generate a MID for a certain <section>. If it DOES send a reply,
>> that would be better. Can you see if it does that?
>
> Rslight logs say, "Removed: 1" ==>
>
> spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
> <uob54a$cqf$1@GRC> Removed: 1
>
> spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article received
> <uob4gi$cjq$1@GRC> Removed: 1
>
> Two of the test messages generated duplicates. Two did not:
>
> https://rslight.sybershock.com/grc/thread.php?group=grc.test
>
> Look for subjects containing, "Testing Duplicate Message Bug".
>
> Here is more logging from my rslight:
>
> nntp.log:Jan 18 12:19:27 grc.test:28 240 Article received OK (posted)
> nntp.log:Jan 18 12:28:20 grc.test:30 240 Article received OK (posted)
> nntp.log.1:Jan 18 12:02:30 grc.test:21 240 Article received OK (posted)
> nntp.log.1:Jan 18 12:02:47 grc.test:22 240 Article received OK (posted)
> nntp.log.1:Jan 18 12:03:16 grc.test:23 240 Article received OK (posted)
> spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
> <uob54a$cqf$1@GRC> Removed: 1
> spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article received
> <uob4gi$cjq$1@GRC> Removed: 1

"Removed: 1" is a leftover debug message. I'll fix that. It simply means
that the file in <section>/outgoing was successfully deleted. (It's the
return value of unlink)

I'm more concerned with: "Posted: .msg". The '.msg' should be a file name,
and I don't see how the file name is '.msg', it should be a message-id. How
was that particular message posted? Via rslight, via rslight nntp server,
or manually maybe?

nntp.log means the message was posted to the rslight nntp server ok.
spoolnews.log means the message was posted to the remote server ok (or at
least received a '240' response from remote server.

On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:

> On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>
>> On Wed, 17 Jan 2024 18:21:51 +0000
>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>
>>> SugarBug wrote:
>>>
>>>> I noticed that with a peer (GRC) that ignores custom Message-ID
>>>> that a duplicate message is created in the newsgroup.
>>>
>>>> Steps:
>>>
>>>> 1. Post a message via the local server. The message posts with the
>>>> Message-ID generated by rslight.
>>>
>>>> 2. The remote INN server accepts the message on sync but strips the
>>>> Message-ID and replaces it with its own Message-ID.
>>>
>>>> This should be reproducible by posting to grc.test.
>>>
>>>> If the remote server strictly generates its own message-id in place
>>>> of the provided message-id, rslight is not aware of this. So
>>>> rslight then stores its original message in the spool with its own
>>>> Message-ID, then pulls the message from remote containing the
>>>> remote-generated Message-ID. This results in the same message
>>>> appearing twice with two Message-ID values.
>>>
>>>> Is there any configuration option to get rslight to accept the
>>>> Message-ID from the remote, or at least discard one of the duplicate
>>>> messages and refrain from re-copying the duplicate to and from the
>>>> remote?
>>>
>>> Interesting. So the remote server is ignoring the supplied MID and
>>> creating it's own.
>>>
>>> Does the remote server send a reply to state this? If it does,
>>> rslight could then modify it's MID. If not, we'll need to code it to
>>> NOT generate a MID for a certain <section>. If it DOES send a reply,
>>> that would be better. Can you see if it does that?
<snip>
>>
>> Two of the test messages generated duplicates. Two did not:

My guess is that two were not inserted immediately and two were. The ones
not inserted immediately were only inserted after receiving back from
remote server (so no duplicate).

Rslight will not insert immediately if spoolnews for the specific section
is running at that moment. In that situation, it will only prepare it for
send to remote server and wait for it's return. (That will explain why some
messages do not appear immediately after posting)

On Thu, 18 Jan 2024 06:38:51 -0700
Retro Guy <retroguy@novabbs.org> wrote:

> On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>
> > On Wed, 17 Jan 2024 18:21:51 +0000
> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
> >
> >> SugarBug wrote:
> >>
> >>> I noticed that with a peer (GRC) that ignores custom Message-ID
> >>> that a duplicate message is created in the newsgroup.
> >>
> >>> Steps:
> >>
> >>> 1. Post a message via the local server. The message posts with the
> >>> Message-ID generated by rslight.
> >>
> >>> 2. The remote INN server accepts the message on sync but strips
> >>> the Message-ID and replaces it with its own Message-ID.
> >>
> >>> This should be reproducible by posting to grc.test.
> >>
> >>> If the remote server strictly generates its own message-id in
> >>> place of the provided message-id, rslight is not aware of this. So
> >>> rslight then stores its original message in the spool with its own
> >>> Message-ID, then pulls the message from remote containing the
> >>> remote-generated Message-ID. This results in the same message
> >>> appearing twice with two Message-ID values.
> >>
> >>> Is there any configuration option to get rslight to accept the
> >>> Message-ID from the remote, or at least discard one of the
> >>> duplicate messages and refrain from re-copying the duplicate to
> >>> and from the remote?
> >>
> >> Interesting. So the remote server is ignoring the supplied MID and
> >> creating it's own.
> >>
> >> Does the remote server send a reply to state this? If it does,
> >> rslight could then modify it's MID. If not, we'll need to code it
> >> to NOT generate a MID for a certain <section>. If it DOES send a
> >> reply, that would be better. Can you see if it does that?
> >
> > Rslight logs say, "Removed: 1" ==>
> >
> > spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
> > <uob54a$cqf$1@GRC> Removed: 1
> >
> > spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article
> > received <uob4gi$cjq$1@GRC> Removed: 1
> >
> > Two of the test messages generated duplicates. Two did not:
> >
> > https://rslight.sybershock.com/grc/thread.php?group=grc.test
> >
> > Look for subjects containing, "Testing Duplicate Message Bug".
> >
> > Here is more logging from my rslight:
> >
> > nntp.log:Jan 18 12:19:27 grc.test:28 240 Article received OK
> > (posted) nntp.log:Jan 18 12:28:20 grc.test:30 240 Article received
> > OK (posted) nntp.log.1:Jan 18 12:02:30 grc.test:21 240 Article
> > received OK (posted) nntp.log.1:Jan 18 12:02:47 grc.test:22 240
> > Article received OK (posted) nntp.log.1:Jan 18 12:03:16 grc.test:23
> > 240 Article received OK (posted) spoolnews.log:Jan 18 12:21:30 grc
> > Posted: .msg: 240 Article received <uob54a$cqf$1@GRC> Removed: 1
> > spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article
> > received <uob4gi$cjq$1@GRC> Removed: 1
>
> "Removed: 1" is a leftover debug message. I'll fix that. It simply
> means that the file in <section>/outgoing was successfully deleted.
> (It's the return value of unlink)
>
> I'm more concerned with: "Posted: .msg". The '.msg' should be a file
> name, and I don't see how the file name is '.msg', it should be a
> message-id. How was that particular message posted? Via rslight, via
> rslight nntp server, or manually maybe?

The messages were all posted using claws-mail news client. The
newsreaders posted the messages to my rslight test server via NNTP. The
test server forwards the messages to GRC. Eventually on synchronizing
with GRC the rslight server pulls back a duplicate message with a
message-id ending with "@GRC".
> nntp.log means the message was posted to the rslight nntp server ok.
> spoolnews.log means the message was posted to the remote server ok
> (or at least received a '240' response from remote server.

--
3883@sugar.bug | sybershock.com | alt.cypher

On Thu, 18 Jan 2024 06:50:04 -0700
Retro Guy <retroguy@novabbs.org> wrote:

> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>
> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
> >
> >> On Wed, 17 Jan 2024 18:21:51 +0000
> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
> >>
> >>> SugarBug wrote:
> >>>
> >>>> I noticed that with a peer (GRC) that ignores custom Message-ID
> >>>> that a duplicate message is created in the newsgroup.
> >>>
> >>>> Steps:
> >>>
> >>>> 1. Post a message via the local server. The message posts with
> >>>> the Message-ID generated by rslight.
> >>>
> >>>> 2. The remote INN server accepts the message on sync but strips
> >>>> the Message-ID and replaces it with its own Message-ID.
> >>>
> >>>> This should be reproducible by posting to grc.test.
> >>>
> >>>> If the remote server strictly generates its own message-id in
> >>>> place of the provided message-id, rslight is not aware of this.
> >>>> So rslight then stores its original message in the spool with
> >>>> its own Message-ID, then pulls the message from remote
> >>>> containing the remote-generated Message-ID. This results in the
> >>>> same message appearing twice with two Message-ID values.
> >>>
> >>>> Is there any configuration option to get rslight to accept the
> >>>> Message-ID from the remote, or at least discard one of the
> >>>> duplicate messages and refrain from re-copying the duplicate to
> >>>> and from the remote?
> >>>
> >>> Interesting. So the remote server is ignoring the supplied MID and
> >>> creating it's own.
> >>>
> >>> Does the remote server send a reply to state this? If it does,
> >>> rslight could then modify it's MID. If not, we'll need to code it
> >>> to NOT generate a MID for a certain <section>. If it DOES send a
> >>> reply, that would be better. Can you see if it does that?
> <snip>
> >>
> >> Two of the test messages generated duplicates. Two did not:
>
> My guess is that two were not inserted immediately and two were. The
> ones not inserted immediately were only inserted after receiving back
> from remote server (so no duplicate).
>
> Rslight will not insert immediately if spoolnews for the specific
> section is running at that moment. In that situation, it will only
> prepare it for send to remote server and wait for it's return. (That
> will explain why some messages do not appear immediately after
> posting)

I ensured the cron job was not running and I posted two messages.

Then I ensured the cron job was running and I posted one message.

I can confirm that a duplicate was created while the cron was running.

Here are the duplicate message-id headers:

<8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com> <uobr41$k2l$1@GRC>

Here is the spoolnews entry:

spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
<uobr41$k2l$1@GRC> Removed: 1

All three messages were posted using claws-mail. All three messages
were posted to my rslight NNTP server, which then synced to the GRC
test group.

--
3883@sugar.bug | Free Usenet Servers | https://sybershock.com/#usenet

On Wed, 17 Jan 2024 21:10:44 +0100
Marco Moock <mm+usenet-es@dorfdsl.de> wrote:

> Am 17.01.2024 um 11:56:33 Uhr schrieb SugarBug:
>
> > 2. The remote INN server accepts the message on sync but strips the
> > Message-ID and replaces it with its own Message-ID.
>
> That is the conceptual problem.
> As message-ID is intended to be an identifier of a message and must
> not be changed because then the message can't be identified anymore.
>
> Tell the operator of that server not to change the Message ID.

I doubt that will change anything. GRC is not peered to Usenet and does
not follow Usenet specs. I have my rslight testing server peered to it.

--
4477@king.bee | Flee the Swarm. Join the horde. | talk.politics.crypto

SugarBug wrote:

> On Thu, 18 Jan 2024 06:50:04 -0700
> Retro Guy <retroguy@novabbs.org> wrote:

>> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>>
>> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>> >
>> >> On Wed, 17 Jan 2024 18:21:51 +0000
>> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>> >>
>> >>> SugarBug wrote:
>> >>>
>> >>>> I noticed that with a peer (GRC) that ignores custom Message-ID
>> >>>> that a duplicate message is created in the newsgroup.
>> >>>
>> >>>> Steps:
>> >>>
>> >>>> 1. Post a message via the local server. The message posts with
>> >>>> the Message-ID generated by rslight.
>> >>>
>> >>>> 2. The remote INN server accepts the message on sync but strips
>> >>>> the Message-ID and replaces it with its own Message-ID.
>> >>>
>> >>>> This should be reproducible by posting to grc.test.
>> >>>
>> >>>> If the remote server strictly generates its own message-id in
>> >>>> place of the provided message-id, rslight is not aware of this.
>> >>>> So rslight then stores its original message in the spool with
>> >>>> its own Message-ID, then pulls the message from remote
>> >>>> containing the remote-generated Message-ID. This results in the
>> >>>> same message appearing twice with two Message-ID values.
>> >>>
>> >>>> Is there any configuration option to get rslight to accept the
>> >>>> Message-ID from the remote, or at least discard one of the
>> >>>> duplicate messages and refrain from re-copying the duplicate to
>> >>>> and from the remote?
>> >>>
>> >>> Interesting. So the remote server is ignoring the supplied MID and
>> >>> creating it's own.
>> >>>
>> >>> Does the remote server send a reply to state this? If it does,
>> >>> rslight could then modify it's MID. If not, we'll need to code it
>> >>> to NOT generate a MID for a certain <section>. If it DOES send a
>> >>> reply, that would be better. Can you see if it does that?
>> <snip>
>> >>
>> >> Two of the test messages generated duplicates. Two did not:
>>
>> My guess is that two were not inserted immediately and two were. The
>> ones not inserted immediately were only inserted after receiving back
>> from remote server (so no duplicate).
>>
>> Rslight will not insert immediately if spoolnews for the specific
>> section is running at that moment. In that situation, it will only
>> prepare it for send to remote server and wait for it's return. (That
>> will explain why some messages do not appear immediately after
>> posting)

> I ensured the cron job was not running and I posted two messages.

> Then I ensured the cron job was running and I posted one message.

> I can confirm that a duplicate was created while the cron was running.

Messages won't immediately insert if the specific spoolnews.php for the specific <section> is running. Cron can be running, just not the specific spoolnews.php.

> Here are the duplicate message-id headers:

> <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com> <uobr41$k2l$1@GRC>

> Here is the spoolnews entry:

> spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
> <uobr41$k2l$1@GRC> Removed: 1

> All three messages were posted using claws-mail. All three messages
> were posted to my rslight NNTP server, which then synced to the GRC
> test group.

Ok, thank you for testing. It does look like anytime a message is immediately inserted, it gets duplicated. I don't see a way around that other than a config option to disable immediate insertion for the section.

--
Retro Guy

On Thu, 18 Jan 2024 12:35:13 -0600, SugarBug wrote:

> On Thu, 18 Jan 2024 06:38:51 -0700
> Retro Guy <retroguy@novabbs.org> wrote:
>
>> On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>>
>>> On Wed, 17 Jan 2024 18:21:51 +0000
>>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>>
>>>> SugarBug wrote:
>>>>
>>>>> I noticed that with a peer (GRC) that ignores custom Message-ID
>>>>> that a duplicate message is created in the newsgroup.
>>>>
>>>>> Steps:
>>>>
>>>>> 1. Post a message via the local server. The message posts with the
>>>>> Message-ID generated by rslight.
>>>>
>>>>> 2. The remote INN server accepts the message on sync but strips
>>>>> the Message-ID and replaces it with its own Message-ID.
>>>>
>>>>> This should be reproducible by posting to grc.test.
>>>>
>>>>> If the remote server strictly generates its own message-id in
>>>>> place of the provided message-id, rslight is not aware of this. So
>>>>> rslight then stores its original message in the spool with its own
>>>>> Message-ID, then pulls the message from remote containing the
>>>>> remote-generated Message-ID. This results in the same message
>>>>> appearing twice with two Message-ID values.
>>>>
>>>>> Is there any configuration option to get rslight to accept the
>>>>> Message-ID from the remote, or at least discard one of the
>>>>> duplicate messages and refrain from re-copying the duplicate to
>>>>> and from the remote?
>>>>
>>>> Interesting. So the remote server is ignoring the supplied MID and
>>>> creating it's own.
>>>>
>>>> Does the remote server send a reply to state this? If it does,
>>>> rslight could then modify it's MID. If not, we'll need to code it
>>>> to NOT generate a MID for a certain <section>. If it DOES send a
>>>> reply, that would be better. Can you see if it does that?
>>>
>>> Rslight logs say, "Removed: 1" ==>
>>>
>>> spoolnews.log:Jan 18 12:21:30 grc Posted: .msg: 240 Article received
>>> <uob54a$cqf$1@GRC> Removed: 1
>>>
>>> spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article
>>> received <uob4gi$cjq$1@GRC> Removed: 1
>>>
>>> Two of the test messages generated duplicates. Two did not:
>>>
>>> https://rslight.sybershock.com/grc/thread.php?group=grc.test
>>>
>>> Look for subjects containing, "Testing Duplicate Message Bug".
>>>
>>> Here is more logging from my rslight:
>>>
>>> nntp.log:Jan 18 12:19:27 grc.test:28 240 Article received OK
>>> (posted) nntp.log:Jan 18 12:28:20 grc.test:30 240 Article received
>>> OK (posted) nntp.log.1:Jan 18 12:02:30 grc.test:21 240 Article
>>> received OK (posted) nntp.log.1:Jan 18 12:02:47 grc.test:22 240
>>> Article received OK (posted) nntp.log.1:Jan 18 12:03:16 grc.test:23
>>> 240 Article received OK (posted) spoolnews.log:Jan 18 12:21:30 grc
>>> Posted: .msg: 240 Article received <uob54a$cqf$1@GRC> Removed: 1
>>> spoolnews.log.1:Jan 18 12:10:58 grc Posted: .msg: 240 Article
>>> received <uob4gi$cjq$1@GRC> Removed: 1
>>
>> "Removed: 1" is a leftover debug message. I'll fix that. It simply
>> means that the file in <section>/outgoing was successfully deleted.
>> (It's the return value of unlink)
>>
>> I'm more concerned with: "Posted: .msg". The '.msg' should be a file
>> name, and I don't see how the file name is '.msg', it should be a
>> message-id. How was that particular message posted? Via rslight, via
>> rslight nntp server, or manually maybe?
>
> The messages were all posted using claws-mail news client. The
> newsreaders posted the messages to my rslight test server via NNTP. The
> test server forwards the messages to GRC. Eventually on synchronizing
> with GRC the rslight server pulls back a duplicate message with a
> message-id ending with "@GRC".
>
>> nntp.log means the message was posted to the rslight nntp server ok.
>> spoolnews.log means the message was posted to the remote server ok
>> (or at least received a '240' response from remote server.

Is it reasonable for you to place all GRC groups in one section?

If so, I'll code a config setting to disable immediate posting and we'll
see how that does.

On Thu, 18 Jan 2024 19:39:59 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> SugarBug wrote:
>
> > On Thu, 18 Jan 2024 06:50:04 -0700
> > Retro Guy <retroguy@novabbs.org> wrote:
>
> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
> >>
> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
> >> >
> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
> >> >>
> >> >>> SugarBug wrote:
> >> >>>
> >> >>>> I noticed that with a peer (GRC) that ignores custom
> >> >>>> Message-ID that a duplicate message is created in the
> >> >>>> newsgroup.
> >> >>>
> >> >>>> Steps:
> >> >>>
> >> >>>> 1. Post a message via the local server. The message posts with
> >> >>>> the Message-ID generated by rslight.
> >> >>>
> >> >>>> 2. The remote INN server accepts the message on sync but
> >> >>>> strips the Message-ID and replaces it with its own
> >> >>>> Message-ID.
> >> >>>
> >> >>>> This should be reproducible by posting to grc.test.
> >> >>>
> >> >>>> If the remote server strictly generates its own message-id in
> >> >>>> place of the provided message-id, rslight is not aware of
> >> >>>> this. So rslight then stores its original message in the
> >> >>>> spool with its own Message-ID, then pulls the message from
> >> >>>> remote containing the remote-generated Message-ID. This
> >> >>>> results in the same message appearing twice with two
> >> >>>> Message-ID values.
> >> >>>
> >> >>>> Is there any configuration option to get rslight to accept the
> >> >>>> Message-ID from the remote, or at least discard one of the
> >> >>>> duplicate messages and refrain from re-copying the duplicate
> >> >>>> to and from the remote?
> >> >>>
> >> >>> Interesting. So the remote server is ignoring the supplied MID
> >> >>> and creating it's own.
> >> >>>
> >> >>> Does the remote server send a reply to state this? If it does,
> >> >>> rslight could then modify it's MID. If not, we'll need to code
> >> >>> it to NOT generate a MID for a certain <section>. If it DOES
> >> >>> send a reply, that would be better. Can you see if it does
> >> >>> that?
> >> <snip>
> >> >>
> >> >> Two of the test messages generated duplicates. Two did not:
> >>
> >> My guess is that two were not inserted immediately and two were.
> >> The ones not inserted immediately were only inserted after
> >> receiving back from remote server (so no duplicate).
> >>
> >> Rslight will not insert immediately if spoolnews for the specific
> >> section is running at that moment. In that situation, it will only
> >> prepare it for send to remote server and wait for it's return.
> >> (That will explain why some messages do not appear immediately
> >> after posting)
>
> > I ensured the cron job was not running and I posted two messages.
>
> > Then I ensured the cron job was running and I posted one message.
>
> > I can confirm that a duplicate was created while the cron was
> > running.
>
> Messages won't immediately insert if the specific spoolnews.php for
> the specific <section> is running. Cron can be running, just not the
> specific spoolnews.php.
>
> > Here are the duplicate message-id headers:
>
> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
> > <uobr41$k2l$1@GRC>
>
> > Here is the spoolnews entry:
>
> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
> > <uobr41$k2l$1@GRC> Removed: 1
>
> > All three messages were posted using claws-mail. All three messages
> > were posted to my rslight NNTP server, which then synced to the GRC
> > test group.
>
> Ok, thank you for testing. It does look like anytime a message is
> immediately inserted, it gets duplicated. I don't see a way around
> that other than a config option to disable immediate insertion for
> the section.

I think you are right in your assessment. Since GRC is opinionated
about using only its own serialized Message-ID header I don't know of
any way to tell the remote server to override that behavior. If you
want to try a config hack I will test it and report back.

I thought about a body+date+from+newsgroups accumulator hash to compare,
but that might take quite a bit more code changing. And since it would
have to be in the database tables, multiple hashes would need to be
computed and compared for every message, slowing things down
consistently.

One other idea I have is something like a alternative to xref:

X-RSLIGHT-SERIAL: newsgroup.name 1234567890 3883@BJM

but that will not work since GRC stripped the custom header in a test.
I presume some other servers would also strip the custom header.

So that leads me to a question: do Usenet hosts and/or servers like INN
have a practice of stripping custom headers from messages posted in
reader mode?

--
CRYP7010G3R | binary bunker | byte base | digit depot | glyph garrison
<b84e067e054611b9e4003dcb8944c7b7$1@sybershock.com>
3883@sugar.bug | sybershock.com | alt.sources.crypto

On Thu, 18 Jan 2024 18:11:57 -0600, SugarBug wrote:

> On Thu, 18 Jan 2024 19:39:59 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>> SugarBug wrote:
>>
>>> On Thu, 18 Jan 2024 06:50:04 -0700
>>> Retro Guy <retroguy@novabbs.org> wrote:
>>
>>>> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>>>>
>>>> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>>>> >
>>>> >> On Wed, 17 Jan 2024 18:21:51 +0000
>>>> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>>> >>
>>>> >>> SugarBug wrote:
>>>> >>>
>>>> >>>> I noticed that with a peer (GRC) that ignores custom
>>>> >>>> Message-ID that a duplicate message is created in the
>>>> >>>> newsgroup.
>>>> >>>
>>>> >>>> Steps:
>>>> >>>
>>>> >>>> 1. Post a message via the local server. The message posts with
>>>> >>>> the Message-ID generated by rslight.
>>>> >>>
>>>> >>>> 2. The remote INN server accepts the message on sync but
>>>> >>>> strips the Message-ID and replaces it with its own
>>>> >>>> Message-ID.
>>>> >>>
>>>> >>>> This should be reproducible by posting to grc.test.
>>>> >>>
>>>> >>>> If the remote server strictly generates its own message-id in
>>>> >>>> place of the provided message-id, rslight is not aware of
>>>> >>>> this. So rslight then stores its original message in the
>>>> >>>> spool with its own Message-ID, then pulls the message from
>>>> >>>> remote containing the remote-generated Message-ID. This
>>>> >>>> results in the same message appearing twice with two
>>>> >>>> Message-ID values.
>>>> >>>
>>>> >>>> Is there any configuration option to get rslight to accept the
>>>> >>>> Message-ID from the remote, or at least discard one of the
>>>> >>>> duplicate messages and refrain from re-copying the duplicate
>>>> >>>> to and from the remote?
>>>> >>>
>>>> >>> Interesting. So the remote server is ignoring the supplied MID
>>>> >>> and creating it's own.
>>>> >>>
>>>> >>> Does the remote server send a reply to state this? If it does,
>>>> >>> rslight could then modify it's MID. If not, we'll need to code
>>>> >>> it to NOT generate a MID for a certain <section>. If it DOES
>>>> >>> send a reply, that would be better. Can you see if it does
>>>> >>> that?
>>>> <snip>
>>>> >>
>>>> >> Two of the test messages generated duplicates. Two did not:
>>>>
>>>> My guess is that two were not inserted immediately and two were.
>>>> The ones not inserted immediately were only inserted after
>>>> receiving back from remote server (so no duplicate).
>>>>
>>>> Rslight will not insert immediately if spoolnews for the specific
>>>> section is running at that moment. In that situation, it will only
>>>> prepare it for send to remote server and wait for it's return.
>>>> (That will explain why some messages do not appear immediately
>>>> after posting)
>>
>>> I ensured the cron job was not running and I posted two messages.
>>
>>> Then I ensured the cron job was running and I posted one message.
>>
>>> I can confirm that a duplicate was created while the cron was
>>> running.
>>
>> Messages won't immediately insert if the specific spoolnews.php for
>> the specific <section> is running. Cron can be running, just not the
>> specific spoolnews.php.
>>
>>> Here are the duplicate message-id headers:
>>
>>> <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
>>> <uobr41$k2l$1@GRC>
>>
>>> Here is the spoolnews entry:
>>
>>> spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
>>> <uobr41$k2l$1@GRC> Removed: 1
>>
>>> All three messages were posted using claws-mail. All three messages
>>> were posted to my rslight NNTP server, which then synced to the GRC
>>> test group.
>>
>> Ok, thank you for testing. It does look like anytime a message is
>> immediately inserted, it gets duplicated. I don't see a way around
>> that other than a config option to disable immediate insertion for
>> the section.
>
> I think you are right in your assessment. Since GRC is opinionated
> about using only its own serialized Message-ID header I don't know of
> any way to tell the remote server to override that behavior. If you
> want to try a config hack I will test it and report back.
>
> I thought about a body+date+from+newsgroups accumulator hash to compare,
> but that might take quite a bit more code changing. And since it would
> have to be in the database tables, multiple hashes would need to be
> computed and compared for every message, slowing things down
> consistently.
>
> One other idea I have is something like a alternative to xref:
>
> X-RSLIGHT-SERIAL: newsgroup.name 1234567890 3883@BJM
>
> but that will not work since GRC stripped the custom header in a test.
> I presume some other servers would also strip the custom header.
>
> So that leads me to a question: do Usenet hosts and/or servers like INN
> have a practice of stripping custom headers from messages posted in
> reader mode?

They shouldn't strip X- headers. INN does not. You can see that X-Rslight-*
headers and X-Spam* headers make it through. But they should strip things
like Injection-Info, Path, stuff like that (unlike giganews).

SugarBug wrote:

> On Thu, 18 Jan 2024 19:39:59 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>> SugarBug wrote:
>>
>> > On Thu, 18 Jan 2024 06:50:04 -0700
>> > Retro Guy <retroguy@novabbs.org> wrote:
>>
>> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>> >>
>> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>> >> >
>> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
>> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>> >> >>
>> >> >>> SugarBug wrote:
>> >> >>>
>> >> >>>> I noticed that with a peer (GRC) that ignores custom
>> >> >>>> Message-ID that a duplicate message is created in the
>> >> >>>> newsgroup.
>> >> >>>
>> >> >>>> Steps:
>> >> >>>
>> >> >>>> 1. Post a message via the local server. The message posts with
>> >> >>>> the Message-ID generated by rslight.
>> >> >>>
>> >> >>>> 2. The remote INN server accepts the message on sync but
>> >> >>>> strips the Message-ID and replaces it with its own
>> >> >>>> Message-ID.
>> >> >>>
>> >> >>>> This should be reproducible by posting to grc.test.
>> >> >>>
>> >> >>>> If the remote server strictly generates its own message-id in
>> >> >>>> place of the provided message-id, rslight is not aware of
>> >> >>>> this. So rslight then stores its original message in the
>> >> >>>> spool with its own Message-ID, then pulls the message from
>> >> >>>> remote containing the remote-generated Message-ID. This
>> >> >>>> results in the same message appearing twice with two
>> >> >>>> Message-ID values.
>> >> >>>
>> >> >>>> Is there any configuration option to get rslight to accept the
>> >> >>>> Message-ID from the remote, or at least discard one of the
>> >> >>>> duplicate messages and refrain from re-copying the duplicate
>> >> >>>> to and from the remote?
>> >> >>>
>> >> >>> Interesting. So the remote server is ignoring the supplied MID
>> >> >>> and creating it's own.
>> >> >>>
>> >> >>> Does the remote server send a reply to state this? If it does,
>> >> >>> rslight could then modify it's MID. If not, we'll need to code
>> >> >>> it to NOT generate a MID for a certain <section>. If it DOES
>> >> >>> send a reply, that would be better. Can you see if it does
>> >> >>> that?
>> >> <snip>
>> >> >>
>> >> >> Two of the test messages generated duplicates. Two did not:
>> >>
>> >> My guess is that two were not inserted immediately and two were.
>> >> The ones not inserted immediately were only inserted after
>> >> receiving back from remote server (so no duplicate).
>> >>
>> >> Rslight will not insert immediately if spoolnews for the specific
>> >> section is running at that moment. In that situation, it will only
>> >> prepare it for send to remote server and wait for it's return.
>> >> (That will explain why some messages do not appear immediately
>> >> after posting)
>>
>> > I ensured the cron job was not running and I posted two messages.
>>
>> > Then I ensured the cron job was running and I posted one message.
>>
>> > I can confirm that a duplicate was created while the cron was
>> > running.
>>
>> Messages won't immediately insert if the specific spoolnews.php for
>> the specific <section> is running. Cron can be running, just not the
>> specific spoolnews.php.
>>
>> > Here are the duplicate message-id headers:
>>
>> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
>> > <uobr41$k2l$1@GRC>
>>
>> > Here is the spoolnews entry:
>>
>> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
>> > <uobr41$k2l$1@GRC> Removed: 1
>>
>> > All three messages were posted using claws-mail. All three messages
>> > were posted to my rslight NNTP server, which then synced to the GRC
>> > test group.
>>
>> Ok, thank you for testing. It does look like anytime a message is
>> immediately inserted, it gets duplicated. I don't see a way around
>> that other than a config option to disable immediate insertion for
>> the section.

> I think you are right in your assessment. Since GRC is opinionated
> about using only its own serialized Message-ID header I don't know of
> any way to tell the remote server to override that behavior. If you
> want to try a config hack I will test it and report back.

I just made a mod to rslight-lib.php and it seems to work fine. It is sending a MID, but doesn't need it for anything, so the message returning with a different MID should be ok.

For a quick test, please add this to the beginning of 'function insert_article' in rslight-lib.php (line 1101):

file_put_contents($logfile, "\n" . format_log_date() . " " . $section . " Insert Disabled... Queuing local post: " . $nntp_group, FILE_APPEND);
$return_val = "240 Article received OK (queued)\r\n";
return ($return_val);

Then kill any running nntp servers and try posting.

It works for me, but let's make sure it's ok and if so, we'll add a config toggle for it.

Here's a nntp.log entry when there is a post:
Jan 19 11:41:51 rocksolid Insert Disabled... Queuing local post: rocksolid.test.test

--
Retro Guy

On Fri, 19 Jan 2024 11:57:06 +0000, Retro Guy wrote:

> SugarBug wrote:
>
>> On Thu, 18 Jan 2024 19:39:59 +0000
>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>>> SugarBug wrote:
>>>
>>> > On Thu, 18 Jan 2024 06:50:04 -0700
>>> > Retro Guy <retroguy@novabbs.org> wrote:
>>>
>>> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>>> >>
>>> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>>> >> >
>>> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
>>> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>> >> >>
>>> >> >>> SugarBug wrote:
>>> >> >>>
>>> >> >>>> I noticed that with a peer (GRC) that ignores custom
>>> >> >>>> Message-ID that a duplicate message is created in the
>>> >> >>>> newsgroup.
>>> >> >>>
>>> >> >>>> Steps:
>>> >> >>>
>>> >> >>>> 1. Post a message via the local server. The message posts with
>>> >> >>>> the Message-ID generated by rslight.
>>> >> >>>
>>> >> >>>> 2. The remote INN server accepts the message on sync but
>>> >> >>>> strips the Message-ID and replaces it with its own
>>> >> >>>> Message-ID.
>>> >> >>>
>>> >> >>>> This should be reproducible by posting to grc.test.
>>> >> >>>
>>> >> >>>> If the remote server strictly generates its own message-id in
>>> >> >>>> place of the provided message-id, rslight is not aware of
>>> >> >>>> this. So rslight then stores its original message in the
>>> >> >>>> spool with its own Message-ID, then pulls the message from
>>> >> >>>> remote containing the remote-generated Message-ID. This
>>> >> >>>> results in the same message appearing twice with two
>>> >> >>>> Message-ID values.
>>> >> >>>
>>> >> >>>> Is there any configuration option to get rslight to accept the
>>> >> >>>> Message-ID from the remote, or at least discard one of the
>>> >> >>>> duplicate messages and refrain from re-copying the duplicate
>>> >> >>>> to and from the remote?
>>> >> >>>
>>> >> >>> Interesting. So the remote server is ignoring the supplied MID
>>> >> >>> and creating it's own.
>>> >> >>>
>>> >> >>> Does the remote server send a reply to state this? If it does,
>>> >> >>> rslight could then modify it's MID. If not, we'll need to code
>>> >> >>> it to NOT generate a MID for a certain <section>. If it DOES
>>> >> >>> send a reply, that would be better. Can you see if it does
>>> >> >>> that?
>>> >> <snip>
>>> >> >>
>>> >> >> Two of the test messages generated duplicates. Two did not:
>>> >>
>>> >> My guess is that two were not inserted immediately and two were.
>>> >> The ones not inserted immediately were only inserted after
>>> >> receiving back from remote server (so no duplicate).
>>> >>
>>> >> Rslight will not insert immediately if spoolnews for the specific
>>> >> section is running at that moment. In that situation, it will only
>>> >> prepare it for send to remote server and wait for it's return.
>>> >> (That will explain why some messages do not appear immediately
>>> >> after posting)
>>>
>>> > I ensured the cron job was not running and I posted two messages.
>>>
>>> > Then I ensured the cron job was running and I posted one message.
>>>
>>> > I can confirm that a duplicate was created while the cron was
>>> > running.
>>>
>>> Messages won't immediately insert if the specific spoolnews.php for
>>> the specific <section> is running. Cron can be running, just not the
>>> specific spoolnews.php.
>>>
>>> > Here are the duplicate message-id headers:
>>>
>>> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
>>> > <uobr41$k2l$1@GRC>
>>>
>>> > Here is the spoolnews entry:
>>>
>>> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article received
>>> > <uobr41$k2l$1@GRC> Removed: 1
>>>
>>> > All three messages were posted using claws-mail. All three messages
>>> > were posted to my rslight NNTP server, which then synced to the GRC
>>> > test group.
>>>
>>> Ok, thank you for testing. It does look like anytime a message is
>>> immediately inserted, it gets duplicated. I don't see a way around
>>> that other than a config option to disable immediate insertion for
>>> the section.
>
>> I think you are right in your assessment. Since GRC is opinionated
>> about using only its own serialized Message-ID header I don't know of
>> any way to tell the remote server to override that behavior. If you
>> want to try a config hack I will test it and report back.
>
> I just made a mod to rslight-lib.php and it seems to work fine. It is sending a MID, but doesn't need it for anything, so the message returning with a different MID should be ok.
>
> For a quick test, please add this to the beginning of 'function insert_article' in rslight-lib.php (line 1101):
>
> file_put_contents($logfile, "\n" . format_log_date() . " " . $section . " Insert Disabled... Queuing local post: " . $nntp_group, FILE_APPEND);
> $return_val = "240 Article received OK (queued)\r\n";
> return ($return_val);
>
> Then kill any running nntp servers and try posting.

That should be, 'kill then RESTART any running nntp servers'.

On Fri, 19 Jan 2024 11:57:06 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> SugarBug wrote:
>
> > On Thu, 18 Jan 2024 19:39:59 +0000
> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
> >> SugarBug wrote:
> >>
> >> > On Thu, 18 Jan 2024 06:50:04 -0700
> >> > Retro Guy <retroguy@novabbs.org> wrote:
> >>
> >> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
> >> >>
> >> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
> >> >> >
> >> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
> >> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
> >> >> >>
> >> >> >>> SugarBug wrote:
> >> >> >>>
> >> >> >>>> I noticed that with a peer (GRC) that ignores custom
> >> >> >>>> Message-ID that a duplicate message is created in the
> >> >> >>>> newsgroup.
> >> >> >>>
> >> >> >>>> Steps:
> >> >> >>>
> >> >> >>>> 1. Post a message via the local server. The message posts
> >> >> >>>> with the Message-ID generated by rslight.
> >> >> >>>
> >> >> >>>> 2. The remote INN server accepts the message on sync but
> >> >> >>>> strips the Message-ID and replaces it with its own
> >> >> >>>> Message-ID.
> >> >> >>>
> >> >> >>>> This should be reproducible by posting to grc.test.
> >> >> >>>
> >> >> >>>> If the remote server strictly generates its own message-id
> >> >> >>>> in place of the provided message-id, rslight is not aware
> >> >> >>>> of this. So rslight then stores its original message in the
> >> >> >>>> spool with its own Message-ID, then pulls the message from
> >> >> >>>> remote containing the remote-generated Message-ID. This
> >> >> >>>> results in the same message appearing twice with two
> >> >> >>>> Message-ID values.
> >> >> >>>
> >> >> >>>> Is there any configuration option to get rslight to accept
> >> >> >>>> the Message-ID from the remote, or at least discard one of
> >> >> >>>> the duplicate messages and refrain from re-copying the
> >> >> >>>> duplicate to and from the remote?
> >> >> >>>
> >> >> >>> Interesting. So the remote server is ignoring the supplied
> >> >> >>> MID and creating it's own.
> >> >> >>>
> >> >> >>> Does the remote server send a reply to state this? If it
> >> >> >>> does, rslight could then modify it's MID. If not, we'll
> >> >> >>> need to code it to NOT generate a MID for a certain
> >> >> >>> <section>. If it DOES send a reply, that would be better.
> >> >> >>> Can you see if it does that?
> >> >> <snip>
> >> >> >>
> >> >> >> Two of the test messages generated duplicates. Two did not:
> >> >> >>
> >> >>
> >> >> My guess is that two were not inserted immediately and two were.
> >> >> The ones not inserted immediately were only inserted after
> >> >> receiving back from remote server (so no duplicate).
> >> >>
> >> >> Rslight will not insert immediately if spoolnews for the
> >> >> specific section is running at that moment. In that situation,
> >> >> it will only prepare it for send to remote server and wait for
> >> >> it's return. (That will explain why some messages do not appear
> >> >> immediately after posting)
> >>
> >> > I ensured the cron job was not running and I posted two
> >> > messages.
> >>
> >> > Then I ensured the cron job was running and I posted one
> >> > message.
> >>
> >> > I can confirm that a duplicate was created while the cron was
> >> > running.
> >>
> >> Messages won't immediately insert if the specific spoolnews.php for
> >> the specific <section> is running. Cron can be running, just not
> >> the specific spoolnews.php.
> >>
> >> > Here are the duplicate message-id headers:
> >>
> >> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
> >> > <uobr41$k2l$1@GRC>
> >>
> >> > Here is the spoolnews entry:
> >>
> >> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article
> >> > received <uobr41$k2l$1@GRC> Removed: 1
> >>
> >> > All three messages were posted using claws-mail. All three
> >> > messages were posted to my rslight NNTP server, which then
> >> > synced to the GRC test group.
> >>
> >> Ok, thank you for testing. It does look like anytime a message is
> >> immediately inserted, it gets duplicated. I don't see a way around
> >> that other than a config option to disable immediate insertion for
> >> the section.
>
> > I think you are right in your assessment. Since GRC is opinionated
> > about using only its own serialized Message-ID header I don't know
> > of any way to tell the remote server to override that behavior. If
> > you want to try a config hack I will test it and report back.
>
> I just made a mod to rslight-lib.php and it seems to work fine. It is
> sending a MID, but doesn't need it for anything, so the message
> returning with a different MID should be ok.
>
> For a quick test, please add this to the beginning of 'function
> insert_article' in rslight-lib.php (line 1101):
>
> file_put_contents($logfile, "\n" . format_log_date() . " " . $section
> . " Insert Disabled... Queuing local post: " . $nntp_group,
> FILE_APPEND); $return_val = "240 Article received OK (queued)\r\n";
> return ($return_val);
>
> Then kill any running nntp servers and try posting.
>
> It works for me, but let's make sure it's ok and if so, we'll add a
> config toggle for it.
>
> Here's a nntp.log entry when there is a post:
> Jan 19 11:41:51 rocksolid Insert Disabled... Queuing local post:
> rocksolid.test.test

The hack is working on this end. The messages are withheld from the
local spool and interface. The proper message is pulled from the GRC
test group on the next cycle.

nntp.log:Jan 19 15:04:50 grc Insert Disabled... Queuing local post:
grc.test

nntp.log:Jan 19 15:05:01 grc Insert Disabled... Queuing local
post: grc.test

nntp.log:Jan 19 15:10:33 allgroups Insert Disabled... Queuing local
post: _all_tests

nntp.log:Jan 19 15:10:42 allgroups Insert Disabled... Queuing local
post: _all_tests

--
1144@code.bug | sybershock.com | alt.sources.crypto
useful sites (not mine) | novabbs.org | tilde.club | grc.com

SugarBug wrote:

> On Fri, 19 Jan 2024 11:57:06 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>> SugarBug wrote:
>>
>> > On Thu, 18 Jan 2024 19:39:59 +0000
>> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>
>> >> SugarBug wrote:
>> >>
>> >> > On Thu, 18 Jan 2024 06:50:04 -0700
>> >> > Retro Guy <retroguy@novabbs.org> wrote:
>> >>
>> >> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>> >> >>
>> >> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>> >> >> >
>> >> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
>> >> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>> >> >> >>
>> >> >> >>> SugarBug wrote:
>> >> >> >>>
>> >> >> >>>> I noticed that with a peer (GRC) that ignores custom
>> >> >> >>>> Message-ID that a duplicate message is created in the
>> >> >> >>>> newsgroup.
>> >> >> >>>
>> >> >> >>>> Steps:
>> >> >> >>>
>> >> >> >>>> 1. Post a message via the local server. The message posts
>> >> >> >>>> with the Message-ID generated by rslight.
>> >> >> >>>
>> >> >> >>>> 2. The remote INN server accepts the message on sync but
>> >> >> >>>> strips the Message-ID and replaces it with its own
>> >> >> >>>> Message-ID.
>> >> >> >>>
>> >> >> >>>> This should be reproducible by posting to grc.test.
>> >> >> >>>
>> >> >> >>>> If the remote server strictly generates its own message-id
>> >> >> >>>> in place of the provided message-id, rslight is not aware
>> >> >> >>>> of this. So rslight then stores its original message in the
>> >> >> >>>> spool with its own Message-ID, then pulls the message from
>> >> >> >>>> remote containing the remote-generated Message-ID. This
>> >> >> >>>> results in the same message appearing twice with two
>> >> >> >>>> Message-ID values.
>> >> >> >>>
>> >> >> >>>> Is there any configuration option to get rslight to accept
>> >> >> >>>> the Message-ID from the remote, or at least discard one of
>> >> >> >>>> the duplicate messages and refrain from re-copying the
>> >> >> >>>> duplicate to and from the remote?
>> >> >> >>>
>> >> >> >>> Interesting. So the remote server is ignoring the supplied
>> >> >> >>> MID and creating it's own.
>> >> >> >>>
>> >> >> >>> Does the remote server send a reply to state this? If it
>> >> >> >>> does, rslight could then modify it's MID. If not, we'll
>> >> >> >>> need to code it to NOT generate a MID for a certain
>> >> >> >>> <section>. If it DOES send a reply, that would be better.
>> >> >> >>> Can you see if it does that?
>> >> >> <snip>
>> >> >> >>
>> >> >> >> Two of the test messages generated duplicates. Two did not:
>> >> >> >>
>> >> >>
>> >> >> My guess is that two were not inserted immediately and two were.
>> >> >> The ones not inserted immediately were only inserted after
>> >> >> receiving back from remote server (so no duplicate).
>> >> >>
>> >> >> Rslight will not insert immediately if spoolnews for the
>> >> >> specific section is running at that moment. In that situation,
>> >> >> it will only prepare it for send to remote server and wait for
>> >> >> it's return. (That will explain why some messages do not appear
>> >> >> immediately after posting)
>> >>
>> >> > I ensured the cron job was not running and I posted two
>> >> > messages.
>> >>
>> >> > Then I ensured the cron job was running and I posted one
>> >> > message.
>> >>
>> >> > I can confirm that a duplicate was created while the cron was
>> >> > running.
>> >>
>> >> Messages won't immediately insert if the specific spoolnews.php for
>> >> the specific <section> is running. Cron can be running, just not
>> >> the specific spoolnews.php.
>> >>
>> >> > Here are the duplicate message-id headers:
>> >>
>> >> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
>> >> > <uobr41$k2l$1@GRC>
>> >>
>> >> > Here is the spoolnews entry:
>> >>
>> >> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article
>> >> > received <uobr41$k2l$1@GRC> Removed: 1
>> >>
>> >> > All three messages were posted using claws-mail. All three
>> >> > messages were posted to my rslight NNTP server, which then
>> >> > synced to the GRC test group.
>> >>
>> >> Ok, thank you for testing. It does look like anytime a message is
>> >> immediately inserted, it gets duplicated. I don't see a way around
>> >> that other than a config option to disable immediate insertion for
>> >> the section.
>>
>> > I think you are right in your assessment. Since GRC is opinionated
>> > about using only its own serialized Message-ID header I don't know
>> > of any way to tell the remote server to override that behavior. If
>> > you want to try a config hack I will test it and report back.
>>
>> I just made a mod to rslight-lib.php and it seems to work fine. It is
>> sending a MID, but doesn't need it for anything, so the message
>> returning with a different MID should be ok.
>>
>> For a quick test, please add this to the beginning of 'function
>> insert_article' in rslight-lib.php (line 1101):
>>
>> file_put_contents($logfile, "\n" . format_log_date() . " " . $section
>> . " Insert Disabled... Queuing local post: " . $nntp_group,
>> FILE_APPEND); $return_val = "240 Article received OK (queued)\r\n";
>> return ($return_val);
>>
>> Then kill any running nntp servers and try posting.
>>
>> It works for me, but let's make sure it's ok and if so, we'll add a
>> config toggle for it.
>>
>> Here's a nntp.log entry when there is a post:
>> Jan 19 11:41:51 rocksolid Insert Disabled... Queuing local post:
>> rocksolid.test.test

> The hack is working on this end. The messages are withheld from the
> local spool and interface. The proper message is pulled from the GRC
> test group on the next cycle.

Great! I will write it up to use a config toggle. I haven't decided yet whether $CONFIG or $OVERRIDES. $CONFIG is more difficult as it requires modifying every <section>.inc.php file if you already have an install, but it still might be better. Not sure yet.

--
Retro Guy

Retro Guy wrote:

> SugarBug wrote:

>> On Fri, 19 Jan 2024 11:57:06 +0000
>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>>> SugarBug wrote:
>>>
>>> > On Thu, 18 Jan 2024 19:39:59 +0000
>>> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>>
>>> >> SugarBug wrote:
>>> >>
>>> >> > On Thu, 18 Jan 2024 06:50:04 -0700
>>> >> > Retro Guy <retroguy@novabbs.org> wrote:
>>> >>
>>> >> >> On Thu, 18 Jan 2024 06:38:51 -0700, Retro Guy wrote:
>>> >> >>
>>> >> >> > On Thu, 18 Jan 2024 07:13:00 -0600, SugarBug wrote:
>>> >> >> >
>>> >> >> >> On Wed, 17 Jan 2024 18:21:51 +0000
>>> >> >> >> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>> >> >> >>
>>> >> >> >>> SugarBug wrote:
>>> >> >> >>>
>>> >> >> >>>> I noticed that with a peer (GRC) that ignores custom
>>> >> >> >>>> Message-ID that a duplicate message is created in the
>>> >> >> >>>> newsgroup.
>>> >> >> >>>
>>> >> >> >>>> Steps:
>>> >> >> >>>
>>> >> >> >>>> 1. Post a message via the local server. The message posts
>>> >> >> >>>> with the Message-ID generated by rslight.
>>> >> >> >>>
>>> >> >> >>>> 2. The remote INN server accepts the message on sync but
>>> >> >> >>>> strips the Message-ID and replaces it with its own
>>> >> >> >>>> Message-ID.
>>> >> >> >>>
>>> >> >> >>>> This should be reproducible by posting to grc.test.
>>> >> >> >>>
>>> >> >> >>>> If the remote server strictly generates its own message-id
>>> >> >> >>>> in place of the provided message-id, rslight is not aware
>>> >> >> >>>> of this. So rslight then stores its original message in the
>>> >> >> >>>> spool with its own Message-ID, then pulls the message from
>>> >> >> >>>> remote containing the remote-generated Message-ID. This
>>> >> >> >>>> results in the same message appearing twice with two
>>> >> >> >>>> Message-ID values.
>>> >> >> >>>
>>> >> >> >>>> Is there any configuration option to get rslight to accept
>>> >> >> >>>> the Message-ID from the remote, or at least discard one of
>>> >> >> >>>> the duplicate messages and refrain from re-copying the
>>> >> >> >>>> duplicate to and from the remote?
>>> >> >> >>>
>>> >> >> >>> Interesting. So the remote server is ignoring the supplied
>>> >> >> >>> MID and creating it's own.
>>> >> >> >>>
>>> >> >> >>> Does the remote server send a reply to state this? If it
>>> >> >> >>> does, rslight could then modify it's MID. If not, we'll
>>> >> >> >>> need to code it to NOT generate a MID for a certain
>>> >> >> >>> <section>. If it DOES send a reply, that would be better.
>>> >> >> >>> Can you see if it does that?
>>> >> >> <snip>
>>> >> >> >>
>>> >> >> >> Two of the test messages generated duplicates. Two did not:
>>> >> >> >>
>>> >> >>
>>> >> >> My guess is that two were not inserted immediately and two were.
>>> >> >> The ones not inserted immediately were only inserted after
>>> >> >> receiving back from remote server (so no duplicate).
>>> >> >>
>>> >> >> Rslight will not insert immediately if spoolnews for the
>>> >> >> specific section is running at that moment. In that situation,
>>> >> >> it will only prepare it for send to remote server and wait for
>>> >> >> it's return. (That will explain why some messages do not appear
>>> >> >> immediately after posting)
>>> >>
>>> >> > I ensured the cron job was not running and I posted two
>>> >> > messages.
>>> >>
>>> >> > Then I ensured the cron job was running and I posted one
>>> >> > message.
>>> >>
>>> >> > I can confirm that a duplicate was created while the cron was
>>> >> > running.
>>> >>
>>> >> Messages won't immediately insert if the specific spoolnews.php for
>>> >> the specific <section> is running. Cron can be running, just not
>>> >> the specific spoolnews.php.
>>> >>
>>> >> > Here are the duplicate message-id headers:
>>> >>
>>> >> > <8b99344ade793ca9dc76e6d34b5e8295$1@sybershock.com>
>>> >> > <uobr41$k2l$1@GRC>
>>> >>
>>> >> > Here is the spoolnews entry:
>>> >>
>>> >> > spoolnews.log:Jan 18 18:36:49 grc Posted: .msg: 240 Article
>>> >> > received <uobr41$k2l$1@GRC> Removed: 1
>>> >>
>>> >> > All three messages were posted using claws-mail. All three
>>> >> > messages were posted to my rslight NNTP server, which then
>>> >> > synced to the GRC test group.
>>> >>
>>> >> Ok, thank you for testing. It does look like anytime a message is
>>> >> immediately inserted, it gets duplicated. I don't see a way around
>>> >> that other than a config option to disable immediate insertion for
>>> >> the section.
>>>
>>> > I think you are right in your assessment. Since GRC is opinionated
>>> > about using only its own serialized Message-ID header I don't know
>>> > of any way to tell the remote server to override that behavior. If
>>> > you want to try a config hack I will test it and report back.
>>>
>>> I just made a mod to rslight-lib.php and it seems to work fine. It is
>>> sending a MID, but doesn't need it for anything, so the message
>>> returning with a different MID should be ok.
>>>
>>> For a quick test, please add this to the beginning of 'function
>>> insert_article' in rslight-lib.php (line 1101):
>>>
>>> file_put_contents($logfile, "\n" . format_log_date() . " " . $section
>>> . " Insert Disabled... Queuing local post: " . $nntp_group,
>>> FILE_APPEND); $return_val = "240 Article received OK (queued)\r\n";
>>> return ($return_val);
>>>
>>> Then kill any running nntp servers and try posting.
>>>
>>> It works for me, but let's make sure it's ok and if so, we'll add a
>>> config toggle for it.
>>>
>>> Here's a nntp.log entry when there is a post:
>>> Jan 19 11:41:51 rocksolid Insert Disabled... Queuing local post:
>>> rocksolid.test.test

>> The hack is working on this end. The messages are withheld from the
>> local spool and interface. The proper message is pulled from the GRC
>> test group on the next cycle.

> Great! I will write it up to use a config toggle. I haven't decided yet whether $CONFIG or $OVERRIDES. $CONFIG is more difficult as it requires modifying every <section>.inc.php file if you already have an install, but it still might be better. Not sure yet.

Decided to go with $OVERRIDES. Example:

// Disable immediate post insertion (wait for remote server)
// for a SECTION. Comma separated list ex: 'rocksolid,section3,section5',
'insert_disable' => 'grc',

So you will be able to disable immediate insertion by <section>.

Retro Guy wrote:

<snip>

>> Great! I will write it up to use a config toggle. I haven't decided yet whether $CONFIG or $OVERRIDES. $CONFIG is more difficult as it requires modifying every <section>.inc.php file if you already have an install, but it still might be better. Not sure yet.

> Decided to go with $OVERRIDES. Example:

> // Disable immediate post insertion (wait for remote server)
> // for a SECTION. Comma separated list ex: 'rocksolid,section3,section5',
> 'insert_disable' => 'grc',

> So you will be able to disable immediate insertion by <section>.

> It's not done yet, just letting you know my plan.

The latest commit (in devel) should have this change.

Use the config shown above in $config_dir/overrides.php (along with the rest of the file)

--
Retro Guy