I think spamc is marking all articles as spam because they don't
contain email headers. Maybe I am missing some module for this? Web
searches return nothing useful about the subject.

Here are the results the tests are returning.

I pulled the latest from gitlab and I installed 'devel' branch.

I enabled spamc.

I set database to '1' in rslight.inc.php.

I set overrides.inc.php to limit messages per run:

// expire.php
'max_articles_per_group' => 444,

// spoolnews.php
'maxarticles_per_run' => 3,
'maxfirstrequest' => 333,

I put some rocksolid groups and two spammy groups into the groups.txt.

I added a few of the addresses to my spamd blacklist_from directives.

I restarted spamd.service.

I truncated syslog to 0 bytes.

I fired the cron job.

The cron was running for a very long time. There are thousands of spamd
entries in syslog. There are hundreds of spam matches in syslog. It does
not appear that anything is actually being synced. The cron is just
running on and on stacking up entries in syslog, far in excess os what
is enabled in overrides.inc.php.

grep "spamd: result: Y" syslog | wc -l
609

Upon further inspection it appears that all messages are being flagged
as spam. In spool/log/spam.log the file is huge and has all of my and
your most recent testing messages in it. The entries look like this,
with some data truncated for brevity:

------------------------------------------
with SpamAssassin (version 4.0.0);
Mon, 25 Dec 2023 21:34:45 -0500
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on nightbulb.net
X-Spam-Flag: YES
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.5 required=5.0
tests=MISSING_DATE,MISSING_FROM,
MISSING_HEADERS,MISSING_MID,MISSING_SUBJECT,NO_HEADERS_MESSAGE,
NO_RECEIVED,NO_RELAYS autolearn=no autolearn_force=no version=4.0.0
MIME-Version: 1.0 Content-Type: multipart/mixed;
boundary="----------=_658A3BC5.12600B3A"

This is a multi-part message in MIME format.

------------=_658A3BC5.12600B3A
Content-Type: text/plain; charset=UTF-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

Spam detection software [...] has identified this incoming email as
possible spam.

Content preview: [truncated]

Content analysis details: (5.5 points, 5.0 required)

pts rule name description
-0.0 NO_RECEIVED Informational: message has no Received headers
0.1 MISSING_MID Missing Message-Id: header
1.8 MISSING_SUBJECT Missing Subject: header
1.4 MISSING_DATE Missing Date: header
1.0 MISSING_FROM Missing From: header
-0.0 NO_RELAYS Informational: message was not relayed via SMTP
1.2 MISSING_HEADERS Missing To: header
0.0 NO_HEADERS_MESSAGE Message appears to be missing most RFC-822
headers

-----------------------------------

So it appears that all messages are being flagged as spam for the same
reasons as this example because they don't look like emails. So it
would appear that rslight is communicating just fine with spamc, and
getting back the "X-Spam-Flag: YES" header for every message pulled in
from remote.

So I shortened the number of messages for first read and got faster
completions of the cron:

I set overrides.inc.php to limit messages per run:

// expire.php
'max_articles_per_group' => 444,

// spoolnews.php
'maxarticles_per_run' => 3,
'maxfirstrequest' => 33, <== nix a 3

Also, when I post via the local server, the fake blacklisted address is
not marked as spam by rslight. When I post from it the articles show up
immediately in the group. It is blacklisted in spamc config and in
banned_names.conf. But if I post to rocksolid.test, then wipe the
spool, then sync, it is flagged as spam when pulled from remote.

Also, on prior and subsequent runs banned_names.conf never seemed to
work, either with or without spamc enabled.

I noticed that when I post locally through the NNTP server that no
entry is made in spam.log even thought the from address is blacklisted
in spamc config.

I also did some more test runs on a re-wiped spool. A few times I
changed spamc path to a script that just cats the data to a file. no
data was sent from rslight to that script at all for about a dozen runs
of cron.

I did the same kind of tests with tradspool instead of database = '1'.
The result are the same. All incoming articles are flagged as spam. All
articles whatever are flagged as spam. All articles are successfully
prevented from being written to the spool articles directory.

All points to rslight getting the X-Spam headers back from spamc and
acting upon them since they are YES for all articles. Do I need some
additional module or setting for spamassasin to work properly with NNTP
articles? Are there config options to skip the checks for the expected
email and SMTP data headers? Do I need to somehow inject fake email
headers that pass muster? What am I missing?

SugarBug wrote:

> I think spamc is marking all articles as spam because they don't
> contain email headers. Maybe I am missing some module for this? Web
> searches return nothing useful about the subject.

Thanks for these test results. Very helpful!

I see why no headers are avail, I can fix that.

The headers that will be added are:
Subject, From, Newsgroups, Message-ID and maybe References

I had removed the above in testing and never put it back.

Spamassassin WILL need some checks removed due to differences between email and news, like NO_RECEIVED, etc. There are quite a few.

> Upon further inspection it appears that all messages are being flagged
> as spam. In spool/log/spam.log the file is huge and has all of my and
> your most recent testing messages in it. The entries look like this,
> with some data truncated for brevity:

It's actually good news. Formatting the message properly (putting the headers back in), and disabling some Spamassassin rules should do the rest.

> Also, when I post via the local server, the fake blacklisted address is
> not marked as spam by rslight. When I post from it the articles show up
> immediately in the group. It is blacklisted in spamc config and in
> banned_names.conf. But if I post to rocksolid.test, then wipe the
> spool, then sync, it is flagged as spam when pulled from remote.

> Also, on prior and subsequent runs banned_names.conf never seemed to
> work, either with or without spamc enabled.

I haven't changed anything with banned_name.conf. but this file is ONLY used when importing articles, not when posting them (as you have noticed). That's the way it's written, it's not broken if that's what it's doing.

I have moved away from using this file and do everything in Spamassassin.

> I did the same kind of tests with tradspool instead of database = '1'.
> The result are the same. All incoming articles are flagged as spam. All
> articles whatever are flagged as spam. All articles are successfully
> prevented from being written to the spool articles directory.

I've been putting a lot of effort into making sure tradspool continues to work when I make changes. Glad it's still working the same either way.

> All points to rslight getting the X-Spam headers back from spamc and
> acting upon them since they are YES for all articles. Do I need some
> additional module or setting for spamassasin to work properly with NNTP
> articles? Are there config options to skip the checks for the expected
> email and SMTP data headers? Do I need to somehow inject fake email
> headers that pass muster? What am I missing?

Yes, the error is mine. I will fix that today and send a commit. I'll let you know when it's done.

Thanks again!

--
Retro Guy

Retro Guy wrote:

> SugarBug wrote:

>> I think spamc is marking all articles as spam because they don't
>> contain email headers. Maybe I am missing some module for this? Web
>> searches return nothing useful about the subject.

<snip>

> Yes, the error is mine. I will fix that today and send a commit. I'll let you know when it's done.

I have made a lot of changes to the code. Including improved logging for troubleshooting.

Spam should be checked no matter how it's posted, newwreader, web interface or spoolnews.

Spam log file only contains SPAM or HAM listing of message-id. Spam itself is saved in $spooldir/spam

Some changes to spoolnews to fix a bug spooling some mailing lists to fix References.

I have just pushed commits to gitlab. Please don't forget to restart nntp server.

After that, let me know how badly I broke everything.

--
Retro Guy

On Tue, 26 Dec 2023 10:46:54 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> I haven't changed anything with banned_name.conf. but this file is
> ONLY used when importing articles, not when posting them (as you have
> noticed). That's the way it's written, it's not broken if that's what
> it's doing.

It's not working for either case. Nothing placed in banned_names.conf
is blocked either locally or pulled in from remote.

I have been placing the email address on one line by itself and name on
another line by itself, and another line combined like this:

Black Hole <black@hole.url>
black@hole.url
Black Hole

What is the exact format for data to be placed in this file? Is it case
sensitive? (I've been ensuring the case of entries matches the target
messages).

On Tue, 26 Dec 2023 10:38:38 -0600, SugarBug wrote:

> On Tue, 26 Dec 2023 10:46:54 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>> I haven't changed anything with banned_name.conf. but this file is
>> ONLY used when importing articles, not when posting them (as you have
>> noticed). That's the way it's written, it's not broken if that's what
>> it's doing.
>
> It's not working for either case. Nothing placed in banned_names.conf
> is blocked either locally or pulled in from remote.
>
> I have been placing the email address on one line by itself and name on
> another line by itself, and another line combined like this:
>
> Black Hole <black@hole.url>
> black@hole.url
> Black Hole
>
> What is the exact format for data to be placed in this file? Is it case
> sensitive? (I've been ensuring the case of entries matches the target
> messages).

Try $config_dir/header_filters.conf

This should work with the following headers:
'Message-ID'
'Subject'
'From'
'Path'

The format should be the header AND header content, and is a regex:

From: /black@hole\.url/
From: /Black Hole \<black@hole\.url\>/
From: /Black Hole/

Subject: /Buy Illegal Crap Today/

On Tue, 26 Dec 2023 16:07:04 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> I have just pushed commits to gitlab. Please don't forget to restart
> nntp server.
>
> After that, let me know how badly I broke everything.

If you don't break something there is no future fun!

SugarBug wrote:

> On Tue, 26 Dec 2023 16:07:04 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>> I have just pushed commits to gitlab. Please don't forget to restart
>> nntp server.
>>
>> After that, let me know how badly I broke everything.

> If you don't break something there is no future fun!

True! I do enjoy the challenge of fixing stuff I think I can't fix :)

Anyway, I now have spoolnews.php getting the correct 'last pulled' article number from saved data, so we don't have all these 'Duplicate' messages in the logs. Took me quite a while to figure out what was going on, but it gives me a chance to improve how it works.

I'll have some commits later today.

--
Retro Guy

Retro Guy wrote:

> SugarBug wrote:

>> On Tue, 26 Dec 2023 16:07:04 +0000
>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:

>>> I have just pushed commits to gitlab. Please don't forget to restart
>>> nntp server.
>>>
>>> After that, let me know how badly I broke everything.

>> If you don't break something there is no future fun!

> True! I do enjoy the challenge of fixing stuff I think I can't fix :)

> Anyway, I now have spoolnews.php getting the correct 'last pulled' article number from saved data, so we don't have all these 'Duplicate' messages in the logs. Took me quite a while to figure out what was going on, but it gives me a chance to improve how it works.

> I'll have some commits later today.

Commits are in. My sites are all running the latest and seem ok. I've been monitoring closely.

--
Retro Guy

I pulled from 'devel' branch commit #
8ecdbdae6f87cad7d9458e198540c2930b0cb679.

I wiped spool and upgraded.

I ran a cron hammer with spamc enabled.

No messages sync and no spam log appears.

There is no spamc activity in syslog.

There is not rslight spam logfile in spool/log.

I switched off the spamc option in rslight.inc.php.

I wiped and ran cronhammer again several times.

Nothing happens. The cron exits quickly and no messages are pulled.

I tried pulling from news.novabbs.org and news.i2pn2.org.

nntp and nntp-ssl socket daemons are running and I can connect to them
via telnet.

I rebooted the server.

I wiped rslight install completely.

I reinstalled from scratch.

Same results.

So at this point, there is no message synchronization, neither with
spamc enabled, nor with spamc disabled.

End of test.

--
CRYP7010G3R | Life is a numbers game. | Heist a hidden jackpot.
<69cd23ef630708e7ad98f2bfc2a159c9$1@sybershock.com>
3883@sugar.bug | sybershock.com | alt.sources.crypto

On Wed, 27 Dec 2023 16:53:27 -0600, SugarBug wrote:

> I pulled from 'devel' branch commit #
> 8ecdbdae6f87cad7d9458e198540c2930b0cb679.
>
> I wiped spool and upgraded.
>
> I ran a cron hammer with spamc enabled.
>
> No messages sync and no spam log appears.
>
> There is no spamc activity in syslog.
>
> There is not rslight spam logfile in spool/log.
>
> I switched off the spamc option in rslight.inc.php.
>
> I wiped and ran cronhammer again several times.
>
> Nothing happens. The cron exits quickly and no messages are pulled.
>
> I tried pulling from news.novabbs.org and news.i2pn2.org.
>
> nntp and nntp-ssl socket daemons are running and I can connect to them
> via telnet.
>
> I rebooted the server.
>
> I wiped rslight install completely.
>
> I reinstalled from scratch.
>
> Same results.
>
> So at this point, there is no message synchronization, neither with
> spamc enabled, nor with spamc disabled.
>
> End of test.

Please grab the latest spoolnews.php. It had a typo, which I've fixed, but
it didn't cause any trouble on my servers.

I'm going to do a clean install in the morning and see what happens. Right
now the latest commits (not including the fix I just mentioned) are running
on all my sites and fine.

My plan is a fresh Debian Bookworm install, then a fresh rslight install.
All software will be from Debian packages. I'll let you know the results.

On Wed, 27 Dec 2023 17:53:21 -0700
Retro Guy <retroguy@novabbs.org> wrote:

> > So at this point, there is no message synchronization, neither with
> > spamc enabled, nor with spamc disabled.
> >
> > End of test.
>
> Please grab the latest spoolnews.php. It had a typo, which I've
> fixed, but it didn't cause any trouble on my servers.
>
> I'm going to do a clean install in the morning and see what happens.
> Right now the latest commits (not including the fix I just mentioned)
> are running on all my sites and fine.
>
> My plan is a fresh Debian Bookworm install, then a fresh rslight
> install. All software will be from Debian packages. I'll let you know
> the results.

I pulled the latest 'devel' branch and did a clean install on Bookworm.

The messages are syncronizing.

The blacklisted addresses are now being caught by spamc and rslight.

None of the spam is making it through.

The good messages are going into spool.

File spool/log/spam.log appears to match /var/log/syslog spamd hits.

At this point rslight and spamc seem to be working correctly.

Git commit 2bbc019daf473b308a187007a7579e1734b68302 is working.

I will keep testing in fits and starts and let you know if I am able to
break anything.

Break all the things!

--
▄▄▄ ▄ ▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄ ▄ ▄ ▄▄▄ https://sybershock.com
▀▄ █ █ █ █ █ █▄▀ █▄▀ █ █ █ alt.sources.crypto
▄▄█ █▄█ █▄█ █▀█ █ █ █▄▀ █▄█ █▄█ @firefly@neon.nightbulb.net

On Wed, 27 Dec 2023 21:29:40 -0600, SugarBug wrote:

> On Wed, 27 Dec 2023 17:53:21 -0700
> Retro Guy <retroguy@novabbs.org> wrote:
>
>>> So at this point, there is no message synchronization, neither with
>>> spamc enabled, nor with spamc disabled.
>>>
>>> End of test.
>>
>> Please grab the latest spoolnews.php. It had a typo, which I've
>> fixed, but it didn't cause any trouble on my servers.
>>
>> I'm going to do a clean install in the morning and see what happens.
>> Right now the latest commits (not including the fix I just mentioned)
>> are running on all my sites and fine.
>>
>> My plan is a fresh Debian Bookworm install, then a fresh rslight
>> install. All software will be from Debian packages. I'll let you know
>> the results.
>
> I pulled the latest 'devel' branch and did a clean install on Bookworm.
>
> The messages are syncronizing.
>
> The blacklisted addresses are now being caught by spamc and rslight.
>
> None of the spam is making it through.
>
> The good messages are going into spool.
>
> File spool/log/spam.log appears to match /var/log/syslog spamd hits.
>
> At this point rslight and spamc seem to be working correctly.
>
> Git commit 2bbc019daf473b308a187007a7579e1734b68302 is working.
>
> I will keep testing in fits and starts and let you know if I am able to
> break anything.
>
> Break all the things!

Wow, I woke up to good news today! Thanks for testing.

Retro Guy wrote:

> On Wed, 27 Dec 2023 21:29:40 -0600, SugarBug wrote:

>> On Wed, 27 Dec 2023 17:53:21 -0700
>> Retro Guy <retroguy@novabbs.org> wrote:
>>
>>>> So at this point, there is no message synchronization, neither with
>>>> spamc enabled, nor with spamc disabled.
>>>>
>>>> End of test.
>>>
>>> Please grab the latest spoolnews.php. It had a typo, which I've
>>> fixed, but it didn't cause any trouble on my servers.
>>>
>>> I'm going to do a clean install in the morning and see what happens.
>>> Right now the latest commits (not including the fix I just mentioned)
>>> are running on all my sites and fine.
>>>
>>> My plan is a fresh Debian Bookworm install, then a fresh rslight
>>> install. All software will be from Debian packages. I'll let you know
>>> the results.
>>
>> I pulled the latest 'devel' branch and did a clean install on Bookworm.
>>
>> The messages are syncronizing.
>>
>> The blacklisted addresses are now being caught by spamc and rslight.
>>
>> None of the spam is making it through.
>>
>> The good messages are going into spool.
>>
>> File spool/log/spam.log appears to match /var/log/syslog spamd hits.
>>
>> At this point rslight and spamc seem to be working correctly.
>>
>> Git commit 2bbc019daf473b308a187007a7579e1734b68302 is working.
>>
>> I will keep testing in fits and starts and let you know if I am able to
>> break anything.
>>
>> Break all the things!

> Wow, I woke up to good news today! Thanks for testing.

Quick question: Would it be useful to log the message-id in spam.log?

If so, for ham and spam? Just spam?

Just looking for an opinion.

--
Retro Guy

On Sat, 30 Dec 2023 13:12:19 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> Quick question: Would it be useful to log the message-id in spam.log?
>
> If so, for ham and spam? Just spam?

Something with a vector flag like this for spam, maybe:

<message-id> <spam> <flag:spamc>
<message-id> <spam> <flag:banned_names>
<message-id> <spam> <flag:header_filters>

And for good ham messages:

<message-id> <ham> <flag:ham-yum-yum>

Or good ham no flag if you really want to save storage in the log:

<message-id> <ham>
<message-id> <ham> <flag:null>

Logging both with category or rejection vector flag would have the
benefit of enabling comparison with syslog and spamd log. It would
simplify deducing if config needs adjusted in header_filters
vs. spamc, etc., and catching false negatives for fine tuning spamc
learner to get the shifty ones.

Having the flag on the same line as message-id with a separator space
or char would be spiffy. Since space should not exist in a message-id I
just opted for that. You could use tab separators for neatness in
viewers, or calculate and pad spaces.

--
4477@king.bee
That which is not good for the swarm,
neither is it good for the bee. ~ Marcus Aurelius

On Sat, 30 Dec 2023 15:26:36 -0600
SugarBug <3883@sugar.bug> wrote:

> On Sat, 30 Dec 2023 13:12:19 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
> > Quick question: Would it be useful to log the message-id in
> > spam.log?
> >
> > If so, for ham and spam? Just spam?
>
> Something with a vector flag like this for spam, maybe:
>
> <message-id> <spam> <flag:spamc>
> <message-id> <spam> <flag:banned_names>
> <message-id> <spam> <flag:header_filters>
>
> And for good ham messages:
>
> <message-id> <ham> <flag:ham-yum-yum>
>
> Or good ham no flag if you really want to save storage in the log:
>
> <message-id> <ham>
> <message-id> <ham> <flag:null>
>
> Logging both with category or rejection vector flag would have the
> benefit of enabling comparison with syslog and spamd log. It would
> simplify deducing if config needs adjusted in header_filters
> vs. spamc, etc., and catching false negatives for fine tuning spamc
> learner to get the shifty ones.
>
> Having the flag on the same line as message-id with a separator space
> or char would be spiffy. Since space should not exist in a message-id
> I just opted for that. You could use tab separators for neatness in
> viewers, or calculate and pad spaces.

It just occurred to me that you could have separate files such as
spam.log and ham.log so only spam.log would need vector flags ensuring
a uniform data table format.

--
CRYP7010G3R | ciphertag | cryptologer army emblem | hushtag | U+2A69
<0068ef508d0d20e76758404e909791e9$1@sybershock.com>
3883@sugar.bug | sybershock.com | alt.sources.crypto

On Sat, 30 Dec 2023 15:26:36 -0600, SugarBug wrote:

> On Sat, 30 Dec 2023 13:12:19 +0000
> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>
>> Quick question: Would it be useful to log the message-id in spam.log?
>>
>> If so, for ham and spam? Just spam?
>
> Something with a vector flag like this for spam, maybe:
>
> <message-id> <spam> <flag:spamc>
> <message-id> <spam> <flag:banned_names>
> <message-id> <spam> <flag:header_filters>
>
> And for good ham messages:
>
> <message-id> <ham> <flag:ham-yum-yum>
>
> Or good ham no flag if you really want to save storage in the log:
>
> <message-id> <ham>
> <message-id> <ham> <flag:null>
>
> Logging both with category or rejection vector flag would have the
> benefit of enabling comparison with syslog and spamd log. It would
> simplify deducing if config needs adjusted in header_filters
> vs. spamc, etc., and catching false negatives for fine tuning spamc
> learner to get the shifty ones.
>
> Having the flag on the same line as message-id with a separator space
> or char would be spiffy. Since space should not exist in a message-id I
> just opted for that. You could use tab separators for neatness in
> viewers, or calculate and pad spaces.

What I have now is:
Dec 31 12:10:47 spamc: HAM
<560aa5be14421dc8cd1927534d9804af@dizum.com> alt.test Echolot
Pinger <echolot@pinger.invalid>
Dec 31 12:10:49 spamc: HAM <wIAiN.38570$woU1.26283@fx13.ams1>
eternal-september.support,alt.test Nospam <-invalid@mail.com>
Dec 31 12:10:50 spamc: HAM <unBiN.27244$c8T6.2555@fx05.ams1>
eternal-september.support,alt.test Nospam <-invalid@mail.com>
Dec 31 12:10:52 spamc: HAM
<c4a7e5db7294d974a9e605be582ab3f4@dizum.com> alt.test Nomen
Nescio <nobody@dizum.com>
Dec 31 12:10:53 spamc: HAM <RFvb5PASXtilFA9w@salmiron.com> alt.test
"Nicholas D. Richards" <nicholas@salmiron.com>

Each field after 'spamc:' is TAB separated, including right after 'spamc'.

The banned filters would have the filter name in place of 'spamc' and spam
would say SPAM instead of HAM.

It this what you have in mine?

Also, I'm working on the actual header_filters code right now, so the above
change is not yet in a commit.

On Sun, 31 Dec 2023 05:46:48 -0700, Retro Guy wrote:

> On Sat, 30 Dec 2023 15:26:36 -0600, SugarBug wrote:
>
>> On Sat, 30 Dec 2023 13:12:19 +0000
>> retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>>
>>> Quick question: Would it be useful to log the message-id in spam.log?
>>>
>>> If so, for ham and spam? Just spam?
>>
>> Something with a vector flag like this for spam, maybe:
>>
>> <message-id> <spam> <flag:spamc>
>> <message-id> <spam> <flag:banned_names>
>> <message-id> <spam> <flag:header_filters>
>>
>> And for good ham messages:
>>
>> <message-id> <ham> <flag:ham-yum-yum>
>>
>> Or good ham no flag if you really want to save storage in the log:
>>
>> <message-id> <ham>
>> <message-id> <ham> <flag:null>
>>
>> Logging both with category or rejection vector flag would have the
>> benefit of enabling comparison with syslog and spamd log. It would
>> simplify deducing if config needs adjusted in header_filters
>> vs. spamc, etc., and catching false negatives for fine tuning spamc
>> learner to get the shifty ones.
>>
>> Having the flag on the same line as message-id with a separator space
>> or char would be spiffy. Since space should not exist in a message-id I
>> just opted for that. You could use tab separators for neatness in
>> viewers, or calculate and pad spaces.
>
> What I have now is:
> Dec 31 12:10:47 spamc: HAM
> <560aa5be14421dc8cd1927534d9804af@dizum.com> alt.test Echolot
> Pinger <echolot@pinger.invalid>
> Dec 31 12:10:49 spamc: HAM <wIAiN.38570$woU1.26283@fx13.ams1>
> eternal-september.support,alt.test Nospam <-invalid@mail.com>
> Dec 31 12:10:50 spamc: HAM <unBiN.27244$c8T6.2555@fx05.ams1>
> eternal-september.support,alt.test Nospam <-invalid@mail.com>
> Dec 31 12:10:52 spamc: HAM
> <c4a7e5db7294d974a9e605be582ab3f4@dizum.com> alt.test Nomen
> Nescio <nobody@dizum.com>
> Dec 31 12:10:53 spamc: HAM <RFvb5PASXtilFA9w@salmiron.com> alt.test
> "Nicholas D. Richards" <nicholas@salmiron.com>
>
> Each field after 'spamc:' is TAB separated, including right after 'spamc'.
>
> The banned filters would have the filter name in place of 'spamc' and spam
> would say SPAM instead of HAM.
>
> It this what you have in mine?
>
> Also, I'm working on the actual header_filters code right now, so the above
> change is not yet in a commit.

Here's a sample with some header_filter.conf matches:
Dec 31 13:00:23 spamc: HAM
<ec3f3b6ee814202f9ae756b52ba279fc@msgid.frell.theremailer.net> alt.test
Fritz Wuehler <fritz@spamexpire-202312.rodent.frell.theremailer.net>
Dec 31 13:00:24 from_filter : SPAM
<b6cfb7ce8223e40eb447abd7d2135620@dizum.com> alt.test Echolot
Pinger <echolot@pinger.invalid>
Dec 31 13:00:24 from_filter : SPAM
<cfda59cf1974369a13419b69d741c47e@msgid.frell.theremailer.net> alt.test
Echolot Pinger <echolot@pinger.invalid>
Dec 31 13:00:24 spamc: HAM
<d0cfc1c0e1d64b38af0bedaf35448510@msgid.frell.theremailer.net> alt.test
Fritz Wuehler <fritz@spamexpire-202312.rodent.frell.theremailer.net>

That's obviously from setting Echolot Pinger in filters to test.

So, contrary to what I may have said recently, header_filters.conf format
is ONLY ONE LINE per header. All regex in one.

From: /Echolot Pinger|sPaMmEr DeLuXe|homer\.simpson/

I'm testing a bit more and will let you know when I push to gitlab.

On Sun, 31 Dec 2023 05:46:48 -0700
Retro Guy <retroguy@novabbs.org> wrote:

> On Sat, 30 Dec 2023 15:26:36 -0600, SugarBug wrote:
>
> > On Sat, 30 Dec 2023 13:12:19 +0000
> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
> >
> >> Quick question: Would it be useful to log the message-id in
> >> spam.log?
> >>
> >> If so, for ham and spam? Just spam?
> >
> > Something with a vector flag like this for spam, maybe:
> >
> > <message-id> <spam> <flag:spamc>
> > <message-id> <spam> <flag:banned_names>
> > <message-id> <spam> <flag:header_filters>
> >
> > And for good ham messages:
> >
> > <message-id> <ham> <flag:ham-yum-yum>
> >
> > Or good ham no flag if you really want to save storage in the log:
> >
> > <message-id> <ham>
> > <message-id> <ham> <flag:null>
> >
> > Logging both with category or rejection vector flag would have the
> > benefit of enabling comparison with syslog and spamd log. It would
> > simplify deducing if config needs adjusted in header_filters
> > vs. spamc, etc., and catching false negatives for fine tuning spamc
> > learner to get the shifty ones.
> >
> > Having the flag on the same line as message-id with a separator
> > space or char would be spiffy. Since space should not exist in a
> > message-id I just opted for that. You could use tab separators for
> > neatness in viewers, or calculate and pad spaces.
>
> What I have now is:
> Dec 31 12:10:47 spamc: HAM
> <560aa5be14421dc8cd1927534d9804af@dizum.com> alt.test
> Echolot Pinger <echolot@pinger.invalid>
> Dec 31 12:10:49 spamc: HAM <wIAiN.38570$woU1.26283@fx13.ams1>
> eternal-september.support,alt.test Nospam <-invalid@mail.com>
> Dec 31 12:10:50 spamc: HAM <unBiN.27244$c8T6.2555@fx05.ams1>
> eternal-september.support,alt.test Nospam <-invalid@mail.com>
> Dec 31 12:10:52 spamc: HAM
> <c4a7e5db7294d974a9e605be582ab3f4@dizum.com> alt.test Nomen
> Nescio <nobody@dizum.com>
> Dec 31 12:10:53 spamc: HAM <RFvb5PASXtilFA9w@salmiron.com>
> alt.test "Nicholas D. Richards" <nicholas@salmiron.com>
>
> Each field after 'spamc:' is TAB separated, including right after
> 'spamc'.
>
> The banned filters would have the filter name in place of 'spamc' and
> spam would say SPAM instead of HAM.
>
> It this what you have in mine?

Yes, it is, and better. Now you're a systemd and spamassassin engineer.
Insert Wayne's World chant here. ;)
> Also, I'm working on the actual header_filters code right now, so the
> above change is not yet in a commit.

I just found this as I was readying to sign off and go saw logs (pun).

The message formatting jumbled the example you pasted. I reformatted it
in my editor to view. Does each line begin with the date? If that is the
case then here is what I am seeing. Correct me if I am wrong:

<date> <spam_engine> : <spam/ham> <newsgroups> <from>

It looks spiffy if my assumption of the date beginning each line
entry is correct. Also your version is more in keeping with log format
with the <date> first in line.

Also, I think it is possible for newsreaders to insert tabs and newlines
into subject, from and maybe some other headers. Perhaps that needs to
be accounted for with an escape or replacement or escape+square
bracketing when flushing to logs? I don't know how to do that with PHP
so I can't really advise what would be a good way to approach it.

I think you made the right choice placing the <from> at the end of the
line, since that is where badchars can exist. There is even a
newsreader that inserts ^M bytes into subjects. I think you
could just use spaces or colons or square brackets for field marking of
the first items, since the server should never accept them with those
chars (I think?). Then maybe just snarf the rest of the line as <from>.

I have serious all-nighter brain fog right now and I will check in
tomorrow. I don't want to try giving any more opinions as blurred up as
I am. I hope some others with good PHP experience chime in with ideas
before I come back.

--
1_|1_|4_|4_ 3883@sugar.bug
3_|8_|8_|3_ https://sybershock.com
7_|7_|6_|6_ news://alt.sources.crypto
4_|4_|7_|7_ #CipherTag #WaffleTag #Cryptologer

SugarBug wrote:

> On Sun, 31 Dec 2023 05:46:48 -0700
> Retro Guy <retroguy@novabbs.org> wrote:

>> On Sat, 30 Dec 2023 15:26:36 -0600, SugarBug wrote:
>>
>> > On Sat, 30 Dec 2023 13:12:19 +0000
>> > retro.guy@rocksolidbbs.com (Retro Guy) wrote:
>> >
>> >> Quick question: Would it be useful to log the message-id in
>> >> spam.log?
>> >>
>> >> If so, for ham and spam? Just spam?
>> >
>> > Something with a vector flag like this for spam, maybe:
>> >
>> > <message-id> <spam> <flag:spamc>
>> > <message-id> <spam> <flag:banned_names>
>> > <message-id> <spam> <flag:header_filters>
>> >
>> > And for good ham messages:
>> >
>> > <message-id> <ham> <flag:ham-yum-yum>
>> >
>> > Or good ham no flag if you really want to save storage in the log:
>> >
>> > <message-id> <ham>
>> > <message-id> <ham> <flag:null>
>> >
>> > Logging both with category or rejection vector flag would have the
>> > benefit of enabling comparison with syslog and spamd log. It would
>> > simplify deducing if config needs adjusted in header_filters
>> > vs. spamc, etc., and catching false negatives for fine tuning spamc
>> > learner to get the shifty ones.
>> >
>> > Having the flag on the same line as message-id with a separator
>> > space or char would be spiffy. Since space should not exist in a
>> > message-id I just opted for that. You could use tab separators for
>> > neatness in viewers, or calculate and pad spaces.
>>
>> What I have now is:
>> Dec 31 12:10:47 spamc: HAM
>> <560aa5be14421dc8cd1927534d9804af@dizum.com> alt.test
>> Echolot Pinger <echolot@pinger.invalid>
>> Dec 31 12:10:49 spamc: HAM <wIAiN.38570$woU1.26283@fx13.ams1>
>> eternal-september.support,alt.test Nospam <-invalid@mail.com>
>> Dec 31 12:10:50 spamc: HAM <unBiN.27244$c8T6.2555@fx05.ams1>
>> eternal-september.support,alt.test Nospam <-invalid@mail.com>
>> Dec 31 12:10:52 spamc: HAM
>> <c4a7e5db7294d974a9e605be582ab3f4@dizum.com> alt.test Nomen
>> Nescio <nobody@dizum.com>
>> Dec 31 12:10:53 spamc: HAM <RFvb5PASXtilFA9w@salmiron.com>
>> alt.test "Nicholas D. Richards" <nicholas@salmiron.com>
>>
>> Each field after 'spamc:' is TAB separated, including right after
>> 'spamc'.
>>
>> The banned filters would have the filter name in place of 'spamc' and
>> spam would say SPAM instead of HAM.
>>
>> It this what you have in mine?

> Yes, it is, and better. Now you're a systemd and spamassassin engineer.
> Insert Wayne's World chant here. ;)

>> Also, I'm working on the actual header_filters code right now, so the
>> above change is not yet in a commit.

> I just found this as I was readying to sign off and go saw logs (pun).

> The message formatting jumbled the example you pasted. I reformatted it
> in my editor to view. Does each line begin with the date? If that is the
> case then here is what I am seeing. Correct me if I am wrong:

> <date> <spam_engine> : <spam/ham> <newsgroups> <from>

<date> <spam_engine>:TAB<spam/ham>TAB<message-id>TAB<newsgroups>TAB<from>

> It looks spiffy if my assumption of the date beginning each line
> entry is correct. Also your version is more in keeping with log format
> with the <date> first in line.

I'm trying to work toward a standard, and make the files easy to read if you're already used to log files.

> Also, I think it is possible for newsreaders to insert tabs and newlines
> into subject, from and maybe some other headers. Perhaps that needs to
> be accounted for with an escape or replacement or escape+square
> bracketing when flushing to logs? I don't know how to do that with PHP
> so I can't really advise what would be a good way to approach it.

It's my plan to clean up the header strings (convert tabs to spaces...)
before logging.

--
Retro Guy