The attached tgz contains rslight-lib.php and post.inc.php

With these two files changed, and nntp.php restarted, it SHOULD reject posted spam, whether through the web interface or newsreader.

Works for me, but you know what that means, lol.

--
Retro Guy

Retro Guy wrote:

> The attached tgz contains rslight-lib.php and post.inc.php

> With these two files changed, and nntp.php restarted, it SHOULD reject posted spam, whether through the web interface or newsreader.

> Works for me, but you know what that means, lol.

Adding spoolnews.php now. Here's what I get for a fresh group with some spam in it:

Dec 21 17:05:11 rocksolid Spooling: comp.arch:1
Dec 21 17:05:12 rocksolid Skipping: news.i2pn2.org comp.arch:38735 Exceeds Spam Score
Dec 21 17:05:13 rocksolid Spooling: comp.arch:2
Dec 21 17:05:15 rocksolid Spooling: comp.arch:3
Dec 21 17:05:16 rocksolid Spooling: comp.arch:4
Dec 21 17:05:17 rocksolid Skipping: news.i2pn2.org rocksolid.spam:38739 Exceeds Spam Score
Dec 21 17:05:18 rocksolid Spooling: comp.arch:5
Dec 21 17:05:19 rocksolid Spooling: comp.arch:6
Dec 21 17:05:20 rocksolid Skipping: news.i2pn2.org rocksolid.spam:38742 Exceeds Spam Score
Dec 21 17:05:20 rocksolid Skipping: news.i2pn2.org rocksolid.spam:38743 Exceeds Spam Score
Dec 21 17:05:21 rocksolid Skipping: news.i2pn2.org rocksolid.spam:38744 Exceeds Spam Score
Dec 21 17:05:21 rocksolid Spooling: comp.arch:7
Dec 21 17:05:22 rocksolid Spooling: comp.arch:8
Dec 21 17:05:23 rocksolid Spooling: comp.arch:9
Dec 21 17:05:24 rocksolid Spooling: comp.arch:10
Dec 21 17:05:26 rocksolid Spooling: comp.arch:11
Dec 21 17:05:26 rocksolid Skipping: news.i2pn2.org rocksolid.spam:38750 Exceeds Spam Score
Dec 21 17:05:27 rocksolid Spooling: comp.arch:12
Dec 21 17:05:30 rocksolid Spooling: comp.arch:13

--
Retro Guy

On Thu, 21 Dec 2023 16:28:36 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> The attached tgz contains rslight-lib.php and post.inc.php
>
> With these two files changed, and nntp.php restarted, it SHOULD
> reject posted spam, whether through the web interface or newsreader.
>
> Works for me, but you know what that means, lol.

No kidding! "Works on my machine" probably appears nearly as much on the
internet as cat memes. I am testing this now.

I'm also putting out feelers to ask others to contribute themes and run
some more peers. Hopefully a few will bite. If your project could get
more attention it could attract more people to Usenet.

On Thu, 21 Dec 2023 16:28:36 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> The attached tgz contains rslight-lib.php and post.inc.php
>
> With these two files changed, and nntp.php restarted, it SHOULD
> reject posted spam, whether through the web interface or newsreader.

I want to be sure I am testing for the right thing. By this do you mean
spam that is posted through the local machine accounts, and not
messages pulled from a peer?
> Works for me, but you know what that means, lol.

On Thu, 21 Dec 2023 11:38:50 -0600, SugarBug wrote:

> On Thu, 21 Dec 2023 16:28:36 +0000 retro.guy@rocksolidbbs.com (Retro
> Guy) wrote:
>
>> The attached tgz contains rslight-lib.php and post.inc.php
>>
>> With these two files changed, and nntp.php restarted, it SHOULD reject
>> posted spam, whether through the web interface or newsreader.
>
> I want to be sure I am testing for the right thing. By this do you mean
> spam that is posted through the local machine accounts, and not messages
> pulled from a peer?
>
>> Works for me, but you know what that means, lol.

Correct. But I did after that send spoolnews.php that may work for
importing from peers.

Retro Guy wrote:

> On Thu, 21 Dec 2023 11:38:50 -0600, SugarBug wrote:

>> On Thu, 21 Dec 2023 16:28:36 +0000 retro.guy@rocksolidbbs.com (Retro
>> Guy) wrote:
>>
>>> The attached tgz contains rslight-lib.php and post.inc.php
>>>
>>> With these two files changed, and nntp.php restarted, it SHOULD reject
>>> posted spam, whether through the web interface or newsreader.
>>
>> I want to be sure I am testing for the right thing. By this do you mean
>> spam that is posted through the local machine accounts, and not messages
>> pulled from a peer?
>>
>>> Works for me, but you know what that means, lol.

> Correct. But I did after that send spoolnews.php that may work for
> importing from peers.

Note: I have made some fixes to spoolnews.php since I sent the above .tgz of same.

--
Retro Guy

On Thu, 21 Dec 2023 17:43:19 -0000 (UTC)
Retro Guy <retroguy@i2pn2.org> wrote:

> On Thu, 21 Dec 2023 11:38:50 -0600, SugarBug wrote:
>
> > On Thu, 21 Dec 2023 16:28:36 +0000 retro.guy@rocksolidbbs.com (Retro
> > Guy) wrote:
> >
> >> The attached tgz contains rslight-lib.php and post.inc.php
> >>
> >> With these two files changed, and nntp.php restarted, it SHOULD
> >> reject posted spam, whether through the web interface or
> >> newsreader.
> >
> > I want to be sure I am testing for the right thing. By this do you
> > mean spam that is posted through the local machine accounts, and
> > not messages pulled from a peer?
> >
> >> Works for me, but you know what that means, lol.
>
> Correct. But I did after that send spoolnews.php that may work for
> importing from peers.

I just found this. I ran tests already without the updated
spoolnews. I'll post the results, then update spoolnews.php and test
that later.

On Thu, 21 Dec 2023 16:28:36 +0000
retro.guy@rocksolidbbs.com (Retro Guy) wrote:

> The attached tgz contains rslight-lib.php and post.inc.php
>
> With these two files changed, and nntp.php restarted, it SHOULD
> reject posted spam, whether through the web interface or newsreader.
>
> Works for me, but you know what that means, lol.
>

I updated the files in place and I am testing with cron runs on a fresh,
empty spool.

Exact order of operations shown below.

[run 1] no errors

[run 2] error:

PHP Fatal error: Uncaught TypeError: fputs(): supplied resource is not
a valid stream resource in .../rocksolid/lib/thread.inc.php:237
Stack trace:
#0 ...//rocksolid/lib/thread.inc.php(237): fputs()
#1 ...//rocksolid/lib/thread.inc.php(304): thread_overview_read()
#2 .../scripts/spoolnews.php(146): thread_load_newsserver()
#3 {main}
thrown in ..//rocksolid/lib/thread.inc.php on line 237

[runs 3-5] no errors

[run 6] same error as run #2

[runs 7-10] no errors

Posting tests:

From: Blocked Address Test <black@hole.url>

first attempt (group 'spam'): message accepted.
next attempt (group 'spam'): connection dropped, spamd in log.

https://rocksolid.sybershock.com/rocksolid/thread.php?group=spam

next attempt (group 'spam.spam'): message accepted.
next attempt (group 'spam.spam'): connection dropped, spamd in log.

https://rocksolid.sybershock.com/rocksolid/thread.php?group=spam.spam

I tested a post via telnet:

[blacklisted address]

281 Authentication succeeded
mode reader
200 Rocksolid Light NNRP Server ready (no posting)
post
340 Send article to be posted
From: black@hole.url
Newsgroups: spam
Subject: Tesing spamd

Testing spamd
..
Connection closed by foreign host.

[not blacklisted address]

281 Authentication succeeded
mode reader
200 Rocksolid Light NNRP Server ready (no posting)
post
340 Send article to be posted
From: good@address.url
Newsgroups: spam
Subject: Testing spamd

Testing spamd
..
Connection closed by foreign host.

This message from good@address.url never appears in the web interface.

[cron run 11] same error as #2 and #6 above.

[cron runs 12-13] no errors.

Web portal shows 78 total messages.

spool/articles$ find . -type f | wc -l
110

Spool shows 110.

Articles are not updating in the web interface:

https://rocksolid.sybershock.com/rocksolid/?

Then I ran the cron sixteen times in a loop with 10-second pause between
each run. Then I checked the spool:

articles$ find . -type f | wc -l
110

The same number of articles as before.

I set these values for testing:

config$ cat overrides.inc.php
<?php
return [
// expire.php
'max_articles_per_group' => 500,

// spoolnews.php
'maxarticles_per_run' => 10,
'maxfirstrequest' => 1000,

I am going to leave it alone right here so you can look at the portal
pages without further changes on my end. The cron is disabled to prevent
(hopefully) updates. I will test the new spoolnews.php later.

Retro Guy wrote:

> Retro Guy wrote:

>> On Thu, 21 Dec 2023 11:38:50 -0600, SugarBug wrote:

>>> On Thu, 21 Dec 2023 16:28:36 +0000 retro.guy@rocksolidbbs.com (Retro
>>> Guy) wrote:
>>>
>>>> The attached tgz contains rslight-lib.php and post.inc.php
>>>>
>>>> With these two files changed, and nntp.php restarted, it SHOULD reject
>>>> posted spam, whether through the web interface or newsreader.
>>>
>>> I want to be sure I am testing for the right thing. By this do you mean
>>> spam that is posted through the local machine accounts, and not messages
>>> pulled from a peer?
>>>
>>>> Works for me, but you know what that means, lol.

>> Correct. But I did after that send spoolnews.php that may work for
>> importing from peers.

> Note: I have made some fixes to spoolnews.php since I sent the above .tgz of same.

Here is the updated spoolnews.php (attached)

--
Retro Guy

On Thu, 21 Dec 2023 12:30:15 -0600, SugarBug wrote:

> On Thu, 21 Dec 2023 16:28:36 +0000 retro.guy@rocksolidbbs.com (Retro
> Guy) wrote:
>
>> The attached tgz contains rslight-lib.php and post.inc.php
>>
>> With these two files changed, and nntp.php restarted, it SHOULD reject
>> posted spam, whether through the web interface or newsreader.
>>
>> Works for me, but you know what that means, lol.
>>
>>
> I updated the files in place and I am testing with cron runs on a fresh,
> empty spool.
>
> Exact order of operations shown below.
>
> [run 1] no errors
>
> [run 2] error:
>
> PHP Fatal error: Uncaught TypeError: fputs(): supplied resource is not
> a valid stream resource in .../rocksolid/lib/thread.inc.php:237 Stack
> trace:
> #0 ...//rocksolid/lib/thread.inc.php(237): fputs()
> #1 ...//rocksolid/lib/thread.inc.php(304): thread_overview_read()
> #2 .../scripts/spoolnews.php(146): thread_load_newsserver()
> #3 {main}
> thrown in ..//rocksolid/lib/thread.inc.php on line 237

This is that same error we had last week. It's not truly fatal in the long
run, just one run. I think I fixed it, but you may not have the latest
code with that fix.

Also, the filtering has nothing to do with cron.php, it's all in the
posting (web interface post, or newsreader post).

I see the post in 'spam' that shouldn't be there. Can you try just wiping
the entire spool dir and starting fresh?

BTW: My current testing is being done on a tradspool setup. Hmmm...

On Thu, 21 Dec 2023 18:40:42 -0000 (UTC)
Retro Guy <retroguy@i2pn2.org> wrote:

> I see the post in 'spam' that shouldn't be there. Can you try just
> wiping the entire spool dir and starting fresh?

Added updated spoolnews.php.

Wiped spool directory.

Looped 10 cron runs.

Syslog is full of spamd activity now:

# grep -c "spamd" /var/log/syslog
1514

Example:

spamd[1249]: spamd: processing message
<hhjaiip0u75iioo8h2u6ukp1qvtk7b6fap@4ax.com> for sybershock.com:1003

Some blacklist addresses are being identified by spamd:

/var/log# grep "spamd: result: Y" syslog | wc -l
115

Interface not updating messages after count of 22 again.

Tradspool files not updating after count of 110ish messages again.

spool/articles$ find . | wc -l
116

After all this, I sent a test message from news reader client address
good@addy.url. This address is not in spamd blacklist.

Message listed in portal but not found:
https://rocksolid.sybershock.com/rocksolid/article-flat.php?id=1&group=spam#1

This single message is listed in 3 newsgroups in portal, although I
posted only to one group:

https://rocksolid.sybershock.com/rocksolid/?

Same message shows subject header via NNTP, but is inaccessible:

This message can't be displayed.
This is probably due to a network error.

Checked via telnet:

200 Rocksolid Light NNTP Server ready (no posting)
mode reader
200 Rocksolid Light NNRP Server ready (no posting)
group spam
211 1 1 1 spam
article 1
430 no such article found
quit
205 closing connection - goodbye!
Connection closed by foreign host.

spool$ ls -a articles/spam/
.. ..

returns null, no files

Then fired cron twice to update everything.

Sent second test message via newsreader over NNTP using from:
newsreader@fakeaddress.com which is not blacklisted. The NNTP server
rejects and drops the connection.

spool/articles$ ls spam/
2

spool/articles$ cat spam/2

spool/articles$ wc -c spam/2
0 spam/2

It is a blank, 0-byte file.

It seems that something is going haywire once a spam match is
activated. Then nothing after that is working quite as usual.

Later on I am going to wipe out the installation and delete all
the files then pull fresh from gitlab and start over by reinstalling
rslight with the updated php files. I must be away for awhile.

For now I leave the cron off so you can see the current state of the
portal.

On Thu, 21 Dec 2023 13:33:55 -0600, SugarBug wrote:

> On Thu, 21 Dec 2023 18:40:42 -0000 (UTC)
> Retro Guy <retroguy@i2pn2.org> wrote:
>
>> I see the post in 'spam' that shouldn't be there. Can you try just
>> wiping the entire spool dir and starting fresh?
>
> Added updated spoolnews.php.
>
> Wiped spool directory.
>
> Looped 10 cron runs.
>
> Syslog is full of spamd activity now:
>
> # grep -c "spamd" /var/log/syslog 1514
>
> Example:
>
> spamd[1249]: spamd: processing message
> <hhjaiip0u75iioo8h2u6ukp1qvtk7b6fap@4ax.com> for sybershock.com:1003
>
> Some blacklist addresses are being identified by spamd:
>
> /var/log# grep "spamd: result: Y" syslog | wc -l 115
>
> Interface not updating messages after count of 22 again.
>
> Tradspool files not updating after count of 110ish messages again.
>
> spool/articles$ find . | wc -l 116
>
> After all this, I sent a test message from news reader client address
> good@addy.url. This address is not in spamd blacklist.
>
> Message listed in portal but not found:
> https://rocksolid.sybershock.com/rocksolid/article-flat.php?
id=1&group=spam#1
>
> This single message is listed in 3 newsgroups in portal, although I
> posted only to one group:
>
> https://rocksolid.sybershock.com/rocksolid/?
>
> Same message shows subject header via NNTP, but is inaccessible:
>
> This message can't be displayed.
> This is probably due to a network error.
>
> Checked via telnet:
>
> 200 Rocksolid Light NNTP Server ready (no posting)
> mode reader 200 Rocksolid Light NNRP Server ready (no posting)
> group spam 211 1 1 1 spam article 1 430 no such article found quit 205
> closing connection - goodbye!
> Connection closed by foreign host.
>
> spool$ ls -a articles/spam/
> . ..
>
> returns null, no files
>
> Then fired cron twice to update everything.
>
> Sent second test message via newsreader over NNTP using from:
> newsreader@fakeaddress.com which is not blacklisted. The NNTP server
> rejects and drops the connection.
>
> spool/articles$ ls spam/
> 2
>
> spool/articles$ cat spam/2
>
> spool/articles$ wc -c spam/2 0 spam/2
>
> It is a blank, 0-byte file.
>
> It seems that something is going haywire once a spam match is activated.
> Then nothing after that is working quite as usual.
>
> Later on I am going to wipe out the installation and delete all the
> files then pull fresh from gitlab and start over by reinstalling rslight
> with the updated php files. I must be away for awhile.
>
> For now I leave the cron off so you can see the current state of the
> portal.

Ok, I need a break also.

The three php files I sent are NOT on github. You will need to apply those
manually. Make sure you use the second spoolnews.php that I sent, not the
first one.

I will convert to database (non tradspool) for my next testing.