So after beating my head against the wall for a few days, I found out that my "reload the group list" implementation in reload_groups.inc.php does not actually work. It succeeds in rewriting the groups.txt files correctly, but it does not actually force the NNTP server to reload its group list internally.

The NNTP servers seems to need to be stopped, and restarted, in order to reload groups.txt. I've verified this by digging deep into newsportal.php and groups_read(), where the server compares its loaded list of groups versus rocksolid/groups.txt. If the nntp server sees a discrepancy between its loaded list of groups and groups.txt, it *ignores* the missing groups.

I want to change this behaviour, by requesting that the nntp service *reloads* its internal list of groups when a change is detected in rocksolid/groups.txt. With inn2, "ctlinnd NEWGROUP somegroupname" accomplishes the task of adding a new group. Does RSL have its own command equivalent for the nntp service?

If not, does RSL at least have an equivalent to "ctlinnd reload" which would force it to reload all its groups from rocksolid/groups.txt?

I'm very, very close to having user-created groups working. I just need the nntp server to reload groups.txt without ignoring the new groups that were added to it.

Thanks again,
vga.

--
Posted on Rocksolid Light

On Tue, 4 Jul 2023 17:10:33 +0000
vga@vga256.com (vga256) wrote:

> So after beating my head against the wall for a few days, I found out
> that my "reload the group list" implementation in
> reload_groups.inc.php does not actually work. It succeeds in
> rewriting the groups.txt files correctly, but it does not actually
> force the NNTP server to reload its group list internally.
>
> The NNTP servers seems to need to be stopped, and restarted, in order
> to reload groups.txt. I've verified this by digging deep into
> newsportal.php and groups_read(), where the server compares its
> loaded list of groups versus rocksolid/groups.txt. If the nntp server
> sees a discrepancy between its loaded list of groups and groups.txt,
> it *ignores* the missing groups.
>
> I want to change this behaviour, by requesting that the nntp service
> *reloads* its internal list of groups when a change is detected in
> rocksolid/groups.txt. With inn2, "ctlinnd NEWGROUP somegroupname"
> accomplishes the task of adding a new group. Does RSL have its own
> command equivalent for the nntp service?
>
> If not, does RSL at least have an equivalent to "ctlinnd reload"
> which would force it to reload all its groups from
> rocksolid/groups.txt?
>
> I'm very, very close to having user-created groups working. I just
> need the nntp server to reload groups.txt without ignoring the new
> groups that were added to it.
>
> Thanks again,
> vga.
>

Try this:

After you change the group lists in the config directories run the
rslight cron. This updates everything in the NNTP backend and the web
front end.

However, the groups may not show up in the web interface until
after 2 or 3 runs of the cron. I don't know where the code is checking
that, or I'd tell you so you could make it effective immediately.

You could create a copy of the cron functions minus the ones that
syncronize with peers, just leaving the functions that update internal
configuration, and set a php trigger to invoke that when a new group is
added.

Let me know if this helps or any problems you encounter with it.

--
SugarBug | https://sybershock.com

Syber Shock wrote:

> syncronize with peers, just leaving the functions that update internal
> configuration, and set a php trigger to invoke that when a new group is
> added.

I should have clarified in the original post - I've already done all of that, including replicating code from both the cron script and the newsportal.php that it calls. It still doesn't work.

There is one absolute guarantee: if you 'killall php' (which kills off the nntp servers) *before* you run the cron script, it 100% guarantees a complete reload of the groups, every single time.

That's not much of a solution, because reloading groups means booting the nntp server offline for a few seconds while it rebuilds. That's no big deal if you run cron.php once a day around midnight, but when there are 1000+ users knocking it offline once a minute, it's a big problem.

So at the moment, I'm trying to figure out how exactly I can tell nntp.php to reload its groups into memory.

--
Posted on Rocksolid Light

This took forever, but I finally pinned down what the issue was.

* The NNTP server loads its group list from $spooldir/spoolnews/groups.txt
* In my reload_groups() function, when groups are added to /section/groups.txt, new entries were not being copied over to $spooldir/spoolnews/groups.txt. It was copying them over to $spooldir."/".$config_name."/groups.txt";
* The solution was to use $spooldir/spoolnews/groups.txt as a path.

This is something to think about for later - but the current method of having multiple groups.txt all over the directory hierarchies is super confusing. They all have the same name, and it's not clear which groups.txt belongs to what.

Anyway, we now have a script for creating groups using a web front-end (create_group.php), which automatically reloads the NNTP server's group list using newsportal.php:reload_groups() once it has added the group. I will probably add an nntp command NEWGROUP to rslight-lib.php in the future so it is more consistent with RFC977, but that's a job for another day.

--
Posted on Rocksolid Light

On Tue, 4 Jul 2023 22:19:33 +0000
vga@vga256.com (vga256) wrote:

> Syber Shock wrote:
>
> > syncronize with peers, just leaving the functions that update
> > internal configuration, and set a php trigger to invoke that when a
> > new group is added.
>
> I should have clarified in the original post - I've already done all
> of that, including replicating code from both the cron script and the
> newsportal.php that it calls. It still doesn't work.
>
> There is one absolute guarantee: if you 'killall php' (which kills
> off the nntp servers) *before* you run the cron script, it 100%
> guarantees a complete reload of the groups, every single time.
>
> That's not much of a solution, because reloading groups means booting
> the nntp server offline for a few seconds while it rebuilds. That's
> no big deal if you run cron.php once a day around midnight, but when
> there are 1000+ users knocking it offline once a minute, it's a big
> problem.
>
> So at the moment, I'm trying to figure out how exactly I can tell
> nntp.php to reload its groups into memory.

I am not able to replicate the error so maybe I am not understanding it
correctly. What I gather is that when you add new groups to a section
then run the cron job, those groups still don't show up when a client
connects to the nntp server. Is this the situation?

Make sure you've done these steps:

1. created a section *.inc.php;
2. created a folder named for the section;
3. placed 'groups.txt' in the folder;
4. updated the group names in 'group.txt';
5. created a folder in the public html dir named after the section;
6. copied or symlinked the web forum php files to the new folder;
7. checked the permissions of ALL files to ensure php can read them.

When this is done, every time the cron job runs it should poll these
files and update your NNTP backend with the added or removed
newsgroups. I have tested it several times and it has always worked
right away to update the NNTP server. However it sometimes takes
multiple cron cycles before the web front end shows the new groups.

If you have done all this and got no joy, then post the section config
*.inc.php (minus passwords!) and groups.txt so I can check. Any error
output from the cron job and nntp.php jobs might also help.

--
SugarBug | https://sybershock.com

The point I was trying to make is that running cron.php every 5 minutes was never a good solution for group reloads. The fact that it "sometimes works" or works on the third or fourth run, is detrimental/annoying behaviour.

My update here provides a method for properly reloading groups, without using cron, and without booting the NNTP server offline while it reloads.

https://novabbs.org/rocksolid/article-flat.php?id=371&group=rocksolid.nodes.help#371

--
Posted on Rocksolid Light

On Wed, 5 Jul 2023 21:00:07 +0000
vga@vga256.com (vga256) wrote:

> The point I was trying to make is that running cron.php every 5
> minutes was never a good solution for group reloads. The fact that it
> "sometimes works" or works on the third or fourth run, is
> detrimental/annoying behaviour.

I see now that you wanted instant update and considered the necessity
of running the cron job a bug needing squashed.
> My update here provides a method for properly reloading groups,
> without using cron, and without booting the NNTP server offline while
> it reloads.
>
> https://novabbs.org/rocksolid/article-flat.php?id=371&group=rocksolid.nodes.help#371

Guud ... keep on squashin'!

--
SugarBug | https://sybershock.com

yup, exactly.

--
Posted on Rocksolid Light

vga256 wrote:

> The point I was trying to make is that running cron.php every 5 minutes was never a good solution for group reloads. The fact that it "sometimes works" or works on the third or fourth run, is detrimental/annoying behaviour.

> My update here provides a method for properly reloading groups, without using cron, and without booting the NNTP server offline while it reloads.

> https://novabbs.org/rocksolid/article-flat.php?id=371&group=rocksolid.nodes.help#371

Your work in getting local groups to work properly is very helpful!

The way groups generally would be added is to edit $config_dir/<section>/groups.txt and they would be added once cron.php calls spoolnews.php (in function create_spool_groups(...)). This should not require nntp.php to be restarted, but of course if you are not running cron.php, it will not happen at all.

I think you've already worked this out, but just to be sure;

$spooldir/<section>/<server:port>.txt is the list of groups for this section and last article retrieved for the remote server.

$spooldir/<section>/local_groups.txt is the same list, but with the last LOCAL article number. This is not for "local" groups, but for the local copy of a group.

The last I left rslight was that LOCAL groups (groups not peered) should be in their own section, or sections. Then, the corresponding $config_dir/<section>.inc.php would list $remote_server='', so no pulling articles. If you mix local groups in a section with also peered groups, rslight will try to poll all the groups in that section with $remote_server.

The above still does not handle if a peer tries to PUSH articles to a LOCAL groups. This I have not completed, but maybe you have by this time.

--
Retro Guy

Syber Shock wrote:

> On Tue, 4 Jul 2023 17:10:33 +0000
> vga@vga256.com (vga256) wrote:

>> So after beating my head against the wall for a few days, I found out
>> that my "reload the group list" implementation in
>> reload_groups.inc.php does not actually work. It succeeds in
>> rewriting the groups.txt files correctly, but it does not actually
>> force the NNTP server to reload its group list internally.
>>
>> The NNTP servers seems to need to be stopped, and restarted, in order
>> to reload groups.txt. I've verified this by digging deep into
>> newsportal.php and groups_read(), where the server compares its
>> loaded list of groups versus rocksolid/groups.txt. If the nntp server
>> sees a discrepancy between its loaded list of groups and groups.txt,
>> it *ignores* the missing groups.
>>
>> I want to change this behaviour, by requesting that the nntp service
>> *reloads* its internal list of groups when a change is detected in
>> rocksolid/groups.txt. With inn2, "ctlinnd NEWGROUP somegroupname"
>> accomplishes the task of adding a new group. Does RSL have its own
>> command equivalent for the nntp service?
>>
>> If not, does RSL at least have an equivalent to "ctlinnd reload"
>> which would force it to reload all its groups from
>> rocksolid/groups.txt?
>>
>> I'm very, very close to having user-created groups working. I just
>> need the nntp server to reload groups.txt without ignoring the new
>> groups that were added to it.
>>
>> Thanks again,
>> vga.
>>

> Try this:

> After you change the group lists in the config directories run the
> rslight cron. This updates everything in the NNTP backend and the web
> front end.

> However, the groups may not show up in the web interface until
> after 2 or 3 runs of the cron. I don't know where the code is checking
> that, or I'd tell you so you could make it effective immediately.

This was due to the caching of the groups list, which can now be disabled
by force_reload=true in groups_read.

--
Retro Guy