Hello!

How many groups can be handled by rslight?
I read in some discussion that a huge amount created problems.
Can somebody confirm/deny that?

I think about adding entire big-8...

--
Gruß
Marco

Spam und Werbung bitte an ichwillgesperrtwerden@nirvana.admins.ws

Marco Moock wrote:

> Hello!

> How many groups can be handled by rslight?
> I read in some discussion that a huge amount created problems.
> Can somebody confirm/deny that?

I can confirm that (Ray, admin of e-s said that, and he's correct).

Each section is not designed to carry hundreds of groups. It's best to just pick the ones you want and serve those.

Break groups into sections. See www.novabbs.com:
arts - interests - computers - sport - tech etc.

> I think about adding entire big-8...

I would pick groups, not just add them all. The groups don't just reach out to an inn server when you click on them, the site syncs with a remote server, and would then sync groups nobody wants. Syncing takes time, and producing the page that shows 'Last Post', etc. takes time to build, even though it is cached for a short time.

The more groups in a section, the slower the site. Trying to use Search or Overboard for 5,000 groups is not going to work well.

Something nice would be for one admin to server de.* Another to handle French groups, etc. A site devoted to programming, one devoted to The Arts. Just an idea.

--
Retro Guy

Thus spake Retro Guy <retroguy@novabbs.com>
>> I read in some discussion that a huge amount created problems.
>> Can somebody confirm/deny that?
> I can confirm that (Ray, admin of e-s said that, and he's correct).
> Each section is not designed to carry hundreds of groups. It's best to just pick the ones you want and serve those.
> Break groups into sections. See www.novabbs.com:
> arts - interests - computers - sport - tech etc.

I have the entire Big 8 available in my test installation, split into 8
sections, plus de.*, fr.*, nl.*, pl.* and uk.* each in a separate
section.

In the long run, I would consider having just one section for a user's
subscribed groups and a list of all newsgroups from which users can
select and subscribe to the groups. This would of course require a
completely different approach to loading overview data from the remote
server, which currently done through static groups.txt files for each
defined section.

>> I think about adding entire big-8...
> I would pick groups, not just add them all. The groups don't just
> reach out to an inn server when you click on them, the site syncs with
> a remote server, and would then sync groups nobody wants. Syncing
> takes time, and producing the page that shows 'Last Post', etc. takes
> time to build, even though it is cached for a short time.
> The more groups in a section, the slower the site. Trying to use Search or Overboard for 5,000 groups is not going to work well.

The real bottleneck seems to be building the thread structure for each
group, more specifically the memory consumption of these functions:

function thread_cache_load($group)
function thread_cache_save($headers, $group)

Allowed memory size of 5444206592 bytes exhausted (tried to allocate 773853184 bytes) in /var/www/html/webclient/rocksolid/lib/thread.inc.php on line 110
Allowed memory size of 3221225472 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/webclient/rocksolid/lib/thread.inc.php on line 77

As you can see, I have already increased the memory_limit from 1.5 to 5
GiB, but this is not enough for some groups with tens of thousands of
articles.

HTH

--
Пу́тін — хуйло́
https://www.eternal-september.org

Ray Banana wrote:

> Thus spake Retro Guy <retroguy@novabbs.com>

>>> I read in some discussion that a huge amount created problems.
>>> Can somebody confirm/deny that?
>> I can confirm that (Ray, admin of e-s said that, and he's correct).
>> Each section is not designed to carry hundreds of groups. It's best to just pick the ones you want and serve those.
>> Break groups into sections. See www.novabbs.com:
>> arts - interests - computers - sport - tech etc.

> I have the entire Big 8 available in my test installation, split into 8
> sections, plus de.*, fr.*, nl.*, pl.* and uk.* each in a separate
> section.

> In the long run, I would consider having just one section for a user's
> subscribed groups and a list of all newsgroups from which users can
> select and subscribe to the groups. This would of course require a
> completely different approach to loading overview data from the remote
> server, which currently done through static groups.txt files for each
> defined section.

This is something I've been thinking about for a while now, but it may be quite a task, and I'm not ready to tackle it at this time (I have real life issues to deal with atm).

>>> I think about adding entire big-8...
>> I would pick groups, not just add them all. The groups don't just
>> reach out to an inn server when you click on them, the site syncs with
>> a remote server, and would then sync groups nobody wants. Syncing
>> takes time, and producing the page that shows 'Last Post', etc. takes
>> time to build, even though it is cached for a short time.
>> The more groups in a section, the slower the site. Trying to use Search or Overboard for 5,000 groups is not going to work well.

> The real bottleneck seems to be building the thread structure for each
> group, more specifically the memory consumption of these functions:

> function thread_cache_load($group)
> function thread_cache_save($headers, $group)

> Allowed memory size of 5444206592 bytes exhausted (tried to allocate 773853184 bytes) in /var/www/html/webclient/rocksolid/lib/thread.inc.php on line 110
> Allowed memory size of 3221225472 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/webclient/rocksolid/lib/thread.inc.php on line 77

These functions are from the original Newsportal, but I changed them to use a database and to not try to rebuild nearly as often as Newsportal did, but they still are very time consuming. I have sections with several hundred thousand articles per section and it works ok, but huge groups will really slow it down. I expire over 100,000 articles per group on novabbs.com.

> As you can see, I have already increased the memory_limit from 1.5 to 5
> GiB, but this is not enough for some groups with tens of thousands of
> articles.

Yep.

Rslight was meant to act like a forum, and use nntp as a backend, which works fine. It's really only since Google Groups is going away that suddenly people want to use it as a web interface to every article in every group on Usenet, and to be a replacement for Deja News and archive all of history :)

That's not really what is was made for, and at the time I started on it (5 or 6 years ago), it never crossed my mind to use it that way.

--
Retro Guy

Thus spake Retro Guy <retroguy@novabbs.com>
{...]
>> In the long run, I would consider having just one section for a user's
>> subscribed groups and a list of all newsgroups from which users can
>> select and subscribe to the groups. This would of course require a
>> completely different approach to loading overview data from the remote
>> server, which currently done through static groups.txt files for each
>> defined section.
> This is something I've been thinking about for a while now, but it may be quite a task, and I'm not ready to tackle it at this time (I have real life issues to deal with atm).

It requires a lot of changes and testing, indeed, so maybe put it on the
backlog for a future iteration ;-)

>> As you can see, I have already increased the memory_limit from 1.5 to 5
>> GiB, but this is not enough for some groups with tens of thousands of
>> articles.
> Yep.

You seem to serialize/unserialize the headers before you store them as
one BLOB/row in the databases. That's the cause of the memory
consumption. (Again: This is just a suggestion for a future iteration
that would need to be analyzed and prioritized).

> Rslight was meant to act like a forum, and use nntp as a backend,
> which works fine. It's really only since Google Groups is going away
> that suddenly people want to use it as a web interface to every
> article in every group on Usenet, and to be a replacement for Deja
> News and archive all of history :)
> That's not really what is was made for, and at the time I started on
> it (5 or 6 years ago), it never crossed my mind to use it that way.

That's the most interesting aspect of the Google Groups dilemma.
I also get lots of inquiries from Google Group users asking for features
I, as a news admin, would never have dreamed of. And archives are indeed
completely different from the traditional news clients and need a
completely different data model and storage backend. I would never mix
web client and archive client access using the same backend.

--
Пу́тін — хуйло́
https://www.eternal-september.org

Ray Banana wrote:

> Thus spake Retro Guy <retroguy@novabbs.com>

> {...]
>>> In the long run, I would consider having just one section for a user's
>>> subscribed groups and a list of all newsgroups from which users can
>>> select and subscribe to the groups. This would of course require a
>>> completely different approach to loading overview data from the remote
>>> server, which currently done through static groups.txt files for each
>>> defined section.
>> This is something I've been thinking about for a while now, but it may be quite a task, and I'm not ready to tackle it at this time (I have real life issues to deal with atm).

> It requires a lot of changes and testing, indeed, so maybe put it on the
> backlog for a future iteration ;-)

>>> As you can see, I have already increased the memory_limit from 1.5 to 5
>>> GiB, but this is not enough for some groups with tens of thousands of
>>> articles.
>> Yep.

> You seem to serialize/unserialize the headers before you store them as
> one BLOB/row in the databases. That's the cause of the memory
> consumption. (Again: This is just a suggestion for a future iteration
> that would need to be analyzed and prioritized).

Yes, that's correct. The threading was written by Florian (Newsportal developer), and I have not attempted to rewrite it. All I've done is stopped saving it to disk back and forth and use a db. His version choked on groups when there were a lot of people posting as it would keep trying to re-thread while it was still saving, and would keep deleting the saved serialized array and starting over. This caused groups to constantly start building groups from the beginning and not displaying anything recent, which didn't make users very happy.

The thought of rewriting the threading is less appealing than cleaning up my yard from my dogs :)

--
Retro Guy

Retro Guy wrote:

> Marco Moock wrote:

>> Hello!

>> How many groups can be handled by rslight?
>> I read in some discussion that a huge amount created problems.
>> Can somebody confirm/deny that?

> I can confirm that (Ray, admin of e-s said that, and he's correct).

> Each section is not designed to carry hundreds of groups. It's best to just pick the ones you want and serve those.

> Break groups into sections. See www.novabbs.com:
> arts - interests - computers - sport - tech etc.

>> I think about adding entire big-8...

> I would pick groups, not just add them all. The groups don't just reach out to an inn server when you click on them, the site syncs with a remote server, and would then sync groups nobody wants. Syncing takes time, and producing the page that shows 'Last Post', etc. takes time to build, even though it is cached for a short time.

> The more groups in a section, the slower the site. Trying to use Search or Overboard for 5,000 groups is not going to work well.

> Something nice would be for one admin to server de.* Another to handle French groups, etc. A site devoted to programming, one devoted to The Arts. Just an idea.

Interesting.

So Jordy should name the specific alt.support newsgroups he requests for adding to Nova BBS, I'll pass that on to him.

Ray Banana wrote:

> Thus spake Retro Guy <retroguy@novabbs.com>

>>> I read in some discussion that a huge amount created problems.
>>> Can somebody confirm/deny that?
>> I can confirm that (Ray, admin of e-s said that, and he's correct).
>> Each section is not designed to carry hundreds of groups. It's best to just pick the ones you want and serve those.
>> Break groups into sections. See www.novabbs.com:
>> arts - interests - computers - sport - tech etc.

> I have the entire Big 8 available in my test installation, split into 8
> sections, plus de.*, fr.*, nl.*, pl.* and uk.* each in a separate
> section.

> In the long run, I would consider having just one section for a user's
> subscribed groups and a list of all newsgroups from which users can
> select and subscribe to the groups. This would of course require a
> completely different approach to loading overview data from the remote
> server, which currently done through static groups.txt files for each
> defined section.

I've made a change that is not really what Ray mentions (though I like his suggestion), but does clean up what a user sees.

Now you can toggle a setting in Configuration (for registered users) to Hide any groups not subscribed. To subscribe to a group, just enter the group from the Grouplist page, then it will show up in the corresponding section. Groups you don't read won't be visible.

Just click 'unsubscribe' to remove a group from your subscription list.

This is live now on www.novabbs.com (until someone says I broke everything, then I'll revert to backup, lol)

--
Retro Guy