>On the bad side, I have apparently trashed my vichan

note to self: rtfm. was setting an option to true where a regex was expected.

Posted on def4

anon wrote:

> a little update:

> the good part: i could integrate your code, and together with the other lib i could build a solution that works locally without interfering with rslight (to be confirmed, of course).
> that could be made into a solution that runs also without a local newsserver if vichan_message_read could get the article from a news server instead of reading it from the spool.

That can be done pretty easily. What I don't know is how do you know what article number you want to retrieve? Do you keep track, or just ask for the next article and if it doesn't exist, quit?

> Also, I managed now to give the articles directly to vichan, without writing it to disk first and I can also capture the reply and do something with it, like syncing again later in case of errors (not implemented yet). All this makes the syncing faster and more reliable.

Are you sending now through nntp commands, or still placing in rslight outgoing spool?

> On the bad side, I have apparently trashed my vichan installation with all the hacking and trying i did, and I can't figure out what I broke exactly. So this will take some more time, until I know all the ins and outs of this software.

Done that before :)

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

Retro Guy wrote:

> anon wrote:

>> a little update:

>> the good part: i could integrate your code, and together with the other lib i could build a solution that works locally without interfering with rslight (to be confirmed, of course).
>> that could be made into a solution that runs also without a local newsserver if vichan_message_read could get the article from a news server instead of reading it from the spool.

> That can be done pretty easily. What I don't know is how do you know what article number you want to retrieve? Do you keep track, or just ask for the next article and if it doesn't exist, quit?

I changed vichan.php to use a news server instead of a spool. It seems to be working fine:

$ ./vichan.php 91 rocksolid.shared.test
O:11:"messageType":2:{s:6:"header";O:10:"headerType":27:{s:6:"number";N;s:2:"id";s:59:"<635faa159efe9999f6f2b469d70207e6$1@z5bqfv5v75kxy7pj.onion>";s:4:"from";s:18:"anonymous@anon.com";s:4:"name";s:9:"anonymous";s:7:"subject";s:34:"testing after upgrade to rslight d";s:10:"newsgroups";s:21:"rocksolid.shared.test";s:8:"followup";N;s:4:"date";i:1590583800;s:11:"date_thread";N;s:12:"organization";s:4:"def2";s:10:"xnoarchive";N;s:10:"references";N;s:13:"bestreference";N;s:25:"content_transfer_encoding";s:4:"8bit";s:12:"mime_version";N;s:12:"content_type";a:1:{i:0;s:10:"text/plain";}s:20:"content_type_charset";a:1:{i:0;s:5:"utf-8";}s:17:"content_type_name";a:1:{i:0;s:7:"unnamed";}s:21:"content_type_boundary";N;s:19:"content_type_format";a:1:{i:0;s:6:"flowed";}s:7:"answers";N;s:8:"isAnswer";b:0;s:8:"username";s:9:"anonymous";s:10:"user_agent";s:45:"Rocksolid Light (news.novabbs.com/getrslight)";s:7:"isReply";N;s:10:"threadsize";N;s:12:"rslight_site";s:60:"$2y$10$iwF363fhTVNz2W/yh0h6kO99Zid/znmcJHaaT5rXt47tlJppHDmo2";}s:4:"body";a:1:{i:0;s:49:"well ?
--
Posted on def2
z5bqfv5v75kxy7pj.onion
";}}

This requests article 91 of rocksolid.shared.test. vichan.php then contacts a nntp server and requests the article, then converts it to output vichan can use. It does not look to a spool at all, so no local spool necessary.

I'll upload it now as vichan_nntp.php to the download directory. You will need to configure the server name and other info in the file before it will work (so it can contact the server).

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

>That can be done pretty easily. What I don't know is how do you know what article number you want to retrieve? Do you keep track, or just ask for the next article and if it doesn't exist, quit?

I ask the server for the group listings and then just walk through all the articles that don't exist already in my local index file (so yes, i keep track of what i received on my end).

>Are you sending now through nntp commands, or still placing in rslight outgoing spool?

Talking about outgoing (so from vichan to rslight), i still use the spool. this is the next thing to change, though. after that change you can run vichan standalone and just connect to a remote news server.
In the text you were referring to I was talking about the opposite, getting articles into vichan. Before, I was just reading the oldest file in a defined directory, which was kind of lame. Now I have recoded the post routine and I post the articles directly (as a json object) to a php script that reads them from php://input and parses them (and then inserts them into vichan of course).
Doing this, you can also push articles from remote (so two instances of vichan could push directly messages to each other). Some work still to be done to achieve this, of course.
But I like the idea very much to have a way to cut out the middle man so to speak and sync messages directly via http, as it is much easier to setup and in some instances would also be faster (of course this cannot fully substitute syncing via news servers, because they are so much more reliable for this, but it could be a nice feature especially for new admins).

I will try out your new code now, and try to build something that I can share. If everything works as planned, it should be enough to add two scripts to an vichan installation to hook it up to our newsgroups.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

>vichan_nntp.php

it gives:

#!/usr/bin/php
b:0;

care to rename :-] ?

can't wait to try that, as it will also make it very easy to use several news servers for updating.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

meaning there is the executable (.php) in the download area, not the source (.txt).
--
Posted on def2
z5bqfv5v75kxy7pj.onion

-------------------------------------------------------------------------------------------------------------
Filenames: NNTP.php epost.php pull_articles.php vichan_mod.php
Download link:
http://ana5icnp3au4qy5i.onion/?n1jO6na7YLJ6eJivMjoZONkKnxu
-------------------------------------------------------------------------------------------------------------

A very early prototype really, just in case you would like to snoop through.

cheers

trw

--
Posted on def2.i2p

anonymous wrote:

>>vichan_nntp.php

> it gives:

> #!/usr/bin/php
> b:0;

> care to rename :-] ?

> can't wait to try that, as it will also make it very easy to use several news servers for updating.

oops, sorry. It's changed now: vichan_nntp.php.txt

--
Posted on: Rocksolid Light
www.novabbs.com

Anonymous wrote:

> -------------------------------------------------------------------------------------------------------------
> Filenames: NNTP.php epost.php pull_articles.php vichan_mod.php
> Download link:
> http://ana5icnp3au4qy5i.onion/?n1jO6na7YLJ6eJivMjoZONkKnxu
> -------------------------------------------------------------------------------------------------------------

> A very early prototype really, just in case you would like to snoop through.

Got it. Looks good, and I see where you are splicing in vichan_message_read.

If you use the new vichan file (vichan_nntp.php), you'll need to set:

$remote_server="127.0.0.1";
$remote_port="119";
$remote_auth_user="username";
$remote_auth_pass="password";

in the file. The vichan_message_read function is different only in that it pulls the article via nntp, not from the spool. There are additional functions in the script which it needs to accomplish this. The other existing functions are not changed.

After setting the values above, you should be able to call vichan_message_read exactly as you are now.

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

i will make it so that the updates come from at least 3 different news servers, that should make it really fast and reliable and lower the load on each individual news server.
if i have relevant updates to my code, i will upload them to the location given earlier.
ideally, i think all code to pull the messages should reside in one file only, and maybe i can get rid of the NNTP.php lib as well, it should be possible to use yours for that.
next thing i will try is to actually use "send" over nntp to push messages to other servers instead of using the spool. when this is accomplished, vichan can run standalone.

cheers

trw

--
Posted on def2.i2p

so now i pull messages to vichan from

-my own inn2 server (def4)
-my own (local) rslight news server (def2)
-your inn2 server at dgqrm3...

i use one common index of threads per newsgroup in order not to retransmit anything, and separate files for all the watermarks (per group and server).

so far seems to be working fine. the server that is used for the current update run is chosen randomly.

of course this setup is still wasteful vs pushing the messages, especially in the case of your inn2 server (tell me if the load should be too heavy, than i have to put in some throttling).

latest code will be uploaded soon.

cheers

trw

--
Posted on def2.i2p

for some reason the almost identical code sequence that works for send.php in rslight for posting messages to a news server (using the post command),
produces a 480 - posting not allowed when i try to use it for vichan (with the same server). guess i do something for the auth.
this is clearly not RESTful :-)

--
Posted on def2

Anonymous wrote:

> for some reason the almost identical code sequence that works for send.php in rslight for posting messages to a news server (using the post command),
> produces a 480 - posting not allowed when i try to use it for vichan (with the same server). guess i do something for the auth.

Are you setting $remote_auth_user and $remote_auth_pass in the file? In rslight these are pulled from config. If that's ok, when exactly do you see the error? After AUTHINFO commands? After POST command?

The function nntp2_open in send.php uses language variables to display some errors. If you're not linked to rslight, you probably won't see the errors from it. You can just change those lines to display the text you want for the error.

BTW, the function nntp2_open in send.php and the modified vichan_nntp.php are identical functions. If you link the two files (include one in the other), you'll need to drop the function from the secondary function.

send.php does not include some functions it uses that are in newsportal.php. If you are running it not including that file, it probably won't work. The best way to handle it for now is to just copy the functions into the send.php you want to use. (vichan_send.php maybe?)

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

>when exactly do you see the error? After AUTHINFO commands? After POST command?

i am not sure, because with telnet everything works, and with the program, i don't see exactly what happens. some screwing up the auth, i think.
actually, ihave or takethis would be better than post.
this will take a but more tinkering.

--
Posted on def2

>Are you setting $remote_auth_user and
>$remote_auth_pass in the file?

i set the vars right (passing them on from the calling script), tested that before.

> when exactly do you see the error? After AUTHINFO commands? After POST command?

i don't know, because i don't see the intermediate stages. i guess i also had a mistake in parsing the the responses from the news server.
doing the same in telnet i get no error at all. guess it is not exactly the same that i do in telnet.

>The best way to handle it for now is to just copy the
>functions into the send.php you want to use.
>(vichan_send.php maybe?)

that is the question. i tried to make nntpchan.php from vichan working, which it did not for some reason. when i do the psoting to the news server there, i can't (or don't want) to drag a lot of libs in, because that will end up messy i guess, and i don't understand the vichan code well enough to really integrate it well.
the other option is to do the posting in the import script i have already (which would also become an export script than), in this case one more lib doesn't hurt. or maybe the posting function could be included in vichan_nntp.php ?
what i don't understand yet is how rslight can use POST to post an article with an existing message id ? i understand that normally the news server generates the number for you when you use POST (like for fudforum for example).
forwarding existing messages should (according to RFC) be done with IHAVE or TAKETHIS.
not that i will complain about this point if i can find a way to make this work, mind you :-).
the advantage of using POST is maybe that more news servers will let you use it vs streaming with IHAVE or TAKETHIS.
but anyway, it is maybe best to take that code outside from vichan and just continue to drop the messages as files in a defined directory. this way i can also implement a followup (post successful or not).

cheers

trw

--
Posted on def2

Anonymous wrote:

>>The best way to handle it for now is to just copy the
>>functions into the send.php you want to use.
>>(vichan_send.php maybe?)

> that is the question. i tried to make nntpchan.php from vichan working, which it did not for some reason. when i do the psoting to the news server there, i can't (or don't want) to drag a lot of libs in, because that will end up messy i guess, and i don't understand the vichan code well enough to really integrate it well.
> the other option is to do the posting in the import script i have already (which would also become an export script than), in this case one more lib doesn't hurt. or maybe the posting function could be included in vichan_nntp.php ?

That shouldn't be a problem, especially since send and retrieving articles share some functions.

> what i don't understand yet is how rslight can use POST to post an article with an existing message id ? i understand that normally the news server generates the number for you when you use POST (like for fudforum for example).
> forwarding existing messages should (according to RFC) be done with IHAVE or TAKETHIS.

inn and rslight news servers will accept articles which include a message-id, and retain that when receiving via the POST command. Both servers will create a m-id if one is not included, but will not change it if it is included.

Consider that rslight spools a message posted to it right away. It can't wait until after it's posted to an upstream server to then get a m-id, it needs one immediately to spool a proper article. It is intentional that when clicking 'Post' when sending a message in rslight, you need not wait for communication with an upstream server before moving along. On i2p especially, this is important.

But when posting to inn or rslight from vichan, including a message-id in your message is not required, it's just supported.

> not that i will complain about this point if i can find a way to make this work, mind you :-).
> the advantage of using POST is maybe that more news servers will let you use it vs streaming with IHAVE or TAKETHIS.

Exactly. Most servers will not allow IHAVE or TAKETHIS (streaming), but any server you can post to with a reader will respond to POST properly.

> but anyway, it is maybe best to take that code outside from vichan and just continue to drop the messages as files in a defined directory. this way i can also implement a followup (post successful or not).

Since you already have a properly formatted message coming out of vichan, it really should be a simple matter of (taken from send.php):

$messages = scandir($outgoing_dir);
foreach($messages as $message) {
if(!is_file($outgoing_dir.$message)) {
continue;
}
echo "Sending: ".$outgoing_dir.$message."rn";
fputs($ns, "MODE READERrn");
$response = line_read($ns);
if (strcmp(substr($response,0,3),"200") != 0) {
file_put_contents($logfile, "n".format_log_date()." ".$config_name." Unexpected response to MODE command: ".$response, FILE_APPEND);
return $response;
}
fputs($ns, "POSTrn");
$response = line_read($ns);
if (strcmp(substr($response,0,3),"340") != 0) {
file_put_contents($logfile, "n".format_log_date()." ".$config_name." Unexpected response to POST command: ".$response, FILE_APPEND);
return $response;
}
$message_fp = fopen($outgoing_dir.$message, "rb");
while (($msgline = fgets($message_fp, 4096)) !== false) {
fputs($ns, $msgline);
}
fputs($ns, ".rn");
fclose($message_fp);
$response = line_read($ns);
if (strcmp(substr($response,0,3),"240") == 0) {
unlink($outgoing_dir.$message);
file_put_contents($logfile, "n".format_log_date()." ".$config_name." Posted: ".$message.": ".$response, FILE_APPEND);
} else {
file_put_contents($logfile, "n".format_log_date()." ".$config_name." Failed to POST: ".$message.": ".$response, FILE_APPEND);
continue;
}
}
return "Messages sentrn";

The above is sending from a file, but could just as easily be sending from a variable.

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

thanks for the code. i had to change a few places, like:
echo "Sending: ".$outgoing_dir.$message."rn";
to
echo "Sending: ".$outgoing_dir.$message."\n\r";
and, more importantly:
fputs($ns, "MODE READERrn");
to
fputs($ns, "MODE READER\r\n");
and like that in all the places sending this sequence.

in the process i discovered that subjects with "\" will break something in bbs :-).

anyway, it is working, and vichan posts directly to news servers now.
so now, it can run without any local news server, meaning you should be able to sync any existing installation to newsgroups.
very cool.
i need to clean up my code, it is still horrible right now, and i guess it can be done in half of the lines.
after that, maybe we should publish that somewhere it can be found (thinking some fork on github or so...). i know i said earlier a fork would be too heavy for me, but maybe we can offer it to some existing fork ? what do you think ?
i just have the feeling it would be unfair to build on something existing and than not sharing your results.

anyway, it is working. very nice. :-)

cheers

trw

--
Posted on def2

btw, this setup should also be a compliant implementation of overchan. i wonder whatever became of this project. it was really huge in terms of numbers of nodes once.

--
Posted on def2

Anonymous wrote:

> thanks for the code. i had to change a few places, like:
> echo "Sending: ".$outgoing_dir.$message."rn";
> to
> echo "Sending: ".$outgoing_dir.$message."nr";
> and, more importantly:
> fputs($ns, "MODE READERrn");
> to
> fputs($ns, "MODE READERrn");
> and like that in all the places sending this sequence.

Yes, rslight removes the backslash when displaying the code. What I sent had the backslashes but they were stripped for display.

> in the process i discovered that subjects with "" will break something in bbs :-).

There's always something new in everybody's code :)

> anyway, it is working, and vichan posts directly to news servers now.
> so now, it can run without any local news server, meaning you should be able to sync any existing installation to newsgroups.
> very cool.

One of the links I sent in r.s.encryption is an rslight nntp server. If you want to test syncing vichan directly to rslight and back, it might be a great test to prove no inn servers are required. I already sync one rslight site to another successfully. (My freebsd test site to rslight.i2p)

> i need to clean up my code, it is still horrible right now, and i guess it can be done in half of the lines.

Uh, yeah, me too :)

> after that, maybe we should publish that somewhere it can be found (thinking some fork on github or so...). i know i said earlier a fork would be too heavy for me, but maybe we can offer it to some existing fork ? what do you think ?
> i just have the feeling it would be unfair to build on something existing and than not sharing your results.

I agree that when it's working well, provide a download at least. That's all I do for rslight. I have no github, cvs or anything, but it can be downloaded. I'd do the same for vichan. If someone wants to fork what you wrote, that's great, but it's nice to keep your own fork for your own mods, etc.

> anyway, it is working. very nice. :-)

I'm amazed at how fast you got this working! Great work!

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

Retro Guy wrote:

> Anonymous wrote:

>> in the process i discovered that subjects with "" will break something in bbs :-).

> There's always something new in everybody's code :)

I noticed earlier today that rslight's overboard did not display koi-8 correctly. The article would display properly, but not in the overboard. That's fixed now :)

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com

Anonymous wrote:

> btw, this setup should also be a compliant implementation of overchan. i wonder whatever became of this project. it was really huge in terms of numbers of nodes once.

It's odd how it just kind of went away. It seemed to work fine.

--
Posted on: Rocksolid Light
www.novabbs.com

>it might be a great test to prove no inn servers are required.

did that with my own instance already, it is working.

--
Posted on def2

anon <anon@anon.com> wrote:
> POST /post.php HTTP/1.1

In case, you can see JSON from my read-only newsreader here:

https://nnrp.alphanet.ch/newsgroup/rocksolid.programming/%3C8cb53de068f68ce971aa235b44272aab%241%40news.novabbs.com%3E?mode=json

Implementation:

if ($self->param('mode') eq 'json') {
my %headers;

my $headers = $a->headers;
# yeah, not simple re
# - to make the loop work, we need to not remove the final \n if any, so we use
# a look-ahead: https://stackoverflow.com/questions/3926451/how-to-match-but-not-capture-part-of-a-regex
# - we accept continuation lines
# - maybe the header re is a bit too strict
# - 'rest' should be empty if all was right
$headers =~ s/(^|\n)([a-zA-Z0-9\-]+):\s?(.*)(\n\s+.*|)(?=\n|$)/$headers{$2} = $3 . $4; ""/eg;

$self->render(json => { 'headers' => \%headers,
'body' => $a->body,
'headers_raw' => $a->headers,
'rest' => $headers
});
}

do you want to start soon, or wait a little longer ?
if you want to start soon, i would also start to work on the direct syncing (so: pushing messages from one instance of vichan to another via http).
if not, i will focus on something else first (like: dealing with attachments in a good way).
and talking about attachments: can we agree on limiting the number of attachments to one for the rocksolid groups ?

cheers

trw

--
Posted on def2

>In case, you can see JSON from my read-only newsreader here:

yes i can see json :-)
what about it ? what do you achieve with that code ?

--
Posted on def2