bbs.tomo.city is not producing a correct hash (used for Message-ID and Notice-ID.

In interBBS_mail.php, here:

$hashtail = hash('crc32', $domain.$organization.$from.$rslight_gpg['nntp_group']);
$thishash = hash('crc32', $body.$diff_days->format("%a").$hashtail).hash('crc32', $signing_key);

it looks like 'hash('crc32', $signing_key)' is not producing a hash, which would indicate that $signing_key may not be set.

Just thought I'd point that out in case there is some issue.

tomo.dialup.cafe is fine.

--
Retro Guy

<snip>

If GPG is buggy or opaque for debugging, you might find solace in
knowing that OpenSSL can replicate much of the GPG functionality:

- generate keys
- verify keys
- sign
- encrypt
- hash
- symmetric cipher
- SSL certificates

You can even duct tape and rubber band your own cert authority for
peers who trust each other instead of using hockeypuck servers.

And conveniently, OpenSSL has a built-in client and server:

- openssl s_client
- openssl s_server

If you need forked simultaneous connections, 'socat' will handle that:

$ apt show socat
Package: socat
......
Description: multipurpose relay for bidirectional data transfer
Socat (for SOcket CAT) establishes two bidirectional byte streams
and transfers data between them. Data channels may be files, pipes,
devices (terminal or modem, etc.), or sockets (Unix, IPv4, IPv6, raw,
UDP, TCP, SSL). It provides forking, logging and tracing, different
modes for interprocess communication and many more options.

It can be used, for example, as a TCP relay (one-shot or daemon),
as an external socksifier, as a shell interface to Unix sockets,
as an IPv6 relay, as a netcat and rinetd replacement, to redirect
TCP-oriented programs to a serial line, or to establish a relatively
secure environment (su and chroot) for running client or server shell
scripts inside network connections. Socat supports sctp as of 1.7.0.

If you need secure, anonymous VPN functionality then 'onioncat' will do
that over both Tor and I2P hidden services.

I think I already mentioned NNCP which also can use Tor.

'nacl-tools' allows scripting against the NACL library:

"NaCl (pronounced "salt") is a new easy-to-use high-speed software
library for network communication, encryption, decryption, signatures,
etc. NaCl's goal is to provide all of the core operations needed to
build higher-level cryptographic tools. Tools include basic programs to
deploy CurveCP message-handling."

'libsodium23' provides scriptable access to the NACL library:

"NaCl (pronounced "salt") is a new easy-to-use high-speed software
library for network communication, encryption, decryption, signatures,
etc.

NaCl's goal is to provide all of the core operations needed to build
higher-level cryptographic tools. Sodium is a portable,
cross-compilable, installable, packageable fork of NaCl, with a
compatible API."

There are so many ways to skin a cat that nine lives are too few.

--
SugarBug | https://sybershock.com

> bbs.tomo.city is not producing a correct hash (used for Message-ID and Notice-ID.

holy crap. how on earth did you figure that out? i had no idea it was producing an incorrect hash, and i would have *never* figured it out.

it turns out $keydir was being set to null due to me forgetting to add a global (I use global $pubkey_dir instead of a hardcoded path for keydir).

--
Posted on Rocksolid Light

vga256 wrote:

>> bbs.tomo.city is not producing a correct hash (used for Message-ID and Notice-ID.

> holy crap. how on earth did you figure that out? i had no idea it was producing an incorrect hash, and i would have *never* figured it out.

Lol. It's the eight zeros at the end of the 'notice' that gave it away. Then figuring why that happened was pretty easy only because I wrote the code that generates it :)

> it turns out $keydir was being set to null due to me forgetting to add a global (I use global $pubkey_dir instead of a hardcoded path for keydir).

Glad that's working. You MAILKEY notices wouldn't offer a key fingerprint with that not working properly.

--
Retro Guy

Spent some more time trying to pin down why mail is showing up in the shared.test group, but is not showing up in user mailboxes...

Replicating odd behaviour here -

Machine 1 (originating):
1. write private message to sysop@bbs.tomo.city (machine 2 - receiving)
2. Run cron to spool the message.
3. Output of spoolnews during the bbsmail copy phase:

Retrieving articles for: tomonet.shared.test...
copying /Users/tomo/tomobbs/www/common/../../spoolnews/articles/tomonet/shared/test/27 to /Users/tomo/tomobbs/www/common/../../spoolnews/bbsmail/in/bbsmail-BBSMAIL notice 940ebbbddedbdfc0
Retrieved: tomonet.shared.test 27<!--openmodus: a-->

4. Check Machine 1 (originating), /spoolnews/bbsmail/in, and there is a file named: bbsmail-BBSMAIL notice 940ebbbddedbdfc0

5. Run cron again, just to see what happens next.

6. Check Machine 1 (originating), /spoolnews/log/mail.log:
Found BBSMAIL message bbsmail-BBSMAIL notice 940ebbbddedbdfc0
Aug 11 20:35:40 spoolnews BAD signature in: "bbsmail-BBSMAIL notice 940ebbbddedbdfc0" decrypt-verify failed: No secret key
Aug 11 20:35:40 spoolnews Let's try to get the key
Aug 11 20:35:40 spoolnews Domain: tomo.dialup.cafe
Aug 11 20:35:40 spoolnews Location: tomo.dialup.cafe/pubkey/server_pubkey.txt
Aug 11 20:35:40 spoolnews IMPORTED: B1849EE67428B6C4C1BC45652E1C769900587C92
Aug 11 20:35:40 spoolnews Domain Match: tomo.dialup.cafe
Aug 11 20:35:40 spoolnews New PGP Key added for: tomo.dialup.cafe Domain: tomo.dialup.cafe

It strikes me as odd that Machine 1 would (a) be putting its *outgoing* mail into its own bbsmail/in/ box. Even weirder that it tries decrypting its own mail, then fails. Is this intended behaviour?

--
Posted on Rocksolid Light

(fwiw, I am running the most recent commits of the RSL-messaging branch)

--
Posted on Rocksolid Light

Addendum to earlier report:

I also noticed that eventually (looks to be after the second run of cron), outgoing mail is being moved from originatingmachine/spoolnews/bbsmail/in/bbsmail-BBSMAIL noticeetcetc, to the /failed/ folder.

On the receiving machine, the post shows up in tomonet.shared.test as expected, but it does *not* get copied to receivingmachine/spoolnews/bbsmail/in

--
Posted on Rocksolid Light

vga256 wrote:

> Addendum to earlier report:

> I also noticed that eventually (looks to be after the second run of cron), outgoing mail is being moved from originatingmachine/spoolnews/bbsmail/in/bbsmail-BBSMAIL noticeetcetc, to the /failed/ folder.

Yes, it tries twice before moving to failed.

> On the receiving machine, the post shows up in tomonet.shared.test as expected, but it does *not* get copied to receivingmachine/spoolnews/bbsmail/in

Ok. That task is handled by spoolnews.php.

--
Retro Guy

vga256 wrote:

> (fwiw, I am running the most recent commits of the RSL-messaging branch)

That's good. It has some more features to make it work closer to properly.

> Spent some more time trying to pin down why mail is showing up in the shared.test group, but is not showing up in user mailboxes...

> It strikes me as odd that Machine 1 would (a) be putting its *outgoing* mail into its own bbsmail/in/ box. Even weirder that it tries decrypting its own mail, then fails. Is this intended behaviour?

Here is how it is supposed to work:

Create a remote Mail message in mail.php.

mail.php checks that it's remote, then writes a copy to your outgoing message dir (default $spooldir/section/outgoing

The message should propagate when you send messages (via cron).

When a server receives the message, and it is in the proper group, it copies it to $spooldir/bbsmail/in

Right now, that also happens on the sending server. I'm not done checking whether an incoming Mail message is from ourself or not.

Then, interBBS_mail.php will see the messages in $spooldir/bbsmail/in and try to act on it.

I would check that the message ends up in $spooldir/bbsmail/in on the target server. If it does, then it's up to interBBS_mail.php to handle it from there.

Do you have any temporarily hardcoded keys or fingerprints? That could explain it failing to import on the originating server.

--
Retro Guy

Retro Guy wrote:

> vga256 wrote:

>> Addendum to earlier report:

>> I also noticed that eventually (looks to be after the second run of cron), outgoing mail is being moved from originatingmachine/spoolnews/bbsmail/in/bbsmail-BBSMAIL noticeetcetc, to the /failed/ folder.

> Yes, it tries twice before moving to failed.

>> On the receiving machine, the post shows up in tomonet.shared.test as expected, but it does *not* get copied to receivingmachine/spoolnews/bbsmail/in

> Ok. That task is handled by spoolnews.php.

Also, make sure the group name is correct in gpg.conf

--
Retro Guy

> Do you have any temporarily hardcoded keys or fingerprints? That could explain it failing to import on the originating server.
None. Keys seem to be correct on both machines.

> Here is how it is supposed to work:
> Create a remote Mail message in mail.php.
> mail.php checks that it's remote, then writes a copy to your outgoing message dir (default $spooldir/section/outgoing
> The message should propagate when you send messages (via cron).
> When a server receives the message, and it is in the proper group, it copies it to $spooldir/bbsmail/in

Thanks for the overview of the intended behaviour - that really helped to pin down the problem. I found the issue, or at least a symptom of the issue:

Sending machine:
[6] => Retrieving articles for: tomonet.shared.test...copying /Users/tomo/tomobbs/www/common/../../spoolnews//articles/tomonet/shared/test/29 to /Users/tomo/tomobbs/www/common/../../spoolnews//bbsmail/in/bbsmail-BBSMAIL notice 29584449dedbdfc0
[7] => Retrieved: tomonet.shared.test 29

Receiving machine:
[10] => Retrieving articles for: tomonet.shared.test...
[11] => Duplicate Message-ID for: tomonet.shared.test:29
[12] => Spoolnews Done

The duplicate message-id on the receiving machine seems to be the culprit.

In spoolnews.php: As soon as it detects the incoming message as a duplicate, it continues the while loop, and script execution never makes it to line 365, where the bbsmail_check is done. I believe this is why mail is making it to the shared group, but isn't making it to the bbsmail inbox on the receiving side.

Now, I noticed that at (Spoolnews.php) 208-217, the article is already being inserted into both the overview and the articles DBs. $msgids is set from $overview_msgid at 223, and then is used to dupe-check the message id at 255.

Why is this happening on my machines, but not yours? Perhaps it is possible that because my server is *always* reloading the articles database every time there is a post (it does not wait for cron for a reload), it is detecting the duplicate msgid.
Conversely, it is possible that your servers are masking/ignoring this bug, because they are not reloading the articles/overview databases after article insertion...

Thinking out loud here:

Effectively, the way the script is written right now, I believe it is inserting the article msgid into the database, grabbing all msgid's from the database, and then finding that msgid already has been posted to the database. So of course, 100% of the time, it is detected as a duplicate.

I could be totally off base here. So let me know if I'm making any sense. I *think* the solution is to just move that chunk of database preparation code (207-217) to *after* the generation of the list of message-ids.

--
Posted on Rocksolid Light

(update - damn, I was wrong - moving that chunk of code before the message-id list creation chunk, didn't help -- still getting dupe message-id errors)

--
Posted on Rocksolid Light

Ignore the last couple of posts. That theory didn't pan out. I figured out that the number of posts in tomonet.shared.test were out of sync, and I believe that was causing the "duplicate message-id" errors. I wiped .shared.test on both machines, started from an empty db, and no more dupe message id errors. Red herring I guess.

--
Posted on Rocksolid Light

OK, shutting it down for the night. One thing is for sure: this while loop in spoolnews->get_articles() is never running:

@mkdir($grouppath,0755,'recursive');
$i=0;
while ($article <= $detail[3]) {

thinking out loud - perhaps because the originating machine's cron job has already done a "Send" articles to the receiving machine, when the receiving machine runs its own cron job, there are no "new" articles to pull from the originating server, so it doesn't run the while loop. since the while loop never runs, bbsmail_check around line 370 is never run.

--
Posted on Rocksolid Light

vga256 wrote:

> OK, shutting it down for the night. One thing is for sure: this while loop in spoolnews->get_articles() is never running:

> @mkdir($grouppath,0755,'recursive');
> $i=0;
> while ($article <= $detail[3]) {

That's the main loop to get articles. If it's not running, it should not be receiving any articles.

$article = what spoolnews thinks the next article number to get from the remote server
$detail[3] = what the remote server says is it's highest article number in the group

> thinking out loud - perhaps because the originating machine's cron job has already done a "Send" articles to the receiving machine, when the receiving machine runs its own cron job, there are no "new" articles to pull from the originating server, so it doesn't run the while loop. since the while loop never runs, bbsmail_check around line 370 is never run.

When rslight receives via 'Send' from another rslight, the article is inserted using the NNTP server. It should be the same as if you posted to your server via Thuderbird or some newsreader.

You might try posting by hand from the command line to see if you can narrow anything down.

--
Retro Guy

> When rslight receives via 'Send' from another rslight, the article is inserted using the NNTP server. It should be the same as if you posted to your server via Thuderbird or some newsreader.

That's right. But if the article were already inserted to the NNTP server from the send of the originating server, would the receiving server's spoolnews even process those posts as potential bbsmail? If I'm not misunderstanding the program logic, as far as spoolnews is concerned, those posts have already been inserted, and there's no work to do.

--
Posted on Rocksolid Light

vga256 wrote:

>> When rslight receives via 'Send' from another rslight, the article is inserted using the NNTP server. It should be the same as if you posted to your server via Thuderbird or some newsreader.

> That's right. But if the article were already inserted to the NNTP server from the send of the originating server, would the receiving server's spoolnews even process those posts as potential bbsmail? If I'm not misunderstanding the program logic, as far as spoolnews is concerned, those posts have already been inserted, and there's no work to do.

I think you are correct! Let me take a look...

--
Retro Guy

Retro Guy wrote:

> vga256 wrote:

>>> When rslight receives via 'Send' from another rslight, the article is inserted using the NNTP server. It should be the same as if you posted to your server via Thuderbird or some newsreader.

>> That's right. But if the article were already inserted to the NNTP server from the send of the originating server, would the receiving server's spoolnews even process those posts as potential bbsmail? If I'm not misunderstanding the program logic, as far as spoolnews is concerned, those posts have already been inserted, and there's no work to do.

> I think you are correct! Let me take a look...

Here is a draft of changes to the function 'prepare_post()' in rslight-lib.php. It also includes trying to resolve the same issue that would happen to nocem messages.

This is a draft. I have NOT tested it:

function prepare_post($filename) {
global $logdir, $spooldir;
$logfile = $logdir.'/nntp.log';
$message = file($filename, FILE_IGNORE_NEW_LINES);
$lines = 0;
$is_header = 1;
$nocem_check="@@NCM";
$bbsmail_check="@@RSL";

foreach($message as $line) {
if(trim($line) == "" || $lines > 0) {
$is_header=0;
$lines++;
}
if($lines > 0 && $is_header = 0) {
$break;
}
if(stripos($line, "Newsgroups: ") === 0) {
$ngroups=explode(': ', $line);
$newsgroups=$ngroups[1];
$lines++;
continue;
}
if(stripos($line, "Subject: ") === 0) {
$sub=explode(': ', $line);
$subject=$sub[1];
$lines++;
continue;
}
}
$ngroups = preg_split("/( |,)/", trim($newsgroups));
// DEBUG
file_put_contents($logfile, "n".format_log_date(). " DEBUG process_post: ".$newsgroups. " ".$ngroups[0], FILE_APPEND);
$ok = 0;
foreach($ngroups as $group) {
$result = process_post($message, $group);
if (substr($result,0,3) == "240") {
$ok = 1;
}
}
if($ok == 1) {
if((strpos($rslight_gpg['nntp_group'], $group) !== false) && ($rslight_gpg['enable'] == '1')) {
if(strpos($subject, $bbsmail_check) !== false) {
$bbsmail_file = preg_replace('/@@RSL /', '', $subject);
$bbsmail_filename = $spooldir."/bbsmail/in/bbsmail-".$bbsmail_file;
copy($filename, $bbsmail_filename);
}
}
if(strpos($subject, $nocem_check) !== false) {
$nocem_file = tempnam($spooldir."/nocem", "nocem-".$group."-");
copy($filename, $nocem_file);
}
$response="240 Article received OKrn";
} else {
$response="441 Posting failedrn";
}
return $response;
}

--
Retro Guy

woo! thanks for digging deep into this one. i can't wait to try this out tomorrow. :D

(i'm also glad that i'm not crazy - i was beginning to worry i had some bizarre system-specific bug that only my machine was experiencing)

--
Posted on Rocksolid Light

The modified script executes, but no change to posting behaviour unfortunately.

I'm going to have to leave this addition alone for a while. Debugging has eaten up about 50 hours of development time so far, and I really need to focus on implementing features. Really appreciate the hard work!

--
Posted on Rocksolid Light

vga256 wrote:

> The modified script executes, but no change to posting behaviour unfortunately.

> I'm going to have to leave this addition alone for a while. Debugging has eaten up about 50 hours of development time so far, and I really need to focus on implementing features. Really appreciate the hard work!

Man, I really thought it would work! I'll pursue this further next week and share results. I'll just set up a couple of vms talking to eachother.

--
Retro Guy

Retro Guy wrote:

> vga256 wrote:

>> The modified script executes, but no change to posting behaviour unfortunately.

>> I'm going to have to leave this addition alone for a while. Debugging has eaten up about 50 hours of development time so far, and I really need to focus on implementing features. Really appreciate the hard work!

> Man, I really thought it would work! I'll pursue this further next week and share results. I'll just set up a couple of vms talking to eachother.

Aha, $rslight_gpg is not set in the function. I can't test it right now but just maybe it's as simple as that?

--
Retro Guy