Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>> On 2023-06-13, Joerg Lorenz wrote:
>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA512
>>>
>>> Troll.
>>
>> Oh? how's a pgp sig make one a troll?
>
> OK: Bragging Troll ...
>
>> Please do tell.
>
> I use OpenPGP signing/encryption only in mails. It makes absolutely no
> sense in the worldwide Usenet. It is just overhead.

BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
accepted).

--
Gutta cavat lapidem (Ovid)

On 2023-06-13, Joerg Lorenz wrote:
> Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
>> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>>> On 2023-06-13, Joerg Lorenz wrote:
>>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA512
>>>>
>>>> Troll.
>>>
>>> Oh? how's a pgp sig make one a troll?
>>
>> OK: Bragging Troll ...

Now I'm bragging? About what, exactly? That I made an error?

> BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
> accepted).

keys.openpgp.org certainly has it available (as do any other keyservers
they share with), it's been there for years.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

Am 13.06.23 um 17:56 schrieb Dan Purgert:
> On 2023-06-13, Joerg Lorenz wrote:
>> Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
>>> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>>>> On 2023-06-13, Joerg Lorenz wrote:
>>>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>> Hash: SHA512
>>>>>
>>>>> Troll.
>>>>
>>>> Oh? how's a pgp sig make one a troll?
>>>
>>> OK: Bragging Troll ...
>
> Now I'm bragging? About what, exactly? That I made an error?
>
>> BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
>> accepted).
>
> keys.openpgp.org certainly has it available (as do any other keyservers
> they share with), it's been there for years.

It is certainly there. I found it and it tells me that it is not
verfied. Think about it.

--
De gustibus non est disputandum

On 2023-06-13 14:30, MarioCPPP wrote:
> On 13/06/23 12:28, Carlos E.R. wrote:
>> On 2023-06-13 12:13, MarioCPPP wrote:
>>> On 12/06/23 20:43, Jim Diamond wrote:
>>>> On 2023-06-12 at 06:37 ADT, Dan Purgert <dan@djph.net> wrote:
>>>>> On 2023-06-11, Jim Diamond wrote:

....

>>>>>> I avoid word processors whenever possible, but I created a .odt
>>>>>> file with libreoffice (there doesn't seem to be a .odf choice
>>>>>> with my program) and that file was apparently already
>>>>>> compressed.  Or, at least, xz didn't make any significant
>>>>>> improvement.
>>>
>>> yes, PURE text .odt are well pre-compressed (also .docx, which I
>>> avoid, not having M$ sw), but I often insert graphics caring little
>>> the kind of sources, and tending to save at top quality internally.
>>> So I think I'll keep compressing such documents
>>
>> LibreOffice files are in fact ZIP archives.
>
> maybe not "monolithic", in that the text and formatting, modules and
> other objects, are surely losslessly compressed, but I suspect (from the
> "quality" parameter, also present in exporting in PDF) that media are
> treated separately and possibly lossy.
> I have not inquiried a lot about it, expecially on audio and video which
> I never add to documents, just a lot of graphics in very heterogeneous
> original formats, and internally managed dunno how, and normally I
> cannot care less :D :D)

The files from LO, like .odt, .odf, .odt, etc... all of them are zip
archives. Even if inside there is multimedia content with lossy
compression, the entire thing is then zipped in a single archive. You
can open them with "unzip" and see the actual files it includes.

--
Cheers, Carlos.

On 2023-06-13 22:24, Joerg Lorenz wrote:
> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>> On 2023-06-13, Joerg Lorenz wrote:
>>> Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
>>>> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>>>>> On 2023-06-13, Joerg Lorenz wrote:
>>>>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>> Hash: SHA512
>>>>>>
>>>>>> Troll.
>>>>>
>>>>> Oh? how's a pgp sig make one a troll?
>>>>
>>>> OK: Bragging Troll ...
>>
>> Now I'm bragging? About what, exactly? That I made an error?
>>
>>> BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
>>> accepted).
>>
>> keys.openpgp.org certainly has it available (as do any other keyservers
>> they share with), it's been there for years.
>
> It is certainly there. I found it and it tells me that it is not
> verfied. Think about it.

That is your fault. :-p

--
Cheers, Carlos.

On Tue, 13 Jun 2023 16:24:57 -0400, Joerg Lorenz <hugybear@gmx.ch> wrote:
> It is certainly there. I found it and it tells me that it is not
> verfied. Think about it.

That's pgp/gpg basics. Keys you've added to your personal keyring are not
marked as verified unless you mark them as verified, or they've been
verified by the web of trust, where you've marked keys that one has been
signed by as being trusted to verify other keys.

Regards, Dave Hodgins

On 2023-06-13, Joerg Lorenz wrote:
> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>> [...]
>> keys.openpgp.org certainly has it available (as do any other keyservers
>> they share with), it's been there for years.
>
> It is certainly there. I found it and it tells me that it is not
> verfied. Think about it.

The only way you'll get automatic verification of my key is if you
have the public keys of other people who've signed my key (and, well,
there are none). I believe the default is that if my key were signed by
one (1) person you "Fully" trust or three (3) people you "Marginally"
trust to validate my identity before they publicly signed my key.

Without that, you would have to do the legwork yourself, and sign my key
yourself, indicating to your copy of GPG that you have "verified" my
key. After that, you can optionally tell gpg whether you trust me to
properly validate other people's keys.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

Am 13.06.23 um 23:16 schrieb Carlos E.R.:
> On 2023-06-13 22:24, Joerg Lorenz wrote:
>> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>>> On 2023-06-13, Joerg Lorenz wrote:
>>>> Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
>>>>> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>>>>>> On 2023-06-13, Joerg Lorenz wrote:
>>>>>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>>> Hash: SHA512
>>>>>>>
>>>>>>> Troll.
>>>>>>
>>>>>> Oh? how's a pgp sig make one a troll?
>>>>>
>>>>> OK: Bragging Troll ...
>>>
>>> Now I'm bragging? About what, exactly? That I made an error?
>>>
>>>> BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
>>>> accepted).
>>>
>>> keys.openpgp.org certainly has it available (as do any other keyservers
>>> they share with), it's been there for years.
>>
>> It is certainly there. I found it and it tells me that it is not
>> verfied. Think about it.
>
> That is your fault. :-p

Then you do not understand what a verified certificate is.

--
De gustibus non est disputandum

Am 13.06.23 um 23:57 schrieb David W. Hodgins:
> On Tue, 13 Jun 2023 16:24:57 -0400, Joerg Lorenz <hugybear@gmx.ch> wrote:
>> It is certainly there. I found it and it tells me that it is not
>> verfied. Think about it.
>
> That's pgp/gpg basics. Keys you've added to your personal keyring are not
> marked as verified unless you mark them as verified, or they've been
> verified by the web of trust, where you've marked keys that one has been
> signed by as being trusted to verify other keys.

Then tell me why I should trust the certificate in a posting in the usenet?

This is basics.

--
De gustibus non est disputandum

On 2023-06-14 07:56, Joerg Lorenz wrote:
> Am 13.06.23 um 23:16 schrieb Carlos E.R.:
>> On 2023-06-13 22:24, Joerg Lorenz wrote:
>>> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>>>> On 2023-06-13, Joerg Lorenz wrote:
>>>>> Am 13.06.23 um 17:38 schrieb Joerg Lorenz:
>>>>>> Am 13.06.23 um 11:49 schrieb Dan Purgert:
>>>>>>> On 2023-06-13, Joerg Lorenz wrote:
>>>>>>>> Am 12.06.23 um 22:46 schrieb Dan Purgert:
>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>>>> Hash: SHA512
>>>>>>>>
>>>>>>>> Troll.
>>>>>>>
>>>>>>> Oh? how's a pgp sig make one a troll?
>>>>>>
>>>>>> OK: Bragging Troll ...
>>>>
>>>> Now I'm bragging? About what, exactly? That I made an error?
>>>>
>>>>> BTW: Your key is not verfied. I checked on the OpenPGP-key-server (not
>>>>> accepted).
>>>>
>>>> keys.openpgp.org certainly has it available (as do any other keyservers
>>>> they share with), it's been there for years.
>>>
>>> It is certainly there. I found it and it tells me that it is not
>>> verfied. Think about it.
>>
>> That is your fault. :-p
>
> Then you do not understand what a verified certificate is.
>

Actually, I do.

--
Cheers, Carlos.

On 2023-06-14 02:01, Dan Purgert wrote:
> On 2023-06-13, Joerg Lorenz wrote:
>> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>>> [...]
>>> keys.openpgp.org certainly has it available (as do any other keyservers
>>> they share with), it's been there for years.
>>
>> It is certainly there. I found it and it tells me that it is not
>> verfied. Think about it.
>
> The only way you'll get automatic verification of my key is if you
> have the public keys of other people who've signed my key (and, well,
> there are none). I believe the default is that if my key were signed by
> one (1) person you "Fully" trust or three (3) people you "Marginally"
> trust to validate my identity before they publicly signed my key.
>
> Without that, you would have to do the legwork yourself, and sign my key
> yourself, indicating to your copy of GPG that you have "verified" my
> key. After that, you can optionally tell gpg whether you trust me to
> properly validate other people's keys.

Or, simply not put any importance on having keys signed, from people on
internet. Even if not signed, they key as it is certifies the post was
made by the same person that signed a decade ago, which is more than
enough on Usenet.

Anyway, Thunderbird can not check inline signatures. The feature was
removed.

--
Cheers, Carlos.

On 2023-06-14, Carlos E.R. wrote:
> On 2023-06-14 02:01, Dan Purgert wrote:
>> On 2023-06-13, Joerg Lorenz wrote:
>>> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>>>> [...]
>>>> keys.openpgp.org certainly has it available (as do any other keyservers
>>>> they share with), it's been there for years.
>>>
>>> It is certainly there. I found it and it tells me that it is not
>>> verfied. Think about it.
>>
>> The only way you'll get automatic verification of my key is if you
>> have the public keys of other people who've signed my key (and, well,
>> there are none). I believe the default is that if my key were signed by
>> one (1) person you "Fully" trust or three (3) people you "Marginally"
>> trust to validate my identity before they publicly signed my key.
>>
>> Without that, you would have to do the legwork yourself, and sign my key
>> yourself, indicating to your copy of GPG that you have "verified" my
>> key. After that, you can optionally tell gpg whether you trust me to
>> properly validate other people's keys.
>
> Or, simply not put any importance on having keys signed, from people on
> internet. Even if not signed, they key as it is certifies the post was
> made by the same person that signed a decade ago, which is more than
> enough on Usenet.

Quite true. Same could be said for mailing lists, etc. Though I do
prefer to not get the warnings (so lsign + "I do not know" trust).

> Anyway, Thunderbird can not check inline signatures. The feature was
> removed.

Yeah, I remember that coming up once in the past. Shame they pulled it
out (although tbird was never really good with GPG in the first place,
needing, oh what was it, enigmail?)

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 6/13/2023 6:13 AM, MarioCPPP wrote:

> my current list of DONT-COMPRESS includes :
>
> .fsa .squashfs .squash .lzw .lz77 .ppm .mp3 .aac .ogg .ogx .vorbis .gif .png .jpeg .jpg .webp .heif .mpeg .mpg .mp4 .m4v .hevc .webm .mkv .mk3d .avi .wmv .asf .DivX .mkv .mov .qt .FLV .F4V .SWF .m2p .mp2t .ps .ts .tsv .m2ts .mts .vob .evo .3gp .jhr .jr .jz .jahr .apk .msi .deb
>
> I am unsure about the .deb and about Java compressed packages

The person here, has provided us with a nice graph.

However, the LZMA is a little old, and it's just possible
there is an LZMA variant closer to ZSTD performance.

https://gregoryszorc.com/images/compression-bundle-modern.png

( https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ )

Using 7Z fast, I can numbers like this for input speed

472MB/sec on compressible data (using a solid-block file).
300MB/sec on incompressible data (output file slightly larger than input file)

and the speed comes from lots of cores in parallel, plus small memory
footprint for "fast". Since the speed exceeds the hard drive storage
rate, it takes the "file type analysis" out of the picture. But the
thing you feed to the compressor, should already have all the files in it.
You cannot compress individual one megabyte files, at 472MB/sec.

From my previous result:

If we compare that to BZ2 ultra (contained in the same program)...

Input W1022H2.vhd 28,492,703,232 bytes
Output W1022H2.7z 12,005,899,854 bytes

Compression rate (best case) 20MB/sec

Using 7Z fast w. 32 threads, I get

Input W1022H2.vhd 28,492,703,232 bytes
Output W1022H2.7z 12,174,518,826 bytes

Compression rate 470-480 MB/sec

[Picture]

https://i.postimg.cc/fLmgQxym/7z-fast.gif

The reason a solid file might matter for that compressor,
is gathering a lot of small files would exact quite a toll,
and would slow it down a lot. With the right modifications
to the command line invocation, you can make it work with
stdin and stdout and pipeline the compressor.

*******

On Windows, this command line invocation, gives the exact same
file for output, as the GUI does.

"7z.exe" a -t7z -m0=LZMA2:d64k:fb32 -ms=16m -mmt=32 -mx=1 -- "D:\W1022H2.vhd.7z" "D:\W1022H2.vhd"

Ubuntu 22.04 has the 7zz package.

apt search 7zz

And with that one, I had hoped the syntax would be

7zz a -t7z -m0=LZMA2:d64k:fb32 -ms=16m -mmt=32 -mx=1 -- "/tmp/W1022H2.vhd.7z" "/tmp/W1022H2.vhd"

but the output is a bit bigger for some reason. The presets
on the two different versions, could be different, and then more
parameters may need to be passed to fix it.

Input W1022H2.vhd 28,492,703,232 bytes
Output W1022H2.7z 12,188,398,690 bytes

In any case, this is an attempt to improve on BZ2 performance.
But with multiple threads, you really need to feed it a tar file,
to ensure the processing speed is available. It took 69 seconds
to compress the 28GB file. My test files were staged on a TMPFS
for /tmp (so that storage speed would not interfere with the results).

[Picture]

https://i.postimg.cc/N07f8PY9/7zfast-timed.gif

Paul

On 2023-06-14 13:42, Dan Purgert wrote:
> On 2023-06-14, Carlos E.R. wrote:
>> On 2023-06-14 02:01, Dan Purgert wrote:
>>> On 2023-06-13, Joerg Lorenz wrote:
>>>> Am 13.06.23 um 17:56 schrieb Dan Purgert:
>>>>> [...]
>>>>> keys.openpgp.org certainly has it available (as do any other keyservers
>>>>> they share with), it's been there for years.
>>>>
>>>> It is certainly there. I found it and it tells me that it is not
>>>> verfied. Think about it.
>>>
>>> The only way you'll get automatic verification of my key is if you
>>> have the public keys of other people who've signed my key (and, well,
>>> there are none). I believe the default is that if my key were signed by
>>> one (1) person you "Fully" trust or three (3) people you "Marginally"
>>> trust to validate my identity before they publicly signed my key.
>>>
>>> Without that, you would have to do the legwork yourself, and sign my key
>>> yourself, indicating to your copy of GPG that you have "verified" my
>>> key. After that, you can optionally tell gpg whether you trust me to
>>> properly validate other people's keys.
>>
>> Or, simply not put any importance on having keys signed, from people on
>> internet. Even if not signed, they key as it is certifies the post was
>> made by the same person that signed a decade ago, which is more than
>> enough on Usenet.
>
> Quite true. Same could be said for mailing lists, etc. Though I do
> prefer to not get the warnings (so lsign + "I do not know" trust).
>
>> Anyway, Thunderbird can not check inline signatures. The feature was
>> removed.
>
> Yeah, I remember that coming up once in the past. Shame they pulled it
> out (although tbird was never really good with GPG in the first place,
> needing, oh what was it, enigmail?)

The "old" Thunderbird did not have internal support for PGP, it used the
external plugin Enigmail for that. At some point they decided to add
internal support and Enigmail got removed. But the internal support does
not include inline signatures, it has to be attached. And attached on
Usenet I don't know if it works.

Conversely, some popular CLI mail clients, like Alpine, only supports
inline singatures, with a "plugin".

Nice situation.

--
Cheers,
Carlos E.R.

On 14/06/23 14:33, Paul wrote:
> On 6/13/2023 6:13 AM, MarioCPPP wrote:
>
>> my current list of DONT-COMPRESS includes :
>>
>> .fsa .squashfs .squash .lzw .lz77 .ppm .mp3 .aac .ogg .ogx
>> .vorbis .gif .png .jpeg .jpg .webp .heif .mpeg .mpg .mp4
>> .m4v .hevc .webm .mkv .mk3d .avi .wmv .asf .DivX .mkv .mov
>> .qt .FLV .F4V .SWF .m2p .mp2t .ps .ts .tsv .m2ts .mts .vob
>> .evo .3gp .jhr .jr .jz .jahr .apk .msi .deb
>>
>> I am unsure about the .deb and about Java compressed packages
>
> The person here, has provided us with a nice graph.
>
> However, the LZMA is a little old, and it's just possible
> there is an LZMA variant closer to ZSTD performance.
>
>
> https://gregoryszorc.com/images/compression-bundle-modern.png
>
>    (
> https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ )
>
> Using 7Z fast, I can numbers like this for input speed
>
> 472MB/sec on compressible data (using a solid-block file).
> 300MB/sec on incompressible data (output file slightly
> larger than input file)
>
> and the speed comes from lots of cores in parallel, plus
> small memory
> footprint for "fast". Since the speed exceeds the hard drive
> storage
> rate, it takes the "file type analysis" out of the picture.
> But the
> thing you feed to the compressor, should already have all
> the files in it.
> You cannot compress individual one megabyte files, at
> 472MB/sec.
>
> From my previous result:
>
> If we compare that to BZ2 ultra (contained in the same
> program)...
>
>    Input  W1022H2.vhd      28,492,703,232 bytes
>    Output W1022H2.7z       12,005,899,854 bytes
>
> Compression rate (best case) 20MB/sec
>
> Using 7Z fast w. 32 threads, I get
>
>    Input  W1022H2.vhd      28,492,703,232 bytes
>    Output W1022H2.7z       12,174,518,826 bytes
>
> Compression rate 470-480 MB/sec
>
>    [Picture]
>
>     https://i.postimg.cc/fLmgQxym/7z-fast.gif
>
> The reason a solid file might matter for that compressor,
> is gathering a lot of small files would exact quite a toll,
> and would slow it down a lot. With the right modifications
> to the command line invocation, you can make it work with
> stdin and stdout and pipeline the compressor.
>
> *******
>
> On Windows, this command line invocation, gives the exact same
> file for output, as the GUI does.
>
> "7z.exe" a -t7z -m0=LZMA2:d64k:fb32 -ms=16m -mmt=32 -mx=1 --
> "D:\W1022H2.vhd.7z" "D:\W1022H2.vhd"
>
> Ubuntu 22.04 has the 7zz package.
>
> apt search 7zz
>
> And with that one, I had hoped the syntax would be
>
> 7zz a -t7z -m0=LZMA2:d64k:fb32 -ms=16m -mmt=32 -mx=1 --
> "/tmp/W1022H2.vhd.7z" "/tmp/W1022H2.vhd"
>
> but the output is a bit bigger for some reason. The presets
> on the two different versions, could be different, and then
> more
> parameters may need to be passed to fix it.
>
>    Input  W1022H2.vhd      28,492,703,232 bytes
>    Output W1022H2.7z       12,188,398,690 bytes
>
> In any case, this is an attempt to improve on BZ2 performance.
> But with multiple threads, you really need to feed it a tar
> file,
> to ensure the processing speed is available. It took 69 seconds
> to compress the 28GB file. My test files were staged on a TMPFS
> for /tmp (so that storage speed would not interfere with the
> results).
>
>    [Picture]
>
>     https://i.postimg.cc/N07f8PY9/7zfast-timed.gif
>
>   Paul
uhm, I have not archived any VM actually, despite the
possibility of huge gains, since I have not backed up them
at all (I keep nothing of any value inside them now).
As far as TAR is concerned, I am hating double container
feature, since for large files, I have to pay overhead (and
overspace) of decompressing the whole tarball before even
parsing the files. Tar also, despite of being access-rights
and ownership respectful, is fragile in recovery on damage.
normally I go for .7z when I pipe manually, but there (on
GAMBAS IDE) I don't have it in the library, and .bz2 was the
better choice ...
intresting experiments. I guess IT is your main job too
--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
MarioCPPP

On 6/14/2023 1:05 PM, MarioCPPP wrote:

>
> intresting experiments. I guess IT is your main job too
>

I happen to be working on the compression problem right now,
so this is just a coincidence. The processor, I got that
running about three or four days ago. I was initially
concerned the processor would burn up the motherboard,
but it's been fine.

The most stressful thing for the hardware, has been memtest,
which as a hardware guy, this was unexpected for me. Even
the Southbridge got hot, and it's not supposed to do that
(it means a CPU voltage is also used by the Southbridge,
and the motherboard firmware turned up that shared voltage on me).
But you can't check the parameters the hardware is using,
while memtest is running.

*******

What you are doing (handling one file at a time), is a good
plan from a flexibility perspective. But what it isn't going
to do, easily, is give good compression speed. It may also
cause the file system on the destination, to fragment a bit
(if you compress twelve files, with twelve separate processes),
making it slower to access later.

Your project is filled with challenges, is the best way to summarize.

Something like squashfs, incorporates a mount-able file system,
and it has some compression options. But the tools to work with
that, I don't know if they parallelize the compression phase.
The attraction to a mount-able file system, is random access later.

Paul

On 2023-06-14 21:00, Paul wrote:
> On 6/14/2023 1:05 PM, MarioCPPP wrote:

> Something like squashfs, incorporates a mount-able file system,
> and it has some compression options. But the tools to work with
> that, I don't know if they parallelize the compression phase.
> The attraction to a mount-able file system, is random access later.

Try compressed btrfs instead :-)

--
Cheers,
Carlos E.R.

On 2023-06-14 19:05, MarioCPPP wrote:
> On 14/06/23 14:33, Paul wrote:
>> On 6/13/2023 6:13 AM, MarioCPPP wrote:

....

> As far as TAR is concerned, I am hating double container feature, since
> for large files, I have to pay overhead (and overspace) of decompressing
> the whole tarball before even parsing the files. Tar also, despite of
> being access-rights and ownership respectful, is fragile in recovery on
> damage.

AFAIK, it is not TAR which is fragile, but the compressor, which is
another program. The entire tar file is compressed as a single entity,
not each separate file inside. So if the compress/decompress fails, the
tar software gets nothing to work on.

--
Cheers,
Carlos E.R.

On 14/06/23 21:00, Paul wrote:
> On 6/14/2023 1:05 PM, MarioCPPP wrote:
>
>>
>> intresting experiments. I guess IT is your main job too
>>
>
> I happen to be working on the compression problem right now,
> so this is just a coincidence. The processor, I got that
> running about three or four days ago. I was initially
> concerned the processor would burn up the motherboard,
> but it's been fine.
>
> The most stressful thing for the hardware, has been memtest,
> which as a hardware guy, this was unexpected for me. Even
> the Southbridge got hot, and it's not supposed to do that
> (it means a CPU voltage is also used by the Southbridge,
> and the motherboard firmware turned up that shared voltage
> on me).
> But you can't check the parameters the hardware is using,
> while memtest is running.
>
> *******
>
> What you are doing (handling one file at a time), is a good
> plan from a flexibility perspective. But what it isn't going
> to do, easily, is give good compression speed. It may also
> cause the file system on the destination, to fragment a bit
> (if you compress twelve files, with twelve separate processes),
> making it slower to access later.
mmm, I have no control on parallelization.
I must admit I did not even think about this as a possible
source of fragmentation.
But, imvho, I must say I see a single huge file to suffer
more likely of fragmentation even on a single-threaded
process, due to intrinsic disk limits (there might be not
such contiguous area, and above all, unlike copy/move
operation, where the SW knows in advance the amount of space
needed, when a large file is generated on the fly without
any possible pre-allocation, large files would more probably
be fragmented than many smaller ones).
That said, on a backup drive, used almost as write once,
read a few, that is not a serious problem anyway. I'd not be
uncompressing all the time, just in case of crashes.
>
> Your project is filled with challenges, is the best way to
> summarize.
>
> Something like squashfs,
I have used only once, with difficulty, and it was AMAZINGLY
efficient (and really slow to compress).
But the gambas compressor is limited and I prefer not to
shell/exec but rely on the components supported. So for now,
i remain on BZIP2.
I just need to avoid really useless compressions, on
precompressed formats.
> incorporates a mount-able file system,
> and it has some compression options. But the tools to work with
> that, I don't know if they parallelize the compression phase.
dunno ... I can recall a huge set of options too, many of
which difficult to understand (about the management of
symlinks and hardlinks and sparse files too).
also mounting was uneasy (sb recommended a loop device)
> The attraction to a mount-able file system, is random access
> later.
>
>    Paul
--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
MarioCPPP

On 14/06/23 22:15, Carlos E. R. wrote:
> On 2023-06-14 21:00, Paul wrote:
>> On 6/14/2023 1:05 PM, MarioCPPP wrote:
>
>
>> Something like squashfs, incorporates a mount-able file
>> system,
>> and it has some compression options. But the tools to work
>> with
>> that, I don't know if they parallelize the compression phase.
>> The attraction to a mount-able file system, is random
>> access later.
>
> Try compressed btrfs instead :-)
>

does brtfs support "transparent" compression at the very
level of file system ? WOW !!!
Sounds intresting, even if I fear the menace of power loss
and damages more, on more complex "layers" (the same also
stands for encryption at fs level, not to say both at the
same time).

--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
MarioCPPP

On 14/06/23 22:15, Carlos E. R. wrote:
> On 2023-06-14 19:05, MarioCPPP wrote:
>> On 14/06/23 14:33, Paul wrote:
>>> On 6/13/2023 6:13 AM, MarioCPPP wrote:
>
> ...
>
>> As far as TAR is concerned, I am hating double container
>> feature, since for large files, I have to pay overhead
>> (and overspace) of decompressing the whole tarball before
>> even parsing the files. Tar also, despite of being
>> access-rights and ownership respectful, is fragile in
>> recovery on damage.
>
> AFAIK, it is not TAR which is fragile, but the compressor,
> which is another program. The entire tar file is compressed
> as a single entity, not each separate file inside. So if the
> compress/decompress fails, the tar software gets nothing to
> work on.
>

well, I was just reporting sb's else opinion, not mine,
according to which TAR format does not store internally any
kind of redundancy or recovery info, so, this I have
understood, in case of a "point failure", maybe all or at
least all stuff after that is damaged and irrecoverably.

I read that .7z is more robust per se, and .RAR (which alas
is not public domain) even more resilient to damages. I wont
be using cracked SW though, i prefer open formats that are
not assumed to change without notice internally.
I mean : i'd like to maintain the access of stored data even
..rar builders change their mind :D

>

--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
MarioCPPP

On 2023-06-15 11:51, MarioCPPP wrote:
> On 14/06/23 22:15, Carlos E. R. wrote:
>> On 2023-06-14 21:00, Paul wrote:
>>> On 6/14/2023 1:05 PM, MarioCPPP wrote:
>>
>>
>>> Something like squashfs, incorporates a mount-able file system,
>>> and it has some compression options. But the tools to work with
>>> that, I don't know if they parallelize the compression phase.
>>> The attraction to a mount-able file system, is random access later.
>>
>> Try compressed btrfs instead :-)
>>
>
> does brtfs support "transparent" compression at the very level of file
> system ? WOW !!!
> Sounds intresting, even if I fear the menace of power loss and damages
> more, on more complex "layers" (the same also stands for encryption at
> fs level, not to say both at the same time).

I am using this feature for some of my external backup hard-disks. I use
both encryption (via LUKS) and compression on btrfs. Then I do backups
using rsync. It is not a huge ratio, but it helps.

But the operating system running those backups is installed on ext4.

--
Cheers, Carlos.

On 2023-06-15 11:55, MarioCPPP wrote:
> On 14/06/23 22:15, Carlos E. R. wrote:
>> On 2023-06-14 19:05, MarioCPPP wrote:
>>> On 14/06/23 14:33, Paul wrote:
>>>> On 6/13/2023 6:13 AM, MarioCPPP wrote:
>>
>> ...
>>
>>> As far as TAR is concerned, I am hating double container feature,
>>> since for large files, I have to pay overhead (and overspace) of
>>> decompressing the whole tarball before even parsing the files. Tar
>>> also, despite of being access-rights and ownership respectful, is
>>> fragile in recovery on damage.
>>
>> AFAIK, it is not TAR which is fragile, but the compressor, which is
>> another program. The entire tar file is compressed as a single entity,
>> not each separate file inside. So if the compress/decompress fails,
>> the tar software gets nothing to work on.
>>
>
> well, I was just reporting sb's else opinion, not mine, according to
> which TAR format does not store internally any kind of redundancy or
> recovery info, so, this I have understood, in case of a "point failure",
> maybe all or at least all stuff after that is damaged and irrecoverably.

Yes, it is true that "compressed tars" have that risk, and is the reason
I do not use tar for backups.

>
> I read that .7z is more robust per se, and .RAR (which alas is not
> public domain) even more resilient to damages. I wont be using cracked
> SW though, i prefer open formats that are not assumed to change without
> notice internally.
> I mean : i'd like to maintain the access of stored data even .rar
> builders change their mind :D

Rar was the first to implement recovery features. It is not open, but
they have been there and stable for decades. The most important snag
about rar in Linux is that it does not support all file attributes and
flags. Permissions etc.

--
Cheers, Carlos.

On 15/06/23 12:50, Carlos E.R. wrote:
> On 2023-06-15 11:55, MarioCPPP wrote:
>> On 14/06/23 22:15, Carlos E. R. wrote:
>>> On 2023-06-14 19:05, MarioCPPP wrote:
>>>> On 14/06/23 14:33, Paul wrote:
>>>>> On 6/13/2023 6:13 AM, MarioCPPP wrote:
>>>
>>> ...
>>>
>>>> As far as TAR is concerned, I am hating double container
>>>> feature, since for large files, I have to pay overhead
>>>> (and overspace) of decompressing the whole tarball
>>>> before even parsing the files. Tar also, despite of
>>>> being access-rights and ownership respectful, is fragile
>>>> in recovery on damage.
>>>
>>> AFAIK, it is not TAR which is fragile, but the
>>> compressor, which is another program. The entire tar file
>>> is compressed as a single entity, not each separate file
>>> inside. So if the compress/decompress fails, the tar
>>> software gets nothing to work on.
>>>
>>
>> well, I was just reporting sb's else opinion, not mine,
>> according to which TAR format does not store internally
>> any kind of redundancy or recovery info, so, this I have
>> understood, in case of a "point failure", maybe all or at
>> least all stuff after that is damaged and irrecoverably.
>
> Yes, it is true that "compressed tars" have that risk, and
> is the reason I do not use tar for backups.
>
>>
>> I read that .7z is more robust per se, and .RAR (which
>> alas is not public domain) even more resilient to damages.
>> I wont be using cracked SW though, i prefer open formats
>> that are not assumed to change without notice internally.
>> I mean : i'd like to maintain the access of stored data
>> even .rar builders change their mind :D
>
> Rar was the first to implement recovery features. It is not
> open, but they have been there and stable for decades. The
> most important snag about rar in Linux is that it does not
> support all file attributes and flags. Permissions etc.
>

true ... alas, neither do .7z, .zip and many more.

Somehow the SW i mentioned (awit dbackup) for per-folder
backups, manage to trace attributes, and I sometime still
use it.

--
1) Resistere, resistere, resistere.
2) Se tutti pagano le tasse, le tasse le pagano tutti
MarioCPPP

On 2023-06-13 at 07:28 ADT, Carlos E.R. <robin_listas@es.invalid> wrote:
> On 2023-06-13 12:13, MarioCPPP wrote:
>> On 12/06/23 20:43, Jim Diamond wrote:
>>> On 2023-06-12 at 06:37 ADT, Dan Purgert <dan@djph.net> wrote:
>>>> On 2023-06-11, Jim Diamond wrote:
>
> ...
>
>> my current list of DONT-COMPRESS includes :
>>
>> .fsa .squashfs .squash .lzw .lz77 .ppm .mp3 .aac .ogg .ogx .vorbis .gif
>> .png .jpeg .jpg .webp .heif .mpeg .mpg .mp4 .m4v .hevc .webm .mkv .mk3d
>> .avi .wmv .asf .DivX .mkv .mov .qt .FLV .F4V .SWF .m2p .mp2t .ps .ts
>> .tsv .m2ts .mts .vob .evo .3gp .jhr .jr .jz .jahr .apk .msi .deb
>>
>> I am unsure about the .deb and about Java compressed packages
>
> A good compressor should know all those and automatically not waste cpu
> cycles on them.

Do you know of any compressor that checks the file types (not the
extension, which could be bogus) and refuses to try compressing?

Clearly (?!) such a thing is possible, but I haven't run into such a
compressor.

Jim