On 2023-11-09 13:57, Frank Slootweg wrote:
> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>> On Mon, 06 Nov 2023 00:28:12 -0600, Char Jackson wrote:
>>>
>>> On Sun, 5 Nov 2023 19:13:43 -0800, Stan Brown <the_stan_brown@fastmail.fm>
>>> wrote:
>>>
>>>> On Sat, 4 Nov 2023 15:46:32 -0700, T wrote:
>>>>> An alternative is SSD Life, but it does not work so well.
>>>>> Try the demo and see if you like it:
>>>>>
>>>>> https://ssd-life.com/
>>>>
>>>> What do you mean by "does not work so well"? What problems did you
>>>> find?
>>>>
>>>> They have a free trial, but if the software is crap then a free trial
>>>> seems otiose, perhaps even nugatory.
>>>
>>> SSD Life has a free version and a Pro version. In the free version, which I use
>>> on a laptop with two SSDs, I'd say the biggest limitation is that it only shows
>>> info for the first SSD in the system. I wouldn't call it crap, by any means.
>>
>> I think we're a victim of thread drift. I asked, and the subject line
>> still says, "check health of a flash drive", but it seems the
>> discussion has changed to SSDs. Such is Usenet.
>>
>> I looked at
>> <https://ssd-life.com/eng/ssdlife-faq-frequently-asked-
>> questions.html>
>> and found this:
>>
>> "What will happen when the lifetime expiration date comes?
>>
>> "... The peculiarity of SSDs is that after their lifetime is over,
>> the data does not disappear from them. They just become read-only,
>> you can read data from it, but you cannot modify it."
>>
>> If that's true, it's good news for SSD users. Is it also true for
>> flash drives? My main concern is not replacing a flash drive, because
>> they're cheap; my concern is not knowing that I _need_ to replace it,
>> and having its contents be corrupt because writes failed.
>
> I am confused! How could the contents be corrupt without you knowing
> it?
>
> If "writes failed", whatever program you used to do the writing would
> give an error message and you could/should take appropriate action.
>
> The only 'silent' failure mode in your workflow would be if the USB
> memory-stick failed *after* the writing (and the subsequent reading
> (copying to the other computer (desktop/laptop)).
>
> But that failure mode will be detected when you start your next
> workflow. If the USB memory-stick fails during that next workflow, the
> known-good files are still on the 'from' computer, so just to be safe,
> you backup those to some media, buy a new USB memory-stick and finish
> your workflow.
>
> So let's turn this around: You apparently think you could have a
> failure without any kind of recovery, so you should be able to describe
> how such a scenario could happen, because I don't see one.

"move" instead of "copy".

Edit documents directly on the stick.

--
Cheers,
Carlos E.R.

On Wed, 8 Nov 2023 10:30:22 -0800, Stan Brown
<the_stan_brown@fastmail.fm> wrote:

>I think we're a victim of thread drift. I asked, and the subject line
>still says, "check health of a flash drive", but it seems the
>discussion has changed to SSDs. Such is Usenet.

Such is all conversations, even spoken ones.

Carlos E. R. <robin_listas@es.invalid> wrote:
> On 2023-11-09 13:57, Frank Slootweg wrote:
> > Stan Brown <the_stan_brown@fastmail.fm> wrote:
[...]
> >> If that's true, it's good news for SSD users. Is it also true for
> >> flash drives? My main concern is not replacing a flash drive, because
> >> they're cheap; my concern is not knowing that I _need_ to replace it,
> >> and having its contents be corrupt because writes failed.
> >
> > I am confused! How could the contents be corrupt without you knowing
> > it?
> >
> > If "writes failed", whatever program you used to do the writing would
> > give an error message and you could/should take appropriate action.
> >
> > The only 'silent' failure mode in your workflow would be if the USB
> > memory-stick failed *after* the writing (and the subsequent reading
> > (copying to the other computer (desktop/laptop)).
> >
> > But that failure mode will be detected when you start your next
> > workflow. If the USB memory-stick fails during that next workflow, the
> > known-good files are still on the 'from' computer, so just to be safe,
> > you backup those to some media, buy a new USB memory-stick and finish
> > your workflow.
> >
> > So let's turn this around: You apparently think you could have a
> > failure without any kind of recovery, so you should be able to describe
> > how such a scenario could happen, because I don't see one.
>
> "move" instead of "copy".
>
> Edit documents directly on the stick.

In a thread of over 50 responses (and then IIRC also some in another
thread), it's easy to lose the context! :-)

My comments are in the context of Stan's workflow [1]. He copies
modified files from one computer to to stick and then from the stick to
another computer (the computers are a desktop and a laptop). So he
doesn't edit documents on the stick, nor does he do a move.

Hence my question.

[1] Message-ID: <MPG.3fa9586b5999eed5990202@news.individual.net>

On 11/9/2023 9:06 AM, Ken Blake wrote:
> On Wed, 8 Nov 2023 10:30:22 -0800, Stan Brown
> <the_stan_brown@fastmail.fm> wrote:
>
>> I think we're a victim of thread drift. I asked, and the subject line
>> still says, "check health of a flash drive", but it seems the
>> discussion has changed to SSDs. Such is Usenet.
>
>
> Such is all conversations, even spoken ones.
>

That's because USB flash drives don't have "health" as a concept.
They die. You move on. Simple really :-/ This is why you *never*
store your only copy of stuff on one.

Only a factory utility could say more, by accessing information
about the flash chip connected to the controller. This is not normally
visible from an OS. It's much better if a controller summarizes the
info, than to attempt to analyze a raw flash chip.

USB flash drives are entirely too simple in design and concept.
And it is taking years and years for them to catch up. Look how
long it took to get SMART passthru on a USB to SATA adapter.

SSDs were mentioned, as having health info.

NVMe were mentioned, as having some info, but not "standards driven health".

Technology with wear out mechanism, should have "wear health" (hello, USB flash).

SCSI/SAS never had SMART on it. I used to use FWB to talk to such drives (custom code per model).

SATA has SMART (ATA has SMART). SATA is Serial ATA. Ribbon cable is ATA.

Modern technology is small enough, you can really have any amount
of logic you would like. The package size determine the price
of the controller (48 pins). If you want to put a CRAY inside a USB stick,
you can. But, you have to pay engineers to put the CRAY in there,
and that has a cost. The USB flash stick industry is not interested.
Making shit, is a way of life for them. That's why I've had wobbly
USB sticks that write at 3MB/sec, and the LED makes an idiotic pattern
while it does so.

Go to your Walmart, if you want to see a "peg board full of ignominy".

At my computer store, you can't "browse" them, as they're stored behind
the counter, like some sort of soft porn.

Sorta like my hardware store yesterday, when I wanted to get some
3/8" drive sockets for a job in the driveway, there was this
huge "cage" around the socket display. And me and another customer
were just standing and looking at the cage. What a way to live.
We had that "OK, now what?" look on our faces.

Paul

On 11/8/2023 1:30 PM, Stan Brown wrote:

> "What will happen when the lifetime expiration date comes?
>
> "... The peculiarity of SSDs is that after their lifetime is over,
> the data does not disappear from them. They just become read-only,
> you can read data from it, but you cannot modify it."
>
> If that's true, it's good news for SSD users.

The statement about SSDs is wrong, and the author of that
paragraph should be shot.

Each manufacturer has a web page with some details on their
policy.

Intel SSDs, when every location has the 600 writes or whatever,
the drives "brick" instantly. Neither reading nor writing work.

This is why, if you own an Intel, your backup schedule should
be changed to make daily backups. Because you're going to get
run over by a steam roller... soon. If your monitoring application
says "100 GB writes remaining", then you would switch to daily backups
and expect some edits to get lost. It depends on your tolerance for
surprises, as to how close you shave that. Clone over using Macrium
and do it from the Macrium CD, not from the OS itself. I have watched
people in years past, play roulette with that, but nobody recently
seems to know about life-monitoring utils. I certainly don't
have one set up, and I should know better. (I have some Intels,
and I bought them knowing the policy. They're sitting in a pile :-)
No daily driver lives on those.)

These are the policies of the different manufacturers:

No read, No write # Intel

Can continue to read # Useful for lazy home users like me...

Can continue to read *and* write! # Hmmm. Bad monkey, bad. The user will
# get hit by a truck. Think how many
# corrupted backups you would make, before
# getting hit by the truck.

That's why you seek out product details, to see
which demonstrated policy is in place for what
you bought. Intel would be "consistent" across
the product range, because "that's how they are".
They developed a technical opinion, and they're
sticking by it.

With an Intel, you monitor "life remaining". Or else.

The middle policy is nice, because the OS croaking on you,
is an immediate hint you've hit the limit, and you can
just clone over using a Macrium CD. Or, if you feel weak
in the knees, do a "dd" from Linux, to be absolutely safe.
That's so you don't lose anything, due to the SSD being
asked to do something it cannot do. You want a quiescent
cloning environment, once you cannot write any more.
The backup software should not be attempting to write
to the source (a backup software could "crash" if it
cannot write its flag file to the source drive).

Paul

On 2023-11-09 18:45, Paul wrote:
> On 11/8/2023 1:30 PM, Stan Brown wrote:
>
>> "What will happen when the lifetime expiration date comes?
>>
>> "... The peculiarity of SSDs is that after their lifetime is over,
>> the data does not disappear from them. They just become read-only,
>> you can read data from it, but you cannot modify it."
>>
>> If that's true, it's good news for SSD users.
>
> The statement about SSDs is wrong, and the author of that
> paragraph should be shot.
>
> Each manufacturer has a web page with some details on their
> policy.
>
> Intel SSDs, when every location has the 600 writes or whatever,
> the drives "brick" instantly. Neither reading nor writing work.
>
> This is why, if you own an Intel, your backup schedule should
> be changed to make daily backups. Because you're going to get
> run over by a steam roller... soon. If your monitoring application
> says "100 GB writes remaining", then you would switch to daily backups
> and expect some edits to get lost. It depends on your tolerance for
> surprises, as to how close you shave that. Clone over using Macrium
> and do it from the Macrium CD, not from the OS itself. I have watched
> people in years past, play roulette with that, but nobody recently
> seems to know about life-monitoring utils. I certainly don't
> have one set up, and I should know better. (I have some Intels,
> and I bought them knowing the policy. They're sitting in a pile :-)
> No daily driver lives on those.)
>
> These are the policies of the different manufacturers:
>
> No read, No write # Intel
>
> Can continue to read # Useful for lazy home users like me...
>
> Can continue to read *and* write! # Hmmm. Bad monkey, bad. The user will
> # get hit by a truck. Think how many
> # corrupted backups you would make, before
> # getting hit by the truck.

Do you know which brands do which?

....

--
Cheers,
Carlos E.R.

On Thu, 9 Nov 2023 11:23:31 -0500, Paul wrote:
> That's because USB flash drives don't have "health" as a concept.
> They die. You move on. Simple really :-/ This is why you *never*
> store your only copy of stuff on one.

Agreed. I never edit the copy of a file that's on the USB stick. And
if the USB stick fails -- provided that I know it's failed -- I toss
it and begin using another for my file copying between computers.

Just what happens when a USB flash drive dies? Do write operations
fail, with a status that Windows reports to the user?

And are already-written files still readable? (Though given my
workflow, that one's not much of a concern since the USB stick will
never contain the only copy of anything. But it would be nice to
know.)

--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

On 9 Nov 2023 12:57:07 GMT, Frank Slootweg wrote:
> Stan Brown <the_stan_brown@fastmail.fm> wrote:
> > [quoted text muted]
> > If that's true, it's good news for SSD users. Is it also true for
> > flash drives? My main concern is not replacing a flash drive, because
> > they're cheap; my concern is not knowing that I _need_ to replace it,
> > and having its contents be corrupt because writes failed.
>
> I am confused! How could the contents be corrupt without you knowing
> it?

Maybe this isn't possible -- I hope it's not. But my fear is that a
file will be written corruptly to a failing USB stick with no error
message, and then that corrupt file will be read and copied to my
other computer. I would love to know whether that's actually
possible, versus how it works for SSDs where we've been told that one
at end of life will refuse to write at all but won't just silently
corrupt the data in the course of writing.

(I know most people are in love with synching via the cloud. But it
seems to me that the same possibility exists there -- if there is a
glitch during transmission your cloud "backup" will be corrupt, and
when it is synched to the other device that will receive the
corrupted copy.)

--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

Stan Brown <the_stan_brown@fastmail.fm> wrote:
> On 9 Nov 2023 12:57:07 GMT, Frank Slootweg wrote:
> > Stan Brown <the_stan_brown@fastmail.fm> wrote:
> > > [quoted text muted]
> > > If that's true, it's good news for SSD users. Is it also true for
> > > flash drives? My main concern is not replacing a flash drive, because
> > > they're cheap; my concern is not knowing that I _need_ to replace it,
> > > and having its contents be corrupt because writes failed.
> >
> > I am confused! How could the contents be corrupt without you knowing
> > it?
>
> Maybe this isn't possible -- I hope it's not. But my fear is that a
> file will be written corruptly to a failing USB stick with no error
> message, and then that corrupt file will be read and copied to my
> other computer.

With proper software and proper procedures - i.e. checking for error
codes / error messages - such a silent corruption cannot happen.

The device reports success/failure to the driver, the driver reports
it to the OS system call, the system call reports it to the program and
the program reports it to the user.

*If* this was a problem, then you could never fully trust any kind of
file on any media - i.e. also not disk, tape, SSD, <whatever> - and
hence also not the backup of such files. Essentially all IT would cease
to exist/function.

Perhaps an example can help: My daily and weekly backups are
automatic/scheduled. Daily to USB memory-stick and NAS. Sometimes these
backups partly or fully fail (for example USB memory-stick not
(properly) inserted or network error to NAS), but the backup utility
always reports that failure and the cause.

> I would love to know whether that's actually
> possible, versus how it works for SSDs where we've been told that one
> at end of life will refuse to write at all but won't just silently
> corrupt the data in the course of writing.

As I mentioned before, the only USB memory-stick failure I had, was a
total failure, i.e. the system didn't even recognize the stick anymore.

> (I know most people are in love with synching via the cloud. But it
> seems to me that the same possibility exists there -- if there is a
> glitch during transmission your cloud "backup" will be corrupt, and
> when it is synched to the other device that will receive the
> corrupted copy.)

Yes, it's the same. The cloud "backup" could be corrupt, but it would
not be a silent corruption, you will get an error message/code.

On 2023-11-09 20:51, Stan Brown wrote:
> On Thu, 9 Nov 2023 11:23:31 -0500, Paul wrote:
>> That's because USB flash drives don't have "health" as a concept.
>> They die. You move on. Simple really :-/ This is why you *never*
>> store your only copy of stuff on one.
>
> Agreed. I never edit the copy of a file that's on the USB stick. And
> if the USB stick fails -- provided that I know it's failed -- I toss
> it and begin using another for my file copying between computers.
>
> Just what happens when a USB flash drive dies? Do write operations
> fail, with a status that Windows reports to the user?
>
> And are already-written files still readable? (Though given my
> workflow, that one's not much of a concern since the USB stick will
> never contain the only copy of anything. But it would be nice to
> know.)

I have one failed stick. Some of the files are corrupt or unreadable,
IIRC. I don't have it here for verification and telling you, but maybe
in a month ask me again.

--
Cheers,
Carlos E.R.

On 11/9/2023 2:51 PM, Stan Brown wrote:
> On Thu, 9 Nov 2023 11:23:31 -0500, Paul wrote:
>> That's because USB flash drives don't have "health" as a concept.
>> They die. You move on. Simple really :-/ This is why you *never*
>> store your only copy of stuff on one.
>
> Agreed. I never edit the copy of a file that's on the USB stick. And
> if the USB stick fails -- provided that I know it's failed -- I toss
> it and begin using another for my file copying between computers.
>
> Just what happens when a USB flash drive dies? Do write operations
> fail, with a status that Windows reports to the user?
>
> And are already-written files still readable? (Though given my
> workflow, that one's not much of a concern since the USB stick will
> never contain the only copy of anything. But it would be nice to
> know.)
>

I just plugged in a defective one, the LED flashed three times,
and the OS did not respond at all. Best guess is, no endpoints
formed, and that is the first step in recognition. The OS has the
option of course, of forming endpoints, probing the device,
not liking what it sees, then removing the endpoints.

A dead one does not seem to respond. I cannot transfer anything,
unless there is some evidence of an entry in Device Manager
(in the namespace).

Read or write might drop to 1.5MB/sec or so (on a USB3 stick),
before it croaks. My guess is, it is using substitute blocks
at that point (replaces defective block with good block).

I think on one occasion, I tried to copy off the data, and it could
not finish. Some of the data came off, but not all of it.

You might also notice, on USB flash sticks with two flash packages
(one on each side of the PCB), the upper half or the lower half,
can have a reasonable transfer rate, while the other half is
in trouble. Devices with just one flash chip, it goes south
and that's it. It does not seem to take long, to go from 1.5MB/sec
to zero. You might have ten minutes. And just about anything you
try, just makes things worse. Until it is dead.

I guess you could say the device is taunting you :-)

Considering it has no rotating media, it could put on a better
show. It could be that it is sparing out everything it touches.
That is just a wild guess at the activity. In some ways, it
almost looks like a firmware bug. What it should do instead,
is go read-only and stop futzing with the spares. I don't even
think these USB sticks can respond with CRC error (for a data
block which is uncorrectable).

Paul

On 11/9/2023 2:12 PM, Carlos E. R. wrote:

>> These are the policies of the different manufacturers:
>>
>>     No read, No write                    # Intel
>>
>>     Can continue to read                 # Useful for lazy home users like me...
>>
>>     Can continue to read *and* write!    # Hmmm. Bad monkey, bad. The user will
>>                                          # get hit by a truck. Think how many
>>                                          # corrupted backups you would make, before
>>                                          # getting hit by the truck.
>
> Do you know which brands do which?

I would have to look them up.

All I have taken to heart, is the policy on my
tiny pile of aluminum Intels.

https://techreport.com/review/the-ssd-endurance-experiment-theyre-all-dead/

"Intel’s 335 Series failed much earlier, though to be fair, it pulled the trigger itself.
The drive’s media wear indicator ran out shortly after 700TB, signaling that the NAND’s
write tolerance had been exceeded. Intel doesn’t have confidence in the drive at that
point, so the 335 Series is designed to shift into read-only mode and then to brick itself
when the power is cycled. Despite suffering

just one reallocated sector, <=== lifespan calc was conservative

our sample dutifully followed the script. Data was accessible until a reboot prompted the
drive to swallow its virtual cyanide pill."

So you could get some data off, if you knew precisely what to do and didn't cycle the power
or something. I simply assume, even if I was there, I would probably panic and
do something stupid and not get my data.

If you read that article, you can see some of the nuanced responses.

The SSD industry started out with dodgy firmware. It was Intel
that re-wrote something it was given, being a bit shocked about
what was in the existing firmware.

That TechReport article was not written in 2023. It was written
quite a while ago. There were several articles, and the article above
in 2015, was the summary when the test was over.

https://www.tomshardware.com/reviews/intel-ssd-600p-nvme-endurance-testing,4826.html

"The last time a reputable website measured endurance, the Intel SSD 335 Series went into a
self-destruct mode. When asked at an Intel flash-focused event in Folsom, California, a
company representative stated the action was by design. Intel designs all consumer SSDs
to go offline in an effort to preserve data after the drive exhausts the spare area
dedicated to replacing failed cells." [Not true, based on the "one reallocated sector"]

Google isn't giving me anything else. Not at the moment.

Paul

On 2023-11-10 00:03, Paul wrote:
> On 11/9/2023 2:12 PM, Carlos E. R. wrote:
>
>>> These are the policies of the different manufacturers:
>>>
>>>     No read, No write                    # Intel
>>>
>>>     Can continue to read                 # Useful for lazy home users like me...
>>>
>>>     Can continue to read *and* write!    # Hmmm. Bad monkey, bad. The user will
>>>                                          # get hit by a truck. Think how many
>>>                                          # corrupted backups you would make, before
>>>                                          # getting hit by the truck.
>>
>> Do you know which brands do which?
>
> I would have to look them up.
>
> All I have taken to heart, is the policy on my
> tiny pile of aluminum Intels.
>
> https://techreport.com/review/the-ssd-endurance-experiment-theyre-all-dead/
>
> "Intel’s 335 Series failed much earlier, though to be fair, it pulled the trigger itself.
> The drive’s media wear indicator ran out shortly after 700TB, signaling that the NAND’s
> write tolerance had been exceeded. Intel doesn’t have confidence in the drive at that
> point, so the 335 Series is designed to shift into read-only mode and then to brick itself
> when the power is cycled. Despite suffering

Wow. So there is no chance to reboot to rescue system and clone the
disk, an operation that can not be done if the system itself is in the
SSD. And the OS will surely crash, so there is no chance at all for
cloning before it bricks.

Remind me to never buy Intel SSDs.

>
> just one reallocated sector, <=== lifespan calc was conservative
>
> our sample dutifully followed the script. Data was accessible until a reboot prompted the
> drive to swallow its virtual cyanide pill."
>
> So you could get some data off, if you knew precisely what to do and didn't cycle the power
> or something. I simply assume, even if I was there, I would probably panic and
> do something stupid and not get my data.
>
> If you read that article, you can see some of the nuanced responses.
>
> The SSD industry started out with dodgy firmware. It was Intel
> that re-wrote something it was given, being a bit shocked about
> what was in the existing firmware.
>
> That TechReport article was not written in 2023. It was written
> quite a while ago. There were several articles, and the article above
> in 2015, was the summary when the test was over.
>
> https://www.tomshardware.com/reviews/intel-ssd-600p-nvme-endurance-testing,4826.html
>
> "The last time a reputable website measured endurance, the Intel SSD 335 Series went into a
> self-destruct mode. When asked at an Intel flash-focused event in Folsom, California, a
> company representative stated the action was by design. Intel designs all consumer SSDs
> to go offline in an effort to preserve data after the drive exhausts the spare area
> dedicated to replacing failed cells." [Not true, based on the "one reallocated sector"]

Going offline to preserve data? That's not the same as bricking, the
data is no longer accessible. That's not "preserving", unless they want
to take the drive to an Intel place to have an expensive recovery operation.

>
> Google isn't giving me anything else. Not at the moment.

Thanks anyway.

--
Cheers,
Carlos E.R.

On 11/9/2023 3:24 PM, Frank Slootweg wrote:
> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>> On 9 Nov 2023 12:57:07 GMT, Frank Slootweg wrote:
>>> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>>>> [quoted text muted]
>>>> If that's true, it's good news for SSD users. Is it also true for
>>>> flash drives? My main concern is not replacing a flash drive, because
>>>> they're cheap; my concern is not knowing that I _need_ to replace it,
>>>> and having its contents be corrupt because writes failed.
>>>
>>> I am confused! How could the contents be corrupt without you knowing
>>> it?
>>
>> Maybe this isn't possible -- I hope it's not. But my fear is that a
>> file will be written corruptly to a failing USB stick with no error
>> message, and then that corrupt file will be read and copied to my
>> other computer.
>
> With proper software and proper procedures - i.e. checking for error
> codes / error messages - such a silent corruption cannot happen.
>
> The device reports success/failure to the driver, the driver reports
> it to the OS system call, the system call reports it to the program and
> the program reports it to the user.
>
> *If* this was a problem, then you could never fully trust any kind of
> file on any media - i.e. also not disk, tape, SSD, <whatever> - and
> hence also not the backup of such files. Essentially all IT would cease
> to exist/function.
>
> Perhaps an example can help: My daily and weekly backups are
> automatic/scheduled. Daily to USB memory-stick and NAS. Sometimes these
> backups partly or fully fail (for example USB memory-stick not
> (properly) inserted or network error to NAS), but the backup utility
> always reports that failure and the cause.
>
>> I would love to know whether that's actually
>> possible, versus how it works for SSDs where we've been told that one
>> at end of life will refuse to write at all but won't just silently
>> corrupt the data in the course of writing.
>
> As I mentioned before, the only USB memory-stick failure I had, was a
> total failure, i.e. the system didn't even recognize the stick anymore.
>
>> (I know most people are in love with synching via the cloud. But it
>> seems to me that the same possibility exists there -- if there is a
>> glitch during transmission your cloud "backup" will be corrupt, and
>> when it is synched to the other device that will receive the
>> corrupted copy.)
>
> Yes, it's the same. The cloud "backup" could be corrupt, but it would
> not be a silent corruption, you will get an error message/code.
>

You could generate a hash, at the source point, and check the hash on
the return trip.

You could upload three copies of the file

file itself
file contained inside a TAR file (which is not compressed)
file contained inside a ZIP file (set compression level to "Store")

and use majority voter logic on the return trip. This would
be in lieu of using PAR2 for enhanced recovery.

TCP/IP is a "reliable" three packet protocol (unlike UDP), the Ethernet
packets are protected by CRC32. Retransmission is requested, on an error.
This enhances the noise floor (independent bit errors) from 10^-6 to around 10^-9.
And those numbers are for shared media (coax and vampire taps), rather
than the point to point wiring used today. SATA interfaces are protected
this way as well (retransmission on packetized data check). There is a counter
in SMART for SATA cable errors, and that counter never resets (it's cumulative
over the lifetime of the drive).

Systematic errors are how we do things today. Such as bugged Ethernet MAC
controllers injecting 32 bits of junk (DMA bug), or network level protocols
incorrectly terminating object transfer and both ends of the transaction
"agreeing" everything is OK.

You could see the same thing with my Macrium backups. I randomly (when bored),
select a backup file .mrimg from a backup device, and run a Verify on it.
And one day, I found several backups to be corrupted. And that was
traceable, to bad RAM on the computer being backed up. Verify on Macrium
does *not* detect all systematic hardware issues, but in this case,
it just happened the bad RAM was somewhere that the problem could
be detected via Verify. Verify being nothing more than a hash check
done by the tool. I could equally have generated hashes for the entire
C: being backed up (hashdeep64, md5deep), as a way of verifying content later.
But that is hard to do properly (I consider NTFS now, to be opaque, which
is a bad rating for a file system to have).

Paul

On 2023-11-10 08:56, Paul wrote:
> On 11/9/2023 3:24 PM, Frank Slootweg wrote:
>> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>>> On 9 Nov 2023 12:57:07 GMT, Frank Slootweg wrote:
>>>> Stan Brown <the_stan_brown@fastmail.fm> wrote:
>>>>> [quoted text muted]
>>>>> If that's true, it's good news for SSD users. Is it also true for
>>>>> flash drives? My main concern is not replacing a flash drive, because
>>>>> they're cheap; my concern is not knowing that I _need_ to replace it,
>>>>> and having its contents be corrupt because writes failed.
>>>>
>>>> I am confused! How could the contents be corrupt without you knowing
>>>> it?
>>>
>>> Maybe this isn't possible -- I hope it's not. But my fear is that a
>>> file will be written corruptly to a failing USB stick with no error
>>> message, and then that corrupt file will be read and copied to my
>>> other computer.
>>
>> With proper software and proper procedures - i.e. checking for error
>> codes / error messages - such a silent corruption cannot happen.
>>
>> The device reports success/failure to the driver, the driver reports
>> it to the OS system call, the system call reports it to the program and
>> the program reports it to the user.
>>
>> *If* this was a problem, then you could never fully trust any kind of
>> file on any media - i.e. also not disk, tape, SSD, <whatever> - and
>> hence also not the backup of such files. Essentially all IT would cease
>> to exist/function.
>>
>> Perhaps an example can help: My daily and weekly backups are
>> automatic/scheduled. Daily to USB memory-stick and NAS. Sometimes these
>> backups partly or fully fail (for example USB memory-stick not
>> (properly) inserted or network error to NAS), but the backup utility
>> always reports that failure and the cause.
>>
>>> I would love to know whether that's actually
>>> possible, versus how it works for SSDs where we've been told that one
>>> at end of life will refuse to write at all but won't just silently
>>> corrupt the data in the course of writing.
>>
>> As I mentioned before, the only USB memory-stick failure I had, was a
>> total failure, i.e. the system didn't even recognize the stick anymore.
>>
>>> (I know most people are in love with synching via the cloud. But it
>>> seems to me that the same possibility exists there -- if there is a
>>> glitch during transmission your cloud "backup" will be corrupt, and
>>> when it is synched to the other device that will receive the
>>> corrupted copy.)
>>
>> Yes, it's the same. The cloud "backup" could be corrupt, but it would
>> not be a silent corruption, you will get an error message/code.
>>
>
> You could generate a hash, at the source point, and check the hash on
> the return trip.
>
> You could upload three copies of the file
>
> file itself
> file contained inside a TAR file (which is not compressed)
> file contained inside a ZIP file (set compression level to "Store")
>
> and use majority voter logic on the return trip. This would
> be in lieu of using PAR2 for enhanced recovery.

As we are on a Windows group, I would then recommend using RAR with
recovery data added, something it can do automatically since the 90's.

It is the only compressor that I know has this technology and is tested
for a long time. Maybe there is another, I would like a Linux one with
it (rar doesn't support Linux attributes and features like sparse files
or links).

I would also recommend against compressed tar like tar.gz, because a
single changed byte in the final archive, and the whole is completely
unrecoverable.

Heh, that could be a way for secret transmissions. Send a corrupted tgz
file, then by different channel transmit the corrections to the file :-D

....

--
Cheers,
Carlos E.R.