On 4/15/2024 10:13 AM, Don Y wrote:
> Is there a general rule of thumb for signalling the likelihood of
> an "imminent" (for some value of "imminent") hardware failure?

This reminded me of some past efforts in this area. It was never
demonstrated to me (given ample opportunity) that this technology
actually worked on intermittently failing hardware I had, so be cautious
in applying it in any future endeavors.

https://radlab.cs.berkeley.edu/classes/cs444a/KGross_CSTH_Stanford.pdf

On 4/18/2024 10:18 AM, Buzz McCool wrote:
> On 4/15/2024 10:13 AM, Don Y wrote:
>> Is there a general rule of thumb for signalling the likelihood of
>> an "imminent" (for some value of "imminent") hardware failure?
>
> This reminded me of some past efforts in this area. It was never demonstrated
> to me (given ample opportunity) that this technology actually worked on
> intermittently failing hardware I had, so be cautious in applying it in any
> future endeavors.

Intermittent failures are the bane of all designers. Until something
is reliably observable, trying to address the problem is largely
wack-a-mole.

> https://radlab.cs.berkeley.edu/classes/cs444a/KGross_CSTH_Stanford.pdf

Thanks for that. I didn't find it in my collection so it's addition will
be welcome.

Sun has historically been aggressive in trying to increase availability,
especially on big iron. In fact, such a "prediction" led me to discard
a small server, yesterday (no time to dick with failing hardware!).

I am now seeing similar features in Dell servers. But, the *actual*
implementation details are always shrouded in mystery.

But, it is obvious (for "always on" systems) that there are many things
that can silently fail that will only manifest some time later -- if at
all and possibly complicated by other failures that may have been
precipitated by it.

Sorting out WHAT to monitor is the tricky part. Then, having the
ability to watch for trends can give you an inkling that something is
headed in the wrong direction -- before it actually exceeds some
baked in "hard limit".

E.g., only the memory that you actively REFERENCE in a product is ever
checked for errors! Bit rot may not be detected until some time after it
has occurred -- when you eventually access that memory (and the memory
controller throws an error).

This is paradoxically amusing; code to HANDLE errors is likely the least
accessed code in a product. So, bit rot IN that code is more likely
to go unnoticed -- until it is referenced (by some error condition)
and the error event complicated by the attendant error in the handler!
The more reliable your code (fewer faults), the more uncertain you
will be of the handlers' abilities to address faults that DO manifest!

The same applies to secondary storage media. How will you know if
some-rarely-accessed-file is intact and ready to be referenced
WHEN NEEDED -- if you aren't doing patrol reads/scrubbing to
verify that it is intact, NOW?

[One common flaw with RAID implementations and naive reliance on that
technology]

On Thu, 18 Apr 2024 15:05:07 -0700, Don Y wrote:

> On 4/18/2024 10:18 AM, Buzz McCool wrote:
>> On 4/15/2024 10:13 AM, Don Y wrote:
>>> Is there a general rule of thumb for signalling the likelihood of an
>>> "imminent" (for some value of "imminent") hardware failure?
>>
>> This reminded me of some past efforts in this area. It was never
>> demonstrated to me (given ample opportunity) that this technology
>> actually worked on intermittently failing hardware I had, so be
>> cautious in applying it in any future endeavors.
>
> Intermittent failures are the bane of all designers. Until something is
> reliably observable, trying to address the problem is largely
> wack-a-mole.
>
>> https://radlab.cs.berkeley.edu/classes/cs444a/KGross_CSTH_Stanford.pdf
>
> Thanks for that. I didn't find it in my collection so it's addition
> will be welcome.
>
> Sun has historically been aggressive in trying to increase availability,
> especially on big iron. In fact, such a "prediction" led me to discard
> a small server, yesterday (no time to dick with failing hardware!).
>
> I am now seeing similar features in Dell servers. But, the *actual*
> implementation details are always shrouded in mystery.
>
> But, it is obvious (for "always on" systems) that there are many things
> that can silently fail that will only manifest some time later -- if at
> all and possibly complicated by other failures that may have been
> precipitated by it.
>
> Sorting out WHAT to monitor is the tricky part. Then, having the
> ability to watch for trends can give you an inkling that something is
> headed in the wrong direction -- before it actually exceeds some baked
> in "hard limit".
>
> E.g., only the memory that you actively REFERENCE in a product is ever
> checked for errors! Bit rot may not be detected until some time after
> it has occurred -- when you eventually access that memory (and the
> memory controller throws an error).
>
> This is paradoxically amusing; code to HANDLE errors is likely the least
> accessed code in a product. So, bit rot IN that code is more likely to
> go unnoticed -- until it is referenced (by some error condition)
> and the error event complicated by the attendant error in the handler!
> The more reliable your code (fewer faults), the more uncertain you will
> be of the handlers' abilities to address faults that DO manifest!
>
> The same applies to secondary storage media. How will you know if
> some-rarely-accessed-file is intact and ready to be referenced WHEN
> NEEDED -- if you aren't doing patrol reads/scrubbing to verify that it
> is intact, NOW?
>
> [One common flaw with RAID implementations and naive reliance on that
> technology]

RAID, even with backups, is unsuited to high reliability storage of large
databases. Distributed storage can be of much higher reliability:

https://telnyx.com/resources/what-is-distributed-storage

<https://towardsdatascience.com/introduction-to-distributed-data-
storage-2ee03e02a11d>

This requires successful retrieval of any n of m data files, normally from
different locations, where n can be arbitrarily smaller than m depending
on your needs. Overkill for small databases but required for high
reliability storage of very large databases.

On 4/18/2024 6:27 PM, Glen Walpert wrote:
> On Thu, 18 Apr 2024 15:05:07 -0700, Don Y wrote:
>
>> The same applies to secondary storage media. How will you know if
>> some-rarely-accessed-file is intact and ready to be referenced WHEN
>> NEEDED -- if you aren't doing patrol reads/scrubbing to verify that it
>> is intact, NOW?
>>
>> [One common flaw with RAID implementations and naive reliance on that
>> technology]
>
> RAID, even with backups, is unsuited to high reliability storage of large
> databases. Distributed storage can be of much higher reliability:
>
> https://telnyx.com/resources/what-is-distributed-storage
>
> <https://towardsdatascience.com/introduction-to-distributed-data-
> storage-2ee03e02a11d>
>
> This requires successful retrieval of any n of m data files, normally from
> different locations, where n can be arbitrarily smaller than m depending
> on your needs. Overkill for small databases but required for high
> reliability storage of very large databases.

This is effectively how I maintain my archive. Except that the
media are all "offline", requiring a human operator (me) to
fetch the required volumes in order to locate the desired files.

Unlike mirroring (or other RAID technologies), my scheme places
no constraints as to the "containers" holding the data. E.g.,

DISK43 /somewhere/in/filesystem/ fileofinterest
DISK21 >some>other>place anothernameforfile
CDROM77 \yet\another\place archive.type /where/in/archive foo

Can all yield the same "content" (as verified by their prestored signatures).
Knowing the hash of each object means you can verify its contents from a
single instance instead of looking for confirmation via other instance(s)

[Hashes take up considerably less space than a duplicate copy would]

This makes it easy to create multiple instances of particular "content"
without imposing constraints on how it is named, stored, located, etc.

I.e., pull a disk out of a system, catalog its contents, slap an adhesive
label on it (to be human-readable) and add it to your store.

(If I could mount all of the volumes -- because I wouldn't know which volume
might be needed -- then access wouldn't require a human operator, regardless
of where the volumes were actually mounted or the peculiarities of the
systems on which they are mounted! But, you can have a daemon that watches to
see WHICH volumes are presently accessible and have it initiate a patrol
read of their contents while the media are being accessed "for whatever OTHER
reason" -- and track the time/date of last "verification" so you know which
volumes haven't been checked, recently)

The inconvenience of requiring human intervention is offset by the lack of
wear on the media (as well as BTUs to keep it accessible) and the ease of
creating NEW content/copies. NOT useful for data that needs to be accessed
frequently but excellent for "archives"/repositories -- that can be mounted,
accessed and DUPLICATED to online/nearline storage for normal use.

On Thu, 18 Apr 2024 15:05:07 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>On 4/18/2024 10:18 AM, Buzz McCool wrote:
>> On 4/15/2024 10:13 AM, Don Y wrote:
>>> Is there a general rule of thumb for signalling the likelihood of
>>> an "imminent" (for some value of "imminent") hardware failure?
>>
>> This reminded me of some past efforts in this area. It was never demonstrated
>> to me (given ample opportunity) that this technology actually worked on
>> intermittently failing hardware I had, so be cautious in applying it in any
>> future endeavors.
>
>Intermittent failures are the bane of all designers. Until something
>is reliably observable, trying to address the problem is largely
>wack-a-mole.
>

The problem I have with troubleshooting intermittent failures is that
they are only intermittend sometimes.

>> https://radlab.cs.berkeley.edu/classes/cs444a/KGross_CSTH_Stanford.pdf
>
>Thanks for that. I didn't find it in my collection so it's addition will
>be welcome.

Yes, neat paper.

boB

>
>Sun has historically been aggressive in trying to increase availability,
>especially on big iron. In fact, such a "prediction" led me to discard
>a small server, yesterday (no time to dick with failing hardware!).
>
>I am now seeing similar features in Dell servers. But, the *actual*
>implementation details are always shrouded in mystery.
>
>But, it is obvious (for "always on" systems) that there are many things
>that can silently fail that will only manifest some time later -- if at
>all and possibly complicated by other failures that may have been
>precipitated by it.
>
>Sorting out WHAT to monitor is the tricky part. Then, having the
>ability to watch for trends can give you an inkling that something is
>headed in the wrong direction -- before it actually exceeds some
>baked in "hard limit".
>
>E.g., only the memory that you actively REFERENCE in a product is ever
>checked for errors! Bit rot may not be detected until some time after it
>has occurred -- when you eventually access that memory (and the memory
>controller throws an error).
>
>This is paradoxically amusing; code to HANDLE errors is likely the least
>accessed code in a product. So, bit rot IN that code is more likely
>to go unnoticed -- until it is referenced (by some error condition)
>and the error event complicated by the attendant error in the handler!
>The more reliable your code (fewer faults), the more uncertain you
>will be of the handlers' abilities to address faults that DO manifest!
>
>The same applies to secondary storage media. How will you know if
>some-rarely-accessed-file is intact and ready to be referenced
>WHEN NEEDED -- if you aren't doing patrol reads/scrubbing to
>verify that it is intact, NOW?
>
>[One common flaw with RAID implementations and naive reliance on that
>technology]

On 4/19/2024 11:16 AM, boB wrote:
>> Intermittent failures are the bane of all designers. Until something
>> is reliably observable, trying to address the problem is largely
>> wack-a-mole.
>
> The problem I have with troubleshooting intermittent failures is that
> they are only intermittend sometimes.

My pet peeve is folks (developers) who OBSERVE FIRST HAND a particular
failure/fault but, because reproducing it is "hard", just pretend it
never happened! Really? Do you think the circuit/code is self-healing???

You're going to "bless" a product that you, personally, know has a fault...

On Fri, 19 Apr 2024 12:10:22 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>On 4/19/2024 11:16 AM, boB wrote:
>>> Intermittent failures are the bane of all designers. Until something
>>> is reliably observable, trying to address the problem is largely
>>> wack-a-mole.
>>
>> The problem I have with troubleshooting intermittent failures is that
>> they are only intermittend sometimes.
>
>My pet peeve is folks (developers) who OBSERVE FIRST HAND a particular
>failure/fault but, because reproducing it is "hard", just pretend it
>never happened! Really? Do you think the circuit/code is self-healing???
>
>You're going to "bless" a product that you, personally, know has a fault...
>

Yes, it may be hard to replicate but you just have to try and try
again sometimes. Or create something that exercises the unit or
software to make it happen and automatically catch it in the act.

I don't care to have to do that very often. When I do, I just try to
make it a challenge.

boB

On 4/21/2024 12:37 PM, boB wrote:
> On Fri, 19 Apr 2024 12:10:22 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 4/19/2024 11:16 AM, boB wrote:
>>>> Intermittent failures are the bane of all designers. Until something
>>>> is reliably observable, trying to address the problem is largely
>>>> wack-a-mole.
>>>
>>> The problem I have with troubleshooting intermittent failures is that
>>> they are only intermittend sometimes.
>>
>> My pet peeve is folks (developers) who OBSERVE FIRST HAND a particular
>> failure/fault but, because reproducing it is "hard", just pretend it
>> never happened! Really? Do you think the circuit/code is self-healing???
>>
>> You're going to "bless" a product that you, personally, know has a fault...
>
> Yes, it may be hard to replicate but you just have to try and try
> again sometimes. Or create something that exercises the unit or
> software to make it happen and automatically catch it in the act.

I think this was the perfect application for Google Glass! It
seems a given that whenever you stumble on one of these "events",
you aren't concentrating on how you GOT there; you didn't expect
the failure to manifest so weren't keeping track of your actions.

If, instead, you could "rewind" a recording of everything that you
had done up to that point, it would likely go a long way towards
helping you recreate the problem!

When you get a "report" of someone encountering some anomalous
behavior, its easy to shrug it off because they are often very
imprecise in describing their actions; details (crucial) are
often missing or a subject of "fantasy". Is the person sure
that the machine wasn't doing exactly what it SHOULD in that
SPECIFIC situation??

OTOH, when it happens to YOU, you know that the report isn't
a fluke. But, are just as weak on the details as those third-party
reporters!

> I don't care to have to do that very often. When I do, I just try to
> make it a challenge.

Being able to break a design into small pieces goes a long way to
improving its quality. Taking "contractual design" to its extreme
lets you build small, validatable modules that stand a greater
chance of working in concert.

Unfortunately, few have the discipline for such detail, hoping,
instead, to test bigger units (if they do ANY formal testing at all!)

Think of how little formal testing goes into a hardware design.
Aside from imposing inputs and outputs at their extremes, what
*really* happens before a design is released to manufacturing?
(I haven't seen a firm that does a rigorous shake-n-bake in
more than 40 years!)

And, how much less goes into software -- where it is relatively easy to
build test scaffolding and implement regression tests to ensure new
releases don't reintroduce old bugs...

When the emphasis (Management) is getting product out the door,
it's easy to see engineering (and manufacturing) disciplines suffer.

:<