Is there a general rule of thumb for signalling the likelihood of
an "imminent" (for some value of "imminent") hardware failure?

I suspect most would involve *relative* changes that would be
suggestive of changing conditions in the components (and not
directly related to environmental influences).

So, perhaps, a good strategy is to just "watch" everything and
notice the sorts of changes you "typically" encounter in the hope
that something of greater magnitude would be a harbinger...

Don Y <blockedofcourse@foo.invalid> Wrote in message:r
> Is there a general rule of thumb for signalling the likelihood ofan "imminent" (for some value of "imminent") hardware failure?I suspect most would involve *relative* changes that would besuggestive of changing conditions in the components (and notdirectly related to environmental influences).So, perhaps, a good strategy is to just "watch" everything andnotice the sorts of changes you "typically" encounter in the hopethat something of greater magnitude would be a harbinger...

Current and voltages outside of normal operation?

Cheers
--

----Android NewsGroup Reader----
https://piaohong.s3-us-west-2.amazonaws.com/usenet/index.html

On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>Is there a general rule of thumb for signalling the likelihood of
>an "imminent" (for some value of "imminent") hardware failure?
>
>I suspect most would involve *relative* changes that would be
>suggestive of changing conditions in the components (and not
>directly related to environmental influences).
>
>So, perhaps, a good strategy is to just "watch" everything and
>notice the sorts of changes you "typically" encounter in the hope
>that something of greater magnitude would be a harbinger...

Checking temperatures is good. An overload or a fan failure can be bad
news.

We put temp sensors on most products. Some parts, like ADCs and FPGAs,
have free built-in temp sensors.

I have tried various ideas to put an air flow sensor on boards, but so
far none have worked very well. We do check fan tachs to be sure they
are still spinning.

Blocking air flow generally makes fan speed *increase*.

On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>Is there a general rule of thumb for signalling the likelihood of
>an "imminent" (for some value of "imminent") hardware failure?
>
>I suspect most would involve *relative* changes that would be
>suggestive of changing conditions in the components (and not
>directly related to environmental influences).
>
>So, perhaps, a good strategy is to just "watch" everything and
>notice the sorts of changes you "typically" encounter in the hope
>that something of greater magnitude would be a harbinger...

There is a standard approach that may work: Measure the level and
trend of very low frequency (around a tenth of a Hertz) flicker noise.
When connections (perhaps within a package) start to fail, the flicker
level rises. The actual frequency monitored isn't all that critical.

Joe Gwinn

On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
wrote:

>On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
><blockedofcourse@foo.invalid> wrote:
>
>>Is there a general rule of thumb for signalling the likelihood of
>>an "imminent" (for some value of "imminent") hardware failure?
>>
>>I suspect most would involve *relative* changes that would be
>>suggestive of changing conditions in the components (and not
>>directly related to environmental influences).
>>
>>So, perhaps, a good strategy is to just "watch" everything and
>>notice the sorts of changes you "typically" encounter in the hope
>>that something of greater magnitude would be a harbinger...
>
>There is a standard approach that may work: Measure the level and
>trend of very low frequency (around a tenth of a Hertz) flicker noise.
>When connections (perhaps within a package) start to fail, the flicker
>level rises. The actual frequency monitored isn't all that critical.
>
>Joe Gwinn

Do connections "start to fail" ?

I don't think I've ever owned a piece of electronic equipment that
warned me of an impending failure.

Cars do, for some failure modes, like low oil level.

Don, what does the thing do?

"Don Y" <blockedofcourse@foo.invalid> wrote in message
news:uvjn74$d54b$1@dont-email.me...
> Is there a general rule of thumb for signalling the likelihood of
> an "imminent" (for some value of "imminent") hardware failure?

My conclusion would be no.
Some of my reasons are given below.

It always puzzled me how HAL could know that the AE-35 would fail in the
near future, but maybe HAL had a motive for lying.

Back in that era I was doing a lot of repair work when I should have been
doing my homework.
So I knew that there were many unrelated kinds of hardware failure.

A component could fail suddenly, such as a short circuit diode, and
everything would work fine after replacing it.
The cause could perhaps have been a manufacturing defect, such as
insufficient cooling due to poor quality assembly, but the exact real cause
would never be known.

A component could fail suddenly as a side effect of another failure.
One short circuit output transistor and several other components could also
burn up.

A component could fail slowly and only become apparent when it got to the
stage of causing an audible or visible effect.
It would often be easy to locate the dried up electrolytic due to it having
already let go of some of its contents.

So I concluded that if I wanted to be sure that I could always watch my
favourite TV show, we would have to have at least two TVs in the house.

If it's not possible to have the equivalent of two TVs then you will want to
be in a position to get the existing TV repaired or replaced as quicky as
possible.

My home wireless Internet system doesn't care if one access point fails, and
I would not expect to be able to do anything to predict a time of failure.
Experience says a dead unit has power supply issues. Usually external but
could be internal.

I don't think it would be possible to "watch" everything because it's rare
that you can properly test a component while it's part of a working system.

These days I would expect to have fun with management asking for software to
be able to diagnose and report any hardware failure.
Not very easy if the power supply has died.

>
> I suspect most would involve *relative* changes that would be
> suggestive of changing conditions in the components (and not
> directly related to environmental influences).
>
> So, perhaps, a good strategy is to just "watch" everything and
> notice the sorts of changes you "typically" encounter in the hope
> that something of greater magnitude would be a harbinger...
>

On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:

>On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>wrote:
>
>>On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>><blockedofcourse@foo.invalid> wrote:
>>
>>>Is there a general rule of thumb for signalling the likelihood of
>>>an "imminent" (for some value of "imminent") hardware failure?
>>>
>>>I suspect most would involve *relative* changes that would be
>>>suggestive of changing conditions in the components (and not
>>>directly related to environmental influences).
>>>
>>>So, perhaps, a good strategy is to just "watch" everything and
>>>notice the sorts of changes you "typically" encounter in the hope
>>>that something of greater magnitude would be a harbinger...
>>
>>There is a standard approach that may work: Measure the level and
>>trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>When connections (perhaps within a package) start to fail, the flicker
>>level rises. The actual frequency monitored isn't all that critical.
>>
>>Joe Gwinn
>
>Do connections "start to fail" ?

Yes, they do, in things like vias. I went through a big drama where a
critical bit of radar logic circuitry would slowly go nuts.

It turned out that the copper plating on the walls of the vias was
suffering from low-cycle fatigue during temperature cycling and slowly
breaking, one little crack at a time, until it went open. If you
measured the resistance to parts per million (6.5 digit DMM), sampling
at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
also measure a copper line, and divide the via-chain resistance by the
no-via resistance, to correct for temperature changes.

The solution was to redesign the vias, mainly to increase the critical
volume of copper. And modern SMD designs have less and less copper
volume.

I bet precision resistors can also be measured this way.

>I don't think I've ever owned a piece of electronic equipment that
>warned me of an impending failure.

Onset of smoke emission is a common sign.

>Cars do, for some failure modes, like low oil level.

The industrial method for big stuff is accelerometers attached near
the bearings, and listen for excessive rotation-correlated (not
necessarily harmonic) noise.

>Don, what does the thing do?

Good question.

Joe Gwinn

On Mon, 15 Apr 2024 18:03:23 -0400, Joe Gwinn <joegwinn@comcast.net>
wrote:

>On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>
>>On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>>wrote:
>>
>>>On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>><blockedofcourse@foo.invalid> wrote:
>>>
>>>>Is there a general rule of thumb for signalling the likelihood of
>>>>an "imminent" (for some value of "imminent") hardware failure?
>>>>
>>>>I suspect most would involve *relative* changes that would be
>>>>suggestive of changing conditions in the components (and not
>>>>directly related to environmental influences).
>>>>
>>>>So, perhaps, a good strategy is to just "watch" everything and
>>>>notice the sorts of changes you "typically" encounter in the hope
>>>>that something of greater magnitude would be a harbinger...
>>>
>>>There is a standard approach that may work: Measure the level and
>>>trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>>When connections (perhaps within a package) start to fail, the flicker
>>>level rises. The actual frequency monitored isn't all that critical.
>>>
>>>Joe Gwinn
>>
>>Do connections "start to fail" ?
>
>Yes, they do, in things like vias. I went through a big drama where a
>critical bit of radar logic circuitry would slowly go nuts.
>
>It turned out that the copper plating on the walls of the vias was
>suffering from low-cycle fatigue during temperature cycling and slowly
>breaking, one little crack at a time, until it went open. If you
>measured the resistance to parts per million (6.5 digit DMM), sampling
>at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
>also measure a copper line, and divide the via-chain resistance by the
>no-via resistance, to correct for temperature changes.

But nobody is going to monitor every via on a PCB, even if it were
possible.

One could instrument a PCB fab test board, I guess. But DC tests would
be fine.

We have one board with over 4000 vias, but they are mostly in
parallel.

>
>The solution was to redesign the vias, mainly to increase the critical
>volume of copper. And modern SMD designs have less and less copper
>volume.
>
>I bet precision resistors can also be measured this way.
>
>
>>I don't think I've ever owned a piece of electronic equipment that
>>warned me of an impending failure.
>
>Onset of smoke emission is a common sign.
>
>
>>Cars do, for some failure modes, like low oil level.
>
>The industrial method for big stuff is accelerometers attached near
>the bearings, and listen for excessive rotation-correlated (not
>necessarily harmonic) noise.

Big ships that I've worked on have a long propeller shaft in the shaft
alley, a long tunnel where nobody often goes. They have magnetic shaft
runout sensors and shaft bearing temperature monitors.

They measure shaft torque and SHP too, from the shaft twist.

I liked hiding out in the shaft alley. It was private and cool, that
giant shaft slowly rotating.

Joe Gwinn <joegwinn@comcast.net> wrote:
> On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>
>> On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>> wrote:
>>
>>> On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>> <blockedofcourse@foo.invalid> wrote:
>>>
>>>> Is there a general rule of thumb for signalling the likelihood of
>>>> an "imminent" (for some value of "imminent") hardware failure?
>>>>
>>>> I suspect most would involve *relative* changes that would be
>>>> suggestive of changing conditions in the components (and not
>>>> directly related to environmental influences).
>>>>
>>>> So, perhaps, a good strategy is to just "watch" everything and
>>>> notice the sorts of changes you "typically" encounter in the hope
>>>> that something of greater magnitude would be a harbinger...
>>>
>>> There is a standard approach that may work: Measure the level and
>>> trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>> When connections (perhaps within a package) start to fail, the flicker
>>> level rises. The actual frequency monitored isn't all that critical.
>>>
>>> Joe Gwinn
>>
>> Do connections "start to fail" ?
>
> Yes, they do, in things like vias. I went through a big drama where a
> critical bit of radar logic circuitry would slowly go nuts.
>
> It turned out that the copper plating on the walls of the vias was
> suffering from low-cycle fatigue during temperature cycling and slowly
> breaking, one little crack at a time, until it went open. If you
> measured the resistance to parts per million (6.5 digit DMM), sampling
> at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
> also measure a copper line, and divide the via-chain resistance by the
> no-via resistance, to correct for temperature changes.
>
> The solution was to redesign the vias, mainly to increase the critical
> volume of copper. And modern SMD designs have less and less copper
> volume.
>
> I bet precision resistors can also be measured this way.
>
>
>> I don't think I've ever owned a piece of electronic equipment that
>> warned me of an impending failure.
>
> Onset of smoke emission is a common sign.
>
>
>> Cars do, for some failure modes, like low oil level.
>
> The industrial method for big stuff is accelerometers attached near
> the bearings, and listen for excessive rotation-correlated (not
> necessarily harmonic) noise.

There are a number of instruments available that look for metal particles
in the lubricating oil.

Cheers

Phil Hobbs
>
>

--
Dr Philip C D Hobbs Principal Consultant ElectroOptical Innovations LLC /
Hobbs ElectroOptics Optics, Electro-optics, Photonics, Analog Electronics

On 4/15/2024 10:32 AM, Martin Rid wrote:
> Don Y <blockedofcourse@foo.invalid> Wrote in message:r
>> Is there a general rule of thumb for signalling the likelihood ofan "imminent" (for some value of "imminent") hardware failure?I suspect most would involve *relative* changes that would besuggestive of changing conditions in the components (and notdirectly related to environmental influences).So, perhaps, a good strategy is to just "watch" everything andnotice the sorts of changes you "typically" encounter in the hopethat something of greater magnitude would be a harbinger...
>
> Current and voltages outside of normal operation?

I think "outside" is (often) likely indicative of
"something is (already) broken".

But, perhaps TRENDS in either/both can be predictive.

E.g., if a (sub)circuit has always been consuming X (which
is nominal for the design) and, over time, starts to consume
1.1X, is that suggestive that something is in the process of
failing?

Note that the goal is not to troubleshoot the particular design
or its components but, rather, act as an early warning that
maintenance may be required (or, that performance may not be
what you are expecting/have become accustomed to).

You can include mechanisms to verify outputs are what you
*intended* them to be (in case the output drivers have shit
the bed).

You can, also, do sanity checks that ensure values are never
what they SHOULDN'T be (this is commonly done within software
products -- if something "can't happen" then noticing that
it IS happening is a sure-fire indication that something
is broken!)

[Limit switches on mechanisms are there to ensure the impossible
is not possible -- like driving a mechanism beyond its extents]

And, where possible, notice second-hand effects of your actions
(e.g., if you switched on a load, you should see an increase
in supplied current).

But, again, these are more helpful in detecting FAILED items.

On Tue, 16 Apr 2024 00:51:03 -0000 (UTC), Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> wrote:

>Joe Gwinn <joegwinn@comcast.net> wrote:
>> On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>>
>>> On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>>> wrote:
>>>
>>>> On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>>> <blockedofcourse@foo.invalid> wrote:
>>>>
>>>>> Is there a general rule of thumb for signalling the likelihood of
>>>>> an "imminent" (for some value of "imminent") hardware failure?
>>>>>
>>>>> I suspect most would involve *relative* changes that would be
>>>>> suggestive of changing conditions in the components (and not
>>>>> directly related to environmental influences).
>>>>>
>>>>> So, perhaps, a good strategy is to just "watch" everything and
>>>>> notice the sorts of changes you "typically" encounter in the hope
>>>>> that something of greater magnitude would be a harbinger...
>>>>
>>>> There is a standard approach that may work: Measure the level and
>>>> trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>>> When connections (perhaps within a package) start to fail, the flicker
>>>> level rises. The actual frequency monitored isn't all that critical.
>>>>
>>>> Joe Gwinn
>>>
>>> Do connections "start to fail" ?
>>
>> Yes, they do, in things like vias. I went through a big drama where a
>> critical bit of radar logic circuitry would slowly go nuts.
>>
>> It turned out that the copper plating on the walls of the vias was
>> suffering from low-cycle fatigue during temperature cycling and slowly
>> breaking, one little crack at a time, until it went open. If you
>> measured the resistance to parts per million (6.5 digit DMM), sampling
>> at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
>> also measure a copper line, and divide the via-chain resistance by the
>> no-via resistance, to correct for temperature changes.
>>
>> The solution was to redesign the vias, mainly to increase the critical
>> volume of copper. And modern SMD designs have less and less copper
>> volume.
>>
>> I bet precision resistors can also be measured this way.
>>
>>
>>> I don't think I've ever owned a piece of electronic equipment that
>>> warned me of an impending failure.
>>
>> Onset of smoke emission is a common sign.
>>
>>
>>> Cars do, for some failure modes, like low oil level.
>>
>> The industrial method for big stuff is accelerometers attached near
>> the bearings, and listen for excessive rotation-correlated (not
>> necessarily harmonic) noise.
>
>There are a number of instruments available that look for metal particles
>in the lubricating oil.
>
>Cheers
>
>Phil Hobbs
>>
>>

And water. Some of our capacitor simulators include a parallel
resistance component.

One customer used to glue bits of metal onto a string and pull it
through the magnetic sensor. We did a simulator for that too.

Jet engines have magnetic eddy-current blade-tip sensors. For
efficiency, they want a tiny clearance between fan blades and the
casing, but not too tiny.

On 4/15/2024 1:32 PM, Edward Rawde wrote:
> "Don Y" <blockedofcourse@foo.invalid> wrote in message
> news:uvjn74$d54b$1@dont-email.me...
>> Is there a general rule of thumb for signalling the likelihood of
>> an "imminent" (for some value of "imminent") hardware failure?
>
> My conclusion would be no.
> Some of my reasons are given below.
>
> It always puzzled me how HAL could know that the AE-35 would fail in the
> near future, but maybe HAL had a motive for lying.

Why does your PC retry failed disk operations? If I ask the drive to give
me LBA 1234, shouldn't it ALWAYS give me LBA1234? Without any data corruption
(CRC error) AND within the normal access time limits defined by the location
of those magnetic domains on the rotating medium?

Why should it attempt to retry this MORE than once?

Now, if you knew your disk drive was repeatedly retrying operations,
would your confidence in it be unchanged from times when it did not
exhibit such behavior?

Assuming you have properly configured a EIA232 interface, why would you
ever get a parity error? (OVERRUN errors can be the result of an i/f
that is running too fast for the system on the receiving end) How would
you even KNOW this was happening?

I suspect everyone who has owned a DVD/CD drive has encountered a
"slow tray" as the mechanism aged. Or, a tray that wouldn't
open (of its own accord) as soon/quickly as it used to.

The controller COULD be watching this (cuz it knows when it
initiated the operation and there is an "end-of-stroke"
sensor available) and KNOW that the drive belt was stretching
to the point where it was impacting operation.

[And, that a stretched belt wasn't going to suddenly decide to
unstretch to fix the problem!]

> Back in that era I was doing a lot of repair work when I should have been
> doing my homework.
> So I knew that there were many unrelated kinds of hardware failure.

The goal isn't to predict ALL failures but, rather, to anticipate
LIKELY failures and treat them before they become an inconvenience
(or worse).

One morning, the (gas) furnace repeatedly tried to light as the
thermostat called for heat. Then, a few moments later, the
safeties would kick in and shut down the gas flow. This attracted my
attention as the LIT furnace should STAY LIT!

The furnace was too stupid to notice its behavior so would repeat
this cycle, endlessly.

I stepped in and overrode the thermostat to eliminate the call
for heat as this behavior couldn't be productive (if something
truly IS wrong, then why let it continue? and, if there is nothing
wrong with the controls/mechanism, then clearly it is unable to meet
my needs so why let it persist in trying?)

[Turns out, there was a city-wide gas shortage so there was enough
gas available to light the furnace but not enough to bring it up to
temperature as quickly as the designers had expected]

> A component could fail suddenly, such as a short circuit diode, and
> everything would work fine after replacing it.
> The cause could perhaps have been a manufacturing defect, such as
> insufficient cooling due to poor quality assembly, but the exact real cause
> would never be known.

You don't care about the real cause. Or, even the failure mode.
You (as user) just don't want to be inconvenienced by the sudden
loss of the functionality/convenience that the the device provided.

> A component could fail suddenly as a side effect of another failure.
> One short circuit output transistor and several other components could also
> burn up.

So, if you could predict the OTHER failure...
Or, that such a failure might occur and lead to the followup failure...

> A component could fail slowly and only become apparent when it got to the
> stage of causing an audible or visible effect.

But, likely, there was something observable *in* the circuit that
just hadn't made it to the level of human perception.

> It would often be easy to locate the dried up electrolytic due to it having
> already let go of some of its contents.
>
> So I concluded that if I wanted to be sure that I could always watch my
> favourite TV show, we would have to have at least two TVs in the house.
>
> If it's not possible to have the equivalent of two TVs then you will want to
> be in a position to get the existing TV repaired or replaced as quicky as
> possible.

Two TVs are affordable. Consider two controllers for a wire-EDM machine.

Or, the cost of having that wire-EDM machine *idle* (because you didn't
have a spare controller!)

> My home wireless Internet system doesn't care if one access point fails, and
> I would not expect to be able to do anything to predict a time of failure.
> Experience says a dead unit has power supply issues. Usually external but
> could be internal.

Again, the goal isn't to predict "time of failure". But, rather, to be
able to know that "this isn't going to end well" -- with some advance notice
that allows for preemptive action to be taken (and not TOO much advance
notice that the user ends up replacing items prematurely).

> I don't think it would be possible to "watch" everything because it's rare
> that you can properly test a component while it's part of a working system.

You don't have to -- as long as you can observe its effects on other
parts of the system. E.g., there's no easy/inexpensive way to
check to see how much the belt on that CD/DVD player has stretched.
But, you can notice that it HAS stretched (or, some less likely
change has occurred that similarly interferes with the tray's actions)
by noting how the activity that it is used for has changed.

> These days I would expect to have fun with management asking for software to
> be able to diagnose and report any hardware failure.
> Not very easy if the power supply has died.

What if the power supply HASN'T died? What if you are diagnosing the
likely upcoming failure *of* the power supply?

You have ECC memory in most (larger) machines. Do you silently
expect it to just fix all the errors? Does it have a way of telling you
how many such errors it HAS corrected? Can you infer the number of
errors that it *hasn't*?

[Why have ECC at all?]

There are (and have been) many efforts to *predict* lifetimes of
components (and, systems). And, some work to examine the state
of systems /in situ/ with an eye towards anticipating their
likelihood of future failure.

[The former has met with poor results -- predicting the future
without a position in its past is difficult. And, knowing how
a device is "stored" when not powered on also plays a role
in its future survival! (is there some reason YOUR devices
can't power themselves on, periodically; notice the environmental
conditions; log them and then power back off)]

The question is one of a practical nature; how much does it cost
you to add this capability to a device and how accurately can it
make those predictions (thus avoiding some future cost/inconvenience).

For small manufacturers, the research required is likely not cost-effective;
just take your best stab at it and let the customer "buy a replacement"
when the time comes (hopefully, outside of your warranty window).

But, anything you can do to minimize this TCO issue gives your product
an edge over competitors. Given that most devices are smart, nowadays,
it seems obvious that they should undertake as much of this task as
they can (conveniently) afford.

<https://www.sciencedirect.com/science/article/abs/pii/S0026271409003667>

<https://www.researchgate.net/publication/3430090_In_Situ_Temperature_Measurement_of_a_Notebook_Computer-A_Case_Study_in_Health_and_Usage_Monitoring_of_Electronics>

<https://www.tandfonline.com/doi/abs/10.1080/16843703.2007.11673148>

<https://www.prognostics.umd.edu/calcepapers/02_V.Shetty_remaingLifeAssesShuttleRemotemanipulatorSystem_22ndSpaceSimulationConf.pdf>

<https://ieeexplore.ieee.org/document/1656125>

<https://journals.sagepub.com/doi/10.1177/0142331208092031>

[Sorry, I can't publish links to the full articles]

"Don Y" <blockedofcourse@foo.invalid> wrote in message
news:uvkn71$ngqi$2@dont-email.me...
> On 4/15/2024 10:32 AM, Martin Rid wrote:
>> Don Y <blockedofcourse@foo.invalid> Wrote in message:r
>>> Is there a general rule of thumb for signalling the likelihood ofan
>>> "imminent" (for some value of "imminent") hardware failure?I suspect
>>> most would involve *relative* changes that would besuggestive of
>>> changing conditions in the components (and notdirectly related to
>>> environmental influences).So, perhaps, a good strategy is to just
>>> "watch" everything andnotice the sorts of changes you "typically"
>>> encounter in the hopethat something of greater magnitude would be a
>>> harbinger...
>>
>> Current and voltages outside of normal operation?
>
> I think "outside" is (often) likely indicative of
> "something is (already) broken".
>
> But, perhaps TRENDS in either/both can be predictive.
>
> E.g., if a (sub)circuit has always been consuming X (which
> is nominal for the design) and, over time, starts to consume
> 1.1X, is that suggestive that something is in the process of
> failing?

That depends on many other unknown factors.
Temperature sensors are common in electronics.
So is current sensing. Voltage sensing too.

>
> Note that the goal is not to troubleshoot the particular design
> or its components but, rather, act as an early warning that
> maintenance may be required (or, that performance may not be
> what you are expecting/have become accustomed to).

If the system is electronic then you can detect whether currents and/or
votages are within expected ranges.
If they are a just a little out of expected range then you might turn on a
warning LED.
If they are way out of range then you might tell the power supply to turn
off quick.
By all means tell the software what has happened, but don't put software
between the current sensor and the emergency turn off.
Be aware that components in monitoring circuits can fail too.

>
> You can include mechanisms to verify outputs are what you
> *intended* them to be (in case the output drivers have shit
> the bed).
>
> You can, also, do sanity checks that ensure values are never
> what they SHOULDN'T be (this is commonly done within software
> products -- if something "can't happen" then noticing that
> it IS happening is a sure-fire indication that something
> is broken!)
>
> [Limit switches on mechanisms are there to ensure the impossible
> is not possible -- like driving a mechanism beyond its extents]
>
> And, where possible, notice second-hand effects of your actions
> (e.g., if you switched on a load, you should see an increase
> in supplied current).
>
> But, again, these are more helpful in detecting FAILED items.

What system would you like to have early warnings for?
Are the warnings needed to indicate operation out of expected limits or to
indicate that maintenance is required, or both?
Without detailed knowledge of the specific sytem, only speculative answers
can be given.

>
>

"Don Y" <blockedofcourse@foo.invalid> wrote in message
news:uvkqqu$o5co$1@dont-email.me...
> On 4/15/2024 1:32 PM, Edward Rawde wrote:
>> "Don Y" <blockedofcourse@foo.invalid> wrote in message
>> news:uvjn74$d54b$1@dont-email.me...
>>> Is there a general rule of thumb for signalling the likelihood of
>>> an "imminent" (for some value of "imminent") hardware failure?
>>
>> My conclusion would be no.
>> Some of my reasons are given below.
>>
>> It always puzzled me how HAL could know that the AE-35 would fail in the
>> near future, but maybe HAL had a motive for lying.
>
> Why does your PC retry failed disk operations?

Because the software designer didn't understand hardware.
The correct approach is to mark that part of the disk as unusable and, if
possible, move any data from it elsewhere quick.

> If I ask the drive to give
> me LBA 1234, shouldn't it ALWAYS give me LBA1234? Without any data
> corruption
> (CRC error) AND within the normal access time limits defined by the
> location
> of those magnetic domains on the rotating medium?
>
> Why should it attempt to retry this MORE than once?
>
> Now, if you knew your disk drive was repeatedly retrying operations,
> would your confidence in it be unchanged from times when it did not
> exhibit such behavior?

I'd have put an SSD in by now, along with an off site backup of the same
data :)

>
> Assuming you have properly configured a EIA232 interface, why would you
> ever get a parity error? (OVERRUN errors can be the result of an i/f
> that is running too fast for the system on the receiving end) How would
> you even KNOW this was happening?
>
> I suspect everyone who has owned a DVD/CD drive has encountered a
> "slow tray" as the mechanism aged. Or, a tray that wouldn't
> open (of its own accord) as soon/quickly as it used to.

If it hasn't been used for some time then I'm ready with a tiny screwdriver
blade to help it open.
But I forget when I last used an optical drive.

>
> The controller COULD be watching this (cuz it knows when it
> initiated the operation and there is an "end-of-stroke"
> sensor available) and KNOW that the drive belt was stretching
> to the point where it was impacting operation.
>
> [And, that a stretched belt wasn't going to suddenly decide to
> unstretch to fix the problem!]
>
>> Back in that era I was doing a lot of repair work when I should have been
>> doing my homework.
>> So I knew that there were many unrelated kinds of hardware failure.
>
> The goal isn't to predict ALL failures but, rather, to anticipate
> LIKELY failures and treat them before they become an inconvenience
> (or worse).
>
> One morning, the (gas) furnace repeatedly tried to light as the
> thermostat called for heat. Then, a few moments later, the
> safeties would kick in and shut down the gas flow. This attracted my
> attention as the LIT furnace should STAY LIT!
>
> The furnace was too stupid to notice its behavior so would repeat
> this cycle, endlessly.
>
> I stepped in and overrode the thermostat to eliminate the call
> for heat as this behavior couldn't be productive (if something
> truly IS wrong, then why let it continue? and, if there is nothing
> wrong with the controls/mechanism, then clearly it is unable to meet
> my needs so why let it persist in trying?)
>
> [Turns out, there was a city-wide gas shortage so there was enough
> gas available to light the furnace but not enough to bring it up to
> temperature as quickly as the designers had expected]

That's why the furnace designers couldn't have anticipated it.
They did not know that such a contition might occur so never tested for it.

>
>> A component could fail suddenly, such as a short circuit diode, and
>> everything would work fine after replacing it.
>> The cause could perhaps have been a manufacturing defect, such as
>> insufficient cooling due to poor quality assembly, but the exact real
>> cause
>> would never be known.
>
> You don't care about the real cause. Or, even the failure mode.
> You (as user) just don't want to be inconvenienced by the sudden
> loss of the functionality/convenience that the the device provided.

There will always be sudden unexpected loss of functionality for reasons
which could not easily be predicted.
People who service lawn mowers in the area where I live are very busy right
now.

>
>> A component could fail suddenly as a side effect of another failure.
>> One short circuit output transistor and several other components could
>> also
>> burn up.
>
> So, if you could predict the OTHER failure...
> Or, that such a failure might occur and lead to the followup failure...
>
>> A component could fail slowly and only become apparent when it got to the
>> stage of causing an audible or visible effect.
>
> But, likely, there was something observable *in* the circuit that
> just hadn't made it to the level of human perception.

Yes a power supply ripple detection circuit could have turned on a warning
LED but that never happened for at least two reasons.
1. The detection circuit would have increased the cost of the equipment and
thus diminished the profit of the manufacturer.
2. The user would not have understood and would have ignored the warning
anyway.

>
>> It would often be easy to locate the dried up electrolytic due to it
>> having
>> already let go of some of its contents.
>>
>> So I concluded that if I wanted to be sure that I could always watch my
>> favourite TV show, we would have to have at least two TVs in the house.
>>
>> If it's not possible to have the equivalent of two TVs then you will want
>> to
>> be in a position to get the existing TV repaired or replaced as quicky as
>> possible.
>
> Two TVs are affordable. Consider two controllers for a wire-EDM machine.
>
> Or, the cost of having that wire-EDM machine *idle* (because you didn't
> have a spare controller!)
>
>> My home wireless Internet system doesn't care if one access point fails,
>> and
>> I would not expect to be able to do anything to predict a time of
>> failure.
>> Experience says a dead unit has power supply issues. Usually external but
>> could be internal.
>
> Again, the goal isn't to predict "time of failure". But, rather, to be
> able to know that "this isn't going to end well" -- with some advance
> notice
> that allows for preemptive action to be taken (and not TOO much advance
> notice that the user ends up replacing items prematurely).

Get feedback from the people who use your equpment.

>
>> I don't think it would be possible to "watch" everything because it's
>> rare
>> that you can properly test a component while it's part of a working
>> system.
>
> You don't have to -- as long as you can observe its effects on other
> parts of the system. E.g., there's no easy/inexpensive way to
> check to see how much the belt on that CD/DVD player has stretched.
> But, you can notice that it HAS stretched (or, some less likely
> change has occurred that similarly interferes with the tray's actions)
> by noting how the activity that it is used for has changed.

Sure but you have to be the operator for that.
So you can be ready to help the tray open when needed.

>
>> These days I would expect to have fun with management asking for software
>> to
>> be able to diagnose and report any hardware failure.
>> Not very easy if the power supply has died.
>
> What if the power supply HASN'T died? What if you are diagnosing the
> likely upcoming failure *of* the power supply?

Then I probably can't, because the power supply may be just a bought in
power supply which was never designed with upcoming failure detection in
mind.

>
> You have ECC memory in most (larger) machines. Do you silently
> expect it to just fix all the errors? Does it have a way of telling you
> how many such errors it HAS corrected? Can you infer the number of
> errors that it *hasn't*?
>
> [Why have ECC at all?]

Things are sometimes done the way they've always been done.
I used to notice a missing chip in the 9th position but now you mention it
the RAM I just looked at has 9 chips each side.

>
> There are (and have been) many efforts to *predict* lifetimes of
> components (and, systems). And, some work to examine the state
> of systems /in situ/ with an eye towards anticipating their
> likelihood of future failure.

On 4/15/2024 8:33 PM, Edward Rawde wrote:

[Shouldn't that be Edwar D rawdE?]

>>> Current and voltages outside of normal operation?
>>
>> I think "outside" is (often) likely indicative of
>> "something is (already) broken".
>>
>> But, perhaps TRENDS in either/both can be predictive.
>>
>> E.g., if a (sub)circuit has always been consuming X (which
>> is nominal for the design) and, over time, starts to consume
>> 1.1X, is that suggestive that something is in the process of
>> failing?
>
> That depends on many other unknown factors.
> Temperature sensors are common in electronics.
> So is current sensing. Voltage sensing too.

Sensors cost money. And, HAVING data but not knowing how to
USE it is a wasted activity (and cost).

Why not monitor every node in the schematic and compare
them (with dedicated hardware -- that is ALSO monitored??)
with expected operational limits?

Then, design some network to weight the individual
observations to make the prediction?

>> Note that the goal is not to troubleshoot the particular design
>> or its components but, rather, act as an early warning that
>> maintenance may be required (or, that performance may not be
>> what you are expecting/have become accustomed to).
>
> If the system is electronic then you can detect whether currents and/or
> votages are within expected ranges.
> If they are a just a little out of expected range then you might turn on a
> warning LED.
> If they are way out of range then you might tell the power supply to turn
> off quick.
> By all means tell the software what has happened, but don't put software
> between the current sensor and the emergency turn off.

Again, the goal is to be an EARLY warning, not an "Oh, Shit! Kill the power!!"

As such, software is invaluable as designing PREDICTIVE hardware is
harder than designing predictive software (algorithms).

You don't want to tell the user "The battery in your smoke detector
is NOW dead (leaving you vulnerable)" but, rather, "The battery in
your smoke detector WILL cease to be able to provide the power necessary
for the smoke detector to provide the level of protection that you
desire."

And, the WAY that you inform the user has to be "productive/useful".
A smoke detector beeping every minute is likely to find itself unplugged,
leading to exactly the situation that the alert was trying to avoid!

A smoke detector that beeps once a day risks not being heard
(what if the occupant "works nights"?). A smoke detector
that beeps a month in advance of the anticipated failure (and
requires acknowledgement) risks being forgotten -- until
it is forced to beep more persistently (see above).

> Be aware that components in monitoring circuits can fail too.

Which is why hardware interlocks are physical switches -- yet
can only be used to protect against certain types of faults
(those that are most costly -- injury or loss of life)

>> But, again, these are more helpful in detecting FAILED items.
>
> What system would you like to have early warnings for?
> Are the warnings needed to indicate operation out of expected limits or to
> indicate that maintenance is required, or both?
> Without detailed knowledge of the specific sytem, only speculative answers
> can be given.

I'm not looking for speculation. I'm looking for folks who have DONE
such things (designing to speculation is more expensive than just letting
the devices fail when they need to fail!)

E.g., when making tablets, it is possible that a bit of air will
get trapped in the granulation during compression. This is dependant
on a lot of factors -- tablet dimensions, location in the die
where the compression event is happening, characteristics of the
granulation, geometry/condition of the tooling, etc.

But, if this happens, some tens of milliseconds later, the top will "pop"
off the tablet. It now is cosmetically damaged as well as likely out
of specification (amount of "active" present in the dose). You want
to either be able to detect this (100% of the time on 100% of the tablets)
and dynamically discard those units (and only those units!). *OR*,
identify the characteristics of the process that most affect this condition
and *monitor* for them to AVOID the problem.

If that means replacing your tooling more frequently (expensive!),
it can save money in the long run (imagine having to "sort" through
a million tablets each hour to determine if any have popped like this?)
Or, throttling down the press so the compression events are "slower"
(more gradual). Or, moving the event up in the die to provide
a better egress for the trapped air. Or...

TELLING the user that this is happening (or likely to happen, soon)
has real $$$ value. Even better if your device can LEARN which
tablets and conditions will likely lead to this -- and when!

On 4/15/2024 9:14 PM, Edward Rawde wrote:
>>> It always puzzled me how HAL could know that the AE-35 would fail in the
>>> near future, but maybe HAL had a motive for lying.
>>
>> Why does your PC retry failed disk operations?
>
> Because the software designer didn't understand hardware.

Actually, he DID understand the hardware which is why he retried
it instead of ASSUMING every operation would proceed correctly.

[Why bother testing the result code if you never expect a failure?]

> The correct approach is to mark that part of the disk as unusable and, if
> possible, move any data from it elsewhere quick.

That only makes sense if the error is *persistent*. "Shit
happens" and you can get an occasional failed operation when
nothing is truly "broken".

(how do you know the HBA isn't the culprit?)

>> If I ask the drive to give
>> me LBA 1234, shouldn't it ALWAYS give me LBA1234? Without any data
>> corruption
>> (CRC error) AND within the normal access time limits defined by the
>> location
>> of those magnetic domains on the rotating medium?
>>
>> Why should it attempt to retry this MORE than once?
>>
>> Now, if you knew your disk drive was repeatedly retrying operations,
>> would your confidence in it be unchanged from times when it did not
>> exhibit such behavior?
>
> I'd have put an SSD in by now, along with an off site backup of the same
> data :)

So, any problems you have with your SSD, today, should be solved by using the
technology that will be invented 10 years hence! Ah, that's a sound strategy!

>> Assuming you have properly configured a EIA232 interface, why would you
>> ever get a parity error? (OVERRUN errors can be the result of an i/f
>> that is running too fast for the system on the receiving end) How would
>> you even KNOW this was happening?
>>
>> I suspect everyone who has owned a DVD/CD drive has encountered a
>> "slow tray" as the mechanism aged. Or, a tray that wouldn't
>> open (of its own accord) as soon/quickly as it used to.
>
> If it hasn't been used for some time then I'm ready with a tiny screwdriver
> blade to help it open.

Why don't they ship such drives with tiny screwdrivers to make it
easier for EVERY customer to address this problem?

> But I forget when I last used an optical drive.

When the firmware in your SSD corrupts your data, what remedy will
you use?

You're missing the forest for the trees.

>> [Turns out, there was a city-wide gas shortage so there was enough
>> gas available to light the furnace but not enough to bring it up to
>> temperature as quickly as the designers had expected]
>
> That's why the furnace designers couldn't have anticipated it.

Really? You can't anticipate the "gas shutoff" not being in the ON
position? (which would yield the same endless retry cycle)

> They did not know that such a contition might occur so never tested for it.

If they planned on ENDLESSLY retrying, then they must have imagined
some condition COULD occur that would lead to such an outcome.
Else, why not just retry *once* and then give up? Or, not
retry at all?

>>> A component could fail suddenly, such as a short circuit diode, and
>>> everything would work fine after replacing it.
>>> The cause could perhaps have been a manufacturing defect, such as
>>> insufficient cooling due to poor quality assembly, but the exact real
>>> cause
>>> would never be known.
>>
>> You don't care about the real cause. Or, even the failure mode.
>> You (as user) just don't want to be inconvenienced by the sudden
>> loss of the functionality/convenience that the the device provided.
>
> There will always be sudden unexpected loss of functionality for reasons
> which could not easily be predicted.

And if they CAN'T be predicted, then they aren't germane to this
discussion, eh?

My concern is for the set of failure modes that can realistically
be anticipated.

I *know* the inverters in my monitors are going to fail. It
would be nice if I knew before I was actively using one when
it went dark!

[But, most users would only use this indication to tell them
to purchase another monitor; "You have been warned!"]

> People who service lawn mowers in the area where I live are very busy right
> now.
>
>>> A component could fail suddenly as a side effect of another failure.
>>> One short circuit output transistor and several other components could
>>> also
>>> burn up.
>>
>> So, if you could predict the OTHER failure...
>> Or, that such a failure might occur and lead to the followup failure...
>>
>>> A component could fail slowly and only become apparent when it got to the
>>> stage of causing an audible or visible effect.
>>
>> But, likely, there was something observable *in* the circuit that
>> just hadn't made it to the level of human perception.
>
> Yes a power supply ripple detection circuit could have turned on a warning
> LED but that never happened for at least two reasons.
> 1. The detection circuit would have increased the cost of the equipment and
> thus diminished the profit of the manufacturer.

That would depend on the market, right? Most of my computers have redundant
"smart" (i.e., internal monitoring and reporting) power supplies. Because
they were marketed to folks who wanted that sort of reliability. Because
a manufacturer who didn't provide that level of AVAILABILITY would quickly
lose market share. The cost of the added components and "handling" is
small compared to the cost of lost opportunity (sales).

> 2. The user would not have understood and would have ignored the warning
> anyway.

That makes assumptions about the market AND the user.

If one of my machines signals a fault, I look to see what it is complaining
about: is it a power supply failure (in which case, I'm now reliant on
a single power supply)? is it a memory failure (in which case, a bank
of memory may have been disabled which means the machine will thrash
more and throughput will drop)? is it a link aggregation error (and
network traffic will suffer)?

If I can't understand these errors, then I either don't buy a product
with that level of reliability *or* have someone on hand who CAN
understand the errors and provide remedies/advice.

Consumers will replace a PC because of malware, trashed registry,
creeping cruft, etc. That's a problem with the consumer buying the
"wrong" sort of computing equipment for his likely method of use.
(buy a Mac?)

>>> My home wireless Internet system doesn't care if one access point fails,
>>> and
>>> I would not expect to be able to do anything to predict a time of
>>> failure.
>>> Experience says a dead unit has power supply issues. Usually external but
>>> could be internal.
>>
>> Again, the goal isn't to predict "time of failure". But, rather, to be
>> able to know that "this isn't going to end well" -- with some advance
>> notice
>> that allows for preemptive action to be taken (and not TOO much advance
>> notice that the user ends up replacing items prematurely).
>
> Get feedback from the people who use your equpment.

Users often don't understand when a device is malfunctioning.
Or, how to report the conditions and symptoms in a meaningful way.

I recall a woman I worked with ~45 years ago sitting, patiently,
waiting for her computer to boot. As I walked past, she asked me how
long it takes for that to happen (floppy based systems). Alarmed
(I had designed the workstations), I asked "How long have you been
waiting?"

Turns out, she had inserted the (8") floppy rotated 90 degrees from
it's proper orientation.

How much longer would she have waited had I not walked past?

>>> I don't think it would be possible to "watch" everything because it's
>>> rare
>>> that you can properly test a component while it's part of a working
>>> system.
>>
>> You don't have to -- as long as you can observe its effects on other
>> parts of the system. E.g., there's no easy/inexpensive way to
>> check to see how much the belt on that CD/DVD player has stretched.
>> But, you can notice that it HAS stretched (or, some less likely
>> change has occurred that similarly interferes with the tray's actions)
>> by noting how the activity that it is used for has changed.
>
> Sure but you have to be the operator for that.
> So you can be ready to help the tray open when needed.

One wouldn't bother with a CD/DVD player -- they are too disposable
and reporting errors won't help the user (even though you have a
big ATTACHED display at your disposal!)

On 15/04/2024 18:13, Don Y wrote:
> Is there a general rule of thumb for signalling the likelihood of
> an "imminent" (for some value of "imminent") hardware failure?

You have to be very careful that the additional complexity doesn't
itself introduce new annoying failure modes. My previous car had
filament bulb failure sensors (new one is LED) of which the one for the
parking light had itself failed - the parking light still worked.
However, the car would great me with "parking light failure" every time
I started the engine and the main dealer refused to cancel it.

Repair of parking light sensor failure required swapping out the
*entire* front light assembly since it was built in one time hot glue.
That would be a very expensive "repair" for a trivial fault.

The parking light is not even a required feature.

> I suspect most would involve *relative* changes that would be
> suggestive of changing conditions in the components (and not
> directly related to environmental influences).
>
> So, perhaps, a good strategy is to just "watch" everything and
> notice the sorts of changes you "typically" encounter in the hope
> that something of greater magnitude would be a harbinger...

Monitoring temperature, voltage supply and current consumption isn't a
bad idea. If they get unexpectedly out of line something is wrong.
Likewise with power on self tests you can catch some latent failures
before they actually affect normal operation.

--
Martin Brown

On 16/04/2024 05:14, Edward Rawde wrote:
> "Don Y" <blockedofcourse@foo.invalid> wrote in message
> news:uvkqqu$o5co$1@dont-email.me...
>> On 4/15/2024 1:32 PM, Edward Rawde wrote:
>>> "Don Y" <blockedofcourse@foo.invalid> wrote in message
>>> news:uvjn74$d54b$1@dont-email.me...
>>>> Is there a general rule of thumb for signalling the likelihood of
>>>> an "imminent" (for some value of "imminent") hardware failure?
>>>
>>> My conclusion would be no.
>>> Some of my reasons are given below.
>>>
>>> It always puzzled me how HAL could know that the AE-35 would fail in the
>>> near future, but maybe HAL had a motive for lying.
>>
>> Why does your PC retry failed disk operations?
>
> Because the software designer didn't understand hardware.
> The correct approach is to mark that part of the disk as unusable and, if
> possible, move any data from it elsewhere quick.
>
>> If I ask the drive to give
>> me LBA 1234, shouldn't it ALWAYS give me LBA1234? Without any data
>> corruption
>> (CRC error) AND within the normal access time limits defined by the
>> location
>> of those magnetic domains on the rotating medium?
>>
>> Why should it attempt to retry this MORE than once?
>>
>> Now, if you knew your disk drive was repeatedly retrying operations,
>> would your confidence in it be unchanged from times when it did not
>> exhibit such behavior?
>
> I'd have put an SSD in by now, along with an off site backup of the same
> data :)
>
>>
>> Assuming you have properly configured a EIA232 interface, why would you
>> ever get a parity error? (OVERRUN errors can be the result of an i/f
>> that is running too fast for the system on the receiving end) How would
>> you even KNOW this was happening?
>>
>> I suspect everyone who has owned a DVD/CD drive has encountered a
>> "slow tray" as the mechanism aged. Or, a tray that wouldn't
>> open (of its own accord) as soon/quickly as it used to.
>
> If it hasn't been used for some time then I'm ready with a tiny screwdriver
> blade to help it open.
> But I forget when I last used an optical drive.
>
>>
>> The controller COULD be watching this (cuz it knows when it
>> initiated the operation and there is an "end-of-stroke"
>> sensor available) and KNOW that the drive belt was stretching
>> to the point where it was impacting operation.
>>
>> [And, that a stretched belt wasn't going to suddenly decide to
>> unstretch to fix the problem!]
>>
>>> Back in that era I was doing a lot of repair work when I should have been
>>> doing my homework.
>>> So I knew that there were many unrelated kinds of hardware failure.
>>
>> The goal isn't to predict ALL failures but, rather, to anticipate
>> LIKELY failures and treat them before they become an inconvenience
>> (or worse).
>>
>> One morning, the (gas) furnace repeatedly tried to light as the
>> thermostat called for heat. Then, a few moments later, the
>> safeties would kick in and shut down the gas flow. This attracted my
>> attention as the LIT furnace should STAY LIT!
>>
>> The furnace was too stupid to notice its behavior so would repeat
>> this cycle, endlessly.
>>
>> I stepped in and overrode the thermostat to eliminate the call
>> for heat as this behavior couldn't be productive (if something
>> truly IS wrong, then why let it continue? and, if there is nothing
>> wrong with the controls/mechanism, then clearly it is unable to meet
>> my needs so why let it persist in trying?)
>>
>> [Turns out, there was a city-wide gas shortage so there was enough
>> gas available to light the furnace but not enough to bring it up to
>> temperature as quickly as the designers had expected]
>
> That's why the furnace designers couldn't have anticipated it.
> They did not know that such a contition might occur so never tested for it.
>
>>
>>> A component could fail suddenly, such as a short circuit diode, and
>>> everything would work fine after replacing it.
>>> The cause could perhaps have been a manufacturing defect, such as
>>> insufficient cooling due to poor quality assembly, but the exact real
>>> cause
>>> would never be known.
>>
>> You don't care about the real cause. Or, even the failure mode.
>> You (as user) just don't want to be inconvenienced by the sudden
>> loss of the functionality/convenience that the the device provided.
>
> There will always be sudden unexpected loss of functionality for reasons
> which could not easily be predicted.
> People who service lawn mowers in the area where I live are very busy right
> now.
>
>>
>>> A component could fail suddenly as a side effect of another failure.
>>> One short circuit output transistor and several other components could
>>> also
>>> burn up.
>>
>> So, if you could predict the OTHER failure...
>> Or, that such a failure might occur and lead to the followup failure...
>>
>>> A component could fail slowly and only become apparent when it got to the
>>> stage of causing an audible or visible effect.
>>
>> But, likely, there was something observable *in* the circuit that
>> just hadn't made it to the level of human perception.
>
> Yes a power supply ripple detection circuit could have turned on a warning
> LED but that never happened for at least two reasons.
> 1. The detection circuit would have increased the cost of the equipment and
> thus diminished the profit of the manufacturer.
> 2. The user would not have understood and would have ignored the warning
> anyway.
>
>>
>>> It would often be easy to locate the dried up electrolytic due to it
>>> having
>>> already let go of some of its contents.
>>>
>>> So I concluded that if I wanted to be sure that I could always watch my
>>> favourite TV show, we would have to have at least two TVs in the house.
>>>
>>> If it's not possible to have the equivalent of two TVs then you will want
>>> to
>>> be in a position to get the existing TV repaired or replaced as quicky as
>>> possible.
>>
>> Two TVs are affordable. Consider two controllers for a wire-EDM machine.
>>
>> Or, the cost of having that wire-EDM machine *idle* (because you didn't
>> have a spare controller!)
>>
>>> My home wireless Internet system doesn't care if one access point fails,
>>> and
>>> I would not expect to be able to do anything to predict a time of
>>> failure.
>>> Experience says a dead unit has power supply issues. Usually external but
>>> could be internal.
>>
>> Again, the goal isn't to predict "time of failure". But, rather, to be
>> able to know that "this isn't going to end well" -- with some advance
>> notice
>> that allows for preemptive action to be taken (and not TOO much advance
>> notice that the user ends up replacing items prematurely).
>
> Get feedback from the people who use your equpment.
>
>>
>>> I don't think it would be possible to "watch" everything because it's
>>> rare
>>> that you can properly test a component while it's part of a working
>>> system.
>>
>> You don't have to -- as long as you can observe its effects on other
>> parts of the system. E.g., there's no easy/inexpensive way to
>> check to see how much the belt on that CD/DVD player has stretched.
>> But, you can notice that it HAS stretched (or, some less likely
>> change has occurred that similarly interferes with the tray's actions)
>> by noting how the activity that it is used for has changed.
>
> Sure but you have to be the operator for that.
> So you can be ready to help the tray open when needed.
>
>>
>>> These days I would expect to have fun with management asking for software
>>> to
>>> be able to diagnose and report any hardware failure.
>>> Not very easy if the power supply has died.
>>
>> What if the power supply HASN'T died? What if you are diagnosing the
>> likely upcoming failure *of* the power supply?
>
> Then I probably can't, because the power supply may be just a bought in
> power supply which was never designed with upcoming failure detection in
> mind.
>
>>
>> You have ECC memory in most (larger) machines. Do you silently
>> expect it to just fix all the errors? Does it have a way of telling you
>> how many such errors it HAS corrected? Can you infer the number of
>> errors that it *hasn't*?
>>
>> [Why have ECC at all?]
>
> Things are sometimes done the way they've always been done.
> I used to notice a missing chip in the 9th position but now you mention it
> the RAM I just looked at has 9 chips each side.
>
>>
>> There are (and have been) many efforts to *predict* lifetimes of
>> components (and, systems). And, some work to examine the state
>> of systems /in situ/ with an eye towards anticipating their
>> likelihood of future failure.
>
> I'm sure that's true.
>
>>
>> [The former has met with poor results -- predicting the future
>> without a position in its past is difficult. And, knowing how
>> a device is "stored" when not powered on also plays a role
>> in its future survival! (is there some reason YOUR devices
>> can't power themselves on, periodically; notice the environmental
>> conditions; log them and then power back off)]
>>
>> The question is one of a practical nature; how much does it cost
>> you to add this capability to a device and how accurately can it
>> make those predictions (thus avoiding some future cost/inconvenience).
>>
>> For small manufacturers, the research required is likely not
>> cost-effective;
>> just take your best stab at it and let the customer "buy a replacement"
>> when the time comes (hopefully, outside of your warranty window).
>>
>> But, anything you can do to minimize this TCO issue gives your product
>> an edge over competitors. Given that most devices are smart, nowadays,
>> it seems obvious that they should undertake as much of this task as
>> they can (conveniently) afford.
>>
>> <https://www.sciencedirect.com/science/article/abs/pii/S0026271409003667>
>>
>> <https://www.researchgate.net/publication/3430090_In_Situ_Temperature_Measurement_of_a_Notebook_Computer-A_Case_Study_in_Health_and_Usage_Monitoring_of_Electronics>
>>
>> <https://www.tandfonline.com/doi/abs/10.1080/16843703.2007.11673148>
>>
>> <https://www.prognostics.umd.edu/calcepapers/02_V.Shetty_remaingLifeAssesShuttleRemotemanipulatorSystem_22ndSpaceSimulationConf.pdf>
>>
>> <https://ieeexplore.ieee.org/document/1656125>
>>
>> <https://journals.sagepub.com/doi/10.1177/0142331208092031>
>>
>> [Sorry, I can't publish links to the full articles]
>>
>
>

On 4/16/2024 1:45 AM, Martin Brown wrote:
> On 15/04/2024 18:13, Don Y wrote:
>> Is there a general rule of thumb for signalling the likelihood of
>> an "imminent" (for some value of "imminent") hardware failure?
>
> You have to be very careful that the additional complexity doesn't itself
> introduce new annoying failure modes.

*Or*, decrease the reliability of the device, in general.

> My previous car had filament bulb failure
> sensors (new one is LED) of which the one for the parking light had itself
> failed - the parking light still worked. However, the car would great me with
> "parking light failure" every time I started the engine and the main dealer
> refused to cancel it.

My goal is to provide *advisories*. You don't want to constrain the
user.

Smoke detectors that nag you with "replace battery" alerts are nags.
A car that refuses to start unless the seat belts are fastened is a nag.

You shouldn't require a third party to enable you to ignore an
advisory. But, it's OK to require the user to acknowledge that
advisory!

> Repair of parking light sensor failure required swapping out the *entire* front
> light assembly since it was built in one time hot glue. That would be a very
> expensive "repair" for a trivial fault.
>
> The parking light is not even a required feature.
>
>> I suspect most would involve *relative* changes that would be
>> suggestive of changing conditions in the components (and not
>> directly related to environmental influences).
>>
>> So, perhaps, a good strategy is to just "watch" everything and
>> notice the sorts of changes you "typically" encounter in the hope
>> that something of greater magnitude would be a harbinger...
>
> Monitoring temperature, voltage supply and current consumption isn't a bad
> idea. If they get unexpectedly out of line something is wrong.

Extremes are easy to detect -- but often indicate failures.
E.g., a short, an open.

The problem is sorting out what magnitude changes are significant
and which are normal variation.

I think being able to track history gives you a leg up in that
it gives you a better idea of what MIGHT be normal instead of
just looking at an instant in time.

> Likewise with
> power on self tests you can catch some latent failures before they actually
> affect normal operation.

POST is seldom executed as devices tend to run 24/7/365.
So, I have to design runtime BIST support that can, hopefully,
coax this information from a *running* system without interfering
with that operation.

This puts constraints on how you operate the hardware
(unless you want to add lots of EXTRA hardware to
extract these observations.

E.g., if you can control N loads, then individually (sequentially)
activating them and noticing the delta power consumption reveals
more than just enabling ALL that need to be enabled and only seeing
the aggregate of those loads.

This can also simplify gross failure detection if part of the
normal control strategy.

E.g., I designed a medical instrument many years ago that had an
external "sensor array". As that could be unplugged at any time,
I had to continually monitor for it's disconnection. At the same
time, individual sensors in the array could be "spoiled" by
spilled reagents. Yet, the other sensors shouldn't be compromised
or voided just because of the failure of certain ones.

Recognizing that this sort of thing COULD happen in normal use
was the biggest part of the design; the hardware and software
to actually handle these exceptions was then straightforward.

Note that some failures may not be possible to recover from
without adding significant cost (and other failure modes).
So, it's a value decision as to what you support and what
you "tolerate".

Don Y wrote:
> Is there a general rule of thumb for signalling the likelihood of
> an "imminent" (for some value of "imminent") hardware failure?
>
> I suspect most would involve *relative* changes that would be
> suggestive of changing conditions in the components (and not
> directly related to environmental influences).
>
> So, perhaps, a good strategy is to just "watch" everything and
> notice the sorts of changes you "typically" encounter in the hope
> that something of greater magnitude would be a harbinger...

A singular speculative spitball - the capacitive marker:

In-situ Prognostic Method of Power MOSFET Based on Miller Effect

... This paper presents a new in-situ prognosis method for
MOSFET based on miller effect. According to the theory
analysis, simulation and experiment results, the miller
platform voltage is identified as a new degradation
precursor ...

(10.1109/PHM.2017.8079139)

Danke,

--
Don, KB7RPU, https://www.qsl.net/kb7rpu
There was a young lady named Bright Whose speed was far faster than light;
She set out one day In a relative way And returned on the previous night.

On Tue, 16 Apr 2024 00:51:03 -0000 (UTC), Phil Hobbs
<pcdhSpamMeSenseless@electrooptical.net> wrote:

>Joe Gwinn <joegwinn@comcast.net> wrote:
>> On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>>
>>> On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>>> wrote:
>>>
>>>> On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>>> <blockedofcourse@foo.invalid> wrote:
>>>>
>>>>> Is there a general rule of thumb for signalling the likelihood of
>>>>> an "imminent" (for some value of "imminent") hardware failure?
>>>>>
>>>>> I suspect most would involve *relative* changes that would be
>>>>> suggestive of changing conditions in the components (and not
>>>>> directly related to environmental influences).
>>>>>
>>>>> So, perhaps, a good strategy is to just "watch" everything and
>>>>> notice the sorts of changes you "typically" encounter in the hope
>>>>> that something of greater magnitude would be a harbinger...
>>>>
>>>> There is a standard approach that may work: Measure the level and
>>>> trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>>> When connections (perhaps within a package) start to fail, the flicker
>>>> level rises. The actual frequency monitored isn't all that critical.
>>>>
>>>> Joe Gwinn
>>>
>>> Do connections "start to fail" ?
>>
>> Yes, they do, in things like vias. I went through a big drama where a
>> critical bit of radar logic circuitry would slowly go nuts.
>>
>> It turned out that the copper plating on the walls of the vias was
>> suffering from low-cycle fatigue during temperature cycling and slowly
>> breaking, one little crack at a time, until it went open. If you
>> measured the resistance to parts per million (6.5 digit DMM), sampling
>> at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
>> also measure a copper line, and divide the via-chain resistance by the
>> no-via resistance, to correct for temperature changes.
>>
>> The solution was to redesign the vias, mainly to increase the critical
>> volume of copper. And modern SMD designs have less and less copper
>> volume.
>>
>> I bet precision resistors can also be measured this way.
>>
>>
>>> I don't think I've ever owned a piece of electronic equipment that
>>> warned me of an impending failure.
>>
>> Onset of smoke emission is a common sign.
>>
>>
>>> Cars do, for some failure modes, like low oil level.
>>
>> The industrial method for big stuff is accelerometers attached near
>> the bearings, and listen for excessive rotation-correlated (not
>> necessarily harmonic) noise.
>
>There are a number of instruments available that look for metal particles
>in the lubricating oil.

Yes.

The old-school version was a magnetic drain plug, which one inspected
for clinging iron chips or dust, also serving to trap those chips. The
newer-school version was to send a sample of the dirty oil to the lab
for microscope and chemical analysis. There are companies that will
take your old lubrication oil and reprocess it, yielding new oil.

If there was an oil filter, inspect the filter surface.

And when one was replacing the oil in the gear case, wipe the bottom
with a white rag, and look at the rag.

Nobody did electronic testing until very recently, because even
expensive electronics were far too unreliable and fragile.

Joe Gwinn

On Mon, 15 Apr 2024 16:26:35 -0700, john larkin <jl@650pot.com> wrote:

>On Mon, 15 Apr 2024 18:03:23 -0400, Joe Gwinn <joegwinn@comcast.net>
>wrote:
>
>>On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>>
>>>On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>>>wrote:
>>>
>>>>On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>>><blockedofcourse@foo.invalid> wrote:
>>>>
>>>>>Is there a general rule of thumb for signalling the likelihood of
>>>>>an "imminent" (for some value of "imminent") hardware failure?
>>>>>
>>>>>I suspect most would involve *relative* changes that would be
>>>>>suggestive of changing conditions in the components (and not
>>>>>directly related to environmental influences).
>>>>>
>>>>>So, perhaps, a good strategy is to just "watch" everything and
>>>>>notice the sorts of changes you "typically" encounter in the hope
>>>>>that something of greater magnitude would be a harbinger...
>>>>
>>>>There is a standard approach that may work: Measure the level and
>>>>trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>>>When connections (perhaps within a package) start to fail, the flicker
>>>>level rises. The actual frequency monitored isn't all that critical.
>>>>
>>>>Joe Gwinn
>>>
>>>Do connections "start to fail" ?
>>
>>Yes, they do, in things like vias. I went through a big drama where a
>>critical bit of radar logic circuitry would slowly go nuts.
>>
>>It turned out that the copper plating on the walls of the vias was
>>suffering from low-cycle fatigue during temperature cycling and slowly
>>breaking, one little crack at a time, until it went open. If you
>>measured the resistance to parts per million (6.5 digit DMM), sampling
>>at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
>>also measure a copper line, and divide the via-chain resistance by the
>>no-via resistance, to correct for temperature changes.
>
>But nobody is going to monitor every via on a PCB, even if it were
>possible.

It was not possible to test the vias on the failing logic board, but
we knew from metallurgical cut, polish, and inspect studies of failed
boards that it was the vias that were failing.

>One could instrument a PCB fab test board, I guess. But DC tests would
>be fine.

What was being tested was a fab test board that had both the series
via chain path and the no-via path of roughly the same DC resistance,
set up so we could do 4-wire Kelvin resistance measurements of each
path independent of the other path.

>We have one board with over 4000 vias, but they are mostly in
>parallel.

This can also be tested , but using a 6.5-digit DMM intended for
measuring very low resistance values. A change of one part in 4,000
is huge to a 6.5-digit instrument. The conductivity will decline
linearly as vias fail one by one.

>>The solution was to redesign the vias, mainly to increase the critical
>>volume of copper. And modern SMD designs have less and less copper
>>volume.
>>
>>I bet precision resistors can also be measured this way.
>>
>>
>>>I don't think I've ever owned a piece of electronic equipment that
>>>warned me of an impending failure.
>>
>>Onset of smoke emission is a common sign.
>>
>>
>>>Cars do, for some failure modes, like low oil level.
>>
>>The industrial method for big stuff is accelerometers attached near
>>the bearings, and listen for excessive rotation-correlated (not
>>necessarily harmonic) noise.
>
>Big ships that I've worked on have a long propeller shaft in the shaft
>alley, a long tunnel where nobody often goes. They have magnetic shaft
>runout sensors and shaft bearing temperature monitors.
>
>They measure shaft torque and SHP too, from the shaft twist.

Yep. And these kinds of things fail slowly. At first.

>I liked hiding out in the shaft alley. It was private and cool, that
>giant shaft slowly rotating.

Probably had a calming flowing water sound as well.

Joe Gwinn

"Don Y" <blockedofcourse@foo.invalid> wrote in message
news:uvl30j$phap$3@dont-email.me...
> On 4/15/2024 9:14 PM, Edward Rawde wrote:
>>>> It always puzzled me how HAL could know that the AE-35 would fail in
>>>> the
>>>> near future, but maybe HAL had a motive for lying.
>>>
>>> Why does your PC retry failed disk operations?
>>
>> Because the software designer didn't understand hardware.
>
> Actually, he DID understand the hardware which is why he retried
> it instead of ASSUMING every operation would proceed correctly.
>
> ....
>
> When the firmware in your SSD corrupts your data, what remedy will
> you use?

Replace drive and restore backup.
It's happened a few times, and a friend had one of those 16 GB but looks
like 1 TB to the OS SSDs from Amazon.

"Don Y" <blockedofcourse@foo.invalid> wrote in message
news:uvl2gr$phap$2@dont-email.me...
> On 4/15/2024 8:33 PM, Edward Rawde wrote:
>
> [Shouldn't that be Edwar D rawdE?]
>

I don't mind how you pronounce it.

>
....
>
> A smoke detector that beeps once a day risks not being heard

Reminds me of a tenant who just removed the battery to stop the annoying
beeping.

>
....

On Tue, 16 Apr 2024 10:19:00 -0400, Joe Gwinn <joegwinn@comcast.net>
wrote:

>On Mon, 15 Apr 2024 16:26:35 -0700, john larkin <jl@650pot.com> wrote:
>
>>On Mon, 15 Apr 2024 18:03:23 -0400, Joe Gwinn <joegwinn@comcast.net>
>>wrote:
>>
>>>On Mon, 15 Apr 2024 13:05:40 -0700, john larkin <jl@650pot.com> wrote:
>>>
>>>>On Mon, 15 Apr 2024 15:41:57 -0400, Joe Gwinn <joegwinn@comcast.net>
>>>>wrote:
>>>>
>>>>>On Mon, 15 Apr 2024 10:13:02 -0700, Don Y
>>>>><blockedofcourse@foo.invalid> wrote:
>>>>>
>>>>>>Is there a general rule of thumb for signalling the likelihood of
>>>>>>an "imminent" (for some value of "imminent") hardware failure?
>>>>>>
>>>>>>I suspect most would involve *relative* changes that would be
>>>>>>suggestive of changing conditions in the components (and not
>>>>>>directly related to environmental influences).
>>>>>>
>>>>>>So, perhaps, a good strategy is to just "watch" everything and
>>>>>>notice the sorts of changes you "typically" encounter in the hope
>>>>>>that something of greater magnitude would be a harbinger...
>>>>>
>>>>>There is a standard approach that may work: Measure the level and
>>>>>trend of very low frequency (around a tenth of a Hertz) flicker noise.
>>>>>When connections (perhaps within a package) start to fail, the flicker
>>>>>level rises. The actual frequency monitored isn't all that critical.
>>>>>
>>>>>Joe Gwinn
>>>>
>>>>Do connections "start to fail" ?
>>>
>>>Yes, they do, in things like vias. I went through a big drama where a
>>>critical bit of radar logic circuitry would slowly go nuts.
>>>
>>>It turned out that the copper plating on the walls of the vias was
>>>suffering from low-cycle fatigue during temperature cycling and slowly
>>>breaking, one little crack at a time, until it went open. If you
>>>measured the resistance to parts per million (6.5 digit DMM), sampling
>>>at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to
>>>also measure a copper line, and divide the via-chain resistance by the
>>>no-via resistance, to correct for temperature changes.
>>
>>But nobody is going to monitor every via on a PCB, even if it were
>>possible.
>
>It was not possible to test the vias on the failing logic board, but
>we knew from metallurgical cut, polish, and inspect studies of failed
>boards that it was the vias that were failing.
>
>
>>One could instrument a PCB fab test board, I guess. But DC tests would
>>be fine.
>
>What was being tested was a fab test board that had both the series
>via chain path and the no-via path of roughly the same DC resistance,
>set up so we could do 4-wire Kelvin resistance measurements of each
>path independent of the other path.

Yes, but the question was whether one could predict the failure of an
operating electronic gadget. The answer is mostly NO.

We had a visit from the quality team from a giant company that you
have heard of. They wanted us to trend analyze all the power supplies
on our boards and apply a complex algotithm to predict failures. It
was total nonsense, basically predicting the future by zooming in on
random noise with a big 1/f component, just like climate prediction.

>
>
>>We have one board with over 4000 vias, but they are mostly in
>>parallel.
>
>This can also be tested , but using a 6.5-digit DMM intended for
>measuring very low resistance values. A change of one part in 4,000
>is huge to a 6.5-digit instrument. The conductivity will decline
>linearly as vias fail one by one.
>
>

Millikelvin temperature changes would make more signal than a failing
via.

>>>The solution was to redesign the vias, mainly to increase the critical
>>>volume of copper. And modern SMD designs have less and less copper
>>>volume.
>>>
>>>I bet precision resistors can also be measured this way.
>>>
>>>
>>>>I don't think I've ever owned a piece of electronic equipment that
>>>>warned me of an impending failure.
>>>
>>>Onset of smoke emission is a common sign.
>>>
>>>
>>>>Cars do, for some failure modes, like low oil level.
>>>
>>>The industrial method for big stuff is accelerometers attached near
>>>the bearings, and listen for excessive rotation-correlated (not
>>>necessarily harmonic) noise.
>>
>>Big ships that I've worked on have a long propeller shaft in the shaft
>>alley, a long tunnel where nobody often goes. They have magnetic shaft
>>runout sensors and shaft bearing temperature monitors.
>>
>>They measure shaft torque and SHP too, from the shaft twist.
>
>Yep. And these kinds of things fail slowly. At first.

They could repair a bearing at sea, given a heads-up about violent
failure. A serious bearing failure on a single-screw machine means
getting a seagoing tug.

The main engine gearbox had padlocks on the covers.

There was also a chem lab to analyze oil and water and such, looking
for contaminamts that might suggest something going on.

>
>
>>I liked hiding out in the shaft alley. It was private and cool, that
>>giant shaft slowly rotating.
>
>Probably had a calming flowing water sound as well.

Yes, cool and beautiful and serene after the heat and noise and
vibration of the engine room. A quiet 32,000 horsepower.

It was fun being an electronic guru on sea trials of a ship full of
big hairy Popeye types. I, skinny gawky kid, got my own stateroom when
other tech reps slept in cots in the hold.

Have you noticed how many lumberjack types are afraid of electricity?
That can be funny.

>
>Joe Gwinn