When a computer recieves an interrupt signal, it needs to save
the complete machine state, so that upon return from the
interrupt, the program thus interrupted is in no way affected.

This is because interrupts can happen at any time, and thus
programs don't prepare for them or expect them. Any disturbance
to the contents of any register would risk causing programs to
crash.

The Concertina II has a potentially large machine state which most
programs do not use. There are vector registers, of the huge
kind found in the Cray I. There are banks of 128 registers to
supplement the banks of 32 registers.

One obvious step in addressing this is for programs that don't
use these registers to run without access to those registers.
If this is indicated in the PSW, then the interrupt routine will
know what it needs to save and restore.

A more elaborate and more automated method is also possible.

Let us imagine the computer speeds up interrupts by having a
second bank of registers that interrupt routines use. But two
register banks aren't enough, as many user programs are running
concurrently.

Here is how I envisage the sequence of events in response to
an interrupt could work:

1) The computer, at the beginning of an area of memory
sufficient to hold all the contents of the computer's
registers, including the PSW and program counter, places
a _restore status_.

2) The computer switches to the interrupt register bank,
and places a pointer to the restore status in one of the
registers, according to a known convention.

3) As the interrupt service routine runs, the computer,
separately and in the background, saves the registers of the
interrupted program into memory. Once this is complete, the
_restore status_ value in memory is changed to reflect this.

4) The restore status value has _two_ uses.

One is, obviously, that when returning from an interrupt,
there will be a 'return from interrupt' routine that will
either just switch register banks, if the registers aren't
saved yet, or re-fill the registers that are actually in
use (the register status also indicating what the complement
of registers available to the interrupted program was) from
memory.

The other is that the restore status can be tested. If the
main register set isn't saved yet, then it's too soon after
the interrupt to *switch to another user program* which also
would use the main register set, but with a different set
of saved values.

5) Some other factors complicate this.

There may be multiple sets of user program registers to
facilitate SMT.

The standard practice in an operating system is to leave
the privileged interrupt service routine as quickly as
possible, and continue handling the interrupt in an
unprivileged portion of the operating system. However, the
return from interrupt instruction is obviously privileged,
as it allows one to put an arbitrary value from memory into
the Program Status Word, including one that would place
the computer into a privileged state after the return.

That last is not unique to the Concertina II, however. So
the obvious solution of allowing the kernel to call
unprivileged subroutines - which terminate in a supervisor
call rather than a normal return - has been found.

John Savard

Quadibloc <quadibloc@servername.invalid> writes:
>When a computer recieves an interrupt signal, it needs to save
>the complete machine state, so that upon return from the
>interrupt, the program thus interrupted is in no way affected.
>
>This is because interrupts can happen at any time, and thus
>programs don't prepare for them or expect them. Any disturbance
>to the contents of any register would risk causing programs to
>crash.

Something needs to preserve state, either the hardware or
the software. Most risc processors lean towards the latter,
generally for good reason - one may not need to save
all the state if the interrupt handler only touchs part of it.

>
>The Concertina II has a potentially large machine state which most
>programs do not use. There are vector registers, of the huge
>kind found in the Cray I. There are banks of 128 registers to
>supplement the banks of 32 registers.
>
>One obvious step in addressing this is for programs that don't
>use these registers to run without access to those registers.
>If this is indicated in the PSW, then the interrupt routine will
>know what it needs to save and restore.

Just like x86 floating point.

>
>A more elaborate and more automated method is also possible.
>
>Let us imagine the computer speeds up interrupts by having a
>second bank of registers that interrupt routines use. But two
>register banks aren't enough, as many user programs are running
>concurrently.
>
>Here is how I envisage the sequence of events in response to
>an interrupt could work:
>
>1) The computer, at the beginning of an area of memory
>sufficient to hold all the contents of the computer's
>registers, including the PSW and program counter, places
>a _restore status_.

Slow DRAM or special SRAMs? The former will add
considerable latency to an interrupt, the later costs
area (on a per-hardware-thread basis) and floorplanning
issues.

Best is to save the minimal amount of state in hardware
and let software deal with the rest, perhaps with
hints from the hardware (e.g. a bit that indicates
whether the FPRs were modified since the last context
switch, etc).

Quadibloc wrote:
> When a computer recieves an interrupt signal, it needs to save
> the complete machine state, so that upon return from the
> interrupt, the program thus interrupted is in no way affected.

It needs to save the portion of the machine state overwritten by that
interrupt. Often this is a small subset of the whole state because many
interrupts only need a few integer registers, maybe 6 or 7.
Additional integer state will be saved and restored as needed by the call ABI
so does not need to be done for every interrupt by the handler prologue.

This allows the OS to only save and restore the full state on thread
switch which only happens after something significant occurs that
changes the highest priority thread on a particular core.
This occurs much less than the frequency of interrupts.

> This is because interrupts can happen at any time, and thus
> programs don't prepare for them or expect them. Any disturbance
> to the contents of any register would risk causing programs to
> crash.
>
> The Concertina II has a potentially large machine state which most
> programs do not use. There are vector registers, of the huge
> kind found in the Cray I. There are banks of 128 registers to
> supplement the banks of 32 registers.

I have trouble imagining what an interrupt handler might use vectors for.

Some OS's deal with this by specifying that drivers can only use integers.
(Graphics drivers get special dispensation but they run in special context.)

> One obvious step in addressing this is for programs that don't
> use these registers to run without access to those registers.
> If this is indicated in the PSW, then the interrupt routine will
> know what it needs to save and restore.
>
> A more elaborate and more automated method is also possible.
>
> Let us imagine the computer speeds up interrupts by having a
> second bank of registers that interrupt routines use. But two
> register banks aren't enough, as many user programs are running
> concurrently.

A consequence of this approach is that the second bank of interrupt
registers are architecturally visible, which means they tie down
resources like physical registers for all those interrupt registers
even when not in use.

And since there are multiple priority interrupt levels each with a bank,
you quickly wind up with hundreds of physical registers mostly sitting
around doing nothing but tied to other contexts.

Quadibloc wrote:

> When a computer recieves an interrupt signal, it needs to save
> the complete machine state, so that upon return from the
> interrupt, the program thus interrupted is in no way affected.

State needs to be saved, whether SW or HW does the save is a free
variable.

> This is because interrupts can happen at any time, and thus
> programs don't prepare for them or expect them. Any disturbance
> to the contents of any register would risk causing programs to
> crash.

Also note:: the ABA problem can happen when an interrupt transpires
in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
before transferring control to the interrupt handler.

> The Concertina II has a potentially large machine state which most
> programs do not use. There are vector registers, of the huge
> kind found in the Cray I. There are banks of 128 registers to
> supplement the banks of 32 registers.

In the past, each register set can be guarded by an in-use bit
and save/restore can be avoided when not in-use.

Then again, My 66000 only has 64 registers, and one reason for this
is to keep context switch times minimal, and the SW path from one
context to the next minimal.

> One obvious step in addressing this is for programs that don't
> use these registers to run without access to those registers.

Or not have them to begin with:: Take Vector Registers:: My 66000
Virtual Vector Method (VVM) enables HW to vectorize a loop with
the property that if an interrupt or exception transpires, the
loop collapses into Scalar mode and the interrupt handler remains
blissfully unaware the the instruction raising the exception is at
a precise point. When control returns the rest of one loop runs in
Scalar mode and when control transfers back to the top of the loop
the loop is re-vectorized. This costs 2 instructions (VEC and LOOP)
and 6-bits of state in PSL

> If this is indicated in the PSW, then the interrupt routine will
> know what it needs to save and restore.

I use PSL instead of PSW because the amount of state is a cache line
not a word, doubleword, or quadword. But space here is at a premium.

> A more elaborate and more automated method is also possible.

> Let us imagine the computer speeds up interrupts by having a
> second bank of registers that interrupt routines use. But two
> register banks aren't enough, as many user programs are running
> concurrently.

You are ignoring several interesting facts with modern interrupt
systems.

a) each GuestOS has its own interrupt table(s) and the Hypervisor
has its own interrupt tables.

b) multiprocessing is a given. There are situations where an interrupt
is sent from a device and a number of cores can respond. The proper
core is the one running at the lowest priority level, and the core
that gets there first.

In My 66000 case, the interrupt is recognized by the core detecting a
write to the raised interrupt register, going out and fetching the
interrupt #. This bus transaction can return "here is the interrupt
#" or it can respond with "someone else already got it". In the later
case, the core goes on doing its current thing. In the former case,
core responds with "I am going to handle this one", and the interrupt
controller acknowledges the interrupt.

Until the interrupt # returns, the core continues processing whatever
it was doing, and then the core goes out and fetches the 5 cache lines
of the interrupt dispatcher thread, as these lines arrive, they displace
the current thread lines. So the restore happens before the save, the
save after the reload, and arriving data pushes out current data.

When control arrives at interrupt dispatcher it has a complete set of
registers (minus 2 used to tell the dispatcher what interrupt it is
to handle.) So, the receiving thread has its SP, its FP, its Root
pointer, its ASID, and for all practical purposes it begins running
as if the first instruction at the dispatcher saw the 30 register
the last instruction performed the last time this thread ran. Thus,
it can hold any variety of pointers to various data structures the
OS deems relevant.

The interrupt dispatcher is 4 instructions long, the first checks
that the interrupt is in bounds, the second directs control out of
if the interrupt is not within bounds, the third transfers control
the the handler (CALL) and when the handler returns control attempts
to return to the interrupted thread.

Many times the handler SoftIRQs (or DPCs) cleanup handlers. The
SuperVisor Return instruction (SVR) checks if there are scheduled
thread above the thread we are attempting to return control to,
and transfers control too them before transferring control to
the interrupted thread. There is no need for SW to sort this out.

> Here is how I envisage the sequence of events in response to
> an interrupt could work:

> 1) The computer, at the beginning of an area of memory
> sufficient to hold all the contents of the computer's
> registers, including the PSW and program counter, places
> a _restore status_.

I found no reason and no rational to assign any of this to
any specific place in memory.

Instead, the entire Supervisor "Stack" is in a set of control
registers, and from those control registers we can find any
thread and all state associated with them.

The Supervisor Stack contains 4 CRs that point at {HV, GuestHV,
GuestOS, Application} a 2-bit "who is currently in charge" a
6-bit priority, an interrupt table pointer, and a dispatcher.
It is (also) 1 cache line long.

> 2) The computer switches to the interrupt register bank,
> and places a pointer to the restore status in one of the
> registers, according to a known convention.

> 3) As the interrupt service routine runs, the computer,
> separately and in the background, saves the registers of the
> interrupted program into memory. Once this is complete, the
> _restore status_ value in memory is changed to reflect this.

> 4) The restore status value has _two_ uses.

> One is, obviously, that when returning from an interrupt,
> there will be a 'return from interrupt' routine that will
> either just switch register banks, if the registers aren't
> saved yet, or re-fill the registers that are actually in
> use (the register status also indicating what the complement
> of registers available to the interrupted program was) from
> memory.

You are going to have a lot of complications with SoftIRQ/DPCs
doing it this way.

> The other is that the restore status can be tested. If the
> main register set isn't saved yet, then it's too soon after
> the interrupt to *switch to another user program* which also
> would use the main register set, but with a different set
> of saved values.

> 5) Some other factors complicate this.

> There may be multiple sets of user program registers to
> facilitate SMT.

> The standard practice in an operating system is to leave
> the privileged interrupt service routine as quickly as
> possible, and continue handling the interrupt in an
> unprivileged portion of the operating system.

This is the SoftIRQs and DPCs.

> However, the
> return from interrupt instruction is obviously privileged,

Why ?? If you are not "in" an interrupt it can raise OPERATION
exception WHITHOUT BEING PRIVILEGEd !! {{Hint: you don't want
a privileged thread to perform a return from interrupt unless
you are IN an interrupt EITHER.}}

> as it allows one to put an arbitrary value from memory into
> the Program Status Word, including one that would place
> the computer into a privileged state after the return.

> That last is not unique to the Concertina II, however. So
> the obvious solution of allowing the kernel to call
> unprivileged subroutines - which terminate in a supervisor
> call rather than a normal return - has been found.

How does privilege get restored on return ??

> John Savard

EricP wrote:

> Quadibloc wrote:
>> When a computer recieves an interrupt signal, it needs to save
>> the complete machine state, so that upon return from the
>> interrupt, the program thus interrupted is in no way affected.

> It needs to save the portion of the machine state overwritten by that
> interrupt. Often this is a small subset of the whole state because many
> interrupts only need a few integer registers, maybe 6 or 7.

Would an interrupt handler not run fewer instructions if its register
state was seeded with pointers of use to the device(s) being serviced ??

> Additional integer state will be saved and restored as needed by the call ABI
> so does not need to be done for every interrupt by the handler prologue.

> This allows the OS to only save and restore the full state on thread
> switch which only happens after something significant occurs that
> changes the highest priority thread on a particular core.

Doesn't running a SoftIRQ (or DPC) require a full register state ??
And don't most device initerrupts need to SoftIRQ ??
{{Yes, I see timer interrupts not needing so much of that}}

> This occurs much less than the frequency of interrupts.

>> This is because interrupts can happen at any time, and thus
>> programs don't prepare for them or expect them. Any disturbance
>> to the contents of any register would risk causing programs to
>> crash.
>>
>> The Concertina II has a potentially large machine state which most
>> programs do not use. There are vector registers, of the huge
>> kind found in the Cray I. There are banks of 128 registers to
>> supplement the banks of 32 registers.

> I have trouble imagining what an interrupt handler might use vectors for.

Memory to memory move from Disk Cache to User Buffer.
SoftIRQ might use Vector arithmetic to verify CRC, Encryption, ...

> Some OS's deal with this by specifying that drivers can only use integers.
> (Graphics drivers get special dispensation but they run in special context.)

On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
> Quadibloc wrote:
>
>> When a computer recieves an interrupt signal, it needs to save
>> the complete machine state, so that upon return from the
>> interrupt, the program thus interrupted is in no way affected.
>
> State needs to be saved, whether SW or HW does the save is a free variable.
>
>> This is because interrupts can happen at any time, and thus
>> programs don't prepare for them or expect them. Any disturbance
>> to the contents of any register would risk causing programs to
>> crash.
>
> Also note:: the ABA problem can happen when an interrupt transpires
> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
> before transferring control to the interrupt handler.
[...]

Just to be clear an interrupt occurring within the hardware
implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
should not effect the outcome of the CAS. Actually, it should not happen
at all, right? CAS does not have any spurious failures.

mitchalsup@aol.com (MitchAlsup1) writes:
>EricP wrote:
>
>> Quadibloc wrote:
>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.
>
>> It needs to save the portion of the machine state overwritten by that
>> interrupt. Often this is a small subset of the whole state because many
>> interrupts only need a few integer registers, maybe 6 or 7.
>
>Would an interrupt handler not run fewer instructions if its register
>state was seeded with pointers of use to the device(s) being serviced ??

I doubt that would have any effect one way or the other on the
number of instructions executed by the handler (a difference of one
instruction isn't significant).

>
>> Additional integer state will be saved and restored as needed by the call ABI
>> so does not need to be done for every interrupt by the handler prologue.
>
>> This allows the OS to only save and restore the full state on thread
>> switch which only happens after something significant occurs that
>> changes the highest priority thread on a particular core.
>
>Doesn't running a SoftIRQ (or DPC) require a full register state ??

Those are run in a separate kernel thread not the interrupt
handler. They havve full context, provided by the thread.

>And don't most device initerrupts need to SoftIRQ ??

Some do, some don't. It really depends on the interrupt
and the operating software (i.e. the linux kernel stack,
which is structured around PCIe semantics).

>> I have trouble imagining what an interrupt handler might use vectors for.
>
>Memory to memory move from Disk Cache to User Buffer.

That's DMA - no cpu intervention required. If you're referring
to a "Soft" disk cache maintained by the kernel (in unix, the
buffer cache or file cache), a dedicated DMA engine that can offload
such transfers would be a better solution than using
vector registers in an interrupt handler which has no business
transferring bulk data.

>SoftIRQ might use Vector arithmetic to verify CRC, Encryption, ...

Both of those are offloaded to on-chip accelerators. Much better
use of area.

Chris M. Thomasson wrote:

> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>> Quadibloc wrote:
>>
>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.
>>
>> State needs to be saved, whether SW or HW does the save is a free variable.
>>
>>> This is because interrupts can happen at any time, and thus
>>> programs don't prepare for them or expect them. Any disturbance
>>> to the contents of any register would risk causing programs to
>>> crash.
>>
>> Also note:: the ABA problem can happen when an interrupt transpires
>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>> before transferring control to the interrupt handler.
> [...]

> Just to be clear an interrupt occurring within the hardware
> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
> should not effect the outcome of the CAS. Actually, it should not happen
> at all, right? CAS does not have any spurious failures.

ABA failure happens BECAUSE one uses the value of data to decide if
something appeared ATOMIC. The CAS instruction (itself and all variants)
is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
to be compared was fetched without any ATOMIC indicator, and someone else
can alter it before CAS. If more than 1 thread alters the location, it
can (seldom) end up with the same data value as the suspended thread
thought it should be.

CAS is ATOMIC, the code leading to CAS was not and this opens up the
hole.

Note:: CAS functionality implemented with LL/SC does not suffer ABA
because the core monitors the LL address until the SC is performed.
It is an addressed based comparison not a data value based one.

MitchAlsup1 wrote:
> EricP wrote:
>
>> Quadibloc wrote:
>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.
>
>> It needs to save the portion of the machine state overwritten by that
>> interrupt. Often this is a small subset of the whole state because many
>> interrupts only need a few integer registers, maybe 6 or 7.
>
> Would an interrupt handler not run fewer instructions if its register
> state was seeded with pointers of use to the device(s) being serviced ??

It would save a couple of load immediate instructions and
costs a set of preserved registers for each interrupt level.
Not worth the trouble.

>> Additional integer state will be saved and restored as needed by the
>> call ABI
>> so does not need to be done for every interrupt by the handler prologue.
>
>> This allows the OS to only save and restore the full state on thread
>> switch which only happens after something significant occurs that
>> changes the highest priority thread on a particular core.
>
> Doesn't running a SoftIRQ (or DPC) require a full register state ??
> And don't most device initerrupts need to SoftIRQ ??
> {{Yes, I see timer interrupts not needing so much of that}}

No, not if you layer the software appropriately.

The ISR prologue saves the non-preserved register subset (say R0 to R7).
The First Level Interrupt Handler (FLIH) determines whether to
restore the saved register subset or jump into the OS.
If it decides to jump into the OS then R0:R7 are already saved on the stack,
and since this was a FLIH that stack must be the prior thread kernel stack.
So that leaves R8:R31 as still containing the prior threads values.

You call whatever routines you like, when they return to this routine
R8:R31 will still contain the prior thread's data. Only when you decide
to switch threads do you needs to spill R8:R31, into the thread header
context save area, plus any float, simd or vector registers,
and then save the kernel stack pointer there so you can find the values
you pushed in the prologue (if you need to edit the thread context).

You then switch thread header pointers to the new thread's.

To load the next thread you pick up from the new thread context R8:R31
and float, simd, vector registers, and the kernel stack pointer.
Its R0:R7 remain on the kernel stack where they were left when saved.

You can now return to the ISR epilogue which pops R0:R7 for this new
thread and REI Return from Exception or Interrupt to run the new thread.

>> This occurs much less than the frequency of interrupts.
>
>>> This is because interrupts can happen at any time, and thus
>>> programs don't prepare for them or expect them. Any disturbance
>>> to the contents of any register would risk causing programs to
>>> crash.
>>>
>>> The Concertina II has a potentially large machine state which most
>>> programs do not use. There are vector registers, of the huge
>>> kind found in the Cray I. There are banks of 128 registers to
>>> supplement the banks of 32 registers.
>
>> I have trouble imagining what an interrupt handler might use vectors for.
>
> Memory to memory move from Disk Cache to User Buffer.
> SoftIRQ might use Vector arithmetic to verify CRC, Encryption, ...

I was thinking of device interrupt service routines but yeah
the DPC/SoftIRQ routines might do this.

I would have those routines that do want this manually save and restore
any non-integer registers. There may be other sync issues to deal with
(pending exceptions intended for the prior thread context).

A set of utility Hardware Abstraction Layer (HAL) subroutines could
handle this for each platform.

>> Some OS's deal with this by specifying that drivers can only use
>> integers.
>> (Graphics drivers get special dispensation but they run in special
>> context.)

On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
> Quadibloc wrote:

>> When a computer recieves an interrupt signal, it needs to save
>> the complete machine state, so that upon return from the
>> interrupt, the program thus interrupted is in no way affected.
>
> It needs to save the portion of the machine state overwritten by that
> interrupt. Often this is a small subset of the whole state because many
> interrupts only need a few integer registers, maybe 6 or 7.
> Additional integer state will be saved and restored as needed by the call ABI
> so does not need to be done for every interrupt by the handler prologue.

Yes, that is true in many cases.

> I have trouble imagining what an interrupt handler might use vectors for.

However, you're apparently forgetting one very important case.

What if the interrupt is a *real-time clock* interrupt, and what is going
to happen is that the computer will _not_ return immediately from that
interrupt to the interrupted program, but instead, regarding it as
compute-bound, proceed to a different program possibly even belonging
to another user?

So you're quite correct that the problem does not _always_ arise. But it
does arise on occasion.

John Savard

Quadibloc wrote:
> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>> Quadibloc wrote:
>
>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.
>> It needs to save the portion of the machine state overwritten by that
>> interrupt. Often this is a small subset of the whole state because many
>> interrupts only need a few integer registers, maybe 6 or 7.
>> Additional integer state will be saved and restored as needed by the call ABI
>> so does not need to be done for every interrupt by the handler prologue.
>
> Yes, that is true in many cases.
>
>> I have trouble imagining what an interrupt handler might use vectors for.
>
> However, you're apparently forgetting one very important case.
>
> What if the interrupt is a *real-time clock* interrupt, and what is going
> to happen is that the computer will _not_ return immediately from that
> interrupt to the interrupted program, but instead, regarding it as
> compute-bound, proceed to a different program possibly even belonging
> to another user?
>
> So you're quite correct that the problem does not _always_ arise. But it
> does arise on occasion.
>
> John Savard

No, I'm not forgetting that. Return from Exception or Interrupt (REI)
has two possible paths, return to what it was doing before or jump into
the OS and do more processing. On many platforms this particular piece
of code is long, complicated, and riddled with race conditions.

But designing an REI mechanism that makes this sequence
simple, efficient and fast is a separate issue.

For interrupts there are two main pieces of code, the Interrupt Service
Routine (ISR), and post processing routine DPC/SoftIrq.

The ISR for a particular device is called by the OS in response
to a hardware priority interrupt. The ISR may decide it needs further
processing but does not want to block other interrupts while doing it
so ISR posts a request for deferred post processing.

There also can be many restrictions on what an ISR is allowed to do
because the OS designers did not want to, say, force every ISR to
sync with the slow x87 FPU just in case someone wanted to use it.

I would not assume that anything other than integer registers would be
available in an ISR.

In a post processing DPC/SoftIrq routine it might be possible but again
there can be limitations. What you don't ever want to happen is to hang
the cpu waiting to sync with a piece of hardware so you can save its state,
as might happen if it was a co-processor. You also don't want to have to
save any state just in case a post routine might want to do something,
but rather save/restore the state on demand and just what is needed.
So it really depends on the device and the platform.

On 1/17/2024 4:01 PM, MitchAlsup1 wrote:
> Chris M. Thomasson wrote:
>
>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>> Quadibloc wrote:
>>>
>>>> When a computer recieves an interrupt signal, it needs to save
>>>> the complete machine state, so that upon return from the
>>>> interrupt, the program thus interrupted is in no way affected.
>>>
>>> State needs to be saved, whether SW or HW does the save is a free
>>> variable.
>>>
>>>> This is because interrupts can happen at any time, and thus
>>>> programs don't prepare for them or expect them. Any disturbance
>>>> to the contents of any register would risk causing programs to
>>>> crash.
>>>
>>> Also note:: the ABA problem can happen when an interrupt transpires
>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>> before transferring control to the interrupt handler.
>> [...]
>
>> Just to be clear an interrupt occurring within the hardware
>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>> should not effect the outcome of the CAS. Actually, it should not
>> happen at all, right? CAS does not have any spurious failures.
>
> ABA failure happens BECAUSE one uses the value of data to decide if
> something appeared ATOMIC. The CAS instruction (itself and all variants)
> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
> to be compared was fetched without any ATOMIC indicator, and someone else
> can alter it before CAS. If more than 1 thread alters the location, it
> can (seldom) end up with the same data value as the suspended thread
> thought it should be.

Yup. Fwiw, some years ago I actually tried to BURN a CAS by creating
several rouge threads that would alter the CAS target using random
numbers at full speed ahead. The interesting part is that forward
progress was damaged for sure, but still occurred. It did not live lock
on me. Interesting.

> CAS is ATOMIC, the code leading to CAS was not and this opens up the hole.

Indeed.

> Note:: CAS functionality implemented with LL/SC does not suffer ABA
> because the core monitors the LL address until the SC is performed.
> It is an addressed based comparison not a data value based one.

Exactly. Actually, I asked myself if I just wrote a stupid question to
you. Sorry Mitch... ;^)

On 1/18/2024 1:05 PM, Chris M. Thomasson wrote:
> On 1/17/2024 4:01 PM, MitchAlsup1 wrote:
>> Chris M. Thomasson wrote:
>>
>>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>>> Quadibloc wrote:
>>>>
>>>>> When a computer recieves an interrupt signal, it needs to save
>>>>> the complete machine state, so that upon return from the
>>>>> interrupt, the program thus interrupted is in no way affected.
>>>>
>>>> State needs to be saved, whether SW or HW does the save is a free
>>>> variable.
>>>>
>>>>> This is because interrupts can happen at any time, and thus
>>>>> programs don't prepare for them or expect them. Any disturbance
>>>>> to the contents of any register would risk causing programs to
>>>>> crash.
>>>>
>>>> Also note:: the ABA problem can happen when an interrupt transpires
>>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>>> before transferring control to the interrupt handler.
>>> [...]
>>
>>> Just to be clear an interrupt occurring within the hardware
>>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>>> should not effect the outcome of the CAS. Actually, it should not
>>> happen at all, right? CAS does not have any spurious failures.
>>
>> ABA failure happens BECAUSE one uses the value of data to decide if
>> something appeared ATOMIC. The CAS instruction (itself and all variants)
>> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
>> to be compared was fetched without any ATOMIC indicator, and someone else
>> can alter it before CAS. If more than 1 thread alters the location, it
>> can (seldom) end up with the same data value as the suspended thread
>> thought it should be.
>
> Yup. Fwiw, some years ago I actually tried to BURN a CAS by creating
> several rouge threads that would alter the CAS target using random
> numbers at full speed ahead. The interesting part is that forward
> progress was damaged for sure, but still occurred. It did not live lock
> on me. Interesting.

I compared CAS successes vs failures. When the rouge threads were turned
on, the failure rate went way up, but not to a point where an actual
sustained livelock occurred.

>> CAS is ATOMIC, the code leading to CAS was not and this opens up the
>> hole.
>
> Indeed.
>
>
>> Note:: CAS functionality implemented with LL/SC does not suffer ABA
>> because the core monitors the LL address until the SC is performed.
>> It is an addressed based comparison not a data value based one.
>
> Exactly. Actually, I asked myself if I just wrote a stupid question to
> you. Sorry Mitch... ;^)

MitchAlsup1 wrote:
> Chris M. Thomasson wrote:
>
>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>> Quadibloc wrote:
>>>
>>>> When a computer recieves an interrupt signal, it needs to save
>>>> the complete machine state, so that upon return from the
>>>> interrupt, the program thus interrupted is in no way affected.
>>>
>>> State needs to be saved, whether SW or HW does the save is a free
>>> variable.
>>>
>>>> This is because interrupts can happen at any time, and thus
>>>> programs don't prepare for them or expect them. Any disturbance
>>>> to the contents of any register would risk causing programs to
>>>> crash.
>>>
>>> Also note:: the ABA problem can happen when an interrupt transpires
>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>> before transferring control to the interrupt handler.
>> [...]
>
>> Just to be clear an interrupt occurring within the hardware
>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>> should not effect the outcome of the CAS. Actually, it should not
>> happen at all, right? CAS does not have any spurious failures.
>
> ABA failure happens BECAUSE one uses the value of data to decide if
> something appeared ATOMIC. The CAS instruction (itself and all variants)
> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
> to be compared was fetched without any ATOMIC indicator, and someone else
> can alter it before CAS. If more than 1 thread alters the location, it
> can (seldom) end up with the same data value as the suspended thread
> thought it should be.
>
> CAS is ATOMIC, the code leading to CAS was not and this opens up the hole.
>
> Note:: CAS functionality implemented with LL/SC does not suffer ABA
> because the core monitors the LL address until the SC is performed.
> It is an addressed based comparison not a data value based one.

Yes but an equal point of view is that LL/SC only emulates atomic and
uses the cache line ownership grab while "locked" to detect possible
interference and infer potential change.

Note that if LL/SC is implemented with temporary line pinning
(as might be done to guarantee forward progress and prevent ping-pong)
then it cannot be interfered with, and CAS and atomic-fetch-op sequences
are semantically identical to the equivalent single instructions
(which may also be implemented with temporary line pinning if their
data must move from cache through the core and back).

Also LL/SC as implemented on Alpha, MIPS, Power, ARM, RISC-V don't allow
any other location loads or stores between them so really aren't useful
for detecting ABA because detecting it requires monitoring two memory
locations for change.

The classic example is the single linked list with items head->A->B->C
Detecting ABA requires monitoring if either head or head->Next change
which LL/SC cannot do as reading head->Next cancels the lock on head.

x86 has cmpxchg8b and ARM has double wide LL/SC which can be used to
implement CASD atomic-double-wide-compare-and-swap. The first word holds
the head pointer and the second word holds a generation counter whose
change is used to infer that head->Next might have changed.

On 1/19/2024 9:10 AM, EricP wrote:
> MitchAlsup1 wrote:
>> Chris M. Thomasson wrote:
>>
>>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>>> Quadibloc wrote:
>>>>
>>>>> When a computer recieves an interrupt signal, it needs to save
>>>>> the complete machine state, so that upon return from the
>>>>> interrupt, the program thus interrupted is in no way affected.
>>>>
>>>> State needs to be saved, whether SW or HW does the save is a free
>>>> variable.
>>>>
>>>>> This is because interrupts can happen at any time, and thus
>>>>> programs don't prepare for them or expect them. Any disturbance
>>>>> to the contents of any register would risk causing programs to
>>>>> crash.
>>>>
>>>> Also note:: the ABA problem can happen when an interrupt transpires
>>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>>> before transferring control to the interrupt handler.
>>> [...]
>>
>>> Just to be clear an interrupt occurring within the hardware
>>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>>> should not effect the outcome of the CAS. Actually, it should not
>>> happen at all, right? CAS does not have any spurious failures.
>>
>> ABA failure happens BECAUSE one uses the value of data to decide if
>> something appeared ATOMIC. The CAS instruction (itself and all variants)
>> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
>> to be compared was fetched without any ATOMIC indicator, and someone else
>> can alter it before CAS. If more than 1 thread alters the location, it
>> can (seldom) end up with the same data value as the suspended thread
>> thought it should be.
>>
>> CAS is ATOMIC, the code leading to CAS was not and this opens up the
>> hole.
>>
>> Note:: CAS functionality implemented with LL/SC does not suffer ABA
>> because the core monitors the LL address until the SC is performed.
>> It is an addressed based comparison not a data value based one.
>
> Yes but an equal point of view is that LL/SC only emulates atomic and
> uses the cache line ownership grab while "locked" to detect possible
> interference and infer potential change.
>
> Note that if LL/SC is implemented with temporary line pinning
> (as might be done to guarantee forward progress and prevent ping-pong)
> then it cannot be interfered with, and CAS and atomic-fetch-op sequences
> are semantically identical to the equivalent single instructions
> (which may also be implemented with temporary line pinning if their
> data must move from cache through the core and back).
>
> Also LL/SC as implemented on Alpha, MIPS, Power, ARM, RISC-V don't allow
> any other location loads or stores between them so really aren't useful
> for detecting ABA because detecting it requires monitoring two memory
> locations for change.
>
> The classic example is the single linked list with items head->A->B->C
> Detecting ABA requires monitoring if either head or head->Next change
> which LL/SC cannot do as reading head->Next cancels the lock on head.
>
> x86 has cmpxchg8b and ARM has double wide LL/SC which can be used to
> implement CASD atomic-double-wide-compare-and-swap. The first word holds
> the head pointer and the second word holds a generation counter whose
> change is used to infer that head->Next might have changed.
>
>

Pretty much it. ABA is explicitly mentioned in an IBM principals of
operation wrt Free-Pool Manipulation in an appendix A-48:

https://www.ibm.com/support/pages/sites/default/files/inline-files/SA22-7832-00.pdf

The problem will destroy the integrity of the list. Perform Locked
Operation (PLO) in the same appendix is pretty interesting as well... ;^)

On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
> Quadibloc wrote:

>> When a computer recieves an interrupt signal, it needs to save
>> the complete machine state, so that upon return from the
>> interrupt, the program thus interrupted is in no way affected.

> It needs to save the portion of the machine state overwritten by that
> interrupt. Often this is a small subset of the whole state because many
> interrupts only need a few integer registers, maybe 6 or 7.
> Additional integer state will be saved and restored as needed by the call ABI
> so does not need to be done for every interrupt by the handler prologue.

Having been so concerned by the large machine state of the Concertina
II, parts of which were rarely used, and not realizing the conventional
approach was entirely adequate... I missed what was the biggest flaw in
interrupts on the Concertina II.

Because in some important ways it is patterned after the IBM System/360,
it shares its biggest problem with interrupts.

On the System/360, it is a *convention* that the last few general registers,
registers 11, 12, 13, 14, and 15 or so, are used as base registers. A
base register *must* be properly set up before a program can write any data
to memory.

So one can't just have an interrupt behave like on an 8-bit microprocessor,
saving only the program counter and the status bits, and leaving any
registers to be saved by software. At least some of the general registers
have to be saved, and set up with new starting values, for the interrupt
routine to be able to save anything else, if need be.

Of course, the System/360 was able to solve this problem, so it's not
intractable. But the fact that the System/360 solved it by saving all
sixteen general registers, and then loading them from an area in memory
allocated to that interrupt type, is what fooled me into thinking I
would need to automatically save _everything_. It didn't save the
floating-point registers - software did, if the need was to move to
a different user's program, and saving the state in two separate pieces
by two different parts of the OS did not cause hopeless confusion.

John Savard

On 1/20/2024 12:54 PM, Quadibloc wrote:
> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>> Quadibloc wrote:
>
>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.
>
>> It needs to save the portion of the machine state overwritten by that
>> interrupt. Often this is a small subset of the whole state because many
>> interrupts only need a few integer registers, maybe 6 or 7.
>> Additional integer state will be saved and restored as needed by the call ABI
>> so does not need to be done for every interrupt by the handler prologue.
>
> Having been so concerned by the large machine state of the Concertina
> II, parts of which were rarely used, and not realizing the conventional
> approach was entirely adequate... I missed what was the biggest flaw in
> interrupts on the Concertina II.
>
> Because in some important ways it is patterned after the IBM System/360,
> it shares its biggest problem with interrupts.
>
> On the System/360, it is a *convention* that the last few general registers,
> registers 11, 12, 13, 14, and 15 or so, are used as base registers. A
> base register *must* be properly set up before a program can write any data
> to memory.
>
> So one can't just have an interrupt behave like on an 8-bit microprocessor,
> saving only the program counter and the status bits, and leaving any
> registers to be saved by software. At least some of the general registers
> have to be saved, and set up with new starting values, for the interrupt
> routine to be able to save anything else, if need be.
>

IIRC, saving off PC, some flags bits, swapping the stack registers, and
doing a computed branch relative to a control register (via bit-slicing,
*). This is effectively the interrupt mechanism I am using on a 64-bit ISA.

*: For a table that is generally one-off in the kernel or similar, it
doesn't ask much to mandate that it has a certain alignment. And if the
required alignment is larger than the size of the table, you have just
saved yourself needing an adder...

If anything, it is a little simpler than the mechanism used on some
8-bit systems, which would have needed a mechanism to push these values
to the stack, and restore them from the stack.

Having side-channels that allow these values to "magically appear" in
certain SPRs is at least simpler, though the cost of the logic itself
could be more up for debate.

> Of course, the System/360 was able to solve this problem, so it's not
> intractable. But the fact that the System/360 solved it by saving all
> sixteen general registers, and then loading them from an area in memory
> allocated to that interrupt type, is what fooled me into thinking I
> would need to automatically save _everything_. It didn't save the
> floating-point registers - software did, if the need was to move to
> a different user's program, and saving the state in two separate pieces
> by two different parts of the OS did not cause hopeless confusion.
>

For ISR convenience, it would make sense to have, say, two SPR's or CR's
designated for ISR use to squirrel off values from GPRs to make the
prolog/epilog easier. Had considered this, but not done so (in this
case, first thing the ISR does is save a few registers to the ISR stack
to free them up to get the rest of the registers saved "more properly",
then has to do a secondary reload to get these registers back into the
correct place).

Assuming interrupts aren't too common, then it isn't a huge issue, and
seemingly a majority of the clock-cycles spent on interrupt entry mostly
have to do with L1 misses (since typically pretty much nothing the ISR
touches is in the L1 cache; and in my implementation, ISR's may not
share L1 cache lines with non-ISR code; so basically it is an
architecturally required "L1 miss" storm).

Only real way to avoid a lot of the L1 misses would be to have multiple
sets of banked registers or similar, but, this is not cheap for the
hardware... Similarly, the ISR would need to do as little, and touch as
little memory, as is possible to perform its task.

> John Savard

EricP wrote:

> There also can be many restrictions on what an ISR is allowed to do
> because the OS designers did not want to, say, force every ISR to
> sync with the slow x87 FPU just in case someone wanted to use it.

What about all the architectures that are not x86 and do not need to synch
to FP, Vectors, SIMD, ..... ?? Why are they constrained by the one badly
designed long life architecture ??

> I would not assume that anything other than integer registers would be
> available in an ISR.

This is quite reasonable: as long as you have a sufficient number that
the ISR can be written in some HLL without a bunch of flags to the compiler.

> In a post processing DPC/SoftIrq routine it might be possible but again
> there can be limitations. What you don't ever want to happen is to hang
> the cpu waiting to sync with a piece of hardware so you can save its state,
> as might happen if it was a co-processor. You also don't want to have to
> save any state just in case a post routine might want to do something,
> but rather save/restore the state on demand and just what is needed.
> So it really depends on the device and the platform.

As long as there are not more than one flag to clue the compiler in,
I am on board.

Chris M. Thomasson wrote:

> On 1/17/2024 4:01 PM, MitchAlsup1 wrote:
>> Chris M. Thomasson wrote:
>>
>>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>>> Quadibloc wrote:
>>>>
>>>>> When a computer recieves an interrupt signal, it needs to save
>>>>> the complete machine state, so that upon return from the
>>>>> interrupt, the program thus interrupted is in no way affected.
>>>>
>>>> State needs to be saved, whether SW or HW does the save is a free
>>>> variable.
>>>>
>>>>> This is because interrupts can happen at any time, and thus
>>>>> programs don't prepare for them or expect them. Any disturbance
>>>>> to the contents of any register would risk causing programs to
>>>>> crash.
>>>>
>>>> Also note:: the ABA problem can happen when an interrupt transpires
>>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>>> before transferring control to the interrupt handler.
>>> [...]
>>
>>> Just to be clear an interrupt occurring within the hardware
>>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>>> should not effect the outcome of the CAS. Actually, it should not
>>> happen at all, right? CAS does not have any spurious failures.
>>
>> ABA failure happens BECAUSE one uses the value of data to decide if
>> something appeared ATOMIC. The CAS instruction (itself and all variants)
>> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
>> to be compared was fetched without any ATOMIC indicator, and someone else
>> can alter it before CAS. If more than 1 thread alters the location, it
>> can (seldom) end up with the same data value as the suspended thread
>> thought it should be.

> Yup. Fwiw, some years ago I actually tried to BURN a CAS by creating
> several rouge threads that would alter the CAS target using random
> numbers at full speed ahead. The interesting part is that forward
> progress was damaged for sure, but still occurred. It did not live lock
> on me. Interesting.

>> CAS is ATOMIC, the code leading to CAS was not and this opens up the hole.

> Indeed.

>> Note:: CAS functionality implemented with LL/SC does not suffer ABA
>> because the core monitors the LL address until the SC is performed.
>> It is an addressed based comparison not a data value based one.

> Exactly. Actually, I asked myself if I just wrote a stupid question to
> you. Sorry Mitch... ;^)

No need to be sorry, this is a NG dedicated to make people think, and
then after they have expressed what they though, to correct and refine
what they think and how.

EricP wrote:

> MitchAlsup1 wrote:
>> Chris M. Thomasson wrote:
>>
>>> On 1/17/2024 2:11 PM, MitchAlsup1 wrote:
>>>> Quadibloc wrote:
>>>>
>>>>> When a computer recieves an interrupt signal, it needs to save
>>>>> the complete machine state, so that upon return from the
>>>>> interrupt, the program thus interrupted is in no way affected.
>>>>
>>>> State needs to be saved, whether SW or HW does the save is a free
>>>> variable.
>>>>
>>>>> This is because interrupts can happen at any time, and thus
>>>>> programs don't prepare for them or expect them. Any disturbance
>>>>> to the contents of any register would risk causing programs to
>>>>> crash.
>>>>
>>>> Also note:: the ABA problem can happen when an interrupt transpires
>>>> in the middle of an ATOMIC sequence. Thus, My 66000 fails the event
>>>> before transferring control to the interrupt handler.
>>> [...]
>>
>>> Just to be clear an interrupt occurring within the hardware
>>> implementation of a CAS operation (e.g, lock cmpxchg over on Intel)
>>> should not effect the outcome of the CAS. Actually, it should not
>>> happen at all, right? CAS does not have any spurious failures.
>>
>> ABA failure happens BECAUSE one uses the value of data to decide if
>> something appeared ATOMIC. The CAS instruction (itself and all variants)
>> is ATOMIC, the the setup to CAS is non-ATOMIC, because the original value
>> to be compared was fetched without any ATOMIC indicator, and someone else
>> can alter it before CAS. If more than 1 thread alters the location, it
>> can (seldom) end up with the same data value as the suspended thread
>> thought it should be.
>>
>> CAS is ATOMIC, the code leading to CAS was not and this opens up the hole.
>>
>> Note:: CAS functionality implemented with LL/SC does not suffer ABA
>> because the core monitors the LL address until the SC is performed.
>> It is an addressed based comparison not a data value based one.

> Yes but an equal point of view is that LL/SC only emulates atomic and
> uses the cache line ownership grab while "locked" to detect possible
> interference and infer potential change.

Which, BTW, opens up a different side channel ...

> Note that if LL/SC is implemented with temporary line pinning
> (as might be done to guarantee forward progress and prevent ping-pong)
> then it cannot be interfered with, and CAS and atomic-fetch-op sequences
> are semantically identical to the equivalent single instructions
> (which may also be implemented with temporary line pinning if their
> data must move from cache through the core and back).

Line pinning requires a NAK in the coherence protocol. As far as I know,
only My 66000 interconnect protocol has such a NaK.

> Also LL/SC as implemented on Alpha, MIPS, Power, ARM, RISC-V don't allow
> any other location loads or stores between them so really aren't useful
> for detecting ABA because detecting it requires monitoring two memory
> locations for change.

> The classic example is the single linked list with items head->A->B->C
> Detecting ABA requires monitoring if either head or head->Next change
> which LL/SC cannot do as reading head->Next cancels the lock on head.

Detecting ABA requires one to monitor addresses not data values.

> x86 has cmpxchg8b and ARM has double wide LL/SC which can be used to
> implement CASD atomic-double-wide-compare-and-swap. The first word holds
> the head pointer and the second word holds a generation counter whose
> change is used to infer that head->Next might have changed.

Quadibloc wrote:

> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>> Quadibloc wrote:

>>> When a computer recieves an interrupt signal, it needs to save
>>> the complete machine state, so that upon return from the
>>> interrupt, the program thus interrupted is in no way affected.

>> It needs to save the portion of the machine state overwritten by that
>> interrupt. Often this is a small subset of the whole state because many
>> interrupts only need a few integer registers, maybe 6 or 7.
>> Additional integer state will be saved and restored as needed by the call ABI
>> so does not need to be done for every interrupt by the handler prologue.

> Having been so concerned by the large machine state of the Concertina
> II, parts of which were rarely used, and not realizing the conventional
> approach was entirely adequate... I missed what was the biggest flaw in
> interrupts on the Concertina II.

The second to last word is misleading and unnecessary::

> interrupts on Concertina II.

The implies there will be only one and it already exists.

> Because in some important ways it is patterned after the IBM System/360,
> it shares its biggest problem with interrupts.

> On the System/360, it is a *convention* that the last few general registers,
> registers 11, 12, 13, 14, and 15 or so, are used as base registers. A
> base register *must* be properly set up before a program can write any data
> to memory.

Captain Obvious strikes again...

> So one can't just have an interrupt behave like on an 8-bit microprocessor,
> saving only the program counter and the status bits, and leaving any
> registers to be saved by software. At least some of the general registers
> have to be saved, and set up with new starting values, for the interrupt
> routine to be able to save anything else, if need be.

Once HW starts saving "a few" it might as well save "enough" of them to mater.

> Of course, the System/360 was able to solve this problem, so it's not
> intractable. But the fact that the System/360 solved it by saving all
> sixteen general registers, and then loading them from an area in memory
> allocated to that interrupt type, is what fooled me into thinking I
> would need to automatically save _everything_. It didn't save the
> floating-point registers - software did, if the need was to move to
> a different user's program, and saving the state in two separate pieces
> by two different parts of the OS did not cause hopeless confusion.

> John Savard

BGB wrote:

> On 1/20/2024 12:54 PM, Quadibloc wrote:
>> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>>
>> So one can't just have an interrupt behave like on an 8-bit microprocessor,
>> saving only the program counter and the status bits, and leaving any
>> registers to be saved by software. At least some of the general registers
>> have to be saved, and set up with new starting values, for the interrupt
>> routine to be able to save anything else, if need be.
>>

> IIRC, saving off PC, some flags bits, swapping the stack registers, and
> doing a computed branch relative to a control register (via bit-slicing,
> *). This is effectively the interrupt mechanism I am using on a 64-bit ISA.

And sounds like the interrupt mechanism for an 8-bit µprocessor...

> *: For a table that is generally one-off in the kernel or similar, it
> doesn't ask much to mandate that it has a certain alignment. And if the
> required alignment is larger than the size of the table, you have just
> saved yourself needing an adder...

In a modern system where you have several HyperVisors and a multiplicity
of GuestOSs, a single interrupt table is unworkable looking forward.
What you want and need is every GuestOS to have its own table, and
every HyperVisor have its own table, some kind of routing mechanism to
route device interrupts to the correct table, and inform appropriate
cores of raised and enabled interrupts. All these tables have to be
concurrently available continuously and simultaneously. The old fixed
mapping will no longer work efficiently--you can make them work with
\a near-Herculean amount of carefully programming.

Or you can look at the problem from a modern viewpoint and fix the
model so the above is manifest.

> If anything, it is a little simpler than the mechanism used on some
> 8-bit systems, which would have needed a mechanism to push these values
> to the stack, and restore them from the stack.

Do you think you mechanism would work "well" with 1024 cores in your
system ??

> Having side-channels that allow these values to "magically appear" in
> certain SPRs is at least simpler, though the cost of the logic itself
> could be more up for debate.

Once you have an FMAC FPU none of the interrupt logic adds any area
to any core.

>> Of course, the System/360 was able to solve this problem, so it's not
>> intractable. But the fact that the System/360 solved it by saving all
>> sixteen general registers, and then loading them from an area in memory
>> allocated to that interrupt type, is what fooled me into thinking I
>> would need to automatically save _everything_. It didn't save the
>> floating-point registers - software did, if the need was to move to
>> a different user's program, and saving the state in two separate pieces
>> by two different parts of the OS did not cause hopeless confusion.
>>

> For ISR convenience, it would make sense to have, say, two SPR's or CR's
> designated for ISR use to squirrel off values from GPRs to make the
> prolog/epilog easier. Had considered this, but not done so (in this
> case, first thing the ISR does is save a few registers to the ISR stack
> to free them up to get the rest of the registers saved "more properly",
> then has to do a secondary reload to get these registers back into the
> correct place).

Or, you could take the point of view that your architecture makes context
switching easy (like 1 instruction from application 1 to application 2)
and when you do this the rest of the model pretty much drops in for free.

> Assuming interrupts aren't too common, then it isn't a huge issue, and
> seemingly a majority of the clock-cycles spent on interrupt entry mostly
> have to do with L1 misses (since typically pretty much nothing the ISR
> touches is in the L1 cache; and in my implementation, ISR's may not
> share L1 cache lines with non-ISR code; so basically it is an
> architecturally required "L1 miss" storm).

The fewer cycles before raising the interrupt at the device, and the
servicing of the interrupt by ISR (prior to scheduling a SoftIRQ/DPC)
is the key number. All these little pieces of state that are obtained
1 at a time by running code not in the ICache, along with the manipulations
on the TLBs, ... get in the way of your goal.

> Only real way to avoid a lot of the L1 misses would be to have multiple
> sets of banked registers or similar, but, this is not cheap for the
> hardware... Similarly, the ISR would need to do as little, and touch as
> little memory, as is possible to perform its task.

You can read up thread-state from DRAM and continue what you are working
on until they arrive, and when they arrive, you ship out the old values
as if the core state was a cache in its own right. Thus, you continue to
make progress on the current thread until you have the state needed to
run the ISR thread and when it arrives you have everything you need to
proceed. ...

>> John Savard

mitchalsup@aol.com (MitchAlsup1) writes:

>
>In a modern system where you have several HyperVisors and a multiplicity
>of GuestOSs, a single interrupt table is unworkable looking forward.
>What you want and need is every GuestOS to have its own table, and
>every HyperVisor have its own table, some kind of routing mechanism to
>route device interrupts to the correct table, and inform appropriate
>cores of raised and enabled interrupts. All these tables have to be
>concurrently available continuously and simultaneously. The old fixed
>mapping will no longer work efficiently--you can make them work with
>\a near-Herculean amount of carefully programming.

For an extant implementation thereof, see

GICv3 Architecture Specification.

https://documentation-service.arm.com/static/6012f2e54ccc190e5e681256

On 1/21/2024 3:18 PM, MitchAlsup1 wrote:
> BGB wrote:
>
>> On 1/20/2024 12:54 PM, Quadibloc wrote:
>>> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>>>
>>> So one can't just have an interrupt behave like on an 8-bit
>>> microprocessor,
>>> saving only the program counter and the status bits, and leaving any
>>> registers to be saved by software. At least some of the general
>>> registers
>>> have to be saved, and set up with new starting values, for the interrupt
>>> routine to be able to save anything else, if need be.
>>>
>
>> IIRC, saving off PC, some flags bits, swapping the stack registers,
>> and doing a computed branch relative to a control register (via
>> bit-slicing, *). This is effectively the interrupt mechanism I am
>> using on a 64-bit ISA.
>
> And sounds like the interrupt mechanism for an 8-bit µprocessor...
>

It was partly a simplification of the design from the SH-4, which was a
32-bit CPU mostly used in embedded systems (and in the Sega Dreamcast...).

Though, the SH-4 did bank out half the registers, which was a feature
that ended up being dropped for cost-saving reasons.

>> *: For a table that is generally one-off in the kernel or similar, it
>> doesn't ask much to mandate that it has a certain alignment. And if
>> the required alignment is larger than the size of the table, you have
>> just saved yourself needing an adder...
>
> In a modern system where you have several HyperVisors and a multiplicity
> of GuestOSs, a single interrupt table is unworkable looking forward.
> What you want and need is every GuestOS to have its own table, and
> every HyperVisor have its own table, some kind of routing mechanism to
> route device interrupts to the correct table, and inform appropriate
> cores of raised and enabled interrupts. All these tables have to be
> concurrently available continuously and simultaneously. The old fixed
> mapping will no longer work efficiently--you can make them work with
> \a near-Herculean amount of carefully programming.
>
> Or you can look at the problem from a modern viewpoint and fix the
> model so the above is manifest.
>

Presumably, only the actual "bare metal" layer has an actual
hardware-level interrupt table, and all of the "guest" tables are faked
in software?...

Much like with MMU:
Only the base level needs to actually handle TLB miss events, and
everything else (nested translation, etc), can be left to software
emulation.

>> If anything, it is a little simpler than the mechanism used on some
>> 8-bit systems, which would have needed a mechanism to push these
>> values to the stack, and restore them from the stack.
>
> Do you think you mechanism would work "well" with 1024 cores in your
> system ??
>

Number of cores should not matter that much.

Presumably, each core gets its own ISR stack, which should not have any
reason to need to interact with each other.

For extra speed, maybe the ISR stacks could be mapped to some sort of
core-local SRAM. This hasn't been done yet though.

Idea here being probably the SRAM region could have a special address
range, and any access to this region would be invisible to any other
cores (and it need not need have backing in external RAM).

One could maybe debate the cost of giving each core 4K or 8K of
dedicated local SRAM though merely for "slightly faster interrupt handling".

>> Having side-channels that allow these values to "magically appear" in
>> certain SPRs is at least simpler, though the cost of the logic itself
>> could be more up for debate.
>
> Once you have an FMAC FPU none of the interrupt logic adds any area
> to any core.
>

I don't have conventional FMA because it would have had too much cost
and latency.

Did experimentally add a "double rounded" FMAC that basically allows
gluing the FMUL and FADD units together, with a timing of roughly 12
clock cycles (non-pipelined).

No SIMD MAC operations mostly because this would also need too much
latency (can't really shoe-horn this into 3 cycles). Similar reasoning
to why the SIMD ops are hard-wired for truncate rounding.

>>> Of course, the System/360 was able to solve this problem, so it's not
>>> intractable. But the fact that the System/360 solved it by saving all
>>> sixteen general registers, and then loading them from an area in memory
>>> allocated to that interrupt type, is what fooled me into thinking I
>>> would need to automatically save _everything_. It didn't save the
>>> floating-point registers - software did, if the need was to move to
>>> a different user's program, and saving the state in two separate pieces
>>> by two different parts of the OS did not cause hopeless confusion.
>>>
>
>> For ISR convenience, it would make sense to have, say, two SPR's or
>> CR's designated for ISR use to squirrel off values from GPRs to make
>> the prolog/epilog easier. Had considered this, but not done so (in
>> this case, first thing the ISR does is save a few registers to the ISR
>> stack to free them up to get the rest of the registers saved "more
>> properly", then has to do a secondary reload to get these registers
>> back into the correct place).
>
> Or, you could take the point of view that your architecture makes context
> switching easy (like 1 instruction from application 1 to application 2)
> and when you do this the rest of the model pretty much drops in for free.
>

This would cost well more, on the hardware side, than having two
non-specialized CRs and being like "ISR's are allowed to stomp these at
will, nothing else may use them".

>> Assuming interrupts aren't too common, then it isn't a huge issue, and
>> seemingly a majority of the clock-cycles spent on interrupt entry
>> mostly have to do with L1 misses (since typically pretty much nothing
>> the ISR touches is in the L1 cache; and in my implementation, ISR's
>> may not share L1 cache lines with non-ISR code; so basically it is an
>> architecturally required "L1 miss" storm).
>
> The fewer cycles before raising the interrupt at the device, and the
> servicing of the interrupt by ISR (prior to scheduling a SoftIRQ/DPC)
> is the key number. All these little pieces of state that are obtained
> 1 at a time by running code not in the ICache, along with the manipulations
> on the TLBs, ... get in the way of your goal.
>

Possibly...

Though, as of yet, I don't have any hardware interrupts either (apart
from the timer IRQ). Pretty much all of the IO at present is polling IO.

Granted, it is possible that it would not scale well to do a full
featured system with exclusively polling IO.

Some other tasks can be handled with the microsecond timer and a loop.

Say:
//void DelayUsec(int usec);
DelayUsec:
CPIUD 30
ADD R4, R0, R6
.L0:
CPUID 30
CMPQGT R0, R6
BT .L0
RTS
Which would create a certain amount of delay (in microseconds) relative
to when the function is called.

>> Only real way to avoid a lot of the L1 misses would be to have
>> multiple sets of banked registers or similar, but, this is not cheap
>> for the hardware... Similarly, the ISR would need to do as little, and
>> touch as little memory, as is possible to perform its task.
>
> You can read up thread-state from DRAM and continue what you are working
> on until they arrive, and when they arrive, you ship out the old values
> as if the core state was a cache in its own right. Thus, you continue to
> make progress on the current thread until you have the state needed to
> run the ISR thread and when it arrives you have everything you need to
> proceed. ...
>

Possibly, if the core were more advanced than a naive strictly-in-order
design...

Or, some way of handling memory other than stalling the pipeline
whenever the L1 cache misses.

But, yeah, for a more advanced system, could maybe justify using a
different interrupt-handler mechanism (and accept that possibly kernel
level code may not be entirely binary compatible between the ISA variants).

>>> John Savard

BGB wrote:

> On 1/21/2024 3:18 PM, MitchAlsup1 wrote:
>> BGB wrote:
>>
>>> On 1/20/2024 12:54 PM, Quadibloc wrote:
>>>> On Wed, 17 Jan 2024 15:35:56 -0500, EricP wrote:
>>>>
>>>> So one can't just have an interrupt behave like on an 8-bit
>>>> microprocessor,
>>>> saving only the program counter and the status bits, and leaving any
>>>> registers to be saved by software. At least some of the general
>>>> registers
>>>> have to be saved, and set up with new starting values, for the interrupt
>>>> routine to be able to save anything else, if need be.
>>>>
>>
>>> IIRC, saving off PC, some flags bits, swapping the stack registers,
>>> and doing a computed branch relative to a control register (via
>>> bit-slicing, *). This is effectively the interrupt mechanism I am
>>> using on a 64-bit ISA.
>>
>> And sounds like the interrupt mechanism for an 8-bit µprocessor...
>>

> It was partly a simplification of the design from the SH-4, which was a
> 32-bit CPU mostly used in embedded systems (and in the Sega Dreamcast...).

> Though, the SH-4 did bank out half the registers, which was a feature
> that ended up being dropped for cost-saving reasons.

>>> *: For a table that is generally one-off in the kernel or similar, it
>>> doesn't ask much to mandate that it has a certain alignment. And if
>>> the required alignment is larger than the size of the table, you have
>>> just saved yourself needing an adder...
>>
>> In a modern system where you have several HyperVisors and a multiplicity
>> of GuestOSs, a single interrupt table is unworkable looking forward.
>> What you want and need is every GuestOS to have its own table, and
>> every HyperVisor have its own table, some kind of routing mechanism to
>> route device interrupts to the correct table, and inform appropriate
>> cores of raised and enabled interrupts. All these tables have to be
>> concurrently available continuously and simultaneously. The old fixed
>> mapping will no longer work efficiently--you can make them work with
>> \a near-Herculean amount of carefully programming.
>>
>> Or you can look at the problem from a modern viewpoint and fix the
>> model so the above is manifest.
>>

> Presumably, only the actual "bare metal" layer has an actual
> hardware-level interrupt table, and all of the "guest" tables are faked
> in software?...

> Much like with MMU:
> Only the base level needs to actually handle TLB miss events, and
> everything else (nested translation, etc), can be left to software
> emulation.

Name a single ISA that fakes the TLB ?? (and has an MMU)

>>> If anything, it is a little simpler than the mechanism used on some
>>> 8-bit systems, which would have needed a mechanism to push these
>>> values to the stack, and restore them from the stack.
>>
>> Do you think you mechanism would work "well" with 1024 cores in your
>> system ??
>>

> Number of cores should not matter that much.

Exactly !! but then try running 1024 cores under differing GuestOSs, and
HyperVisors under one set of system-wide Tables !!

> Presumably, each core gets its own ISR stack, which should not have any
> reason to need to interact with each other.

I presume an interrupt can be serviced by any number of cores.
I presume that there are a vast number of devices. Each device assigned
to a few GuestOSs.
I presume the core that services the interrupt (ISR) is running the same
GuestOS under the same HyperVisor that initiated the device.
I presume the core that services the interrupt was of the lowest priority
of all the cores then running that GuestOS.
I presume the core that services the interrupt wasted no time in doing so.

And the GuestOS decides on how its ISR stack is {formatted, allocated, used,
serviced, ...} which can be different for each GuestOS.

> For extra speed, maybe the ISR stacks could be mapped to some sort of
> core-local SRAM. This hasn't been done yet though.

Caches either work or they don't.

Wasting cycles fetching instructions, translations, and data are genuine
overhead that can be avoided if one treats thread-state as a cache.

If the interrupt occurs often enough to mater, its instructions, data,
and translations will be in the cache hierarchy.

HW that knows what it is doing can start fetching these things even
BEFORE it can execute the first instruction on behalf of the interrupt
dispatcher. SW can NEVER do any of this prior to starting to run instr.

> Idea here being probably the SRAM region could have a special address
> range, and any access to this region would be invisible to any other
> cores (and it need not need have backing in external RAM).

> One could maybe debate the cost of giving each core 4K or 8K of
> dedicated local SRAM though merely for "slightly faster interrupt handling".

Yech: end of debate.....

>>> Having side-channels that allow these values to "magically appear" in
>>> certain SPRs is at least simpler, though the cost of the logic itself
>>> could be more up for debate.
>>
>> Once you have an FMAC FPU none of the interrupt logic adds any area
>> to any core.
>>

> I don't have conventional FMA because it would have had too much cost
> and latency.

Because you are measuring from the wrong implementation technology,
using macros ill suited to the problem at hand.

>>> For ISR convenience, it would make sense to have, say, two SPR's or
>>> CR's designated for ISR use to squirrel off values from GPRs to make
>>> the prolog/epilog easier. Had considered this, but not done so (in
>>> this case, first thing the ISR does is save a few registers to the ISR
>>> stack to free them up to get the rest of the registers saved "more
>>> properly", then has to do a secondary reload to get these registers
>>> back into the correct place).
>>
>> Or, you could take the point of view that your architecture makes context
>> switching easy (like 1 instruction from application 1 to application 2)
>> and when you do this the rest of the model pretty much drops in for free.
>>

> This would cost well more, on the hardware side, than having two
> non-specialized CRs and being like "ISR's are allowed to stomp these at
> will, nothing else may use them".

The sequencer is surprisingly small. Everything else already exists and
is just waiting around for a signal to capture this or emit that.

> Some other tasks can be handled with the microsecond timer and a loop.

> Say:
> //void DelayUsec(int usec);
> DelayUsec:
> CPIUD 30
> ADD R4, R0, R6
> .L0:
> CPUID 30
> CMPQGT R0, R6
> BT .L0
> RTS
> Which would create a certain amount of delay (in microseconds) relative
> to when the function is called.

CPUID n Opteron's time frame took 200-600 cycles. Do you really want to
talk to your timer with those kinds of delay ??