Looking for a term here for fire-and-forget, FAF, stores.

For the Thor core usually the memory operation waits for ack or error status back before being marked complete. This can take eight or ten cycles depending how many hops the store makes for the round trip to memory and back. Decided to implement ‘FAF’, fire-and-forget, stores that do not wait for an ack or error status. I think this may be useful for updating displays where perhaps it is not absolutely necessary that the store worked. For instance, streaming data to a graphics display. While the store operation will likely almost always work, if there is the occasional pixel that is not set properly this may not be noticeable.

robf...@gmail.com wrote:
> Looking for a term here for fire-and-forget, FAF, stores.
>
> For the Thor core usually the memory operation waits for ack or error status back before being marked complete. This can take eight or ten cycles depending how many hops the store makes for the round trip to memory and back. Decided to implement ‘FAF’, fire-and-forget, stores that do not wait for an ack or error status. I think this may be useful for updating displays where perhaps it is not absolutely necessary that the store worked. For instance, streaming data to a graphics display. While the store operation will likely almost always work, if there is the occasional pixel that is not set properly this may not be noticeable.

Your question is really about error detection, reporting, and logging.
This is inherently error and model dependent. Errors can be instruction
synchronous, asynchronous, both or neither; can be recoverable or
non-recoverable; they can leave instructions restartable or not.

I imagine that most cores implement store as what I call an asynchronous
write-behind operation. The write operation is initiated when the store
instruction retires and instruction pointer is incremented.

If an error was subsequently detected it could be hundreds of clocks
after the store retires so the RIP and data virtual address are long gone.

If there is a cache then the store error detection could be completely
disconnected from program execution - your programs' store evicts
my programs' cache line and my line write back detects an error
long after my thread switched out.

Memory can be shared between programs - who do you tell and
what do you say if you have no process RIP or data virtual addresses?
By the way, something happened someplace and I thought you should know?

Read errors are not necessarily program synchronous as cache hardware
prefetchers can be reading ahead based on physical address patterns.
DRAM error scrubbers run asynchronously in the memory controller or DIMM.

So the question is how to report such an error, who should be informed,
what information is available to log, how should the logger work, etc.
And what to do if the error reporter or loggers themselves get an error.

On Sunday, June 25, 2023 at 9:21:03 AM UTC-5, EricP wrote:
> robf...@gmail.com wrote:
> > Looking for a term here for fire-and-forget, FAF, stores.
> >
> > For the Thor core usually the memory operation waits for ack or error status back before being marked complete. This can take eight or ten cycles depending how many hops the store makes for the round trip to memory and back. Decided to implement ‘FAF’, fire-and-forget, stores that do not wait for an ack or error status. I think this may be useful for updating displays where perhaps it is not absolutely necessary that the store worked. For instance, streaming data to a graphics display. While the store operation will likely almost always work, if there is the occasional pixel that is not set properly this may not be noticeable.
> Your question is really about error detection, reporting, and logging.
> This is inherently error and model dependent. Errors can be instruction
> synchronous, asynchronous, both or neither; can be recoverable or
> non-recoverable; they can leave instructions restartable or not.
>
> I imagine that most cores implement store as what I call an asynchronous
> write-behind operation. The write operation is initiated when the store
> instruction retires and instruction pointer is incremented.
>
> If an error was subsequently detected it could be hundreds of clocks
> after the store retires so the RIP and data virtual address are long gone..
>
> If there is a cache then the store error detection could be completely
> disconnected from program execution - your programs' store evicts
> my programs' cache line and my line write back detects an error
> long after my thread switched out.
>
> Memory can be shared between programs - who do you tell and
> what do you say if you have no process RIP or data virtual addresses?
> By the way, something happened someplace and I thought you should know?
>
> Read errors are not necessarily program synchronous as cache hardware
> prefetchers can be reading ahead based on physical address patterns.
> DRAM error scrubbers run asynchronously in the memory controller or DIMM.
>
> So the question is how to report such an error, who should be informed,
> what information is available to log, how should the logger work, etc.
> And what to do if the error reporter or loggers themselves get an error.
<
I agree with most of what EricP says above, with a subtle correction::
<
If the µArchitecture has miss buffers, the latent store has passed through
one of them and the bus protocol NaK knows which one it was. So, you
have a physical address but no IP and you may also have another copy of
the data to be stored, so the core could try-again®; but by this time you
will have lost memory order so recovery depends on whether you need
sequential consistency or can live with causal consistency.

On Sunday, June 25, 2023 at 12:37:09 PM UTC-4, MitchAlsup wrote:
> On Sunday, June 25, 2023 at 9:21:03 AM UTC-5, EricP wrote:
> > robf...@gmail.com wrote:
> > > Looking for a term here for fire-and-forget, FAF, stores.
> > >
> > > For the Thor core usually the memory operation waits for ack or error status back before being marked complete. This can take eight or ten cycles depending how many hops the store makes for the round trip to memory and back. Decided to implement ‘FAF’, fire-and-forget, stores that do not wait for an ack or error status. I think this may be useful for updating displays where perhaps it is not absolutely necessary that the store worked. For instance, streaming data to a graphics display. While the store operation will likely almost always work, if there is the occasional pixel that is not set properly this may not be noticeable.
> > Your question is really about error detection, reporting, and logging.
> > This is inherently error and model dependent. Errors can be instruction
> > synchronous, asynchronous, both or neither; can be recoverable or
> > non-recoverable; they can leave instructions restartable or not.
> >
> > I imagine that most cores implement store as what I call an asynchronous
> > write-behind operation. The write operation is initiated when the store
> > instruction retires and instruction pointer is incremented.
> >
> > If an error was subsequently detected it could be hundreds of clocks
> > after the store retires so the RIP and data virtual address are long gone.
> >
> > If there is a cache then the store error detection could be completely
> > disconnected from program execution - your programs' store evicts
> > my programs' cache line and my line write back detects an error
> > long after my thread switched out.
> >
> > Memory can be shared between programs - who do you tell and
> > what do you say if you have no process RIP or data virtual addresses?
> > By the way, something happened someplace and I thought you should know?
> >
> > Read errors are not necessarily program synchronous as cache hardware
> > prefetchers can be reading ahead based on physical address patterns.
> > DRAM error scrubbers run asynchronously in the memory controller or DIMM.
> >
> > So the question is how to report such an error, who should be informed,
> > what information is available to log, how should the logger work, etc.
> > And what to do if the error reporter or loggers themselves get an error..
> <
> I agree with most of what EricP says above, with a subtle correction::
> <
> If the µArchitecture has miss buffers, the latent store has passed through
> one of them and the bus protocol NaK knows which one it was. So, you
> have a physical address but no IP and you may also have another copy of
> the data to be stored, so the core could try-again®; but by this time you
> will have lost memory order so recovery depends on whether you need
> sequential consistency or can live with causal consistency.

Okay, it would really be an error reporting option to ignore errors on a store.
Now to refactor all the code from FAF to IE. There is a way to pass load or store
instruction qualifiers to instructions from the high-level language for specific
variables; cache-ability attributes are already passed. Maybe an additional
language construct would help. Something like try..catch comes to mind. Or it
may be better to restrict usage to assembly language.

for (n = 0; n < 16384; n++)
__no_error(screen[n] = 0);

I had all memory operations synchronous requiring an ack back before being
marked done. SO, it would be possible to repeat operations. But with
operations taking possibly hundreds of cycles this would really limit
performance.

On Sunday, June 25, 2023 at 4:50:10 PM UTC-5, robf...@gmail.com wrote:
> On Sunday, June 25, 2023 at 12:37:09 PM UTC-4, MitchAlsup wrote:
> > On Sunday, June 25, 2023 at 9:21:03 AM UTC-5, EricP wrote:
> > > robf...@gmail.com wrote:
> > > > Looking for a term here for fire-and-forget, FAF, stores.
> > > >
> > > > For the Thor core usually the memory operation waits for ack or error status back before being marked complete. This can take eight or ten cycles depending how many hops the store makes for the round trip to memory and back. Decided to implement ‘FAF’, fire-and-forget, stores that do not wait for an ack or error status. I think this may be useful for updating displays where perhaps it is not absolutely necessary that the store worked. For instance, streaming data to a graphics display. While the store operation will likely almost always work, if there is the occasional pixel that is not set properly this may not be noticeable.
> > > Your question is really about error detection, reporting, and logging..
> > > This is inherently error and model dependent. Errors can be instruction
> > > synchronous, asynchronous, both or neither; can be recoverable or
> > > non-recoverable; they can leave instructions restartable or not.
> > >
> > > I imagine that most cores implement store as what I call an asynchronous
> > > write-behind operation. The write operation is initiated when the store
> > > instruction retires and instruction pointer is incremented.
> > >
> > > If an error was subsequently detected it could be hundreds of clocks
> > > after the store retires so the RIP and data virtual address are long gone.
> > >
> > > If there is a cache then the store error detection could be completely
> > > disconnected from program execution - your programs' store evicts
> > > my programs' cache line and my line write back detects an error
> > > long after my thread switched out.
> > >
> > > Memory can be shared between programs - who do you tell and
> > > what do you say if you have no process RIP or data virtual addresses?
> > > By the way, something happened someplace and I thought you should know?
> > >
> > > Read errors are not necessarily program synchronous as cache hardware
> > > prefetchers can be reading ahead based on physical address patterns.
> > > DRAM error scrubbers run asynchronously in the memory controller or DIMM.
> > >
> > > So the question is how to report such an error, who should be informed,
> > > what information is available to log, how should the logger work, etc..
> > > And what to do if the error reporter or loggers themselves get an error.
> > <
> > I agree with most of what EricP says above, with a subtle correction::
> > <
> > If the µArchitecture has miss buffers, the latent store has passed through
> > one of them and the bus protocol NaK knows which one it was. So, you
> > have a physical address but no IP and you may also have another copy of
> > the data to be stored, so the core could try-again®; but by this time you
> > will have lost memory order so recovery depends on whether you need
> > sequential consistency or can live with causal consistency.
> Okay, it would really be an error reporting option to ignore errors on a store.
> Now to refactor all the code from FAF to IE. There is a way to pass load or store
> instruction qualifiers to instructions from the high-level language for specific
> variables; cache-ability attributes are already passed. Maybe an additional
> language construct would help. Something like try..catch comes to mind. Or it
> may be better to restrict usage to assembly language.
>
> for (n = 0; n < 16384; n++)
> __no_error(screen[n] = 0);
>
> I had all memory operations synchronous requiring an ack back before being
> marked done. SO, it would be possible to repeat operations. But with
> operations taking possibly hundreds of cycles this would really limit
> performance.
<
In about 1991, I was working on a 6-wide GBOoO machine µArchiecture.
<
Late in the project, we realized that "with a little logic" we could implement
a "paranoid" memory model where the store data had to arrive at the DRAM
controller without ECC failure for the store to be considered successful.
<
The added latency (on retirement) cost about 2% in performance (compared
to without the paranoid memory model.) The low overall cost is largely because
we had 48 buffers in which to store ongoing memory activity (with a 96 inst
instruction window.)
<
But rather than the error being associated with an individual instruction, the
error detection was applied to a whole application (process, task, thread,....).
Either the whole application is willing to pay to check all its stores get to memory
correctly or it doesn't wait for any of them. That is scientific applications might
not bother with store correctness, while banking applications definitely will.

robf...@gmail.com <robfi680@gmail.com> wrote:
> Looking for a term here for fire-and-forget, FAF, stores.

PCI(e) calls this 'posted transactions'. 'Posted' by analogy with posting a
letter (which is largely 'fire and forget').

Most writes are posted, meaning that you don't get an acknowledgement that
the write completed. This makes I/O quicker because you don't have to worry
about I/O latency, you can just pipeline them out.

https://en.wikipedia.org/wiki/Posted_write

I think you may still be able to get an error indication message, but it
would be asynchronous.

Theo

On Monday, June 26, 2023 at 6:41:04 AM UTC-4, Theo Markettos wrote:
> robf...@gmail.com <robf...@gmail.com> wrote:
> > Looking for a term here for fire-and-forget, FAF, stores.
> PCI(e) calls this 'posted transactions'. 'Posted' by analogy with posting a
> letter (which is largely 'fire and forget').
>
> Most writes are posted, meaning that you don't get an acknowledgement that
> the write completed. This makes I/O quicker because you don't have to worry
> about I/O latency, you can just pipeline them out.
>
> https://en.wikipedia.org/wiki/Posted_write
>
> I think you may still be able to get an error indication message, but it
> would be asynchronous.
>
> Theo

I get the impression now that most writes are posted writes in a modern CPU; I
thought it would be the other way around. I added error record writes to the ISA
though specified as ‘.erc’ for error record. There was some extra opcode room
in the indexed addressing instructions. The test system is almost entirely
pipelined with one or two bridges to synchronous busses.

It is interesting to see the adoption of language terms to probably slightly
different meanings than originally intended. Posting a transaction meant to
me that the transaction is placed on a bus, regardless of whether there would
be an ack. I have been calling this a transaction pulse. What are posted
transactions that expect an ack called then?

Theo Markettos <theom+news@chiark.greenend.org.uk> writes:
>robf...@gmail.com <robfi680@gmail.com> wrote:
>> Looking for a term here for fire-and-forget, FAF, stores.
>
>PCI(e) calls this 'posted transactions'. 'Posted' by analogy with posting a
>letter (which is largely 'fire and forget').
>
>Most writes are posted, meaning that you don't get an acknowledgement that
>the write completed. This makes I/O quicker because you don't have to worry
>about I/O latency, you can just pipeline them out.
>
>https://en.wikipedia.org/wiki/Posted_write
>
>I think you may still be able to get an error indication message, but it
>would be asynchronous.

Most higher-end CPU's now support poisoning throughout the datapath, so
errors involving corrupt data (parity/ecc detected) will mark the data
as poisoned[*]. If and/or when the data is consumed, the poison indication
will follow the data to the consumer, raising a RAS error; synchronously
for some accesses (mainly reads) and asynchronously otherwise.

Some hardware will raise an interrupt when the poisoned data is detected,
not just when it is consumed.

[*] for example, by corrupting the corresponding ECC/parity bits such that
a subsequent access will get an error.

Figuring out which process/thread owns the poisoned data can be complicated.

robf...@gmail.com wrote:
>
> Okay, it would really be an error reporting option to ignore errors on a store.
> Now to refactor all the code from FAF to IE. There is a way to pass load or store
> instruction qualifiers to instructions from the high-level language for specific
> variables; cache-ability attributes are already passed. Maybe an additional
> language construct would help. Something like try..catch comes to mind. Or it
> may be better to restrict usage to assembly language.
>
> for (n = 0; n < 16384; n++)
> __no_error(screen[n] = 0);
>
> I had all memory operations synchronous requiring an ack back before being
> marked done. SO, it would be possible to repeat operations. But with
> operations taking possibly hundreds of cycles this would really limit
> performance.
>

Your focus seems to be on detecting errors in single data value stores
to a range of device addresses.

I like x86's way of cache control where each PTE has a 3-bit field which
is an index into an 8 entry table of bits specifying the control flags.

You could have an option as non-cacheable, write through, synchronous
which waits for end-to-end ack/nak for each individual store instruction.
Then you can define that stores to such locations that detect an error
raise a synchronous exception, similar to a page fault.

Being write-through removes cache as a source of write errors
so that would seem to leave only bus errors as a source in this
particular scenario (ignoring core internal errors).

Address parity errors should not decode to any device so the bus
needs a default device at each end to signal BUSERR (wired-OR pulldown?).
Non-existent devices have a valid address but also do not decode to any
device so they are detected by a missing DTAK data transfer ack.
Data parity errors are signaled by the addressed device on BUSERR wire.

Bus parity errors can be transient or persistent. Transient would be
fixed by just retrying the operation once or twice, but if it occurs
repeatedly then the transfer should be aborted (a "stuck-at" fault).

Retrying assumes the addressed device does not undergo any state change
for each attempt. A bus write data parity error should not cause a
device state change. Some devices change state when read.
A bus read data parity error should not re-read a device register so
retrying for bus transmission errors also should not cause repeated
state changes (if any).

In any case the physical address and any other error info should
be logged. There needs to be a fifo someplace you can dump error
info into and interrupt the cpu to drain the fifo.

On Monday, June 26, 2023 at 2:59:57 PM UTC+3, robf...@gmail.com wrote:
> On Monday, June 26, 2023 at 6:41:04 AM UTC-4, Theo Markettos wrote:
> > robf...@gmail.com <robf...@gmail.com> wrote:
> > > Looking for a term here for fire-and-forget, FAF, stores.
> > PCI(e) calls this 'posted transactions'. 'Posted' by analogy with posting a
> > letter (which is largely 'fire and forget').
> >
> > Most writes are posted, meaning that you don't get an acknowledgement that
> > the write completed. This makes I/O quicker because you don't have to worry
> > about I/O latency, you can just pipeline them out.
> >
> > https://en.wikipedia.org/wiki/Posted_write
> >
> > I think you may still be able to get an error indication message, but it
> > would be asynchronous.
> >
> > Theo
> I get the impression now that most writes are posted writes in a modern CPU;

Nowadays even IBM z has write-back caches in their cache hierarchy. They resisted
them for longer than most, but gave up more than decade ago.
With write-back cache, most write are "posted" even if not called such officially.

> I
> thought it would be the other way around. I added error record writes to the ISA
> though specified as ‘.erc’ for error record. There was some extra opcode room
> in the indexed addressing instructions. The test system is almost entirely
> pipelined with one or two bridges to synchronous busses.
>
> It is interesting to see the adoption of language terms to probably slightly
> different meanings than originally intended. Posting a transaction meant to
> me that the transaction is placed on a bus, regardless of whether there would
> be an ack. I have been calling this a transaction pulse. What are posted
> transactions that expect an ack called then?

On Saturday, July 1, 2023 at 3:59:47 PM UTC-5, Michael S wrote:
> On Monday, June 26, 2023 at 2:59:57 PM UTC+3, robf...@gmail.com wrote:
> > On Monday, June 26, 2023 at 6:41:04 AM UTC-4, Theo Markettos wrote:
> > > robf...@gmail.com <robf...@gmail.com> wrote:
> > > > Looking for a term here for fire-and-forget, FAF, stores.
> > > PCI(e) calls this 'posted transactions'. 'Posted' by analogy with posting a
> > > letter (which is largely 'fire and forget').
> > >
> > > Most writes are posted, meaning that you don't get an acknowledgement that
> > > the write completed. This makes I/O quicker because you don't have to worry
> > > about I/O latency, you can just pipeline them out.
> > >
> > > https://en.wikipedia.org/wiki/Posted_write
> > >
> > > I think you may still be able to get an error indication message, but it
> > > would be asynchronous.
> > >
> > > Theo
> > I get the impression now that most writes are posted writes in a modern CPU;
> Nowadays even IBM z has write-back caches in their cache hierarchy. They resisted
> them for longer than most, but gave up more than decade ago.
> With write-back cache, most write are "posted" even if not called such officially.
<
With a transport protocol like HyperTransport, all writes are essentially posted,
but the sequence ID cannot be reused until an ack from the target arrives.
<
Which means one could call then non-posted as they are not fire and forget,
there are fire and wait until ack at least at the transport level.
<
> > I
> > thought it would be the other way around. I added error record writes to the ISA
> > though specified as ‘.erc’ for error record. There was some extra opcode room
> > in the indexed addressing instructions. The test system is almost entirely
> > pipelined with one or two bridges to synchronous busses.
> >
> > It is interesting to see the adoption of language terms to probably slightly
> > different meanings than originally intended. Posting a transaction meant to
> > me that the transaction is placed on a bus, regardless of whether there would
> > be an ack. I have been calling this a transaction pulse. What are posted
> > transactions that expect an ack called then?

MitchAlsup <MitchAlsup@aol.com> writes:
>On Saturday, July 1, 2023 at 3:59:47=E2=80=AFPM UTC-5, Michael S wrote:

>> > I get the impression now that most writes are posted writes in a modern=
> CPU;
>> Nowadays even IBM z has write-back caches in their cache hierarchy. They =
>resisted=20
>> them for longer than most, but gave up more than decade ago.=20
>> With write-back cache, most write are "posted" even if not called such of=
>ficially.
><
>With a transport protocol like HyperTransport, all writes are essentially p=
>osted,
>but the sequence ID cannot be reused until an ack from the target arrives.
><
>Which means one could call then non-posted as they are not fire and forget,
>there are fire and wait until ack at least at the transport level.

Hypertransport does have some support for non-posted sized writes with some
rules (starting with the originator knowing a priori that the receiver
can accomodate the request). The target needs to send TARGET DONE to
ensure any intermediate (e.g. bridge) request queue entries get deallocated.

On Sunday, July 2, 2023 at 5:08:27 PM UTC-5, Scott Lurndal wrote:
> MitchAlsup <Mitch...@aol.com> writes:
> >On Saturday, July 1, 2023 at 3:59:47=E2=80=AFPM UTC-5, Michael S wrote:
>
> >> > I get the impression now that most writes are posted writes in a modern=
> > CPU;
> >> Nowadays even IBM z has write-back caches in their cache hierarchy. They =
> >resisted=20
> >> them for longer than most, but gave up more than decade ago.=20
> >> With write-back cache, most write are "posted" even if not called such of=
> >ficially.
> ><
> >With a transport protocol like HyperTransport, all writes are essentially p> >osted,
> >but the sequence ID cannot be reused until an ack from the target arrives.
> ><
> >Which means one could call then non-posted as they are not fire and forget,
> >there are fire and wait until ack at least at the transport level.
<
> Hypertransport does have some support for non-posted sized writes with some
> rules (starting with the originator knowing a priori that the receiver
> can accomodate the request). The target needs to send TARGET DONE to
> ensure any intermediate (e.g. bridge) request queue entries get deallocated.
<
Posted in this sense means that there are multiple steps from source to
destination and that each step is ack-ed individually. But you still COULD
have the property that the ack is not sent until address+command+data
have passed ECC checks at receiver.
<
So you don't know when it arrives, but you do know it arrived without error..

Slightly OT.

Massive number of edits about to happen in the Thor documentation. I decided
to remove the cache-ability field from load and store instructions. Cache-ability
is now controlled strictly by memory page and region. It turns out that the
hardware just ignored the cache-ability attributes coming from the instruction
anyway. Removing the bits from the instruction format gives two more bits for
a displacement or other use.

Thinking of adding a block construct to the high-level language to indicate
cache-ability of variables. Some languages for embedded apps have an ‘at’
attribute allowing the memory address to be specified which would indirectly
allow the cache-ability to be controlled. One possibility is to add a section
specification to the language so specific sections may be selected for
variables. Then define non-cacheable bss, ncbss, non-cacheable data, etc. as
sections in the linker script. C has an ugly way to do this using __attribute__.
But given control over the HLL perhaps a prettier syntax can be created.

On Saturday, July 8, 2023 at 9:08:13 PM UTC-5, robf...@gmail.com wrote:
> Slightly OT.
>
> Massive number of edits about to happen in the Thor documentation. I decided
> to remove the cache-ability field from load and store instructions. Cache-ability
> is now controlled strictly by memory page and region. It turns out that the
> hardware just ignored the cache-ability attributes coming from the instruction
> anyway. Removing the bits from the instruction format gives two more bits for
> a displacement or other use.
<
Why is this not something for the MMU tables ??
<
For examples My 66000 has 4 spaces and 5 kinds of accesses.
There is cacheable DRAM and these use the used and modified bits normally.
There is the unCacheable domain that use the used and Modified bits differently.
00 -> unCacheable DRAM access (actually cached in L3 cache)
01 -> Memory Mapped I/O space
10 -> Configuration Space
11 -> ROM
<
Cacheable RWE DRAM is causally ordered, except within an ATOMIC event
where it is sequentially consistent.
Cacheable RO or EO DRAM is weakly ordered
Cacheable WO is sequentially consistent
<
unCacheable DRAM is sequentially Consistent
Memory Mapped I/O is sequentially Consistent
Configuration is strongly ordered
ROM is weakly ordered.
<
Solving several utility functions in one fell swoop.
>
> Thinking of adding a block construct to the high-level language to indicate
> cache-ability of variables. Some languages for embedded apps have an ‘at’
> attribute allowing the memory address to be specified which would indirectly
> allow the cache-ability to be controlled. One possibility is to add a section
> specification to the language so specific sections may be selected for
> variables. Then define non-cacheable bss, ncbss, non-cacheable data, etc. as
> sections in the linker script. C has an ugly way to do this using __attribute__.
> But given control over the HLL perhaps a prettier syntax can be created.

On Saturday, July 8, 2023 at 10:58:45 PM UTC-4, MitchAlsup wrote:
> On Saturday, July 8, 2023 at 9:08:13 PM UTC-5, robf...@gmail.com wrote:
> > Slightly OT.
> >
> > Massive number of edits about to happen in the Thor documentation. I decided
> > to remove the cache-ability field from load and store instructions. Cache-ability
> > is now controlled strictly by memory page and region. It turns out that the
> > hardware just ignored the cache-ability attributes coming from the instruction
> > anyway. Removing the bits from the instruction format gives two more bits for
> > a displacement or other use.
> <
> Why is this not something for the MMU tables ??

It is in the MMU tables. By memory page I meant in the MMU tables. There is a
cache-ability field in the PTE along with a 3-bit region identifier field.

> <
> For examples My 66000 has 4 spaces and 5 kinds of accesses.
> There is cacheable DRAM and these use the used and modified bits normally..
> There is the unCacheable domain that use the used and Modified bits differently.
> 00 -> unCacheable DRAM access (actually cached in L3 cache)
> 01 -> Memory Mapped I/O space
> 10 -> Configuration Space
> 11 -> ROM
> <
> Cacheable RWE DRAM is causally ordered, except within an ATOMIC event
> where it is sequentially consistent.
> Cacheable RO or EO DRAM is weakly ordered
> Cacheable WO is sequentially consistent
> <
> unCacheable DRAM is sequentially Consistent
> Memory Mapped I/O is sequentially Consistent
> Configuration is strongly ordered
> ROM is weakly ordered.
> <
> Solving several utility functions in one fell swoop.
> >
> > Thinking of adding a block construct to the high-level language to indicate
> > cache-ability of variables. Some languages for embedded apps have an ‘at’
> > attribute allowing the memory address to be specified which would indirectly
> > allow the cache-ability to be controlled. One possibility is to add a section
> > specification to the language so specific sections may be selected for
> > variables. Then define non-cacheable bss, ncbss, non-cacheable data, etc. as
> > sections in the linker script. C has an ugly way to do this using __attribute__.
> > But given control over the HLL perhaps a prettier syntax can be created..

MitchAlsup <MitchAlsup@aol.com> writes:
>On Saturday, July 8, 2023 at 9:08:13=E2=80=AFPM UTC-5, robf...@gmail.com wr=
>ote:
>> Slightly OT.=20
>>=20
>> Massive number of edits about to happen in the Thor documentation. I deci=
>ded=20
>> to remove the cache-ability field from load and store instructions. Cache=
>-ability=20
>> is now controlled strictly by memory page and region. It turns out that t=
>he=20
>> hardware just ignored the cache-ability attributes coming from the instru=
>ction=20
>> anyway. Removing the bits from the instruction format gives two more bits=
> for=20
>> a displacement or other use.=20
><
>Why is this not something for the MMU tables ??
><
>For examples My 66000 has 4 spaces and 5 kinds of accesses.
>There is cacheable DRAM and these use the used and modified bits normally.
>There is the unCacheable domain that use the used and Modified bits differe=
>ntly.
>00 -> unCacheable DRAM access (actually cached in L3 cache)
>01 -> Memory Mapped I/O space
>10 -> Configuration Space
>11 -> ROM

How do you handle cacheable DRAM behind a PCIe BAR (e.g. CXL-Cache)?

On Sunday, July 9, 2023 at 10:29:02 AM UTC-5, Scott Lurndal wrote:
> MitchAlsup <Mitch...@aol.com> writes:
> >On Saturday, July 8, 2023 at 9:08:13=E2=80=AFPM UTC-5, robf...@gmail.com wr=
> >ote:
> >> Slightly OT.=20
> >>=20
> >> Massive number of edits about to happen in the Thor documentation. I deci=
> >ded=20
> >> to remove the cache-ability field from load and store instructions. Cache=
> >-ability=20
> >> is now controlled strictly by memory page and region. It turns out that t=
> >he=20
> >> hardware just ignored the cache-ability attributes coming from the instru=
> >ction=20
> >> anyway. Removing the bits from the instruction format gives two more bits=
> > for=20
> >> a displacement or other use.=20
> ><
> >Why is this not something for the MMU tables ??
> ><
> >For examples My 66000 has 4 spaces and 5 kinds of accesses.
> >There is cacheable DRAM and these use the used and modified bits normally.
> >There is the unCacheable domain that use the used and Modified bits differe> >ntly.
> >00 -> unCacheable DRAM access (actually cached in L3 cache)
> >01 -> Memory Mapped I/O space
> >10 -> Configuration Space
> >11 -> ROM
<
> How do you handle cacheable DRAM behind a PCIe BAR (e.g. CXL-Cache)?
<
Haven't gotten that far:: But to a first order::
<
Core makes an access which gets translated to cacheable DRAM. The
request wanders through the interconnect and arrives at L3 Cache.
L3 cache talks to cores with My 66000 coherence protocol.
L3 has certain physical address regions mapped to CXL:Bus,Dev;Fctn
L3 talks to CXL caches using CXL protocol.
L3 miss unmapped initiates DRAM read
L3 miss ....mapped initiates CXL ....read
L3 miss ....mapped initiates CXL ....read miss initiates DRAM read