The topic came up elsewhere, where people were arguing that
software-managed TLB was bad/useless and that (supposedly) no modern CPU
architecture would consider using it.

Whereas, I had been having OK enough results thus far with BJX2 using a
software-managed TLB.

In my current workloads, it is typically under around 100 TLB
misses/second, but it is only around 1000 TLB misses/second or more when
it starts to noticeably eat into CPU time in the benchmarks.

Thus far, only around 0.1% of the total clock-cycle budget is going into
TLB Miss handling (with the current TLB and page size), so I don't
really see what the big issue is in this case...

Trying to use a large memory footprint at present will tend to either
cause lots of L2 cache-misses, or page faults, either of which will
"ruin ones' day" a lot faster then the TLB misses.

So, it looks like for architectures I can find information on:
Hardware Managed (Page-Table):
x86 / x86-64
ARM / ARM64
RISC-V (Privileged ISA Spec)
Software Managed TLB:
SuperH (SH-4 and SH-5)
MIPS
SPARC
Alpha
Power and PowerPC
PA-RISC
Itanium / IA-64
(Hybrid, Supported RAM-Backed TLB)
BJX2
...
Unknown:
M68K
PDP / VAX
None (Physical only):
SH-2
MSP430
AVR8
Cortex-M0
TMS320?
...
None / External Bank Switching:
6502
65C816
Z80
...

On the surface, it looks like SW TLB is the majority case, never-mind if
many of the latter ones have effectively died off.

There are still a few (modern) projects built around the 6502 (such as
the "Commander X16"), which continue to use bank-switched memory (Why?
Dunno...). Though, these guys generally like doing everything as
socketed DIP, apparently doing the bank-switching with 74xx series logic
chips and similar, ...
Though, personally, I don't really get it (Eg: Sorta like the
Commodore64, but not entirely binary compatible with C64 software due to
using different hardware interfaces?...).

As for tradeoffs:
Hardware TLB:
Faster
Less flexible
Pretty much every use-case needs hardware support.
You can *only* use page tables
Say, one can't just go and fake segmented addressing in an ISR
...
Not really friendly to more advanced memory security schemes.
Something like ACL checks on page access would be less practical.
...
Software TLB:
Slower
Can do lots of crazy stuff in the ISR if one wants.
I have B-Tree page-tables, ACL checking, etc.

Pretty much everything I can find online (apart from the occasional
random arguments online), is mostly people describing a high-level
overview of what they do, rather than why one might choose one or the other.

Though, in my case, I did end up going and adding a hash-table check to
the Hybrid-B-Tree case, so the main part of the B-Tree lookup can be
skipped over if the lookup hits in a hash-table.

Though, IA-64 did have an intermediate option, in the form of a larger
RAM-backed TLB.

Say, if going this route:
L1 TLB: 32 entry
These caching prior responses from the main TLB.
This is an optional feature (slightly reduces miss latency).
L2 TLB: 256x 4-way (as-is, Main TLB)
RAM Backed TLB: Say, 4096x 4-way.

Say, if configured to do so, the TLB-Miss event could be delayed, with
the CPU first trying to fetch an entry from the RAM-Backed TLB, and then
raising a fault if it misses.

I would not particularly want to go with a page-walker, as I wouldn't
really want to give up on the flexibility the current scheme can offer.

However, the "Large RAM-backed TLB" option does still seem to allow a
similar level of flexibility (while, theoretically, one could have 256MB
or 1GB or more of TLB coverage without needing to trigger a TLB Miss ISR).

As-is, the TLB covers a working set of around 16MB.

However... This does complicate things slightly, as now there needs to
be a mechanism to:
Detect a TLB miss happened (TLB emits exception code);
Stall the pipeline;
But, delay forwarding the exception.
Try to handle the TLB Miss;
Send cache-line requests to RAM;
Check returned lines for a match with the TLB Miss;
If success:
Feed into TLB and then signal L1s to retry their requests;
Release Stall signal several cycles later.
Else:
Forward TLB Miss to Execute-Unit.

Possibly the mechanism could have a timeout delay, using a prefetch
and/or falling back to the ISR if it takes more than 200 cycles or so
for the L2 response to arrive (say, the TLBE had fallen out to external
DRAM).

Though, generally, a DRAM request should usually get a response within
100-150 clock-cycles regardless of whether there is a timeout in place.

It could almost make more sense to try to handle this "external TLB" as
a special-case in the TLB Miss ISR handler. Though, to be worthwhile
would likely need to handle the ISR entry point in ASM (as-is, the ISR
is handled in C, with the entry/exit logic handled by my C compiler).

Could shave around 500 clock-cycles off the ISR handlier if there was a
way to side-step saving and restoring all of the registers (a special
entry point could maybe only save/restore "some" of the registers).

But, say, the full save/restore is still needed in the general case,
say, if one wants to trigger a context-switch into the kernel to handle
a page-fault.

Well, and/or spend resources to bank out some of the the registers.

Or (more lazy) add a few special CRs whose sole purpose is to be special
scratchpad registers for ISRs to use to free up GPRs for their own use.

Could just go for using R0U and R1U for this purpose, but this would
mostly just (maybe) allow making ISR entry and exit slightly less
awkward (as-is, it is a slightly awkward process of trying to get a few
registers shuffled around into RAM so that one can free up some working
registers to begin the process of saving off all the rest of the registers).

Well, and/or go and define C48..C63 or similar as special scratch
registers (but, unclear how much this would save over "just copy them to
RAM", as-is currently the case). Better if I could access them 128 bits
at a time.

Doesn't really seem worth it at the moment...

More so, as generally page-faults tend to start wrecking things well
before TLB misses become an issue (with only 32MB or 64MB of a 128MB or
256MB RAM chip currently being used for pagefile backed memory).

Or, one could take the "simpler option" and just do a 1024x 4-way TLB in
Block-RAM (would cover a 64MB of working set), but would likely require
adding an extra clock-cycle to the TLB handling (in addition to eating
more Block-RAM).

Major architectural redesign in the case of BJX2 is unlikely at the
moment, so if anyone wants to use it, they will need to live with the
software managed TLB.

Any thoughts?...

BGB <cr88192@gmail.com> writes:
>The topic came up elsewhere, where people were arguing that
>software-managed TLB was bad/useless and that (supposedly) no modern CPU
>architecture would consider using it.
>
>
>So, it looks like for architectures I can find information on:
> Hardware Managed (Page-Table):
> x86 / x86-64
> ARM / ARM64
> RISC-V (Privileged ISA Spec)

Modern CPUs.

> Software Managed TLB:
> SuperH (SH-4 and SH-5)
> MIPS
> SPARC
> Alpha
> Power and PowerPC
> PA-RISC
> Itanium / IA-64
> (Hybrid, Supported RAM-Backed TLB)

Obsolete CPUs, all designed three decades ago.

>On the surface, it looks like SW TLB is the majority case, never-mind if
>many of the latter ones have effectively died off.

For any non-microcontroller CPU being sold today, not one supports S/W TLB.

You're stretching logic to confirm your conclusions.

In article <Ia3cM.3440031$iU59.2338510@fx14.iad>,
Scott Lurndal <slp53@pacbell.net> wrote:
>BGB <cr88192@gmail.com> writes:
>>The topic came up elsewhere, where people were arguing that
>>software-managed TLB was bad/useless and that (supposedly) no modern CPU
>>architecture would consider using it.
>>
>>
>>So, it looks like for architectures I can find information on:
>> Hardware Managed (Page-Table):
>> x86 / x86-64
>> ARM / ARM64
>> RISC-V (Privileged ISA Spec)
>
>Modern CPUs.
>
>> Software Managed TLB:
>> SuperH (SH-4 and SH-5)
>> MIPS
>> SPARC
>> Alpha
>> Power and PowerPC
>> PA-RISC
>> Itanium / IA-64
>> (Hybrid, Supported RAM-Backed TLB)
>
>Obsolete CPUs, all designed three decades ago.
>
>>On the surface, it looks like SW TLB is the majority case, never-mind if
>>many of the latter ones have effectively died off.
>
>For any non-microcontroller CPU being sold today, not one supports S/W TLB.
>
>You're stretching logic to confirm your conclusions.

Indeed, a classic example of shooting an arrow and then drawing
a target around where it hits.

- Dan C.

On 5/26/2023 9:16 AM, Scott Lurndal wrote:
> BGB <cr88192@gmail.com> writes:
>> The topic came up elsewhere, where people were arguing that
>> software-managed TLB was bad/useless and that (supposedly) no modern CPU
>> architecture would consider using it.
>>
>>
>> So, it looks like for architectures I can find information on:
>> Hardware Managed (Page-Table):
>> x86 / x86-64
>> ARM / ARM64
>> RISC-V (Privileged ISA Spec)
>
> Modern CPUs.
>

x86 is older than I am.
ARM is around the same age.

x86-64 came out 20 years ago...

Only RISC-V is "newer".

RISC-V still predates my BJX2 project by around a decade...

>> Software Managed TLB:
>> SuperH (SH-4 and SH-5)
>> MIPS
>> SPARC
>> Alpha
>> Power and PowerPC
>> PA-RISC
>> Itanium / IA-64
>> (Hybrid, Supported RAM-Backed TLB)
>
> Obsolete CPUs, all designed three decades ago.
>

Same with x86 and ARM...

>> On the surface, it looks like SW TLB is the majority case, never-mind if
>> many of the latter ones have effectively died off.
>
> For any non-microcontroller CPU being sold today, not one supports S/W TLB.
>
> You're stretching logic to confirm your conclusions.

IBM still sells Power ISA CPUs.

PowerPC is seemingly effectively dead now (in a commercial sense), but
Power ISA lives on.

But, in any case, changing this aspect of the design in my case would be
too big of a design change, so is unlikely to happen (well, at least
short of figuring out how to get as similar feature-set from a hardware
page walker with a cost of less than around 5k LUTs).

This is like asking for it to not be a VLIW, for that matter...

Like, I have my reasons...

On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
> Any thoughts?...
<
A Software managed TLB (i.e., MIPS) is not viable in a system
that uses both HyperVisors and Guest OSs.
<
A Single HW lookup to a big hash table followed by a software
management of the big hash table remains viable--but only so
long as the HyperVisor performs the hash table updates.
<
Once you get more than 1 set of mapping tables (HV and OS) you
really want to take software out of the TLB update process.
<
I, personally, went through a phase where I enjoyed software
managed TLBs (with a modicum of HW updates), but the
advent of HyperVisors stuck a stick into that heart.

On Friday, May 26, 2023 at 11:50:57 AM UTC-5, BGB wrote:
> On 5/26/2023 9:16 AM, Scott Lurndal wrote:
> > BGB <cr8...@gmail.com> writes:
> >> The topic came up elsewhere, where people were arguing that
> >> software-managed TLB was bad/useless and that (supposedly) no modern CPU
> >> architecture would consider using it.
> >>
> >>
> >> So, it looks like for architectures I can find information on:
> >> Hardware Managed (Page-Table):
> >> x86 / x86-64
> >> ARM / ARM64
> >> RISC-V (Privileged ISA Spec)
> >
> > Modern CPUs.
> >
> x86 is older than I am.
> ARM is around the same age.
<
Obviously you are not old enough -:)
>
> x86-64 came out 20 years ago...
>
> Only RISC-V is "newer".
>
> RISC-V still predates my BJX2 project by around a decade...
<
RISC-V dates from 2012. You have been doing BJX stuff for
more than 4 years.
<
> >> Software Managed TLB:
> >> SuperH (SH-4 and SH-5)
> >> MIPS
> >> SPARC
<
Depends on which SPARC; SPARC V8 had page tables that one
could build HW table walkers for, I believe SPARC V9 did similarly
but my memory on that has faded. The earliest gate array) SPARC
did have SW managed TLBs.

On 5/26/2023 12:10 PM, MitchAlsup wrote:
> On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
>> Any thoughts?...
> <
> A Software managed TLB (i.e., MIPS) is not viable in a system
> that uses both HyperVisors and Guest OSs.
> <
> A Single HW lookup to a big hash table followed by a software
> management of the big hash table remains viable--but only so
> long as the HyperVisor performs the hash table updates.
> <
> Once you get more than 1 set of mapping tables (HV and OS) you
> really want to take software out of the TLB update process.
> <
> I, personally, went through a phase where I enjoyed software
> managed TLBs (with a modicum of HW updates), but the
> advent of HyperVisors stuck a stick into that heart.

I guess maybe one could do something where:
Only the top-level does Software TLB;
All of the lower levels need to use page-tables / etc;
It is left ambiguous as to whether the CPU does HW or SW TLB.

Maybe the ISA Spec specifies enough table formats that it can remain
flexible. In this case, following one of the specified table formats
becomes a mandate.

Granted, B-Tree page-tables seem like a stretch for a hardware
implementation.

Need a way around the "large address space + ASLR => huge amounts of
wasted pages" issue. Partly as hash-tables don't scale very well.

Well, unless maybe:
1-level hash-table + N-level page table.

Whenever a hash collision would occur in the top-level hash table, the
page-table depth increases by 1 and the hash-table is repacked. With
each hash-table entry being 128-bit (partial VA then 64-bit PDE).

So:
H+2 levels
H+3 levels
H+4 levels
H+5 levels
H+6 levels
H+7 levels
8 levels, back to a plain page-table.

Well, unless it is 4K pages and then it goes up to 10, but starts at H+3:
H+3 .. H+9, then 10-level page table.

Could at least address the worst form of the problem.

Would also need a good way to express the ACL table in a more hardware
friendly way.

I guess one possible way would be to organize the ACL table as a
page-table like structure, say:
(31:16): KRR_ID
(15: 0): ACLID

Though, this would need a 4-level table, and (for a non-trivial number
of ACLs) would eat a significant amount of memory.

Though, could possibly make it more space-efficient (in the common case
of sequentially increasing ACLs) by Morton-shuffling the bits.

Maybe also the leaf-table has 8-bit entries. This would reduce the table
to 3 levels (for a 16K page size).

With, the final entries being, say:
URWX-000V
V: Valid
U: User/Supervisor
R: Read
W: Write
X: Execute

Sort of a bit brute-force though, as traditionally ACLs have a more
hierarchical structure.

This probably excludes any ab(use) the TLB to emulate segmented
addressing schemes though...

Still not entirely obvious how one would pull off something like
emulating an 8086 style addressing mode using only page-tables (at
least, short of massive memory duplication).

But, even with a SW TLB, how exactly one would pull off funky modes like
"Big Real Mode" in a TLB are uncertain (seems like likely, a segment
register-version would need to be encoded in the virtual address, so
every time one reloads DS or ES, it moves to a different part of the
extended virtual address space).

Well, and/or, parts of the address space are defined, say:
xxxx_xxx0ssss_0000_0000aaaa (Real-Mode)
xxxx_xxx1ssss_0000_aaaaaaaa (Protected-Mode, 32-bit)
xxxx_xxx2aaaa_aaaa_aaaaaaaa (Long-Mode, 64-bit)

Which, if the segment registers encode a few extra bits, could
approximate the behavior of "Big-Real Mode" (by leaving the segment in
protected mode).

But, as-is, my "emulate x86 on BJX2" sub-project is still kinda stalled
at present, and no obvious ways to make it "not suck" apart from adding
a bunch of helper mechanisms (such as figuring out some effective way to
fake the FLAGS register bits, etc).

Short of getting the emulated code running at near native speed, it
would be "basically useless" (well, unless maybe the end-goal was to try
to emulate the MS-DOS version of Sim City or similar...).

....

BGB <cr88192@gmail.com> writes:
>On 5/26/2023 9:16 AM, Scott Lurndal wrote:
>> BGB <cr88192@gmail.com> writes:
>>> The topic came up elsewhere, where people were arguing that
>>> software-managed TLB was bad/useless and that (supposedly) no modern CPU
>>> architecture would consider using it.
>>>
>>>
>>> So, it looks like for architectures I can find information on:
>>> Hardware Managed (Page-Table):
>>> x86 / x86-64
>>> ARM / ARM64
>>> RISC-V (Privileged ISA Spec)
>>
>> Modern CPUs.
>>
>
>x86 is older than I am.
>ARM is around the same age.
>
>x86-64 came out 20 years ago...
>
>Only RISC-V is "newer".

Unlike the CPUs below, x86 and arm are continuously
being enhanced with new features (e.g. secure conclaves,
realms, new instructions, etc).

>
>RISC-V still predates my BJX2 project by around a decade...
>
>
>>> Software Managed TLB:
>>> SuperH (SH-4 and SH-5)
>>> MIPS
>>> SPARC
>>> Alpha
>>> Power and PowerPC
>>> PA-RISC
>>> Itanium / IA-64
>>> (Hybrid, Supported RAM-Backed TLB)
>>
>> Obsolete CPUs, all designed three decades ago.

On 5/26/2023 12:15 PM, MitchAlsup wrote:
> On Friday, May 26, 2023 at 11:50:57 AM UTC-5, BGB wrote:
>> On 5/26/2023 9:16 AM, Scott Lurndal wrote:
>>> BGB <cr8...@gmail.com> writes:
>>>> The topic came up elsewhere, where people were arguing that
>>>> software-managed TLB was bad/useless and that (supposedly) no modern CPU
>>>> architecture would consider using it.
>>>>
>>>>
>>>> So, it looks like for architectures I can find information on:
>>>> Hardware Managed (Page-Table):
>>>> x86 / x86-64
>>>> ARM / ARM64
>>>> RISC-V (Privileged ISA Spec)
>>>
>>> Modern CPUs.
>>>
>> x86 is older than I am.
>> ARM is around the same age.
> <
> Obviously you are not old enough -:)

I am scarily close to 40, by many definitions this is "old"...

>>
>> x86-64 came out 20 years ago...
>>
>> Only RISC-V is "newer".
>>
>> RISC-V still predates my BJX2 project by around a decade...
> <
> RISC-V dates from 2012. You have been doing BJX stuff for
> more than 4 years.
> <

Probably true. Looks like it had its origins ~ 2010, and BJX2 got
started ~ 2017 (from the ashes of my BJX1 project), but the ISA design
mutated a fair bit early on.

My first "actually runs on an FPGA" versions were from around 2019/2020.
Most of the core ISA has been relatively stable since then.

Then it has been an endless uphill battle of debugging and similar since
then...

Not exactly rapid progress, sadly...

>>>> Software Managed TLB:
>>>> SuperH (SH-4 and SH-5)
>>>> MIPS
>>>> SPARC
> <
> Depends on which SPARC; SPARC V8 had page tables that one
> could build HW table walkers for, I believe SPARC V9 did similarly
> but my memory on that has faded. The earliest gate array) SPARC
> did have SW managed TLBs.

Stuff I read had said that UltraSPARC and SPARC64 had also used SW TLB.

Granted, I hadn't looked too deeply.

Looking more, it seems that for Power and PowerPC, it was more
implementation dependent rather than a core architectural feature.

On Friday, May 26, 2023 at 1:14:03 PM UTC-5, BGB wrote:
> On 5/26/2023 12:10 PM, MitchAlsup wrote:
> > On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
> >> Any thoughts?...
> > <
> > A Software managed TLB (i.e., MIPS) is not viable in a system
> > that uses both HyperVisors and Guest OSs.
> > <
> > A Single HW lookup to a big hash table followed by a software
> > management of the big hash table remains viable--but only so
> > long as the HyperVisor performs the hash table updates.
> > <
> > Once you get more than 1 set of mapping tables (HV and OS) you
> > really want to take software out of the TLB update process.
> > <
> > I, personally, went through a phase where I enjoyed software
> > managed TLBs (with a modicum of HW updates), but the
> > advent of HyperVisors stuck a stick into that heart.
<
> I guess maybe one could do something where:
> Only the top-level does Software TLB;
> All of the lower levels need to use page-tables / etc;
> It is left ambiguous as to whether the CPU does HW or SW TLB.
>
> Maybe the ISA Spec specifies enough table formats that it can remain
> flexible. In this case, following one of the specified table formats
> becomes a mandate.
>
>
> Granted, B-Tree page-tables seem like a stretch for a hardware
> implementation.
>
> Need a way around the "large address space + ASLR => huge amounts of
> wasted pages" issue. Partly as hash-tables don't scale very well.
<
The need for ASLR is a symptom that the OS tables remain visible
while running application threads (and bad µArchitecture.) If the
application ALWAYS got PageFault when touching anything with
the bit<63> set, then ASLR would not be needed.
<
But the Global bit--used to optimize TLB across user<->super
boundary opened the door for Spectré like attacks. Bad µArch,
then, widened the door such that one could train the cache (or
other predictors) to make a dependent access on an access that
is not allowed AND to watch such an event with a high precision
timer, well, then, we did it to ourselves.......
<
My 66000 is not attackable by those means--and probably can
avoid needing ASLR although there is no reason to to use it as
it slows nothing down.
>
<snip>
>
> I guess one possible way would be to organize the ACL table as a
> page-table like structure, say:
> (31:16): KRR_ID
> (15: 0): ACLID
<
In principle, you could have a different page hash table in memory for each
ASID.
>
> Though, this would need a 4-level table, and (for a non-trivial number
> of ACLs) would eat a significant amount of memory.
<
The thing with the HW assisted software TLB is the HW makes exactly
1 memory access. If it succeeds, we get along with running code, if it
fails we trap to SW and let SW sort it out. An efficient HW hash function
is key.
>
> Though, could possibly make it more space-efficient (in the common case
> of sequentially increasing ACLs) by Morton-shuffling the bits.
>
> Maybe also the leaf-table has 8-bit entries. This would reduce the table
> to 3 levels (for a 16K page size).
>
> With, the final entries being, say:
> URWX-000V
> V: Valid
> U: User/Supervisor
> R: Read
> W: Write
> X: Execute
>
> Sort of a bit brute-force though, as traditionally ACLs have a more
> hierarchical structure.
>
>
> This probably excludes any ab(use) the TLB to emulate segmented
> addressing schemes though...
<
Why would anyone want to do that ?
>
> Still not entirely obvious how one would pull off something like
> emulating an 8086 style addressing mode using only page-tables (at
> least, short of massive memory duplication).
<
Again, Why would any new architecture want to do that ??
>
> But, even with a SW TLB, how exactly one would pull off funky modes like
> "Big Real Mode" in a TLB are uncertain (seems like likely, a segment
> register-version would need to be encoded in the virtual address, so
> every time one reloads DS or ES, it moves to a different part of the
> extended virtual address space).
<
If you want x86 you have to support A20 addressing

On Friday, May 26, 2023 at 1:29:07 PM UTC-5, Scott Lurndal wrote:
> BGB <cr8...@gmail.com> writes:
> >On 5/26/2023 9:16 AM, Scott Lurndal wrote:
> >> BGB <cr8...@gmail.com> writes:
> >>> The topic came up elsewhere, where people were arguing that
> >>> software-managed TLB was bad/useless and that (supposedly) no modern CPU
> >>> architecture would consider using it.
> >>>
> >>>
> >>> So, it looks like for architectures I can find information on:
> >>> Hardware Managed (Page-Table):
> >>> x86 / x86-64
> >>> ARM / ARM64
> >>> RISC-V (Privileged ISA Spec)
> >>
> >> Modern CPUs.
> >>
> >
> >x86 is older than I am.
> >ARM is around the same age.
> >
> >x86-64 came out 20 years ago...
> >
> >Only RISC-V is "newer".
> Unlike the CPUs below, x86 and arm are continuously
> being enhanced with new features (e.g. secure conclaves,
> realms, new instructions, etc).
<
You use the word "enhance" in a way contrary to the dictionary definition......

MitchAlsup <MitchAlsup@aol.com> writes:
>On Friday, May 26, 2023 at 1:29:07=E2=80=AFPM UTC-5, Scott Lurndal wrote:
>> BGB <cr8...@gmail.com> writes:=20
>> >On 5/26/2023 9:16 AM, Scott Lurndal wrote:=20
>> >> BGB <cr8...@gmail.com> writes:=20
>> >>> The topic came up elsewhere, where people were arguing that=20
>> >>> software-managed TLB was bad/useless and that (supposedly) no modern =
>CPU=20
>> >>> architecture would consider using it.=20
>> >>>=20
>> >>>=20
>> >>> So, it looks like for architectures I can find information on:=20
>> >>> Hardware Managed (Page-Table):=20
>> >>> x86 / x86-64=20
>> >>> ARM / ARM64=20
>> >>> RISC-V (Privileged ISA Spec)=20
>> >>=20
>> >> Modern CPUs.=20
>> >>=20
>> >=20
>> >x86 is older than I am.=20
>> >ARM is around the same age.=20
>> >=20
>> >x86-64 came out 20 years ago...=20
>> >=20
>> >Only RISC-V is "newer".
>> Unlike the CPUs below, x86 and arm are continuously=20
>> being enhanced with new features (e.g. secure conclaves,=20
>> realms, new instructions, etc).
><
>You use the word "enhance" in a way contrary to the dictionary definition..=

There must have been demand for them from someone.

On Friday, May 26, 2023 at 2:19:40 PM UTC-5, Scott Lurndal wrote:
> MitchAlsup <Mitch...@aol.com> writes:
> >On Friday, May 26, 2023 at 1:29:07=E2=80=AFPM UTC-5, Scott Lurndal wrote:
> >> BGB <cr8...@gmail.com> writes:=20
> >> >On 5/26/2023 9:16 AM, Scott Lurndal wrote:=20
> >> >> BGB <cr8...@gmail.com> writes:=20
> >> >>> The topic came up elsewhere, where people were arguing that=20
> >> >>> software-managed TLB was bad/useless and that (supposedly) no modern =
> >CPU=20
> >> >>> architecture would consider using it.=20
> >> >>>=20
> >> >>>=20
> >> >>> So, it looks like for architectures I can find information on:=20
> >> >>> Hardware Managed (Page-Table):=20
> >> >>> x86 / x86-64=20
> >> >>> ARM / ARM64=20
> >> >>> RISC-V (Privileged ISA Spec)=20
> >> >>=20
> >> >> Modern CPUs.=20
> >> >>=20
> >> >=20
> >> >x86 is older than I am.=20
> >> >ARM is around the same age.=20
> >> >=20
> >> >x86-64 came out 20 years ago...=20
> >> >=20
> >> >Only RISC-V is "newer".
> >> Unlike the CPUs below, x86 and arm are continuously=20
> >> being enhanced with new features (e.g. secure conclaves,=20
> >> realms, new instructions, etc).
> ><
> >You use the word "enhance" in a way contrary to the dictionary definition..=
>
> There must have been demand for them from someone.
<
A billion demands from a billion different people does not a Mona Lisa make..

BGB <cr88192@gmail.com> writes:
>On 5/26/2023 12:15 PM, MitchAlsup wrote:
>> On Friday, May 26, 2023 at 11:50:57 AM UTC-5, BGB wrote:
>>> On 5/26/2023 9:16 AM, Scott Lurndal wrote:
>>>> BGB <cr8...@gmail.com> writes:
>>>>> The topic came up elsewhere, where people were arguing that
>>>>> software-managed TLB was bad/useless and that (supposedly) no modern CPU
>>>>> architecture would consider using it.
>>>>>
>>>>>
>>>>> So, it looks like for architectures I can find information on:
>>>>> Hardware Managed (Page-Table):
>>>>> x86 / x86-64
>>>>> ARM / ARM64
>>>>> RISC-V (Privileged ISA Spec)
>>>>
>>>> Modern CPUs.
>>>>
>>> x86 is older than I am.
>>> ARM is around the same age.
>> <
>> Obviously you are not old enough -:)
>
>I am scarily close to 40, by many definitions this is "old"...

At my first job out of college, this was our main production
system:

http://bitsavers.org/pdf/burroughs/MediumSystems/brochures/1120318_B4890_Central_System_Mar1979_1.jpg

Although ours was the first engineering prototype system, so instead of the white
metal covers, they were all smoked plexiglas exposing the thousands
of diagnostic LEDs on the edge of every card. Pretty impressive in a darkened
lab.

A year later, the new B4900 engineering prototype replaced it. Less
than a quarter of the floorspace used by the B4800 and no diagnostic
LEDs (but we did rig up a set of channel activity lights, something
customers kept asking for).

MitchAlsup <MitchAlsup@aol.com> writes:
>On Friday, May 26, 2023 at 1:14:03=E2=80=AFPM UTC-5, BGB wrote:
>> On 5/26/2023 12:10 PM, MitchAlsup wrote:=20
>> > On Friday, May 26, 2023 at 3:00:03=E2=80=AFAM UTC-5, BGB wrote:=20
>> >> Any thoughts?...=20
>> > <=20
>> > A Software managed TLB (i.e., MIPS) is not viable in a system=20
>> > that uses both HyperVisors and Guest OSs.=20
>> > <=20
>> > A Single HW lookup to a big hash table followed by a software=20
>> > management of the big hash table remains viable--but only so=20
>> > long as the HyperVisor performs the hash table updates.=20
>> > <=20
>> > Once you get more than 1 set of mapping tables (HV and OS) you=20
>> > really want to take software out of the TLB update process.=20
>> > <=20
>> > I, personally, went through a phase where I enjoyed software=20
>> > managed TLBs (with a modicum of HW updates), but the=20
>> > advent of HyperVisors stuck a stick into that heart.
><
>> I guess maybe one could do something where:=20
>> Only the top-level does Software TLB;=20
>> All of the lower levels need to use page-tables / etc;=20
>> It is left ambiguous as to whether the CPU does HW or SW TLB.=20
>>=20
>> Maybe the ISA Spec specifies enough table formats that it can remain=20
>> flexible. In this case, following one of the specified table formats=20
>> becomes a mandate.=20
>>=20
>>=20
>> Granted, B-Tree page-tables seem like a stretch for a hardware=20
>> implementation.=20
>>=20
>> Need a way around the "large address space + ASLR =3D> huge amounts of=20
>> wasted pages" issue. Partly as hash-tables don't scale very well.=20
><
>The need for ASLR is a symptom that the OS tables remain visible=20
>while running application threads (and bad =C2=B5Architecture.) If the
>application ALWAYS got PageFault when touching anything with
>the bit<63> set, then ASLR would not be needed.

ARMv8 has a feature that causes a translation fault if user mode
code accesses the upper half of the shared (by kernel) address space
(although they key on bit 56 rather than bit 63 - the maximum usable
VA space is either 48 or 52 bits, depending on the implementation)
and with canonical (AMD's term) addresses, addresses are sign-extended
from the bit 47 or bit 51 for translation purposes). (bit 48/52 in
the IOMMU to support full access to the VA space).

On 5/26/2023 1:46 PM, MitchAlsup wrote:
> On Friday, May 26, 2023 at 1:14:03 PM UTC-5, BGB wrote:
>> On 5/26/2023 12:10 PM, MitchAlsup wrote:
>>> On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
>>>> Any thoughts?...
>>> <
>>> A Software managed TLB (i.e., MIPS) is not viable in a system
>>> that uses both HyperVisors and Guest OSs.
>>> <
>>> A Single HW lookup to a big hash table followed by a software
>>> management of the big hash table remains viable--but only so
>>> long as the HyperVisor performs the hash table updates.
>>> <
>>> Once you get more than 1 set of mapping tables (HV and OS) you
>>> really want to take software out of the TLB update process.
>>> <
>>> I, personally, went through a phase where I enjoyed software
>>> managed TLBs (with a modicum of HW updates), but the
>>> advent of HyperVisors stuck a stick into that heart.
> <
>> I guess maybe one could do something where:
>> Only the top-level does Software TLB;
>> All of the lower levels need to use page-tables / etc;
>> It is left ambiguous as to whether the CPU does HW or SW TLB.
>>
>> Maybe the ISA Spec specifies enough table formats that it can remain
>> flexible. In this case, following one of the specified table formats
>> becomes a mandate.
>>
>>
>> Granted, B-Tree page-tables seem like a stretch for a hardware
>> implementation.
>>
>> Need a way around the "large address space + ASLR => huge amounts of
>> wasted pages" issue. Partly as hash-tables don't scale very well.
> <
> The need for ASLR is a symptom that the OS tables remain visible
> while running application threads (and bad µArchitecture.) If the
> application ALWAYS got PageFault when touching anything with
> the bit<63> set, then ASLR would not be needed.
> <
> But the Global bit--used to optimize TLB across user<->super
> boundary opened the door for Spectré like attacks. Bad µArch,
> then, widened the door such that one could train the cache (or
> other predictors) to make a dependent access on an access that
> is not allowed AND to watch such an event with a high precision
> timer, well, then, we did it to ourselves.......
> <
> My 66000 is not attackable by those means--and probably can
> avoid needing ASLR although there is no reason to to use it as
> it slows nothing down.

I had assumed ASLR as a "standard line of defense".

>>
> <snip>
>>
>> I guess one possible way would be to organize the ACL table as a
>> page-table like structure, say:
>> (31:16): KRR_ID
>> (15: 0): ACLID
> <
> In principle, you could have a different page hash table in memory for each
> ASID.

To clarify, ASID and ACLID are two different features:
ASID is used for separating address spaces;
ACLID is for per-page per-task access-rights checking.

But, yeah, a hash-table works, but would still need a handler ISR to
fill entries in the table; so isn't likely to be that much different
than the existing mechanism (of using a small cache for ACL / Keyring
pairs).

I may at some point formally deprecate the original VUGID system in
place of an "ACLs Only" approach.

>>
>> Though, this would need a 4-level table, and (for a non-trivial number
>> of ACLs) would eat a significant amount of memory.
> <
> The thing with the HW assisted software TLB is the HW makes exactly
> 1 memory access. If it succeeds, we get along with running code, if it
> fails we trap to SW and let SW sort it out. An efficient HW hash function
> is key.

OK.

If I were to use a strategy similar to IA-64, it would likely hammer out
a series of requests to the L2 cache and probably fetch either 64 or 128
bytes of data (the former should have ~ 150-200 cycles on L2 miss, the
latter ~ 300-400 cycles).

Question is partly how to best mesh this in with my existing caches and
ring-bus.

It might almost make more sense for the TLB to be able to remove
requests from the bus and re-insert them later. In this case, this could
avoid all the pipeline-stall and exception-intercepting wonk (so, from
the L1 caches' POV, it just looks like the L1 miss took an extra long time).

Probably preferable in that it would not be a massive mess...

>>
>> Though, could possibly make it more space-efficient (in the common case
>> of sequentially increasing ACLs) by Morton-shuffling the bits.
>>
>> Maybe also the leaf-table has 8-bit entries. This would reduce the table
>> to 3 levels (for a 16K page size).
>>
>> With, the final entries being, say:
>> URWX-000V
>> V: Valid
>> U: User/Supervisor
>> R: Read
>> W: Write
>> X: Execute
>>
>> Sort of a bit brute-force though, as traditionally ACLs have a more
>> hierarchical structure.
>>
>>
>> This probably excludes any ab(use) the TLB to emulate segmented
>> addressing schemes though...
> <
> Why would anyone want to do that ?

Something like DOSBox or similar...

>>
>> Still not entirely obvious how one would pull off something like
>> emulating an 8086 style addressing mode using only page-tables (at
>> least, short of massive memory duplication).
> <
> Again, Why would any new architecture want to do that ??

Again, something like DOSBox...

Granted, if one is running at GHz speeds, this is unnecessary since one
can just fake it in software and still have it be faster than the
original 8088 PCs.

But, on a 50MHz CPU, it is unclear if a pure software approach could
match the performance of the 8088 at this task.

>>
>> But, even with a SW TLB, how exactly one would pull off funky modes like
>> "Big Real Mode" in a TLB are uncertain (seems like likely, a segment
>> register-version would need to be encoded in the virtual address, so
>> every time one reloads DS or ES, it moves to a different part of the
>> extended virtual address space).
> <
> If you want x86 you have to support A20 addressing

A20 shouldn't be particularly hard to fake, once one has the logic in
place to support faking 8086 style addressing (in general).

Though, it does mean potentially the funkiness that the pages would be
organized so that the FFFF:xxxx range is actually the one present in the
physical page map, and the 0000 range mirrors the FFFF range (well, and
then remap the pages above this point once the A20 bit is toggled).

So, say:
Low 1MB map expresses 00010000..0010FFFF;
High map expresses 00100000..00BFFFFF.
00C00000..00FFFFFF IIRC, ISA-Bus stuff goes here...

....

In article <d4319038-c204-43f5-a093-d00d914f59d8n@googlegroups.com>,
MitchAlsup <MitchAlsup@aol.com> wrote:
>On Friday, May 26, 2023 at 2:19:40 PM UTC-5, Scott Lurndal wrote:
>> >You use the word "enhance" in a way contrary to the dictionary definition..=
>>
>> There must have been demand for them from someone.
>
>A billion demands from a billion different people does not a Mona Lisa make.

While I'm sure we can all agree that the x86 is a dog's
breakfast, that does not imply that all of its features are bad.
Nor does it imply that a hardware page-table walker is bad.

The OP seems to think so, but has yet to provide a particularly
compelling argument beyond some measurements from very
unrepresentative workloads running on a hobby ISA on an FPGA

- Dan C.

On Friday, May 26, 2023 at 2:45:04 PM UTC-5, BGB wrote:
> On 5/26/2023 1:46 PM, MitchAlsup wrote:
> > On Friday, May 26, 2023 at 1:14:03 PM UTC-5, BGB wrote:
> >> On 5/26/2023 12:10 PM, MitchAlsup wrote:
> >>> On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
> >>>> Any thoughts?...
> >>> <
> >>> A Software managed TLB (i.e., MIPS) is not viable in a system
> >>> that uses both HyperVisors and Guest OSs.
> >>> <
> >>> A Single HW lookup to a big hash table followed by a software
> >>> management of the big hash table remains viable--but only so
> >>> long as the HyperVisor performs the hash table updates.
> >>> <
> >>> Once you get more than 1 set of mapping tables (HV and OS) you
> >>> really want to take software out of the TLB update process.
> >>> <
> >>> I, personally, went through a phase where I enjoyed software
> >>> managed TLBs (with a modicum of HW updates), but the
> >>> advent of HyperVisors stuck a stick into that heart.
> > <
> >> I guess maybe one could do something where:
> >> Only the top-level does Software TLB;
> >> All of the lower levels need to use page-tables / etc;
> >> It is left ambiguous as to whether the CPU does HW or SW TLB.
> >>
> >> Maybe the ISA Spec specifies enough table formats that it can remain
> >> flexible. In this case, following one of the specified table formats
> >> becomes a mandate.
> >>
> >>
> >> Granted, B-Tree page-tables seem like a stretch for a hardware
> >> implementation.
> >>
> >> Need a way around the "large address space + ASLR => huge amounts of
> >> wasted pages" issue. Partly as hash-tables don't scale very well.
> > <
> > The need for ASLR is a symptom that the OS tables remain visible
> > while running application threads (and bad µArchitecture.) If the
> > application ALWAYS got PageFault when touching anything with
> > the bit<63> set, then ASLR would not be needed.
> > <
> > But the Global bit--used to optimize TLB across user<->super
> > boundary opened the door for Spectré like attacks. Bad µArch,
> > then, widened the door such that one could train the cache (or
> > other predictors) to make a dependent access on an access that
> > is not allowed AND to watch such an event with a high precision
> > timer, well, then, we did it to ourselves.......
> > <
> > My 66000 is not attackable by those means--and probably can
> > avoid needing ASLR although there is no reason to to use it as
> > it slows nothing down.
<
> I had assumed ASLR as a "standard line of defense".
<
If/when the unprivileged cannot access super-address-space no mater
the bit pattern created at AGEN, ASLR is not needed, as there is no way
the user can access memory not mapped by his page tables.
> >>
> > <snip>
> >>
> >> I guess one possible way would be to organize the ACL table as a
> >> page-table like structure, say:
> >> (31:16): KRR_ID
> >> (15: 0): ACLID
> > <
> > In principle, you could have a different page hash table in memory for each
> > ASID.
<
> To clarify, ASID and ACLID are two different features:
> ASID is used for separating address spaces;
> ACLID is for per-page per-task access-rights checking.
<
Most people place ACLID in the PTE and some in the hierarchy of
PTPs and PTE.

On Friday, May 26, 2023 at 2:59:13 PM UTC-5, Dan Cross wrote:
> In article <d4319038-c204-43f5...@googlegroups.com>,
> MitchAlsup <Mitch...@aol.com> wrote:
> >On Friday, May 26, 2023 at 2:19:40 PM UTC-5, Scott Lurndal wrote:
> >> >You use the word "enhance" in a way contrary to the dictionary definition..=
> >>
> >> There must have been demand for them from someone.
> >
> >A billion demands from a billion different people does not a Mona Lisa make.
> While I'm sure we can all agree that the x86 is a dog's
> breakfast, that does not imply that all of its features are bad.
<
All the features of an ugly woman are not bad, either.
<
> Nor does it imply that a hardware page-table walker is bad.
<
It has always been my argument that HW tableWalkers are best.
>
> The OP seems to think so, but has yet to provide a particularly
> compelling argument beyond some measurements from very
> unrepresentative workloads running on a hobby ISA on an FPGA
<
One must rate BGBs architecture is the less-than-even-academic
category; far from industrial quality.
<
But we enjoy watching him stumble across industrial problems
making the same mistakes we made 40 years ago.
>
> - Dan C.

On 5/26/2023 2:29 PM, Scott Lurndal wrote:
> MitchAlsup <MitchAlsup@aol.com> writes:
>> On Friday, May 26, 2023 at 1:14:03=E2=80=AFPM UTC-5, BGB wrote:
>>> On 5/26/2023 12:10 PM, MitchAlsup wrote:=20
>>>> On Friday, May 26, 2023 at 3:00:03=E2=80=AFAM UTC-5, BGB wrote:=20
>>>>> Any thoughts?...=20
>>>> <=20
>>>> A Software managed TLB (i.e., MIPS) is not viable in a system=20
>>>> that uses both HyperVisors and Guest OSs.=20
>>>> <=20
>>>> A Single HW lookup to a big hash table followed by a software=20
>>>> management of the big hash table remains viable--but only so=20
>>>> long as the HyperVisor performs the hash table updates.=20
>>>> <=20
>>>> Once you get more than 1 set of mapping tables (HV and OS) you=20
>>>> really want to take software out of the TLB update process.=20
>>>> <=20
>>>> I, personally, went through a phase where I enjoyed software=20
>>>> managed TLBs (with a modicum of HW updates), but the=20
>>>> advent of HyperVisors stuck a stick into that heart.
>> <
>>> I guess maybe one could do something where:=20
>>> Only the top-level does Software TLB;=20
>>> All of the lower levels need to use page-tables / etc;=20
>>> It is left ambiguous as to whether the CPU does HW or SW TLB.=20
>>> =20
>>> Maybe the ISA Spec specifies enough table formats that it can remain=20
>>> flexible. In this case, following one of the specified table formats=20
>>> becomes a mandate.=20
>>> =20
>>> =20
>>> Granted, B-Tree page-tables seem like a stretch for a hardware=20
>>> implementation.=20
>>> =20
>>> Need a way around the "large address space + ASLR =3D> huge amounts of=20
>>> wasted pages" issue. Partly as hash-tables don't scale very well.=20
>> <
>> The need for ASLR is a symptom that the OS tables remain visible=20
>> while running application threads (and bad =C2=B5Architecture.) If the
>> application ALWAYS got PageFault when touching anything with
>> the bit<63> set, then ASLR would not be needed.
>
> ARMv8 has a feature that causes a translation fault if user mode
> code accesses the upper half of the shared (by kernel) address space
> (although they key on bit 56 rather than bit 63 - the maximum usable
> VA space is either 48 or 52 bits, depending on the implementation)
> and with canonical (AMD's term) addresses, addresses are sign-extended
> from the bit 47 or bit 51 for translation purposes). (bit 48/52 in
> the IOMMU to support full access to the VA space).

(From my alt/shop account).

BJX2 is similar here, in that:
0000_00000000..7FFF_FFFFFFFF are user-accessible.
8000_00000000..FFFF_FFFFFFFF are immediate trap in Quadrant 0.

In non-zero quadrants, 8000..FFFF may represent additional virtual
address space.

8000..BFFF were intended for OS virtual memory, though as-is in
TestKern, the kernel is located at low addresses and shares the same
address space as user applications (this will need to change for a more
"proper" OS).

ASLR'ing the kernel sorta works, but is also sorta crap...

But, no canonical addresses here, since the upper bits were left for
type tags, they are generally ignored for most operations (except for a
few specialized instructions which make use of type-tags).

In article <66610ab9-0c63-4b8e-adfc-3acf89a56dc4n@googlegroups.com>,
MitchAlsup <MitchAlsup@aol.com> wrote:
>On Friday, May 26, 2023 at 2:59:13 PM UTC-5, Dan Cross wrote:
>> Nor does it imply that a hardware page-table walker is bad.
>
>It has always been my argument that HW tableWalkers are best.

I concur. OP does not, but OP doesn't seem to be talking from a
particularly knowledgable position.

>> The OP seems to think so, but has yet to provide a particularly
>> compelling argument beyond some measurements from very
>> unrepresentative workloads running on a hobby ISA on an FPGA
>
>One must rate BGBs architecture is the less-than-even-academic
>category; far from industrial quality.

Agreed.

>But we enjoy watching him stumble across industrial problems
>making the same mistakes we made 40 years ago.

It seems like many of those could be avoided if OP were a bit
more open-minded and, dare I say it, self-aware.

- Dan C.

On 5/26/2023 3:31 PM, MitchAlsup wrote:
> On Friday, May 26, 2023 at 2:45:04 PM UTC-5, BGB wrote:
>> On 5/26/2023 1:46 PM, MitchAlsup wrote:
>>> On Friday, May 26, 2023 at 1:14:03 PM UTC-5, BGB wrote:
>>>> On 5/26/2023 12:10 PM, MitchAlsup wrote:
>>>>> On Friday, May 26, 2023 at 3:00:03 AM UTC-5, BGB wrote:
>>>>>> Any thoughts?...
>>>>> <
>>>>> A Software managed TLB (i.e., MIPS) is not viable in a system
>>>>> that uses both HyperVisors and Guest OSs.
>>>>> <
>>>>> A Single HW lookup to a big hash table followed by a software
>>>>> management of the big hash table remains viable--but only so
>>>>> long as the HyperVisor performs the hash table updates.
>>>>> <
>>>>> Once you get more than 1 set of mapping tables (HV and OS) you
>>>>> really want to take software out of the TLB update process.
>>>>> <
>>>>> I, personally, went through a phase where I enjoyed software
>>>>> managed TLBs (with a modicum of HW updates), but the
>>>>> advent of HyperVisors stuck a stick into that heart.
>>> <
>>>> I guess maybe one could do something where:
>>>> Only the top-level does Software TLB;
>>>> All of the lower levels need to use page-tables / etc;
>>>> It is left ambiguous as to whether the CPU does HW or SW TLB.
>>>>
>>>> Maybe the ISA Spec specifies enough table formats that it can remain
>>>> flexible. In this case, following one of the specified table formats
>>>> becomes a mandate.
>>>>
>>>>
>>>> Granted, B-Tree page-tables seem like a stretch for a hardware
>>>> implementation.
>>>>
>>>> Need a way around the "large address space + ASLR => huge amounts of
>>>> wasted pages" issue. Partly as hash-tables don't scale very well.
>>> <
>>> The need for ASLR is a symptom that the OS tables remain visible
>>> while running application threads (and bad µArchitecture.) If the
>>> application ALWAYS got PageFault when touching anything with
>>> the bit<63> set, then ASLR would not be needed.
>>> <
>>> But the Global bit--used to optimize TLB across user<->super
>>> boundary opened the door for Spectré like attacks. Bad µArch,
>>> then, widened the door such that one could train the cache (or
>>> other predictors) to make a dependent access on an access that
>>> is not allowed AND to watch such an event with a high precision
>>> timer, well, then, we did it to ourselves.......
>>> <
>>> My 66000 is not attackable by those means--and probably can
>>> avoid needing ASLR although there is no reason to to use it as
>>> it slows nothing down.
> <
>> I had assumed ASLR as a "standard line of defense".
> <
> If/when the unprivileged cannot access super-address-space no mater
> the bit pattern created at AGEN, ASLR is not needed, as there is no way
> the user can access memory not mapped by his page tables.

ASLR can help protect against things like userspace code doing
buffer-overflow exploits against system calls or intentionally mangled
data structures (along with things like marking stack and heap memory
and similar as non-executable, ...).

Partly as buffer-overflow and many forms of "confused deputy" attacks
are not stopped by conventional memory-access protections, but can be
made ineffective by using ASLR.

Similar goes for the "compiler shuffles all the functions on each
rebuild", etc.

In my case, there is a hardware RNG that can keep its random seed state
on the SDcard. A "better" option would be some sort of NVRAM, but (much
like a real-time clock), pretty much no normal FPGA boards have this.

Would "almost" be nice if capabilities could be supported as well,
except that there are some serious drawbacks with capabilities as well
(to make them "actually effective" adds drawbacks, otherwise one could
sidestep them, which would severely limit their effectiveness).

There is a feature for bounds-checked pointers in BJX2 (encoding the
bounds in the tag bits), but it falls well short of what would be needed
for an actual capability architecture.

>>>>
>>> <snip>
>>>>
>>>> I guess one possible way would be to organize the ACL table as a
>>>> page-table like structure, say:
>>>> (31:16): KRR_ID
>>>> (15: 0): ACLID
>>> <
>>> In principle, you could have a different page hash table in memory for each
>>> ASID.
> <
>> To clarify, ASID and ACLID are two different features:
>> ASID is used for separating address spaces;
>> ACLID is for per-page per-task access-rights checking.
> <
> Most people place ACLID in the PTE and some in the hierarchy of
> PTPs and PTE.
>

There are some protection flags in the PTEs as well, but they only cover
the traditional User/Supervisor and "Global RWX" state.

ACLID is for more fine-grained use, such as being able to segment
individual threads or similar off into their own sandboxes within a
shared virtual address space.

On Friday, May 26, 2023 at 4:43:19 PM UTC-5, BGB-Alt wrote:
> On 5/26/2023 3:31 PM, MitchAlsup wrote:

> >>> My 66000 is not attackable by those means--and probably can
> >>> avoid needing ASLR although there is no reason to to use it as
> >>> it slows nothing down.
> > <
> >> I had assumed ASLR as a "standard line of defense".
> > <
> > If/when the unprivileged cannot access super-address-space no mater
> > the bit pattern created at AGEN, ASLR is not needed, as there is no way
> > the user can access memory not mapped by his page tables.
<
> ASLR can help protect against things like userspace code doing
> buffer-overflow exploits against system calls or intentionally mangled
> data structures (along with things like marking stack and heap memory
> and similar as non-executable, ...).
<
What if the control-flow information is not accessible to the LDs and STs
of an ISA ?? That is, you can buffer overflow all you want, but when you
execute RET you end up back at who called that subroutine ???
<
My 66000 architecture has 2 stack pointers, one for the data stack and
one for the control flow and protected registers stack. The one for the
protected register stack is not accessible to the program except through
ENTER and EXIT and RET instructions and the pages are marked RWE = 000
and HW verifies that those pages are so marked (at least at user privilege)..
<
So, you can damage as much data memory as you like, but return from a
subroutine will end up at the instruction after the call AND all of the
preserved registers have their old values.
>
> Partly as buffer-overflow and many forms of "confused deputy" attacks
> are not stopped by conventional memory-access protections, but can be
> made ineffective by using ASLR.
>
Look, I am NOT arguing that ASLR is bad, just that if the ISA and MMU
is properly defined ASLR brings nothing MORE to the table. I am arguing
the the need for ASLR is simply indicative of bad architecture.
>
> Similar goes for the "compiler shuffles all the functions on each
> rebuild", etc.
<
Oh, and BTW, the user privilege is not allowed to write GOT and My 66000
does not need a PLT, either. Cutting off even more attack vectors.
>
> In my case, there is a hardware RNG that can keep its random seed state
> on the SDcard. A "better" option would be some sort of NVRAM, but (much
> like a real-time clock), pretty much no normal FPGA boards have this.
>
Look, if you NEED ASLR to have a modicum of safety and protection,
GO FO IT.
<
My 66000 has no such need.
>
> Would "almost" be nice if capabilities could be supported as well,
> except that there are some serious drawbacks with capabilities as well
> (to make them "actually effective" adds drawbacks, otherwise one could
> sidestep them, which would severely limit their effectiveness).
<
One way to think about My 66000 is that the user address space is a
capability, and that the user does not have a capability to GuestOS
address space, GuestOS does not have a capability to HyperVisor
address space,.....
<
It does not mater what bit pattern AGEN spits out, you cannot access
the greater privilege level's address spaces. Not from ISA, not from TLB,
not from Cache lines, not from watching a high precision timer.
<
Computer architecture is hard enough, there is no need to add even more
fuzz across a myriad of problem-spaces when you can simply design them
out before hand.
>
> There is a feature for bounds-checked pointers in BJX2 (encoding the
> bounds in the tag bits), but it falls well short of what would be needed
> for an actual capability architecture.
<
I access bounds checking via the CMP instruction.
> >>>>
> >>> <snip>
> >>>>
> >>>> I guess one possible way would be to organize the ACL table as a
> >>>> page-table like structure, say:
> >>>> (31:16): KRR_ID
> >>>> (15: 0): ACLID
> >>> <
> >>> In principle, you could have a different page hash table in memory for each
> >>> ASID.
> > <
> >> To clarify, ASID and ACLID are two different features:
> >> ASID is used for separating address spaces;
> >> ACLID is for per-page per-task access-rights checking.
> > <
> > Most people place ACLID in the PTE and some in the hierarchy of
> > PTPs and PTE.
> >
> There are some protection flags in the PTEs as well, but they only cover
> the traditional User/Supervisor and "Global RWX" state.
<
Consider a HyperVisor hosting 2 GuestOSs. Both Guest OSs want to run
their applications (the normal way) by having the user program share
page tables and "Optimizing" the TLB using the G-bit. Now that you have
2 GuestOSs, this G-bit is now allowing leaks between GuestOSs, confusing
the TLB mappings, and causing a host of other problems.
<
Now, consider an ASID system where the GuestOSs use different ASIDs
and now all those G-bit problems disappear ! the TLB remains unconfused,
and the memory hierarchy knows what to do. HyperVisors do this to the
old MMU models, new architectures should solve these problems without
creating new ones.
>
> ACLID is for more fine-grained use, such as being able to segment
> individual threads or similar off into their own sandboxes within a
> shared virtual address space.

On 5/26/2023 5:17 PM, MitchAlsup wrote:
> On Friday, May 26, 2023 at 4:43:19 PM UTC-5, BGB-Alt wrote:
>> On 5/26/2023 3:31 PM, MitchAlsup wrote:
>
>>>>> My 66000 is not attackable by those means--and probably can
>>>>> avoid needing ASLR although there is no reason to to use it as
>>>>> it slows nothing down.
>>> <
>>>> I had assumed ASLR as a "standard line of defense".
>>> <
>>> If/when the unprivileged cannot access super-address-space no mater
>>> the bit pattern created at AGEN, ASLR is not needed, as there is no way
>>> the user can access memory not mapped by his page tables.
> <
>> ASLR can help protect against things like userspace code doing
>> buffer-overflow exploits against system calls or intentionally mangled
>> data structures (along with things like marking stack and heap memory
>> and similar as non-executable, ...).
> <
> What if the control-flow information is not accessible to the LDs and STs
> of an ISA ?? That is, you can buffer overflow all you want, but when you
> execute RET you end up back at who called that subroutine ???
> <
> My 66000 architecture has 2 stack pointers, one for the data stack and
> one for the control flow and protected registers stack. The one for the
> protected register stack is not accessible to the program except through
> ENTER and EXIT and RET instructions and the pages are marked RWE = 000
> and HW verifies that those pages are so marked (at least at user privilege).
> <
> So, you can damage as much data memory as you like, but return from a
> subroutine will end up at the instruction after the call AND all of the
> preserved registers have their old values.

Yeah. Only one stack in my case.

There are stack canaries though, which can detect if a buffer overflow
had overwritten the canary value.

Had at one point considered a feature to hash the state of all of the
preserved registers, and then verify that everything was intact. No real
practical way to do this check in-hardware though.

>>
>> Partly as buffer-overflow and many forms of "confused deputy" attacks
>> are not stopped by conventional memory-access protections, but can be
>> made ineffective by using ASLR.
>>
> Look, I am NOT arguing that ASLR is bad, just that if the ISA and MMU
> is properly defined ASLR brings nothing MORE to the table. I am arguing
> the the need for ASLR is simply indicative of bad architecture.

OK.

Most systems use it at least, and it is reasonably cheap.

In an ideal world, maybe it would be unnecessary, but as I see it, it is
likely well worth its cost on this front.

>>
>> Similar goes for the "compiler shuffles all the functions on each
>> rebuild", etc.
> <
> Oh, and BTW, the user privilege is not allowed to write GOT and My 66000
> does not need a PLT, either. Cutting off even more attack vectors.

OK.

I am mostly using direct branches for local calls, and had intended
Abs48 branches for DLL imports (but, with the drawback that Abs48
branches can't encode Inter-ISA branches).

Though, it is possible I could consider also allowing a Jumbo-Load +
Register Branch sequence for DLL imports, which is capable of InterISA,
and/or always leave at least 128 bits.

It seems reasonable that class VTable's and similar could be put into
read-only pages.

Still leaves a concern though if a program could compose "malicious COM
objects" or similar and then put them somewhere where they are not
supposed to go.

Though, this is an area where (in theory) keyring/ACL checks could help,
since (ideally) if neither party can execute each others' ".text"
sections, then one can stop programs from sneaking bad COM objects into
OS APIs.

Well, that or run lint-checks on any user-supplied objects and make
disallow ones with VTables or method pointers into writable memory. But,
this requires the APIs to be "well designed", which is possibly asking a
lot.

>>
>> In my case, there is a hardware RNG that can keep its random seed state
>> on the SDcard. A "better" option would be some sort of NVRAM, but (much
>> like a real-time clock), pretty much no normal FPGA boards have this.
>>
> Look, if you NEED ASLR to have a modicum of safety and protection,
> GO FO IT.
> <
> My 66000 has no such need.

It isn't strictly needed in an architectural sense, but I don't trust
security without it.

Like, seeing how easily x86 systems have been to hack due to these sorts
of issues (and the epic failure of "just write code that is not weak
against buffer overflow").

Well, and the pros/cons that trying to invoke buffer-overflow exploits
over USB was one of the major strategies for "jail breaking" or
"rooting" cell-phones.

Well, and as-is, TestKern is still kinda pathetic on this front.

>>
>> Would "almost" be nice if capabilities could be supported as well,
>> except that there are some serious drawbacks with capabilities as well
>> (to make them "actually effective" adds drawbacks, otherwise one could
>> sidestep them, which would severely limit their effectiveness).
> <
> One way to think about My 66000 is that the user address space is a
> capability, and that the user does not have a capability to GuestOS
> address space, GuestOS does not have a capability to HyperVisor
> address space,.....
> <
> It does not mater what bit pattern AGEN spits out, you cannot access
> the greater privilege level's address spaces. Not from ISA, not from TLB,
> not from Cache lines, not from watching a high precision timer.
> <
> Computer architecture is hard enough, there is no need to add even more
> fuzz across a myriad of problem-spaces when you can simply design them
> out before hand.

Hmm...

Hard to keep things both "cheap" and "wont interfere with normal C
coding practices".

>>
>> There is a feature for bounds-checked pointers in BJX2 (encoding the
>> bounds in the tag bits), but it falls well short of what would be needed
>> for an actual capability architecture.
> <
> I access bounds checking via the CMP instruction.

I have the LEAT and BNDCHK instructions.

Where:
LEAT.x (Rm, Ri), Rn
Behaves like a LEA, but also adjusts the bounds.
BNDCHK.x Rm, Rn
Will raise a fault if the access is out-of-bounds.

Whereas, the normal LEA.x will zero the tag bits.

For the XLEA.x and XMOV.x instructions, bound-adjustment and checking
are the default behaviors (if bounds-checking is enabled in the ISA).

But, unlike a capability machine, nothing prevents a program from
twiddling the tag bits. A true capability machine would require a way to
tag the registers and memory to prevent the program from twiddling these
bits.

But, at the moment one is like "well, we can assign 2 bits to every
128-bit pair to separate pointers from other data", a massive crap-storm
is unleashed.

Sadly, not enough bits to make them "not suck" or to encode both bounds
and an element type.

>>>>>>
>>>>> <snip>
>>>>>>
>>>>>> I guess one possible way would be to organize the ACL table as a
>>>>>> page-table like structure, say:
>>>>>> (31:16): KRR_ID
>>>>>> (15: 0): ACLID
>>>>> <
>>>>> In principle, you could have a different page hash table in memory for each
>>>>> ASID.
>>> <
>>>> To clarify, ASID and ACLID are two different features:
>>>> ASID is used for separating address spaces;
>>>> ACLID is for per-page per-task access-rights checking.
>>> <
>>> Most people place ACLID in the PTE and some in the hierarchy of
>>> PTPs and PTE.
>>>
>> There are some protection flags in the PTEs as well, but they only cover
>> the traditional User/Supervisor and "Global RWX" state.
> <
> Consider a HyperVisor hosting 2 GuestOSs. Both Guest OSs want to run
> their applications (the normal way) by having the user program share
> page tables and "Optimizing" the TLB using the G-bit. Now that you have
> 2 GuestOSs, this G-bit is now allowing leaks between GuestOSs, confusing
> the TLB mappings, and causing a host of other problems.
> <
> Now, consider an ASID system where the GuestOSs use different ASIDs
> and now all those G-bit problems disappear ! the TLB remains unconfused,
> and the memory hierarchy knows what to do. HyperVisors do this to the
> old MMU models, new architectures should solve these problems without
> creating new ones.

OK.

I guess it came up elsewhere that apparently hypervisors differ more
from emulators than initially thought.

On 5/26/2023 4:15 PM, Dan Cross wrote:
> In article <66610ab9-0c63-4b8e-adfc-3acf89a56dc4n@googlegroups.com>,
> MitchAlsup <MitchAlsup@aol.com> wrote:
>> On Friday, May 26, 2023 at 2:59:13 PM UTC-5, Dan Cross wrote:
>>> Nor does it imply that a hardware page-table walker is bad.
>>
>> It has always been my argument that HW tableWalkers are best.
>
> I concur. OP does not, but OP doesn't seem to be talking from a
> particularly knowledgable position.
>

If I had no idea what I was doing, I would not likely have gotten as far
as I have...

But, admittedly, some amount of what information I had came from
quick-and-dirty web-searches and gathering information from things like
CS/EE PowerPoint slides and similar.

Some amount of reading PDFs and similar as well.

>>> The OP seems to think so, but has yet to provide a particularly
>>> compelling argument beyond some measurements from very
>>> unrepresentative workloads running on a hobby ISA on an FPGA
>>
>> One must rate BGBs architecture is the less-than-even-academic
>> category; far from industrial quality.
>
> Agreed.
>

How many people have done much better with their hobby CPU ISA projects?...

>> But we enjoy watching him stumble across industrial problems
>> making the same mistakes we made 40 years ago.
>
> It seems like many of those could be avoided if OP were a bit
> more open-minded and, dare I say it, self-aware.
>

?...

I am aware of my own existence, and am able to recognize my own
reflection in a mirror, etc.

I am not always the best in "intuitive" contexts.
Nor necessarily at noticing or thinking about "obvious" things.

Nor, particularly skilled with "top down" thinking.

I am not entirely sure how to describe my thought processes, or how they
compare to others.

More, it is like mentally stringing random thoughts and ideas together,
and seeing which (if any) go in seemingly interesting directions.

....