Branch miss logic versus clock frequency.

The branch miss logic for the current OoO version of Thor is quite
involved. It needs to back out the register source indexes to the last
valid source before the branch instruction. To do this in a single
cycle, the logic is about 25+ logic levels deep. I find this somewhat
unacceptable.

I can remove a lot of logic improving the clock frequency substantially
by removing the branch miss logic that resets the registers source id to
the last valid source. Instead of stomping on the instruction on a miss
and flushing the instructions in a single cycle, I think the predicate
for the instructions can be cleared which will effectively turn them
into a NOP. The value of the target register will be propagated in the
reorder buffer meaning the registers source id need not be reset. The
reorder buffer is only eight entries. So, on average four entries would
be turned into NOPs. The NOPs would still propagate through the reorder
buffer so it may take several clock cycles for them to be flushed from
the buffer. Meaning the branch latency for miss-predicted branches would
be quite high. However, if the clock frequency can be improved by 20%
for all instructions, much of the lost performance on the branches may
be made up.

Robert Finch wrote:
> Branch miss logic versus clock frequency.
>
> The branch miss logic for the current OoO version of Thor is quite
> involved. It needs to back out the register source indexes to the last
> valid source before the branch instruction. To do this in a single
> cycle, the logic is about 25+ logic levels deep. I find this somewhat
> unacceptable.
>
> I can remove a lot of logic improving the clock frequency substantially
> by removing the branch miss logic that resets the registers source id to
> the last valid source. Instead of stomping on the instruction on a miss
> and flushing the instructions in a single cycle, I think the predicate
> for the instructions can be cleared which will effectively turn them
> into a NOP. The value of the target register will be propagated in the
> reorder buffer meaning the registers source id need not be reset. The
> reorder buffer is only eight entries. So, on average four entries would
> be turned into NOPs. The NOPs would still propagate through the reorder
> buffer so it may take several clock cycles for them to be flushed from
> the buffer. Meaning the branch latency for miss-predicted branches would
> be quite high. However, if the clock frequency can be improved by 20%
> for all instructions, much of the lost performance on the branches may
> be made up.

Basically it sounds like you want to eliminate the checkpoint and rollback,
and instead let resources be recovered at Retire. That could work.

However you are not restoring the Renamer's future Register Alias Table (RAT)
to its state at the point of the mispredicted branch instruction, which is
what the rollback would have done, so its state will be whatever it was at
the end of the mispredicted sequence. That needs to be re-sync'ed with the
program state as of the branch.

That can be accomplished by stalling the front end, waiting until the
mispredicted branch reaches Retire and then copying the committed RAT,
maintained by Retire, to the future RAT at Rename, and restart front end.
The list of free physical registers is then all those that are not
marked as architectural registers.
This is partly how I handle exceptions.

Also you still need a mechanism to cancel start of execution of the
subset of pending uOps for the purged set. You don't want to launch
a LD or DIV from the mispredicted set if it has not already started.
If you are using a reservation station design then you need some way
to distribute the cancel request to the various FU's and RS's,
and wait for them to clean themselves up.

Note that some things might not be able to cancel immediately,
like an in-flight MUL in a pipeline or an outstanding LD to the cache.
So some of this will be asynchronous (send cancel request, wait for ACK).

There are some other things that might need cleanup.
A Return Stack Predictor might be manipulated by the mispredicted path.
Not sure how to handle that without a checkpoint.
Maybe have two copies like RAT, a future one maintained by Decode and
a committed one maintained by Retire, and copy the committed to future.

Robert Finch wrote:

> Branch miss logic versus clock frequency.

> The branch miss logic for the current OoO version of Thor is quite
> involved. It needs to back out the register source indexes to the last
> valid source before the branch instruction. To do this in a single
> cycle, the logic is about 25+ logic levels deep. I find this somewhat
> unacceptable.
<
When you launch a predicted branch into execution (prelude to signaling
recovery is required), while the branch is determining whether to backup
(or not) have the branch recovery logic setup the register indexes such
that::
a) if the branch succeeds keep the current map
b) if the branch fails, you are 1 multiplexer delay from having the state
you want.
<
That is move the setup to repair the previous clock.
<
> I can remove a lot of logic improving the clock frequency substantially
> by removing the branch miss logic that resets the registers source id to
> the last valid source. Instead of stomping on the instruction on a miss
> and flushing the instructions in a single cycle, I think the predicate
> for the instructions can be cleared which will effectively turn them
> into a NOP. The value of the target register will be propagated in the
> reorder buffer meaning the registers source id need not be reset. The
> reorder buffer is only eight entries. So, on average four entries would
> be turned into NOPs. The NOPs would still propagate through the reorder
> buffer so it may take several clock cycles for them to be flushed from
> the buffer. Meaning the branch latency for miss-predicted branches would
> be quite high. However, if the clock frequency can be improved by 20%
> for all instructions, much of the lost performance on the branches may
> be made up.

EricP wrote:

> Robert Finch wrote:
>> Branch miss logic versus clock frequency.
>>
>> The branch miss logic for the current OoO version of Thor is quite
>> involved. It needs to back out the register source indexes to the last
>> valid source before the branch instruction. To do this in a single
>> cycle, the logic is about 25+ logic levels deep. I find this somewhat
>> unacceptable.
>>
>> I can remove a lot of logic improving the clock frequency substantially
>> by removing the branch miss logic that resets the registers source id to
>> the last valid source. Instead of stomping on the instruction on a miss
>> and flushing the instructions in a single cycle, I think the predicate
>> for the instructions can be cleared which will effectively turn them
>> into a NOP. The value of the target register will be propagated in the
>> reorder buffer meaning the registers source id need not be reset. The
>> reorder buffer is only eight entries. So, on average four entries would
>> be turned into NOPs. The NOPs would still propagate through the reorder
>> buffer so it may take several clock cycles for them to be flushed from
>> the buffer. Meaning the branch latency for miss-predicted branches would
>> be quite high. However, if the clock frequency can be improved by 20%
>> for all instructions, much of the lost performance on the branches may
>> be made up.

> Basically it sounds like you want to eliminate the checkpoint and rollback,
> and instead let resources be recovered at Retire. That could work.

> However you are not restoring the Renamer's future Register Alias Table (RAT)
> to its state at the point of the mispredicted branch instruction, which is
> what the rollback would have done, so its state will be whatever it was at
> the end of the mispredicted sequence. That needs to be re-sync'ed with the
> program state as of the branch.
<
I, personally, don't use a RAT--I use a CAM based architectural decoder
for operand read and a standard physical equality decoder for writes.
<
Every cycle the CAM.valid bits are block loaded into a history table
and if you need to return the CAMs to the checkpointed mappings, you
take the valid bits from the history table and write the CAM.valid
bits back into the physical register file. Presto, the map is how it
used to be.
<
Can even be made to be performed in 0-cycles. {yes: 0 not 1 cycles}
<
> That can be accomplished by stalling the front end, waiting until the
> mispredicted branch reaches Retire and then copying the committed RAT,
> maintained by Retire, to the future RAT at Rename, and restart front end.
> The list of free physical registers is then all those that are not
> marked as architectural registers.
<
Sounds slow.
<
> This is partly how I handle exceptions.

> Also you still need a mechanism to cancel start of execution of the
> subset of pending uOps for the purged set. You don't want to launch
> a LD or DIV from the mispredicted set if it has not already started.
> If you are using a reservation station design then you need some way
> to distribute the cancel request to the various FU's and RS's,
> and wait for them to clean themselves up.
<
I use the concept of an execution window to do this both at the reservation
station and function units. There is an insert pointer and a consistent
pointer RS is only allowed to launch when the instruction is between.
FU are only allowed to calculate so long as the instruction remains
between these 2 pointers. The 2 pointers (4-bits each) are broadcast
around the machine every cycle. Each station and unit decide for themselves.

> Note that some things might not be able to cancel immediately,
> like an in-flight MUL in a pipeline or an outstanding LD to the cache.
> So some of this will be asynchronous (send cancel request, wait for ACK).
<
If an instruction that should not have its result delivered is delivered,
it is delivered to the physical register it was assigned at its issue time.
But since the value had not been delivered, that register is not in the
pool of assignable registers, so no dependency has been created.
<
> There are some other things that might need cleanup.
> A Return Stack Predictor might be manipulated by the mispredicted path.
<
Do these with a linked list and you can backup a misprediced return
to a mispredicted call.
<
> Not sure how to handle that without a checkpoint.
<
Every (non exceptional) flow altering instruction needs a checkpoint.
Predicated strings of instructions use a light weight checkpoint;
predicted branches use a heavy weight version.
<
> Maybe have two copies like RAT, a future one maintained by Decode and
> a committed one maintained by Retire, and copy the committed to future.

Decided to shelve Thor2024 and begin work on Thor2025. While Thor2024 is
very good there are a few issues with it. The ROB is used to store
register values and that is effectively a CAM. It is not very resource
efficient in an FPGA. I have been researching an x86 OoO implementation
(https://www.stuffedcow.net/files/henry-thesis-phd.pdf ) done in an FPGA
and it turns out to be considerably smaller than Thor. There are more
efficient implementations for components than what is currently in use.

Thor2025 will use a PRF approach although using a PRF seems large to me.
To reduce the size and complexity of the register file, separate
register files will be used for float and integer operations, along with
separate register files for vector mask registers and subroutine link
registers. This set of register files limits the GPR file to only 3
write ports and 18 read ports to support all the functional units.
Currently the register file is 10r2w.

The trade-off is block RAM usage instead of LUTs.

While having separate registers files seems like a step backwards, it
should ultimately make the hardware more resource efficient. It does
impact the ISA spec.

Robert Finch wrote:

> Decided to shelve Thor2024 and begin work on Thor2025. While Thor2024 is
> very good there are a few issues with it. The ROB is used to store
> register values and that is effectively a CAM. It is not very resource
> efficient in an FPGA. I have been researching an x86 OoO implementation
> (https://www.stuffedcow.net/files/henry-thesis-phd.pdf ) done in an FPGA
> and it turns out to be considerably smaller than Thor. There are more
> efficient implementations for components than what is currently in use.

> Thor2025 will use a PRF approach although using a PRF seems large to me.
<
I have a PRF design I could show you--way to big for comp.arch and
with the requisite figures.
<
> To reduce the size and complexity of the register file, separate
> register files will be used for float and integer operations, along with
> separate register files for vector mask registers and subroutine link
> registers. This set of register files limits the GPR file to only 3
> write ports and 18 read ports to support all the functional units.
> Currently the register file is 10r2w.

> The trade-off is block RAM usage instead of LUTs.

> While having separate registers files seems like a step backwards, it
> should ultimately make the hardware more resource efficient. It does
> impact the ISA spec.

On 2023-11-15 2:11 p.m., MitchAlsup wrote:
> Robert Finch wrote:
>
>> Decided to shelve Thor2024 and begin work on Thor2025. While Thor2024
>> is very good there are a few issues with it. The ROB is used to store
>> register values and that is effectively a CAM. It is not very resource
>> efficient in an FPGA. I have been researching an x86 OoO
>> implementation (https://www.stuffedcow.net/files/henry-thesis-phd.pdf
>> ) done in an FPGA and it turns out to be considerably smaller than
>> Thor. There are more efficient implementations for components than
>> what is currently in use.
>
>> Thor2025 will use a PRF approach although using a PRF seems large to me.
> <
> I have a PRF design I could show you--way to big for comp.arch and
> with the requisite figures.
> <
>> To reduce the size and complexity of the register file, separate
>> register files will be used for float and integer operations, along
>> with separate register files for vector mask registers and subroutine
>> link registers. This set of register files limits the GPR file to only
>> 3 write ports and 18 read ports to support all the functional units.
>> Currently the register file is 10r2w.
>
>> The trade-off is block RAM usage instead of LUTs.
>
>> While having separate registers files seems like a step backwards, it
>> should ultimately make the hardware more resource efficient. It does
>> impact the ISA spec.

Still digesting the PRF diagram.

Decided to go with a unified register file, 27r3w so far. Having
separate register files would not reduce the number of read ports
required and would add complexity to the processor.

Loads, FPU operations and flow control (FCU) operations all share the
third write port of the register file. The other two write ports are
dedicated to the ALU results. I think this will be okay given <1% of
instructions would be FCU updates. Loads are about 25%, and FPU depends
on the application.

The ALUs/FPU/Loads have five input operands including the 3 source
operands, a target operand, and a mask register. Stores do not need a
target operand. FCU ops are non-masked so do not need a mask register or
target operand input.

Not planning to implement the vector register file as it would be immense.

On 2023-11-17 10:39 p.m., Robert Finch wrote:
> On 2023-11-15 2:11 p.m., MitchAlsup wrote:
>> Robert Finch wrote:
>>
>>> Decided to shelve Thor2024 and begin work on Thor2025. While Thor2024
>>> is very good there are a few issues with it. The ROB is used to store
>>> register values and that is effectively a CAM. It is not very
>>> resource efficient in an FPGA. I have been researching an x86 OoO
>>> implementation (https://www.stuffedcow.net/files/henry-thesis-phd.pdf
>>> ) done in an FPGA and it turns out to be considerably smaller than
>>> Thor. There are more efficient implementations for components than
>>> what is currently in use.
>>
>>> Thor2025 will use a PRF approach although using a PRF seems large to me.
>> <
>> I have a PRF design I could show you--way to big for comp.arch and
>> with the requisite figures.
>> <
>>> To reduce the size and complexity of the register file, separate
>>> register files will be used for float and integer operations, along
>>> with separate register files for vector mask registers and subroutine
>>> link registers. This set of register files limits the GPR file to
>>> only 3 write ports and 18 read ports to support all the functional
>>> units. Currently the register file is 10r2w.
>>
>>> The trade-off is block RAM usage instead of LUTs.
>>
>>> While having separate registers files seems like a step backwards, it
>>> should ultimately make the hardware more resource efficient. It does
>>> impact the ISA spec.
>
> Still digesting the PRF diagram.
>
> Decided to go with a unified register file, 27r3w so far. Having
> separate register files would not reduce the number of read ports
> required and would add complexity to the processor.
>
> Loads, FPU operations and flow control (FCU) operations all share the
> third write port of the register file. The other two write ports are
> dedicated to the ALU results. I think this will be okay given <1% of
> instructions would be FCU updates. Loads are about 25%, and FPU depends
> on the application.
>
> The ALUs/FPU/Loads have five input operands including the 3 source
> operands, a target operand, and a mask register. Stores do not need a
> target operand. FCU ops are non-masked so do not need a mask register or
> target operand input.
>
> Not planning to implement the vector register file as it would be immense.
>
Changed the moniker of my current processor project from Thor to Qupls
(Q-Plus). I wanted a five- letter name beginning with ‘Q’. For a moment
I thought of calling it Quake but thought that would be too confusing.
One must understand the magic behind name choices.

The current design uses instruction postfixes of 32, 48, 80, and 144
bits which provide constants of 23, 39, 64 and 128 bits. Two bits in the
instruction indicate the postfix size. 64 and 128-bit constants have
seven extra unused bits available. The fields available being 71 and 135
bits.

Somewhat ugly, but it is desired to keep instructions a multiple of
16-bits in size. The shortest instruction is a NOP which is 16-bits so
that it may be used for alignment.

I almost switched to 96-bit floats which seem appealing, but once again
remembered that the progression of 32, 64, 128-bit floats work very well
for the float approximations.

Branches are 48-bit, being a combination of a compare and a branch with
a 24-bit target address field. Other flow control ops like JSR and JMP
are also 48-bit to keep all flow controls at 48-bit for simplified decoding.

Most instructions are 32-bits in size.

Sticking with a 64-register unified register file.

Removed the vector operations. There is enough play in the ISA to add
them at a later date if desired.

Loads and stores support two address mode, d(Rn) and d(Rn+Rm*Sc). The
scaled index address mode will likely be a 48-bit op.

Robert Finch wrote:

> On 2023-11-15 2:11 p.m., MitchAlsup wrote:
>> Robert Finch wrote:
>>
>>> Decided to shelve Thor2024 and begin work on Thor2025. While Thor2024
>>> is very good there are a few issues with it. The ROB is used to store
>>> register values and that is effectively a CAM. It is not very resource
>>> efficient in an FPGA. I have been researching an x86 OoO
>>> implementation (https://www.stuffedcow.net/files/henry-thesis-phd.pdf
>>> ) done in an FPGA and it turns out to be considerably smaller than
>>> Thor. There are more efficient implementations for components than
>>> what is currently in use.
>>
>>> Thor2025 will use a PRF approach although using a PRF seems large to me.
>> <
>> I have a PRF design I could show you--way to big for comp.arch and
>> with the requisite figures.
>> <
>>> To reduce the size and complexity of the register file, separate
>>> register files will be used for float and integer operations, along
>>> with separate register files for vector mask registers and subroutine
>>> link registers. This set of register files limits the GPR file to only
>>> 3 write ports and 18 read ports to support all the functional units.
>>> Currently the register file is 10r2w.
>>
>>> The trade-off is block RAM usage instead of LUTs.
>>
>>> While having separate registers files seems like a step backwards, it
>>> should ultimately make the hardware more resource efficient. It does
>>> impact the ISA spec.

> Still digesting the PRF diagram.

The diagram is for a 6R6W PRF with a history table, ARN->PRN translation,
Free pool pickers, and register ports. The X with a ½ box is a latch
or flip-flop depending on the clocking that is put around the figure.
It also includes the renamer {history table and free pool pickers}.

> Decided to go with a unified register file, 27r3w so far. Having
> separate register files would not reduce the number of read ports
> required and would add complexity to the processor.

9 Reads per 1 write ?!?!?

> Loads, FPU operations and flow control (FCU) operations all share the
> third write port of the register file. The other two write ports are
> dedicated to the ALU results. I think this will be okay given <1% of
> instructions would be FCU updates. Loads are about 25%, and FPU depends
> on the application.

> The ALUs/FPU/Loads have five input operands including the 3 source
> operands, a target operand, and a mask register. Stores do not need a
> target operand. FCU ops are non-masked so do not need a mask register or
> target operand input.

> Not planning to implement the vector register file as it would be immense.

On 2023-11-18 12:27 p.m., MitchAlsup wrote:
> Robert Finch wrote:
>
>> On 2023-11-15 2:11 p.m., MitchAlsup wrote:
>>> Robert Finch wrote:
>>>
>>>> Decided to shelve Thor2024 and begin work on Thor2025. While
>>>> Thor2024 is very good there are a few issues with it. The ROB is
>>>> used to store register values and that is effectively a CAM. It is
>>>> not very resource efficient in an FPGA. I have been researching an
>>>> x86 OoO implementation
>>>> (https://www.stuffedcow.net/files/henry-thesis-phd.pdf ) done in an
>>>> FPGA and it turns out to be considerably smaller than Thor. There
>>>> are more efficient implementations for components than what is
>>>> currently in use.
>>>
>>>> Thor2025 will use a PRF approach although using a PRF seems large to
>>>> me.
>>> <
>>> I have a PRF design I could show you--way to big for comp.arch and
>>> with the requisite figures.
>>> <
>>>> To reduce the size and complexity of the register file, separate
>>>> register files will be used for float and integer operations, along
>>>> with separate register files for vector mask registers and
>>>> subroutine link registers. This set of register files limits the GPR
>>>> file to only 3 write ports and 18 read ports to support all the
>>>> functional units. Currently the register file is 10r2w.
>>>
>>>> The trade-off is block RAM usage instead of LUTs.
>>>
>>>> While having separate registers files seems like a step backwards,
>>>> it should ultimately make the hardware more resource efficient. It
>>>> does impact the ISA spec.
>
>> Still digesting the PRF diagram.
>
> The diagram is for a 6R6W PRF with a history table, ARN->PRN translation,
> Free pool pickers, and register ports. The X with a ½ box is a latch
> or flip-flop depending on the clocking that is put around the figure.
> It also includes the renamer {history table and free pool pickers}.
>
>> Decided to go with a unified register file, 27r3w so far. Having
>> separate register files would not reduce the number of read ports
>> required and would add complexity to the processor.
>
> 9 Reads per 1 write ?!?!?
>
>> Loads, FPU operations and flow control (FCU) operations all share the
>> third write port of the register file. The other two write ports are
>> dedicated to the ALU results. I think this will be okay given <1% of
>> instructions would be FCU updates. Loads are about 25%, and FPU
>> depends on the application.
>
>> The ALUs/FPU/Loads have five input operands including the 3 source
>> operands, a target operand, and a mask register. Stores do not need a
>> target operand. FCU ops are non-masked so do not need a mask register
>> or target operand input.
>
>> Not planning to implement the vector register file as it would be
>> immense.
Freelist:

I just used the find-first/last-one’s trick on a bit-list to pick a PR
for an AR. It can provide PRs for two ARs per cycle. I have all the PRs
from the ROB feeding into the list manager so that on a branch miss the
PRs can be freed up. (Just the portion of the PRs associated with the
miss are freed). Three discarded PRs from commit also feed into the list
manager so they can be freed. It seems like a lot of logic translating
the PR to a bit. It seems a bit impractical to me to feed all the PRs
from the ROB to the list manager. It can be done with the smallish 16
entry ROB, but for a larger ROB the free may have to be split up or
another means found.

RAT:

A register alias table is being used to track the mappings of ARs to
PRs. It uses two maps; speculative and committed. On instruction enqueue
speculative mappings are updated. On commit committed mappings are
updated, and on pipeline flush commit is copied to speculative.

Register file:

I’ve reduced the number of read ports, by not supporting the vector
stuff. There are only 18 read ports. Six groups of three.

ROB:
The ROB acts like a CAM to store both the aRN and pRN for the target
register. The aRN is needed to know which previous pRN to free on
commit. For source operands only the pRN is stored.

On 2023-11-18 2:41 p.m., Robert Finch wrote:
Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
64kB page can handle 512MB of mappings. Tonight’s trade-off is how many
root pointers to support. With a 12-bit ASID, 4096 root pointers are
required to link to the mapping tables with one root pointer for each
address space. A 512 MB space is probably sufficient for a large number
of apps. Meaning access for a TLB update is via a single root pointer
lookup and then looking up the translation from a single memory page.
Not much for the table walker to do. The 4096 root pointers use two
block RAMs and require an 8192-byte address space for update assuming a
32-bit physical address space (a 16-bit root page number).

An IO mapped area of 64kB is available for root pointer memory. 16 block
RAMs could be setup in this area, that would allow 8 root pointers for
each address space. Three bits of the virtual address space could then
be mapped using root pointers. If the root pointer just points to a
single level of page tables, then a 4GB (32-bit) space could be mapped.
I am mulling over whether it is worth it to support the additional root
pointers. It is a chunk of block RAM memory that might be better spent
elsewhere.

If I use an 11-bit ASID, all the root pointers could be present in a
single block RAM. So, design choices are 11 or 12-bits ASID, 1 or 8 root
pointers per address space.

My thought is to have only a single root pointer per space, and organize
the root pointer table as if there were 32-bits for the pointer. This
would allow a 48-bit physical address space to place the mapping tables
in. The RAM could be mapped so that the high order bits of the pointer
are assumed to be zero. The system could get by using a single block RAM
if the mapping tables location were restricted to a 16MB address range.
Eight-bit pointers could be used then.

Given that it is a small system, with only 512MB of DRAM, I think it
best to keep the page-table-walker simple, and use the minimum amount of
BRAM (1).

Robert Finch wrote:

> On 2023-11-18 2:41 p.m., Robert Finch wrote:
> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how many
> root pointers to support. With a 12-bit ASID, 4096 root pointers are
> required to link to the mapping tables with one root pointer for each
> address space.

So, you associate a single ROOT pointer VALUE with an ASID, and manage
in SW who gets that ROOT pointer VALUE; using ASID as an index into
Virtual Address Spaces.

How is this usefully different that only using the ASID to qualify TLB
results ?? <Was this TLB entry installed from the same ASID as is accessing
right now>. And using ASID as an index into any array might lead to some
conundrum down the road a apiece.

Secondarily, SUN started out with 12-bit ASID and went to 16-bits just about
as fast as they could--even before main memories went bigger than 4GB.

On 2023-11-24 8:00 p.m., MitchAlsup wrote:
> Robert Finch wrote:
>
>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>> are required to link to the mapping tables with one root pointer for
>> each address space.
>
> So, you associate a single ROOT pointer VALUE with an ASID, and manage
> in SW who gets that ROOT pointer VALUE; using ASID as an index into
> Virtual Address Spaces.
>
> How is this usefully different that only using the ASID to qualify TLB
> results ?? <Was this TLB entry installed from the same ASID as is accessing
> right now>. And using ASID as an index into any array might lead to some
> conundrum down the road a apiece.
>
> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
> about
> as fast as they could--even before main memories went bigger than 4GB.

I view the address space as an entity in it own right to be managed by
the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
identifies the address space has a life outside of just the TLB. I may
be increasing the typical scope of an ASID.

It is the same idea as using the ASID to qualify TLB entries, except
that it qualifies the root pointer as well. So, the root pointer does
not need to be switched by software. Once the root pointer is set for
the AS it simply sits there statically until the AS is reused.

I am using the ASID like a process ID. So, the root pointer register
does not need to be reset on a task switch. Address spaces may not be
mapped 1:1 with processes. An address space may outlive a task if it is
shared with another task. So, I do not want to use the PID to
distinguish tables. This assumes the address space will not be freed up
and reused by another task, if there are tasks using the ASID.

4096 address spaces is a lot. But if using a 16-bit ASID it would no
longer be practical to store a root pointer per ASID in a table.
Instead, the root pointer would have to be managed by software as is
normally done.

I am wondering why the 16-bit ASID? 256 address spaces in 256 process? I
suspect it is just because 16-bit is easier to pass around/calculate in
a HLL than some other value like 14-bits. Are 65536 address spaces
really needed?

On 11/24/2023 8:28 PM, Robert Finch wrote:
> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>> Robert Finch wrote:
>>
>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A
>>> single 64kB page can handle 512MB of mappings. Tonight’s trade-off is
>>> how many root pointers to support. With a 12-bit ASID, 4096 root
>>> pointers are required to link to the mapping tables with one root
>>> pointer for each address space.
>>
>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>> Virtual Address Spaces.
>>
>> How is this usefully different that only using the ASID to qualify TLB
>> results ?? <Was this TLB entry installed from the same ASID as is
>> accessing
>> right now>. And using ASID as an index into any array might lead to some
>> conundrum down the road a apiece.
>>
>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>> about
>> as fast as they could--even before main memories went bigger than 4GB.
>
> I view the address space as an entity in it own right to be managed by
> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
> identifies the address space has a life outside of just the TLB. I may
> be increasing the typical scope of an ASID.
>
> It is the same idea as using the ASID to qualify TLB entries, except
> that it qualifies the root pointer as well. So, the root pointer does
> not need to be switched by software. Once the root pointer is set for
> the AS it simply sits there statically until the AS is reused.
>
> I am using the ASID like a process ID. So, the root pointer register
> does not need to be reset on a task switch. Address spaces may not be
> mapped 1:1 with processes. An address space may outlive a task if it is
> shared with another task. So, I do not want to use the PID to
> distinguish tables. This assumes the address space will not be freed up
> and reused by another task, if there are tasks using the ASID.
>
> 4096 address spaces is a lot. But if using a 16-bit ASID it would no
> longer be practical to store a root pointer per ASID in a table.
> Instead, the root pointer would have to be managed by software as is
> normally done.
>
> I am wondering why the 16-bit ASID? 256 address spaces in 256 process? I
> suspect it is just because 16-bit is easier to pass around/calculate in
> a HLL than some other value like 14-bits. Are 65536 address spaces
> really needed?
>

If one assumes one address space per PID, then one is going to hit a
limit of 4K a lot faster than 64K, and when one hits the limit, there is
no good way to "reclaim" previously used address spaces short of
flushing the TLB to be sure that no entries from that space remain in
the TLB (ASID thrashing is likely to be relatively expensive to deal
with as a result).

Well, along with other things, like if/how to allow "Global" pages:
True global pages are likely a foot gun, as there is no way to exclude
them from a given process (where there may be a need to do so);
Disallowing global pages entirely means higher TLB miss rates because no
processes can share TLB entries.

One option seems to be, say, that a few of the high-order bits of the
ASID could be used as a "page group", with global pages only applying
within a single page-group (possibly with one of the page groups being
designated as "No global pages allowed").

....

On 2023-11-24 10:16 p.m., BGB wrote:
> On 11/24/2023 8:28 PM, Robert Finch wrote:
>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>>> Robert Finch wrote:
>>>
>>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A
>>>> single 64kB page can handle 512MB of mappings. Tonight’s trade-off
>>>> is how many root pointers to support. With a 12-bit ASID, 4096 root
>>>> pointers are required to link to the mapping tables with one root
>>>> pointer for each address space.
>>>
>>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>>> Virtual Address Spaces.
>>>
>>> How is this usefully different that only using the ASID to qualify TLB
>>> results ?? <Was this TLB entry installed from the same ASID as is
>>> accessing
>>> right now>. And using ASID as an index into any array might lead to some
>>> conundrum down the road a apiece.
>>>
>>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits
>>> just about
>>> as fast as they could--even before main memories went bigger than 4GB.
>>
>> I view the address space as an entity in it own right to be managed by
>> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
>> identifies the address space has a life outside of just the TLB. I may
>> be increasing the typical scope of an ASID.
>>
>> It is the same idea as using the ASID to qualify TLB entries, except
>> that it qualifies the root pointer as well. So, the root pointer does
>> not need to be switched by software. Once the root pointer is set for
>> the AS it simply sits there statically until the AS is reused.
>>
>> I am using the ASID like a process ID. So, the root pointer register
>> does not need to be reset on a task switch. Address spaces may not be
>> mapped 1:1 with processes. An address space may outlive a task if it
>> is shared with another task. So, I do not want to use the PID to
>> distinguish tables. This assumes the address space will not be freed
>> up and reused by another task, if there are tasks using the ASID.
>>
>> 4096 address spaces is a lot. But if using a 16-bit ASID it would no
>> longer be practical to store a root pointer per ASID in a table.
>> Instead, the root pointer would have to be managed by software as is
>> normally done.
>>
>> I am wondering why the 16-bit ASID? 256 address spaces in 256 process?
>> I suspect it is just because 16-bit is easier to pass around/calculate
>> in a HLL than some other value like 14-bits. Are 65536 address spaces
>> really needed?
>>
>
> If one assumes one address space per PID, then one is going to hit a
> limit of 4K a lot faster than 64K, and when one hits the limit, there is
> no good way to "reclaim" previously used address spaces short of
> flushing the TLB to be sure that no entries from that space remain in
> the TLB (ASID thrashing is likely to be relatively expensive to deal
> with as a result).
>
I see after reading several webpages that the root pointer is used to
point to only a single table for a process. This is not how I was doing
things. I have a MMU tables for each address space as opposed to having
a table for the process. The process may have only a single address
space, or it may use several address spaces.

I am wondering why there is only a single table per process.
>
>
> Well, along with other things, like if/how to allow "Global" pages:
> True global pages are likely a foot gun, as there is no way to exclude
> them from a given process (where there may be a need to do so);
> Disallowing global pages entirely means higher TLB miss rates because no
> processes can share TLB entries.
>
Global space can be assigned by designating an address space as a global
space and giving it an ASID. All process wanting access to the global
space need only then use the MMU table for that ASID. Eg. use ASID 0 for
the global address space.

> One option seems to be, say, that a few of the high-order bits of the
> ASID could be used as a "page group", with global pages only applying
> within a single page-group (possibly with one of the page groups being
> designated as "No global pages allowed").
>
> ...
>

mitchalsup@aol.com (MitchAlsup) writes:
>Robert Finch wrote:
>
>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how many
>> root pointers to support. With a 12-bit ASID, 4096 root pointers are
>> required to link to the mapping tables with one root pointer for each
>> address space.
>
>So, you associate a single ROOT pointer VALUE with an ASID, and manage
>in SW who gets that ROOT pointer VALUE; using ASID as an index into
>Virtual Address Spaces.
>
>How is this usefully different that only using the ASID to qualify TLB
>results ?? <Was this TLB entry installed from the same ASID as is accessing
>right now>. And using ASID as an index into any array might lead to some
>conundrum down the road a apiece.
>
>Secondarily, SUN started out with 12-bit ASID and went to 16-bits just about
>as fast as they could--even before main memories went bigger than 4GB.

Yeah, armv8 was originally 8-bit, and added 16 even before the spec was dry.

I don't see a benefit to tying the ASID (or VMID for that matter) to
the root of the page table. Especially with the common split
address spaces (ARMv8 has a root pointer for each half of the VA space,
for example, where the upper half is shared by all schedulable entities).

Robert Finch <robfi680@gmail.com> writes:
>On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>> Robert Finch wrote:
>>
>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>>> are required to link to the mapping tables with one root pointer for
>>> each address space.
>>
>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>> Virtual Address Spaces.
>>
>> How is this usefully different that only using the ASID to qualify TLB
>> results ?? <Was this TLB entry installed from the same ASID as is accessing
>> right now>. And using ASID as an index into any array might lead to some
>> conundrum down the road a apiece.
>>
>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>> about
>> as fast as they could--even before main memories went bigger than 4GB.
>
>I view the address space as an entity in it own right to be managed by
>the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
>identifies the address space has a life outside of just the TLB. I may
>be increasing the typical scope of an ASID.
>
>It is the same idea as using the ASID to qualify TLB entries, except
>that it qualifies the root pointer as well. So, the root pointer does
>not need to be switched by software. Once the root pointer is set for
>the AS it simply sits there statically until the AS is reused.
>
>I am using the ASID like a process ID. So, the root pointer register
>does not need to be reset on a task switch. Address spaces may not be
>mapped 1:1 with processes. An address space may outlive a task if it is
>shared with another task. So, I do not want to use the PID to
>distinguish tables. This assumes the address space will not be freed up
>and reused by another task, if there are tasks using the ASID.
>
>4096 address spaces is a lot. But if using a 16-bit ASID it would no
>longer be practical to store a root pointer per ASID in a table.
>Instead, the root pointer would have to be managed by software as is
>normally done.
>
>I am wondering why the 16-bit ASID? 256 address spaces in 256 process? I
>suspect it is just because 16-bit is easier to pass around/calculate in
>a HLL than some other value like 14-bits. Are 65536 address spaces
>really needed?
>

256 is far too small.

$ ps -ef | wc -l
709

Every time the ASID overflows, the system must basically flush
all the caches system-wide. On an 80 processor system, that's a lot of
overhead.

Robert Finch <robfi680@gmail.com> writes:
>On 2023-11-24 10:16 p.m., BGB wrote:
>> On 11/24/2023 8:28 PM, Robert Finch wrote:
>>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:

>>
>> If one assumes one address space per PID, then one is going to hit a
>> limit of 4K a lot faster than 64K, and when one hits the limit, there is
>> no good way to "reclaim" previously used address spaces short of
>> flushing the TLB to be sure that no entries from that space remain in
>> the TLB (ASID thrashing is likely to be relatively expensive to deal
>> with as a result).
>>
>I see after reading several webpages that the root pointer is used to
>point to only a single table for a process. This is not how I was doing
>things. I have a MMU tables for each address space as opposed to having
>a table for the process. The process may have only a single address
>space, or it may use several address spaces.
>
>I am wondering why there is only a single table per process.

There is actually two in most operating systems - the lower half
of the VA space is owned by the user-mode code in the process and
the upper-half is shared by all processors and used by the
operating system on behalf of the process. For Intel/AMD, the
kernel manages both halves, for ARMv8, each half has a completely
distinct and separate root pointer (at each exeception level).

On 11/24/2023 9:48 PM, Robert Finch wrote:
> On 2023-11-24 10:16 p.m., BGB wrote:
>> On 11/24/2023 8:28 PM, Robert Finch wrote:
>>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>>>> Robert Finch wrote:
>>>>
>>>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A
>>>>> single 64kB page can handle 512MB of mappings. Tonight’s trade-off
>>>>> is how many root pointers to support. With a 12-bit ASID, 4096 root
>>>>> pointers are required to link to the mapping tables with one root
>>>>> pointer for each address space.
>>>>
>>>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>>>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>>>> Virtual Address Spaces.
>>>>
>>>> How is this usefully different that only using the ASID to qualify TLB
>>>> results ?? <Was this TLB entry installed from the same ASID as is
>>>> accessing
>>>> right now>. And using ASID as an index into any array might lead to
>>>> some
>>>> conundrum down the road a apiece.
>>>>
>>>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits
>>>> just about
>>>> as fast as they could--even before main memories went bigger than 4GB.
>>>
>>> I view the address space as an entity in it own right to be managed
>>> by the MMU. ASIDs and address spaces should be mapped 1:1. The ASID
>>> that identifies the address space has a life outside of just the TLB.
>>> I may be increasing the typical scope of an ASID.
>>>
>>> It is the same idea as using the ASID to qualify TLB entries, except
>>> that it qualifies the root pointer as well. So, the root pointer does
>>> not need to be switched by software. Once the root pointer is set for
>>> the AS it simply sits there statically until the AS is reused.
>>>
>>> I am using the ASID like a process ID. So, the root pointer register
>>> does not need to be reset on a task switch. Address spaces may not be
>>> mapped 1:1 with processes. An address space may outlive a task if it
>>> is shared with another task. So, I do not want to use the PID to
>>> distinguish tables. This assumes the address space will not be freed
>>> up and reused by another task, if there are tasks using the ASID.
>>>
>>> 4096 address spaces is a lot. But if using a 16-bit ASID it would no
>>> longer be practical to store a root pointer per ASID in a table.
>>> Instead, the root pointer would have to be managed by software as is
>>> normally done.
>>>
>>> I am wondering why the 16-bit ASID? 256 address spaces in 256
>>> process? I suspect it is just because 16-bit is easier to pass
>>> around/calculate in a HLL than some other value like 14-bits. Are
>>> 65536 address spaces really needed?
>>>
>>
>> If one assumes one address space per PID, then one is going to hit a
>> limit of 4K a lot faster than 64K, and when one hits the limit, there
>> is no good way to "reclaim" previously used address spaces short of
>> flushing the TLB to be sure that no entries from that space remain in
>> the TLB (ASID thrashing is likely to be relatively expensive to deal
>> with as a result).
>>
> I see after reading several webpages that the root pointer is used to
> point to only a single table for a process. This is not how I was doing
> things. I have a MMU tables for each address space as opposed to having
> a table for the process. The process may have only a single address
> space, or it may use several address spaces.
>
> I am wondering why there is only a single table per process.

I went the opposite route of one big address space, with the idea of
allowing memory protection within this address space via the VUGID/ACL
mechanism. There is a KRR, or Keyring Register, which holds up to 4 keys
that may be used for ACL checking, granting an access if it is allowed
by at least one of the keys; triggering an ISR on miss similar to the
TLB. In this case, the conceptual model is more similar to that
typically used in filesystems.

But, I also have a 16-bit ASID...

As-is, there is at most one set of page tables per address space, or
per-process if processes are given different address spaces.

>>
>>
>> Well, along with other things, like if/how to allow "Global" pages:
>> True global pages are likely a foot gun, as there is no way to exclude
>> them from a given process (where there may be a need to do so);
>> Disallowing global pages entirely means higher TLB miss rates because
>> no processes can share TLB entries.
>>
> Global space can be assigned by designating an address space as a global
> space and giving it an ASID. All process wanting access to the global
> space need only then use the MMU table for that ASID. Eg. use ASID 0 for
> the global address space.
>

Had considered this, but there is a problem:
What if you have a process that you *don't* want to be able to see into
this global space?...

Though, this is where the idea of page-grouping can come in, say, the
ASID becomes:
gggg-pppp-pppp-pppp

Where:
0000 is visible to all of 0zzz
1000 is visible to all of 1zzz
...
Except:
Fzzz, this group does not have any global pages (all one-off ASIDs).

Or, possible also, is a 2.14 bit split.

>> One option seems to be, say, that a few of the high-order bits of the
>> ASID could be used as a "page group", with global pages only applying
>> within a single page-group (possibly with one of the page groups being
>> designated as "No global pages allowed").
>>
>> ...
>>
>

Meanwhile:
I went and bought 128GB of RAM, only to realize my PC doesn't work if
one tries to install the full 128GB (the BIOS boot-loops a bunch of
times, and then apparently concludes that there is only 3.5GB ...).

Does work at least if I install 3x 32GB sticks and 1x 16GB stick, giving
112GB. This breaks the pairing rules, but seems to be working.

....

Had I known this, could have spent half as much, and only upgraded to 96GB.

Seemingly MOBO/BIOS/... designers didn't anticipate someone sticking a
full 128GB in this thing?... (BIOS is dated from 2018).

Well, either this, or a hardware compatibility issue with one of the
cards?...

Robert Finch wrote:

> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>> Robert Finch wrote:
>>
>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>>> are required to link to the mapping tables with one root pointer for
>>> each address space.
>>
>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>> Virtual Address Spaces.
>>
>> How is this usefully different that only using the ASID to qualify TLB
>> results ?? <Was this TLB entry installed from the same ASID as is accessing
>> right now>. And using ASID as an index into any array might lead to some
>> conundrum down the road a apiece.
>>
>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>> about
>> as fast as they could--even before main memories went bigger than 4GB.

> I view the address space as an entity in it own right to be managed by
> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
> identifies the address space has a life outside of just the TLB. I may
> be increasing the typical scope of an ASID.

Consider the case where two different processes MMAP the same area
of memory.
Should they both end up using the same ASID ??
Should they both take extra TLB walks because they use different ASIDs ??
Should they uses their own ASIDs for their own memory but a different ASID
for the shared memory ?? And How do you expect this to happen ??

> It is the same idea as using the ASID to qualify TLB entries, except
> that it qualifies the root pointer as well. So, the root pointer does
> not need to be switched by software. Once the root pointer is set for
> the AS it simply sits there statically until the AS is reused.

> I am using the ASID like a process ID. So, the root pointer register
> does not need to be reset on a task switch. Address spaces may not be
> mapped 1:1 with processes. An address space may outlive a task if it is
> shared with another task. So, I do not want to use the PID to
> distinguish tables. This assumes the address space will not be freed up
> and reused by another task, if there are tasks using the ASID.

> 4096 address spaces is a lot. But if using a 16-bit ASID it would no
> longer be practical to store a root pointer per ASID in a table.
> Instead, the root pointer would have to be managed by software as is
> normally done.

> I am wondering why the 16-bit ASID? 256 address spaces in 256 process? I
> suspect it is just because 16-bit is easier to pass around/calculate in
> a HLL than some other value like 14-bits. Are 65536 address spaces
> really needed?

Scott Lurndal wrote:

> Robert Finch <robfi680@gmail.com> writes:
>>On 2023-11-24 10:16 p.m., BGB wrote:
>>> On 11/24/2023 8:28 PM, Robert Finch wrote:
>>>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:

>>>
>>> If one assumes one address space per PID, then one is going to hit a
>>> limit of 4K a lot faster than 64K, and when one hits the limit, there is
>>> no good way to "reclaim" previously used address spaces short of
>>> flushing the TLB to be sure that no entries from that space remain in
>>> the TLB (ASID thrashing is likely to be relatively expensive to deal
>>> with as a result).
>>>
>>I see after reading several webpages that the root pointer is used to
>>point to only a single table for a process. This is not how I was doing
>>things. I have a MMU tables for each address space as opposed to having
>>a table for the process. The process may have only a single address
>>space, or it may use several address spaces.
>>
>>I am wondering why there is only a single table per process.

> There is actually two in most operating systems - the lower half
> of the VA space is owned by the user-mode code in the process and
> the upper-half is shared by all processors and used by the
> operating system on behalf of the process. For Intel/AMD, the
> kernel manages both halves, for ARMv8, each half has a completely
> distinct and separate root pointer (at each exeception level).

My 66000 Architecture has 4 Root Pointers available at all instants
of time. The above was designed before the rise of HyperVisors and is
now showing its age problems. All 4 Root Pointers are used based on
privilege level::

HOB=0 HOB=1
Application:: Application 2-level No Access
Guest OS :: Application 2-level Guest OS 2-level
Guest HV :: Guest HV 1-level Guest OS 2-level
Real HV :: Guest HV 1-level Real HV 1-level

The overhead of Application to Application is no higher than that
of Guest OS to a different Guest OS--whereas on the machines with
VMENTER and VMEXIT it takes 10,000 cycles whereas Application to
Application is closer to 1,000 cycles. I want this down in the
10-100 cycle range.

The exception <stack> system is designed to allow Guest HV to
recover a Guest OS that takes page faults while servicing ISRs
(and the like).

The interrupt <stack> system is designed to allow the ISR to
RPC or softIRQ without having to look at the pending stack on
the way out. RTI looks at the pending stack and services the
highest pending PRC/softIRQ affinitized to the CPU with control.

The Interrupt dispatch system allows the CPU to continue running
instructions until the contending CPUs decide which interrupt
is claimed by which CPU (1::1) and then context switch do the
interrupt dispatcher.

mitchalsup@aol.com (MitchAlsup) writes:
>Robert Finch wrote:
>
>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>>> Robert Finch wrote:
>>>
>>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>>>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>>>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>>>> are required to link to the mapping tables with one root pointer for
>>>> each address space.
>>>
>>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>>> Virtual Address Spaces.
>>>
>>> How is this usefully different that only using the ASID to qualify TLB
>>> results ?? <Was this TLB entry installed from the same ASID as is accessing
>>> right now>. And using ASID as an index into any array might lead to some
>>> conundrum down the road a apiece.
>>>
>>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>>> about
>>> as fast as they could--even before main memories went bigger than 4GB.
>
>> I view the address space as an entity in it own right to be managed by
>> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
>> identifies the address space has a life outside of just the TLB. I may
>> be increasing the typical scope of an ASID.
>
>Consider the case where two different processes MMAP the same area
>of memory.

In which case, the area of memory would be mapped to different
virtual address ranges in each process, and thus naturally
consume two TLBs.

FWIW, MAP_FIXED is specified as an optional feature by POSIX
and may not be supported by the OS at all.

Given various forms of ASLR being used, it's unlikely even in
two instances of the same executable that a call to mmap
with MAP_SHARED without MAP_FIXED would map the region at
the same virtual address in both processes.

>Should they both end up using the same ASID ??

They couldn't share an ASID assuming the TLB looks up by VA.

>Should they both take extra TLB walks because they use different ASIDs ??

Given the above, yes. It's likely they'll each be scheduled
on different cores anyway in any modern system.

Scott Lurndal wrote:

> mitchalsup@aol.com (MitchAlsup) writes:
>>Robert Finch wrote:
>>
>>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>>>> Robert Finch wrote:
>>>>
>>>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>>>>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>>>>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>>>>> are required to link to the mapping tables with one root pointer for
>>>>> each address space.
>>>>
>>>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>>>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>>>> Virtual Address Spaces.
>>>>
>>>> How is this usefully different that only using the ASID to qualify TLB
>>>> results ?? <Was this TLB entry installed from the same ASID as is accessing
>>>> right now>. And using ASID as an index into any array might lead to some
>>>> conundrum down the road a apiece.
>>>>
>>>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>>>> about
>>>> as fast as they could--even before main memories went bigger than 4GB.
>>
>>> I view the address space as an entity in it own right to be managed by
>>> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
>>> identifies the address space has a life outside of just the TLB. I may
>>> be increasing the typical scope of an ASID.
>>
>>Consider the case where two different processes MMAP the same area
>>of memory.

> In which case, the area of memory would be mapped to different
> virtual address ranges in each process, and thus naturally
> consume two TLBs.

MMAP() first, fork() second. Now we have 2 processes with the
memory mapped shared memory at the same address.

> FWIW, MAP_FIXED is specified as an optional feature by POSIX
> and may not be supported by the OS at all.

> Given various forms of ASLR being used, it's unlikely even in
> two instances of the same executable that a call to mmap
> with MAP_SHARED without MAP_FIXED would map the region at
> the same virtual address in both processes.

>>Should they both end up using the same ASID ??

> They couldn't share an ASID assuming the TLB looks up by VA.

>>Should they both take extra TLB walks because they use different ASIDs ??

> Given the above, yes. It's likely they'll each be scheduled
> on different cores anyway in any modern system.

mitchalsup@aol.com (MitchAlsup) writes:
>Scott Lurndal wrote:
>
>> mitchalsup@aol.com (MitchAlsup) writes:
>>>Robert Finch wrote:
>>>
>>>> On 2023-11-24 8:00 p.m., MitchAlsup wrote:
>>>>> Robert Finch wrote:
>>>>>
>>>>>> On 2023-11-18 2:41 p.m., Robert Finch wrote:
>>>>>> Q+ uses 64kB memory pages containing 8192 PTEs to map memory. A single
>>>>>> 64kB page can handle 512MB of mappings. Tonight’s trade-off is how
>>>>>> many root pointers to support. With a 12-bit ASID, 4096 root pointers
>>>>>> are required to link to the mapping tables with one root pointer for
>>>>>> each address space.
>>>>>
>>>>> So, you associate a single ROOT pointer VALUE with an ASID, and manage
>>>>> in SW who gets that ROOT pointer VALUE; using ASID as an index into
>>>>> Virtual Address Spaces.
>>>>>
>>>>> How is this usefully different that only using the ASID to qualify TLB
>>>>> results ?? <Was this TLB entry installed from the same ASID as is accessing
>>>>> right now>. And using ASID as an index into any array might lead to some
>>>>> conundrum down the road a apiece.
>>>>>
>>>>> Secondarily, SUN started out with 12-bit ASID and went to 16-bits just
>>>>> about
>>>>> as fast as they could--even before main memories went bigger than 4GB.
>>>
>>>> I view the address space as an entity in it own right to be managed by
>>>> the MMU. ASIDs and address spaces should be mapped 1:1. The ASID that
>>>> identifies the address space has a life outside of just the TLB. I may
>>>> be increasing the typical scope of an ASID.
>>>
>>>Consider the case where two different processes MMAP the same area
>>>of memory.
>
>> In which case, the area of memory would be mapped to different
>> virtual address ranges in each process, and thus naturally
>> consume two TLBs.
>
>MMAP() first, fork() second. Now we have 2 processes with the
>memory mapped shared memory at the same address.

Yes, in that case, they'll be mapped at the same VA. All
the below points still apply so long as TLB's are per core.

>
>> FWIW, MAP_FIXED is specified as an optional feature by POSIX
>> and may not be supported by the OS at all.
>
>> Given various forms of ASLR being used, it's unlikely even in
>> two instances of the same executable that a call to mmap
>> with MAP_SHARED without MAP_FIXED would map the region at
>> the same virtual address in both processes.
>
>>>Should they both end up using the same ASID ??
>
>> They couldn't share an ASID assuming the TLB looks up by VA.
>
>>>Should they both take extra TLB walks because they use different ASIDs ??
>
>> Given the above, yes. It's likely they'll each be scheduled
>> on different cores anyway in any modern system.

Are top-level page directory pages shared between tasks? Suppose a task
needs a 32-bit address space. With one level of page maps, 27 bits is
accommodated, that leaves 5 bits of address translation to be done by
the page directory. Using a whole page which can handle 11 address bits
would be wasteful. But if root pointers could point into the same page
directory page then the space would not be wasted. For instance, root
pointer for task #1 could point the first 32 entries, root pointer for
task #2 could point into the next 32 entries, and so on.