In article <fbed57b4-1553-4b63-b39e-c130754b3aa8n@googlegroups.com>,
jsavard@ecn.ab.ca (Quadibloc) wrote:

> hardware support for packed decimal
> hardware support for IBM System/360 hexadecimal floating point
>
> because people do run Hercules on their computers and so on.

I read the Hercules mailing list. Nobody on there uses it for serious
work. It seems to be mainly used to evoke memories of youth. Running it
on low-powered hardware, such as Raspberry Pi, attracts more notice than
running it on something powerful. Hercules is written in portable C,
because portability is considered more important than performance.

> I have now added, at the bottom of the page, a scheme, involving
> having dual-channel memory where each channel is 192 bits wide,
> that permits the operating system to allocate blocks of 384-bit
> wide memory, 288-bit wide memory, 240-bit wide memory, and 256-bit
> wide memory.

That's an interesting new way to have your system run short of the right
kind of memory.

John

On Sunday, September 24, 2023 at 11:59:41 AM UTC-6, Quadibloc wrote:

> On the page
>
> http://www.quadibloc.com/arch/per14.htm

Upon review of the contents of the two pages involved, I have now
moved the new material from that page to the bottom of this
page instead:

http://www.quadibloc.com/arch/per04.htm

John Savard

jgd@cix.co.uk (John Dallman) writes:
>In article <fbed57b4-1553-4b63-b39e-c130754b3aa8n@googlegroups.com>,
>jsavard@ecn.ab.ca (Quadibloc) wrote:
>
>> hardware support for packed decimal
>> hardware support for IBM System/360 hexadecimal floating point
>>
>> because people do run Hercules on their computers and so on.
>
>I read the Hercules mailing list. Nobody on there uses it for serious
>work. It seems to be mainly used to evoke memories of youth. Running it
>on low-powered hardware, such as Raspberry Pi, attracts more notice than
>running it on something powerful. Hercules is written in portable C,
>because portability is considered more important than performance.
>
>> I have now added, at the bottom of the page, a scheme, involving
>> having dual-channel memory where each channel is 192 bits wide,
>> that permits the operating system to allocate blocks of 384-bit
>> wide memory, 288-bit wide memory, 240-bit wide memory, and 256-bit
>> wide memory.
>
>That's an interesting new way to have your system run short of the right
>kind of memory.

Indeed. It's not the path from memory to the core complex that is
currently most interesting (although 256-bit wide (and higher) mesh or crossbars
aren't uncommon), but rather the data path widths from I/O
subsystems. 512-bit wide paths from network controllers and on-board
non-coherent (or coherent, see CXL) coprocessors has become
common. Supporting 80gbit/sec of network traffic into memory
or the networking subsystem isn't trivial.

The memory bandwidth grows by adding controllers and striping across them
for the most part.

I think intermediate-sized FPs are unlikely to be worth the effort other
than in very niche places where the few percents gained can be justified
by a very large volume.

In memory management libraries (malloc, GCs, ...) we do try and minimize
the amount of memory that's "wasted" of course, but we generally
consider that having an upper bound of ~100% overhead is acceptable
(i.e. using twice as much memory as actually needed in the worst case).

Stefan

On 9/25/2023 10:41 AM, Scott Lurndal wrote:
> jgd@cix.co.uk (John Dallman) writes:
>> In article <fbed57b4-1553-4b63-b39e-c130754b3aa8n@googlegroups.com>,
>> jsavard@ecn.ab.ca (Quadibloc) wrote:
>>
>>> hardware support for packed decimal
>>> hardware support for IBM System/360 hexadecimal floating point
>>>
>>> because people do run Hercules on their computers and so on.
>>
>> I read the Hercules mailing list. Nobody on there uses it for serious
>> work. It seems to be mainly used to evoke memories of youth. Running it
>> on low-powered hardware, such as Raspberry Pi, attracts more notice than
>> running it on something powerful. Hercules is written in portable C,
>> because portability is considered more important than performance.
>>
>>> I have now added, at the bottom of the page, a scheme, involving
>>> having dual-channel memory where each channel is 192 bits wide,
>>> that permits the operating system to allocate blocks of 384-bit
>>> wide memory, 288-bit wide memory, 240-bit wide memory, and 256-bit
>>> wide memory.
>>
>> That's an interesting new way to have your system run short of the right
>> kind of memory.
>
> Indeed. It's not the path from memory to the core complex that is
> currently most interesting (although 256-bit wide (and higher) mesh or crossbars
> aren't uncommon), but rather the data path widths from I/O
> subsystems. 512-bit wide paths from network controllers and on-board
> non-coherent (or coherent, see CXL) coprocessors has become
> common. Supporting 80gbit/sec of network traffic into memory
> or the networking subsystem isn't trivial.
>
> The memory bandwidth grows by adding controllers and striping across them
> for the most part.

?...

AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
have 4 DIMM slots with DIMMs being filled in pairs.

This would seemingly imply that RAM would be mostly limited to a 128-bit
datapath (or 64-bit in unganged mode).

Similarly, PCIe slots are effectively multiple serial lanes running in
parallel, etc...

Typical onboard peripherals being connected either with PCIe lanes or
via an onboard LPC bus.

Similarly, a typical motherboard only has a single CPU socket.

....

Outside of the CPU itself, unclear where any of these wide interconnects
would be, or where they would be going.

Similarly, most things much smaller than a PC, will have 16-bit or
32-bit RAM interfaces (with typically 1 or 2 RAM chips soldered onto the
motherboard somewhere, often also with an eMMC Flash chip and other things).

Granted, on the other hand, I have some ~ 18 year old Xeon based rack
servers, which have a pair of CPUs surrounded by a small island of RAM
modules, and comparably still rather impressive multi-core "memcpy()"
performance.

Say, while each core individually is still limited to a few GB/sec, one
can have all of the cores running memcpy at the same time without any
significant drop (vs, say, on a PC where once the total exceeds around 8
to 10GB/sec or so, per-thread performance drops off).

Well, and both still significantly beat out the "memcpy()" performance
on a 20 year old laptop (as does a RasPi...).

....

BGB <cr88192@gmail.com> writes:
>On 9/25/2023 10:41 AM, Scott Lurndal wrote:
>> jgd@cix.co.uk (John Dallman) writes:

>>> That's an interesting new way to have your system run short of the right
>>> kind of memory.
>>
>> Indeed. It's not the path from memory to the core complex that is
>> currently most interesting (although 256-bit wide (and higher) mesh or crossbars
>> aren't uncommon), but rather the data path widths from I/O
>> subsystems. 512-bit wide paths from network controllers and on-board
>> non-coherent (or coherent, see CXL) coprocessors has become
>> common. Supporting 80gbit/sec of network traffic into memory
>> or the networking subsystem isn't trivial.
>>
>> The memory bandwidth grows by adding controllers and striping across them
>> for the most part.
>
>?...
>
>
>AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
>have 4 DIMM slots with DIMMs being filled in pairs.

"typical" in what context? Home desktops? That's certainly not
typical for the data center or cloud servers. One chip I'm aware of
has 20 dual-channel DDR5 memory controllers (one per every four
cores).

>
>This would seemingly imply that RAM would be mostly limited to a 128-bit
>datapath (or 64-bit in unganged mode).

That's sufficient for the home desktop windows user, I suppose. It's
certainly not sufficient for cloud servers, enterprise data center servers,
high-end networking appliances, et alia.

>
>Similarly, PCIe slots are effectively multiple serial lanes running in
>parallel, etc...

Gen 6 x16 has a boatload of bandwidth (128 gigabytes per second).
Note that the serial lanes are only downstream (towards the endpoint)
from the root complex. The root complex itself uses a parallel
interconnect to the cache/memory subsystem on the host side.

>
>Typical onboard peripherals being connected either with PCIe lanes or
>via an onboard LPC bus.

There is nothing useful (or high performance) connected to
LPC bus in decades. Even intel is deprecating it in modern chipsets
with a nod for backward compatability (i.e. supporting in/out
instructions to a subset of standard ISA peripherals like keyboard
controllers). They're even planning on getting rid of most of it entirely
in the future and boot the processor directly into long mode so
all the legacy compatibility stuff like the original PIC can
be removed.

>
>Similarly, a typical motherboard only has a single CPU socket.

Typical in what context?

>
>
>Outside of the CPU itself, unclear where any of these wide interconnects
>would be, or where they would be going.

Did you read the post you responded to? How do you get 40gbytes/sec
into your memory subsystem from an onboard 400gbit nic? Or 128 gbytes/sec
from a PCIe root complex? Or from a PCIe CXL-cache memory extender?

On 9/25/2023 10:06 AM, Scott Lurndal wrote:
> BGB <cr88192@gmail.com> writes:

>> AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
>> have 4 DIMM slots with DIMMs being filled in pairs.
>
> "typical" in what context? Home desktops? That's certainly not
> typical for the data center or cloud servers. One chip I'm aware of
> has 20 dual-channel DDR5 memory controllers (one per every four
> cores).

Wow! How many pins on the package? It must be massive. is there a
latency penalty for getting that many DIMMs "close" to the CPU?

--
- Stephen Fuld
(e-mail address disguised to prevent spam)

On 9/25/2023 12:06 PM, Scott Lurndal wrote:
> BGB <cr88192@gmail.com> writes:
>> On 9/25/2023 10:41 AM, Scott Lurndal wrote:
>>> jgd@cix.co.uk (John Dallman) writes:
>
>>>> That's an interesting new way to have your system run short of the right
>>>> kind of memory.
>>>
>>> Indeed. It's not the path from memory to the core complex that is
>>> currently most interesting (although 256-bit wide (and higher) mesh or crossbars
>>> aren't uncommon), but rather the data path widths from I/O
>>> subsystems. 512-bit wide paths from network controllers and on-board
>>> non-coherent (or coherent, see CXL) coprocessors has become
>>> common. Supporting 80gbit/sec of network traffic into memory
>>> or the networking subsystem isn't trivial.
>>>
>>> The memory bandwidth grows by adding controllers and striping across them
>>> for the most part.
>>
>> ?...
>>
>>
>> AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
>> have 4 DIMM slots with DIMMs being filled in pairs.
>
> "typical" in what context? Home desktops? That's certainly not
> typical for the data center or cloud servers. One chip I'm aware of
> has 20 dual-channel DDR5 memory controllers (one per every four
> cores).
>

Mostly desktop PCs.

I don't have any data center or cloud servers, or personally know anyone
who does around here, ...

So, it seems reasonable to assume, most people don't have them, thus,
they are not typical.

>>
>> This would seemingly imply that RAM would be mostly limited to a 128-bit
>> datapath (or 64-bit in unganged mode).
>
> That's sufficient for the home desktop windows user, I suppose. It's
> certainly not sufficient for cloud servers, enterprise data center servers,
> high-end networking appliances, et alia.
>

It seemed like the context was "common", which would mostly imply "stuff
people actually have", not necessarily high-end data-center servers,
which pretty much no one has apart from the companies running the data
centers.

But, then if say, a datacenter has 1 sever per 10k users (each user with
a desktop PC or similar); this would mean still only mean 0.01% of the
total computers were servers.

And, your typical desktop PC is only going to have 1 CPU and 2 or 4 RAM
sticks.

>
>>
>> Similarly, PCIe slots are effectively multiple serial lanes running in
>> parallel, etc...
>
> Gen 6 x16 has a boatload of bandwidth (128 gigabytes per second).
> Note that the serial lanes are only downstream (towards the endpoint)
> from the root complex. The root complex itself uses a parallel
> interconnect to the cache/memory subsystem on the host side.
>

OK.

>>
>> Typical onboard peripherals being connected either with PCIe lanes or
>> via an onboard LPC bus.
>
> There is nothing useful (or high performance) connected to
> LPC bus in decades. Even intel is deprecating it in modern chipsets
> with a nod for backward compatability (i.e. supporting in/out
> instructions to a subset of standard ISA peripherals like keyboard
> controllers). They're even planning on getting rid of most of it entirely
> in the future and boot the processor directly into long mode so
> all the legacy compatibility stuff like the original PIC can
> be removed.
>

There is typically still an RS232 port and PS/2 keyboard and mouse ports
and similar...

But, yeah.

>>
>> Similarly, a typical motherboard only has a single CPU socket.
>
> Typical in what context?
>

Standard home desktop PC.

That is what most people are using, at least, excluding laptops,
tablets, and cell-phones.

>>
>>
>> Outside of the CPU itself, unclear where any of these wide interconnects
>> would be, or where they would be going.
>
> Did you read the post you responded to? How do you get 40gbytes/sec
> into your memory subsystem from an onboard 400gbit nic? Or 128 gbytes/sec
> from a PCIe root complex? Or from a PCIe CXL-cache memory extender?
>

Generally you don't...

I think, if one has a 1GbE Ethernet port, they could maybe get 120 MB/s
or similar if it is going "full tilt".

On 9/25/2023 11:30 AM, Stefan Monnier wrote:
> I think intermediate-sized FPs are unlikely to be worth the effort other
> than in very niche places where the few percents gained can be justified
> by a very large volume.
>
> In memory management libraries (malloc, GCs, ...) we do try and minimize
> the amount of memory that's "wasted" of course, but we generally
> consider that having an upper bound of ~100% overhead is acceptable
> (i.e. using twice as much memory as actually needed in the worst case).
>

I am now evaluating the possible use of a 48-bit floating-point format,
but this is (merely) in terms of memory storage (in registers, it will
still use Binary64).

Still to be determined whether being able to save 2-bytes in struct or
array storage (at the cost of a few extra clock-cycles per load/store
operation; and a non-standard C type) will be a worthwhile tradeoff.

Would also need to determine the relative cost tradeoff between having
dedicated 48-bit load/store ops, and "faking it" via multi-instruction
sequences.

Similarly, while I was at it, went and added a few "proper" byte-swap
instructions (to potentially help with the efficiency of endian
conversion, at least slightly...).

As for memory management, I generally aim for around a 25% space
overhead per small/medium "malloc()".

Sometimes, this does mean needing to switch between strategies, say:
Cell-oriented allocator for small objects (say, under 256 or 384B);
Memory-block-list allocator for medium objects;
Page-based allocation for larger objects.

....

On Fri, 22 Sep 2023 08:10:27 -0700 (PDT), Quadibloc
<jsavard@ecn.ab.ca> wrote:

> ..., if you've got enough transistors to put multiple CPUs on one die,
>and you believe parallelism is far, far inferior to speeding up the clock or
>reducing latency by other means, then you might actually think that throwing
>transistors at reducing latency in this fashion is rational.
>
>Essentially, it is true that I don't buy the idea that programmers are just
>lazy, and if they did things right they could exploit parallelism effectively.
>I will grant that they can probably do much better in many cases, but I
>also feel there are fundamental limits.

Compute resources always are limited, and most programmers are far
better at figuring out what CAN be done in parallel than what SHOULD
be done in parallel given available resources.

Then there is Amdahl's law.

There always will be a need for faster single thread serial execution.

>John Savard

YMMV,
George

> I am now evaluating the possible use of a 48-bit floating-point format, but
> this is (merely) in terms of memory storage (in registers, it will still use
> Binary64).

I suspect this is indeed the only sane way to go about it.
Also, I suspect that such 48bit floats would only be worthwhile when you
have some large vectors/matrices and care about the 33% bandwidth
overhead of using 64bit rather than 48bit. So maybe the focus should be
on "load 3 chunks, then spread turn them into 4" since the limiting
factor would presumably be the memory bandwidth.

E.g. load 3 chunks (C1, C2, and C3) of 256bits each using standard SIMD
load, and then add an instruction to turn C1+C2 into two 256bit vectors
of 4x64bit floats, and another to do the same with C2+C3 (basically, the
same instruction except it uses the other half of the bits of C2).

Stefan

BGB <cr88192@gmail.com> writes:
>On 9/25/2023 12:06 PM, Scott Lurndal wrote:
>> BGB <cr88192@gmail.com> writes:
>>> On 9/25/2023 10:41 AM, Scott Lurndal wrote:
>
>
>It seemed like the context was "common", which would mostly imply "stuff
>people actually have", not necessarily high-end data-center servers,
>which pretty much no one has apart from the companies running the data
>centers.
>
>But, then if say, a datacenter has 1 sever per 10k users (each user with
>a desktop PC or similar); this would mean still only mean 0.01% of the
>total computers were servers.

Last year, there were 29 million servers, and 290 million PCs shipped,
so servers are about 10%.

>>
>> Did you read the post you responded to? How do you get 40gbytes/sec
>> into your memory subsystem from an onboard 400gbit nic? Or 128 gbytes/sec
>> from a PCIe root complex? Or from a PCIe CXL-cache memory extender?
>>
>
>Generally you don't...
>
>I think, if one has a 1GbE Ethernet port, they could maybe get 120 MB/s
>or similar if it is going "full tilt".

2.5Gbit Ethernet is making it's way into desktop chipsets. It won't be
long before 10Gbit Ethernet is present in high-end PC mainboards.

On Monday, September 25, 2023 at 1:39:09 PM UTC-4, BGB wrote:
> On 9/25/2023 11:30 AM, Stefan Monnier wrote:
> > I think intermediate-sized FPs are unlikely to be worth the effort other
> > than in very niche places where the few percents gained can be justified
> > by a very large volume.
> >
> > In memory management libraries (malloc, GCs, ...) we do try and minimize
> > the amount of memory that's "wasted" of course, but we generally
> > consider that having an upper bound of ~100% overhead is acceptable
> > (i.e. using twice as much memory as actually needed in the worst case).
> >
> I am now evaluating the possible use of a 48-bit floating-point format,
> but this is (merely) in terms of memory storage (in registers, it will
> still use Binary64).
>
> Still to be determined whether being able to save 2-bytes in struct or
> array storage (at the cost of a few extra clock-cycles per load/store
> operation; and a non-standard C type) will be a worthwhile tradeoff.
>
> Would also need to determine the relative cost tradeoff between having
> dedicated 48-bit load/store ops, and "faking it" via multi-instruction
> sequences.

If your CPU supports unaligned data, you could store 64-bit floats every six bytes,
instead of eight bytes, overlapping previously stored values. But you would need
to be careful the order the stores take place.

With a 48-bit load/store, scaled indexed addressing for arrays would need to
scale by six.
>
>
>
> Similarly, while I was at it, went and added a few "proper" byte-swap
> instructions (to potentially help with the efficiency of endian
> conversion, at least slightly...).
>
> As for memory management, I generally aim for around a 25% space
> overhead per small/medium "malloc()".
>
> Sometimes, this does mean needing to switch between strategies, say:
> Cell-oriented allocator for small objects (say, under 256 or 384B);
> Memory-block-list allocator for medium objects;
> Page-based allocation for larger objects.
>
> ...

On 9/25/2023 1:11 PM, Stefan Monnier wrote:
>> I am now evaluating the possible use of a 48-bit floating-point format, but
>> this is (merely) in terms of memory storage (in registers, it will still use
>> Binary64).
>
> I suspect this is indeed the only sane way to go about it.
> Also, I suspect that such 48bit floats would only be worthwhile when you
> have some large vectors/matrices and care about the 33% bandwidth
> overhead of using 64bit rather than 48bit. So maybe the focus should be
> on "load 3 chunks, then spread turn them into 4" since the limiting
> factor would presumably be the memory bandwidth.
>

Yeah, memory bandwidth tends to be one of the major limiting factors for
performance in my experience for many algorithms.

This is partly why I had some wonk like 3x Float21 vectors (with 64-bit
storage). And, part of why I do a lot of stuff using Binary16 (where, in
this case, both Binary16 and Binary32 have the same latency).

Well, and for a lot of my 3D rendering is using RGB555 framebuffers,
16-bit Z-buffer, and texture compression...

As noted in some past examples, even desktop PCs are not immune to this,
and saving some memory via bit-twiddly can often be cheaper than using a
"less memory dense" strategy (that results in a higher number of L1 misses).

Ironically, this seems to run counter to the conventional wisdom of
saving calculations via lookup tables (depending on the algorithm, the
lookup table may only "win" if it is less than 1/4 or 1/8 the size of
the L1 cache).

Many people seem to evaluate efficiency solely in terms of how many
clock-cycles it would take to evaluate a sequence of instructions,
rather than necessarily how much memory is touched by the algorithm or
its probability of resulting in L1 misses and similar.

Granted, OoO CPUs can sort of hide away L1 miss costs to some extent
(they are a little more obvious with a strictly in-order CPU).

> E.g. load 3 chunks (C1, C2, and C3) of 256bits each using standard SIMD
> load, and then add an instruction to turn C1+C2 into two 256bit vectors
> of 4x64bit floats, and another to do the same with C2+C3 (basically, the
> same instruction except it uses the other half of the bits of C2).
>

I don't really have a good way to do 256-bit loads or work with 256-bit
vectors in my core.

But, yeah, this is how it works for things like the 3x Float21 (64-bit)
and 3x Float42 vectors (128-bit). Where, Float42 vectors would be split
up into three Binary64 values for doing math on them, and then repacked
later.

The Float48 format is basically special case Load/Store ops which pad
the value to 64 bits on Load, and narrow it to 48 bits on Store. These
would be more intended for scalar operations on arrays.

There is a separate penalty for using it in arrays, in that it needs an
extra LEA.W instruction to handle the 48-bit element size (well, along
with the drawback of the Disp5 direct-displacement ops only having a
60-byte range in this case).

But, emulating this without these special instructions, would take a
somewhat longer instruction sequence (and ends up needing to stomp R16
and R17 for the pseudo-instructions, ...), so special ops seem
justifiable. I also added the fallback case to BGBCC as well.

>
> Stefan

>> E.g. load 3 chunks (C1, C2, and C3) of 256bits each using standard SIMD
>> load, and then add an instruction to turn C1+C2 into two 256bit vectors
>> of 4x64bit floats, and another to do the same with C2+C3 (basically, the
>> same instruction except it uses the other half of the bits of C2).
> I don't really have a good way to do 256-bit loads or work with 256-bit
> vectors in my core.

Then do 3x 64bit loads which you then split into 4 64bit floats:

64bit 64bit 64bit
+---------+ +-------------------------+ +---------+
A0-31 B0-31 A32-47 B32-47 C32-47 D32-47 C0-31 D0-31

It's an admittedly unusual layout, but lets you load&store in standard
sized chunks, and hence full-bandwidth. And lets you "reshuffle" things
using only "2-in" operations: if your pipeline can do "2-in 2-out" you
can do it two (parallel) instructions and otherwise you can do it in
4 (parallel) instructions. Of course, if your pipeline can accommodate
"3-in 4-out", then you can use a more standard layout.

> The Float48 format is basically special case Load/Store ops which pad the
> value to 64 bits on Load, and narrow it to 48 bits on Store.

So you're wasting the extra L1 bandwidth since you're using a load which
can fetch 64bit but only use 48 of those bits (you admittedly still
gain w.r.t to the bandwidth of higher levels of the memory hierarchy).

Stefan

Stephen Fuld <sfuld@alumni.cmu.edu.invalid> writes:
>On 9/25/2023 10:06 AM, Scott Lurndal wrote:
>> BGB <cr88192@gmail.com> writes:
>
>>> AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
>>> have 4 DIMM slots with DIMMs being filled in pairs.
>>
>> "typical" in what context? Home desktops? That's certainly not
>> typical for the data center or cloud servers. One chip I'm aware of
>> has 20 dual-channel DDR5 memory controllers (one per every four
>> cores).
>
>Wow! How many pins on the package? It must be massive. is there a
>latency penalty for getting that many DIMMs "close" to the CPU?

Further deponent sayeth not.

On Monday, September 25, 2023 at 11:43:45 AM UTC-5, BGB wrote:
> On 9/25/2023 10:41 AM, Scott Lurndal wrote:
> > j...@cix.co.uk (John Dallman) writes:
> >> In article <fbed57b4-1553-4b63...@googlegroups.com>,
> >> jsa...@ecn.ab.ca (Quadibloc) wrote:
> >>
> >>> hardware support for packed decimal
> >>> hardware support for IBM System/360 hexadecimal floating point
> >>>
> >>> because people do run Hercules on their computers and so on.
> >>
> >> I read the Hercules mailing list. Nobody on there uses it for serious
> >> work. It seems to be mainly used to evoke memories of youth. Running it
> >> on low-powered hardware, such as Raspberry Pi, attracts more notice than
> >> running it on something powerful. Hercules is written in portable C,
> >> because portability is considered more important than performance.
> >>
> >>> I have now added, at the bottom of the page, a scheme, involving
> >>> having dual-channel memory where each channel is 192 bits wide,
> >>> that permits the operating system to allocate blocks of 384-bit
> >>> wide memory, 288-bit wide memory, 240-bit wide memory, and 256-bit
> >>> wide memory.
> >>
> >> That's an interesting new way to have your system run short of the right
> >> kind of memory.
> >
> > Indeed. It's not the path from memory to the core complex that is
> > currently most interesting (although 256-bit wide (and higher) mesh or crossbars
> > aren't uncommon), but rather the data path widths from I/O
> > subsystems. 512-bit wide paths from network controllers and on-board
> > non-coherent (or coherent, see CXL) coprocessors has become
> > common. Supporting 80gbit/sec of network traffic into memory
> > or the networking subsystem isn't trivial.
> >
> > The memory bandwidth grows by adding controllers and striping across them
> > for the most part.
> ?...
>
>
> AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
> have 4 DIMM slots with DIMMs being filled in pairs.
>
> This would seemingly imply that RAM would be mostly limited to a 128-bit
> datapath (or 64-bit in unganged mode).
>
> Similarly, PCIe slots are effectively multiple serial lanes running in
> parallel, etc...
>
> Typical onboard peripherals being connected either with PCIe lanes or
> via an onboard LPC bus.
>
> Similarly, a typical motherboard only has a single CPU socket.
>
> ...
>
>
> Outside of the CPU itself, unclear where any of these wide interconnects
> would be, or where they would be going.
>
>
> Similarly, most things much smaller than a PC, will have 16-bit or
> 32-bit RAM interfaces (with typically 1 or 2 RAM chips soldered onto the
> motherboard somewhere, often also with an eMMC Flash chip and other things).
>
>
> Granted, on the other hand, I have some ~ 18 year old Xeon based rack
> servers, which have a pair of CPUs surrounded by a small island of RAM
> modules, and comparably still rather impressive multi-core "memcpy()"
> performance.
>
> Say, while each core individually is still limited to a few GB/sec, one
> can have all of the cores running memcpy at the same time without any
> significant drop (vs, say, on a PC where once the total exceeds around 8
> to 10GB/sec or so, per-thread performance drops off).
>
> Well, and both still significantly beat out the "memcpy()" performance
> on a 20 year old laptop (as does a RasPi...).
>
> ...

Ugh,
|> This would seemingly imply that RAM would be mostly limited to a 128-bit
|> datapath (or 64-bit in unganged mode).

https://en.wikipedia.org/wiki/CAS_latency
Has a table of DIMM performance for various DIMM generations and various burst lengths.
And by extension to some number of directly soldered DRAM chips which each can have up to 16 data pins.
(e.g. two chips DRAM with 32 total data pins and a burst size of eight = 256 bit cache line)

BGB-Alt wrote:
> On 9/25/2023 1:11 PM, Stefan Monnier wrote:
>>> I am now evaluating the possible use of a 48-bit floating-point
>>> format, but
>>> this is (merely) in terms of memory storage (in registers, it will
>>> still use
>>> Binary64).
>>
>> I suspect this is indeed the only sane way to go about it.
>> Also, I suspect that such 48bit floats would only be worthwhile when you
>> have some large vectors/matrices and care about the 33% bandwidth
>> overhead of using 64bit rather than 48bit.  So maybe the focus should be
>> on "load 3 chunks, then spread turn them into 4" since the limiting
>> factor would presumably be the memory bandwidth.
>>
>
> Yeah, memory bandwidth tends to be one of the major limiting factors for
> performance in my experience for many algorithms.
>
> This is partly why I had some wonk like 3x Float21 vectors (with 64-bit
> storage). And, part of why I do a lot of stuff using Binary16 (where, in
> this case, both Binary16 and Binary32 have the same latency).
>
> Well, and for a lot of my 3D rendering is using RGB555 framebuffers,
> 16-bit Z-buffer, and texture compression...
>
>
>
> As noted in some past examples, even desktop PCs are not immune to this,
> and saving some memory via bit-twiddly can often be cheaper than using a
> "less memory dense" strategy (that results in a higher number of L1
> misses).
>
> Ironically, this seems to run counter to the conventional wisdom of
> saving calculations via lookup tables (depending on the algorithm, the
> lookup table may only "win" if it is less than 1/4 or 1/8 the size of
> the L1 cache).

I used to be the master of lookup tables (i.e. a Word Count clone that
ran in 1.5 clock cycles/byte on a Pentium, using two dependent lookup
tables and zero branching), but more lately vector operations and the
increasing cost of storage means that re-calculating stuff can be faster
and/or more power-efficient.
>
> Many people seem to evaluate efficiency solely in terms of how many
> clock-cycles it would take to evaluate a sequence of instructions,
> rather than necessarily how much memory is touched by the algorithm or
> its probability of resulting in L1 misses and similar.

BTDT, lots of times!

Any serious optimization effort have to measure how each intended
improvement actually work as part of the full program, and not in
isolation. I have been in a situation where I really wanted to beat the
best closed-source ogg vorbis decoder, and I was getting desperate near
the end: About 9 out of 10 new ideas that would all work in
theory/isolation ended up either a wash or with a slowdown in the full
program.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

Scott Lurndal <slp53@pacbell.net> wrote:
>BGB <cr88192@gmail.com> writes:
>>On 9/25/2023 12:06 PM, Scott Lurndal wrote:
>>> Did you read the post you responded to? How do you get 40gbytes/sec
>>> into your memory subsystem from an onboard 400gbit nic? Or 128 gbytes/sec
>>> from a PCIe root complex? Or from a PCIe CXL-cache memory extender?
>>
>>Generally you don't...
>>
>>I think, if one has a 1GbE Ethernet port, they could maybe get 120 MB/s
>>or similar if it is going "full tilt".
>
>2.5Gbit Ethernet is making it's way into desktop chipsets. It won't be
>long before 10Gbit Ethernet is present in high-end PC mainboards.

10G is and has been a common feature in high-end PC motherboards for a
few years now (as part of the "differentiate from the cheap stuff").
"Creator" and certain other PC motherboard segments also often include
them.

Today I'd argue it's NOT a high-end PC motherboard if it doesn't have
at least one 10G network controller, some even have a second network
controller, this used to be an Intel 1G (everyone has drivers) but
lately there's a lot of 10G+2.5G or 10G+5G combos. And some 2x10G.

A single 1G controller is mostly relegated to the cheapest
motherboards which sometimes can't even provide full power to the top
processors!

Recently even some budget models (as opposed to "cheapest possible")
have 2.5G and in the various segment above that it's very common and
becomes almost a given as you move up. The 2.5G network controller has
become *cheap*.

Obviously it'll take time for all PCs to be replaced and all network
equipment to be replaced... OTOH, 2.5G will run on (almost) all
wiring/connectors that can handle 1G.

Scott Lurndal <scott@slp53.sl.home> schrieb:
> One chip I'm aware of
> has 20 dual-channel DDR5 memory controllers (one per every four
> cores).

That is seriously beefy.

Hm, let's see. For Power10, Wikipedia gives 410 GB/s for DDR4
and 800 GB/s for DDR6, DDR5 is probably in the middle, let's
call it 600 GB/s.

So, Power10 seems to have more memory bandwidth per core (equivalent
to ~12 DDR5 channels for 15 cores), unless you factor in the
eight-way SMT.

Still, that system you describe is _seriously_ beefy.

On Tuesday, September 26, 2023 at 9:41:36 PM UTC+3, Thomas Koenig wrote:
> Scott Lurndal <sc...@slp53.sl.home> schrieb:
> > One chip I'm aware of
> > has 20 dual-channel DDR5 memory controllers (one per every four
> > cores).
> That is seriously beefy.
>
> Hm, let's see. For Power10, Wikipedia gives 410 GB/s for DDR4
> and 800 GB/s for DDR6,

By now, DDR6 is not finalized even as specification. First modules are
~2 years away. Something as slow moving as POWER? I don't believe we
will see something even full 3 years from now.

Wikipadia is talking about GDDR6 rather than DDR6. GDDR6 is high bandwidth,
high latency, low capacity variant of DDR4 intended primarily for GPUs.

I would think that in case of POWER10 GDDR6 is applicable only for
rare specialty systems.

> DDR5 is probably in the middle, let's
> call it 600 GB/s.
>
> So, Power10 seems to have more memory bandwidth per core (equivalent
> to ~12 DDR5 channels for 15 cores), unless you factor in the
> eight-way SMT.
>
> Still, that system you describe is _seriously_ beefy.

If you want high bandwidth, don't care about high capacity or field upgradability,
but still want DDR5-like latency then LDDR5 looks like better path than GDDR7.

On Tuesday, September 26, 2023 at 11:43:48 PM UTC+3, Michael S wrote:
> On Tuesday, September 26, 2023 at 9:41:36 PM UTC+3, Thomas Koenig wrote:
> > Scott Lurndal <sc...@slp53.sl.home> schrieb:
> > > One chip I'm aware of
> > > has 20 dual-channel DDR5 memory controllers (one per every four
> > > cores).
> > That is seriously beefy.
> >
> > Hm, let's see. For Power10, Wikipedia gives 410 GB/s for DDR4
> > and 800 GB/s for DDR6,
> By now, DDR6 is not finalized even as specification. First modules are
> ~2 years away. Something as slow moving as POWER? I don't believe we
> will see something even full 3 years from now.
>
> Wikipadia is talking about GDDR6 rather than DDR6. GDDR6 is high bandwidth,
> high latency, low capacity variant of DDR4 intended primarily for GPUs.
>
> I would think that in case of POWER10 GDDR6 is applicable only for
> rare specialty systems.
> > DDR5 is probably in the middle, let's
> > call it 600 GB/s.
> >
> > So, Power10 seems to have more memory bandwidth per core (equivalent
> > to ~12 DDR5 channels for 15 cores), unless you factor in the
> > eight-way SMT.
> >
> > Still, that system you describe is _seriously_ beefy.
> If you want high bandwidth, don't care about high capacity or field upgradability,
> but still want DDR5-like latency then LDDR5 looks like better path than GDDR7.

LPDDR5, of course.

On 9/25/2023 5:08 PM, Stefan Monnier wrote:
>>> E.g. load 3 chunks (C1, C2, and C3) of 256bits each using standard SIMD
>>> load, and then add an instruction to turn C1+C2 into two 256bit vectors
>>> of 4x64bit floats, and another to do the same with C2+C3 (basically, the
>>> same instruction except it uses the other half of the bits of C2).
>> I don't really have a good way to do 256-bit loads or work with 256-bit
>> vectors in my core.
>
> Then do 3x 64bit loads which you then split into 4 64bit floats:
>
> 64bit 64bit 64bit
> +---------+ +-------------------------+ +---------+
> A0-31 B0-31 A32-47 B32-47 C32-47 D32-47 C0-31 D0-31
>
> It's an admittedly unusual layout, but lets you load&store in standard
> sized chunks, and hence full-bandwidth. And lets you "reshuffle" things
> using only "2-in" operations: if your pipeline can do "2-in 2-out" you
> can do it two (parallel) instructions and otherwise you can do it in
> 4 (parallel) instructions. Of course, if your pipeline can accommodate
> "3-in 4-out", then you can use a more standard layout.
>

Possible at least.

>> The Float48 format is basically special case Load/Store ops which pad the
>> value to 64 bits on Load, and narrow it to 48 bits on Store.
>
> So you're wasting the extra L1 bandwidth since you're using a load which
> can fetch 64bit but only use 48 of those bits (you admittedly still
> gain w.r.t to the bandwidth of higher levels of the memory hierarchy).
>

This is little different from normal load/store:
The L1 cache is wide enough to support 128 bits in 1 cycle;
But, anything narrower than 128 bits, still takes 1 cycle.

Typically, 64 and 32 bit access tends to dominate, followed by 128 bit,
with 8 and 16 bit access a little further down the list.

The Load path basically just performs an unaligned 64-bit load and then
extracts 48 bits from it. The Store path needed to add a special case to
support a 48-bit store.

Theoretically, 96-bit could also be possible, but would need some
additional logic (thus cost).

At least if moving data around with 64 or 128 bit operations, the
limiting factor tends to be L1<->L2 transfers, but if doing 8 or 16 bit
Load/Store, then the loads/stores themselves can become the bottleneck.

Doing stuff with 32-bit ops can go either way.

On 9/25/2023 7:06 PM, JimBrakefield wrote:
> On Monday, September 25, 2023 at 11:43:45 AM UTC-5, BGB wrote:
>> On 9/25/2023 10:41 AM, Scott Lurndal wrote:
>>> j...@cix.co.uk (John Dallman) writes:
>>>> In article <fbed57b4-1553-4b63...@googlegroups.com>,
>>>> jsa...@ecn.ab.ca (Quadibloc) wrote:
>>>>
>>>>> hardware support for packed decimal
>>>>> hardware support for IBM System/360 hexadecimal floating point
>>>>>
>>>>> because people do run Hercules on their computers and so on.
>>>>
>>>> I read the Hercules mailing list. Nobody on there uses it for serious
>>>> work. It seems to be mainly used to evoke memories of youth. Running it
>>>> on low-powered hardware, such as Raspberry Pi, attracts more notice than
>>>> running it on something powerful. Hercules is written in portable C,
>>>> because portability is considered more important than performance.
>>>>
>>>>> I have now added, at the bottom of the page, a scheme, involving
>>>>> having dual-channel memory where each channel is 192 bits wide,
>>>>> that permits the operating system to allocate blocks of 384-bit
>>>>> wide memory, 288-bit wide memory, 240-bit wide memory, and 256-bit
>>>>> wide memory.
>>>>
>>>> That's an interesting new way to have your system run short of the right
>>>> kind of memory.
>>>
>>> Indeed. It's not the path from memory to the core complex that is
>>> currently most interesting (although 256-bit wide (and higher) mesh or crossbars
>>> aren't uncommon), but rather the data path widths from I/O
>>> subsystems. 512-bit wide paths from network controllers and on-board
>>> non-coherent (or coherent, see CXL) coprocessors has become
>>> common. Supporting 80gbit/sec of network traffic into memory
>>> or the networking subsystem isn't trivial.
>>>
>>> The memory bandwidth grows by adding controllers and striping across them
>>> for the most part.
>> ?...
>>
>>
>> AFAIK, typical DIMMs have a 64-bit wide interface, and typical MOBOs
>> have 4 DIMM slots with DIMMs being filled in pairs.
>>
>> This would seemingly imply that RAM would be mostly limited to a 128-bit
>> datapath (or 64-bit in unganged mode).
>>
>> Similarly, PCIe slots are effectively multiple serial lanes running in
>> parallel, etc...
>>
>> Typical onboard peripherals being connected either with PCIe lanes or
>> via an onboard LPC bus.
>>
>> Similarly, a typical motherboard only has a single CPU socket.
>>
>> ...
>>
>>
>> Outside of the CPU itself, unclear where any of these wide interconnects
>> would be, or where they would be going.
>>
>>
>> Similarly, most things much smaller than a PC, will have 16-bit or
>> 32-bit RAM interfaces (with typically 1 or 2 RAM chips soldered onto the
>> motherboard somewhere, often also with an eMMC Flash chip and other things).
>>
>>
>> Granted, on the other hand, I have some ~ 18 year old Xeon based rack
>> servers, which have a pair of CPUs surrounded by a small island of RAM
>> modules, and comparably still rather impressive multi-core "memcpy()"
>> performance.
>>
>> Say, while each core individually is still limited to a few GB/sec, one
>> can have all of the cores running memcpy at the same time without any
>> significant drop (vs, say, on a PC where once the total exceeds around 8
>> to 10GB/sec or so, per-thread performance drops off).
>>
>> Well, and both still significantly beat out the "memcpy()" performance
>> on a 20 year old laptop (as does a RasPi...).
>>
>> ...
>
> Ugh,
> |> This would seemingly imply that RAM would be mostly limited to a 128-bit
> |> datapath (or 64-bit in unganged mode).
>
> https://en.wikipedia.org/wiki/CAS_latency
> Has a table of DIMM performance for various DIMM generations and various burst lengths.
> And by extension to some number of directly soldered DRAM chips which each can have up to 16 data pins.
> (e.g. two chips DRAM with 32 total data pins and a burst size of eight = 256 bit cache line)

I was talking about the width of the datapath, not the size of a burst
transfer.

So, for example, with a DDR3 chip with a 16-bit interface, 8x burst, you
can move 16B in 4 cycles (from the RAM chip's POV); but this doesn't
change that the interface is only 16-bit wide.

Internally, in my case, there is a 512-bit wide interface between my DDR
RAM module and L2 cache, but I don't really count it as it is fairly
special purpose: Move data in big enough chunks that I can get full RAM
bandwidth, with a DDR controller that can only issue one request at a
time at a 50MHz RAM clock (logic internally operating at 100MHz and thus
driving the high and low sides as separate clock-cycles as far as the
FPGA is concerned).

If I were able to FIFO the requests in the RAM controller, wouldn't
necessarily need the 512-bit bursts; but there are annoyances (and
latency) here, due to crossing clock domains.

On 9/26/2023 1:09 AM, Terje Mathisen wrote:
> BGB-Alt wrote:
>> On 9/25/2023 1:11 PM, Stefan Monnier wrote:
>>>> I am now evaluating the possible use of a 48-bit floating-point
>>>> format, but
>>>> this is (merely) in terms of memory storage (in registers, it will
>>>> still use
>>>> Binary64).
>>>
>>> I suspect this is indeed the only sane way to go about it.
>>> Also, I suspect that such 48bit floats would only be worthwhile when you
>>> have some large vectors/matrices and care about the 33% bandwidth
>>> overhead of using 64bit rather than 48bit.  So maybe the focus should be
>>> on "load 3 chunks, then spread turn them into 4" since the limiting
>>> factor would presumably be the memory bandwidth.
>>>
>>
>> Yeah, memory bandwidth tends to be one of the major limiting factors
>> for performance in my experience for many algorithms.
>>
>> This is partly why I had some wonk like 3x Float21 vectors (with
>> 64-bit storage). And, part of why I do a lot of stuff using Binary16
>> (where, in this case, both Binary16 and Binary32 have the same latency).
>>
>> Well, and for a lot of my 3D rendering is using RGB555 framebuffers,
>> 16-bit Z-buffer, and texture compression...
>>
>>
>>
>> As noted in some past examples, even desktop PCs are not immune to
>> this, and saving some memory via bit-twiddly can often be cheaper than
>> using a "less memory dense" strategy (that results in a higher number
>> of L1 misses).
>>
>> Ironically, this seems to run counter to the conventional wisdom of
>> saving calculations via lookup tables (depending on the algorithm, the
>> lookup table may only "win" if it is less than 1/4 or 1/8 the size of
>> the L1 cache).
>
> I used to be the master of lookup tables (i.e. a Word Count clone that
> ran in 1.5 clock cycles/byte on a Pentium, using two dependent lookup
> tables and zero branching), but more lately vector operations and the
> increasing cost of storage means that re-calculating stuff can be faster
> and/or more power-efficient.

Yeah.

Or, in my core, because of the comparably high cost of L1 misses...

Using lookup tables to sidestep a divide operation can be a win though.
Software divide loop is slow.
The hardware DIVx.x ops are also slow.

Except that DIVS.L / DIVU.L can optimize a similar subset to the lookup
tables (internally handling it as a multiply-by-reciprocal). So, say,
for divisors in the range of 0..63, the DIVS.L op can handle it in 3
cycles (vs 36 cycles for most everything else).

>>
>> Many people seem to evaluate efficiency solely in terms of how many
>> clock-cycles it would take to evaluate a sequence of instructions,
>> rather than necessarily how much memory is touched by the algorithm or
>> its probability of resulting in L1 misses and similar.
>
> BTDT, lots of times!
>
> Any serious optimization effort have to measure how each intended
> improvement actually work as part of the full program, and not in
> isolation. I have been in a situation where I really wanted to beat the
> best closed-source ogg vorbis decoder, and I was getting desperate near
> the end: About 9 out of 10 new ideas that would all work in
> theory/isolation ended up either a wash or with a slowdown in the full
> program.
>

Yeah.

This is related sort of to some of my codecs using Rice and other
non-standard entropy encoders (rather than Huffman), as while in
isolation, Huffman is short and fast, once has multiple contexts (and a
lot of L1 misses), performance quickly takes a hit.

So, there may be wonky encoders, like ranking all the symbols by
probability and then Rice-encoding the index (so, only need ~ 256 bytes
per context in this case), usually with the Rice-coding being
length-limited (usually, Q=7 as a limit, where a case of 8x 1-bits or
similar is followed by a raw symbol).

More complex logic, but fewer L1 misses, ...

Though, Huffman with a 12-bit length-limit is also a promising
compromise, as it is generally easier for an 8K lookup table to fit into
the L1 cache... And, 12-bit Huffman does still lead to better
compression than the permutation-table + Rice variants.

....

> Terje
>