On 3/24/23 8:12 PM, MitchAlsup wrote:
> On Friday, March 24, 2023 at 4:44:26 PM UTC-5, Kent Dickey wrote:
[snip]
>> CXL also adds a way to add memory on PCIe which host CPUs can use as more
>> coherent memory (which may be in another computer), and I wasn't referring
>> to that.
> <
> This is mostly being sold as:: you can eliminate the DRAM I/O pins and
> put everything on PCIe. For something like a modern x86-64 chip, you
> save 200 DRAM pins and add a couple (4) CXL popcorn parts to access
> DRAM.

While PCIe might well be very bandwidth efficient for a commodity
standard for general use, I suspect one could get a little more
effective bandwidth with a specialized interface, especially if
one can assume tighter integration and higher quality
implementation.

I am probably also a little of a latency fanboi — promoting
latency as important not because it has been measured to be so but
because I happened to land on that side of the conflict. Adding a
translation layer and a chip hop seems likely to increase latency.

(Caches do work surprisingly well and avoiding the cache checking
latency overhead is difficult and some on-chip network delay is
unavoidable in the general case — in the special case of a core
being close to the memory controller it primarily (or urgently)
uses. Checking an L3 filter or predictor in parallel with L2 tags
might be practical, but I suspect the cost would not be worth the
benefit.)

> It also decouples the CPU chips from DRAM I/O protocols. Enabling more
> use of HBM........

Buffered memory (e.g., FB-DIMM) had a similar benefit but did not
seem to lead to diversity.

An intermediate chip could also provide caching, prefetch
buffering, and coherence (if connected to more than one node).
Extra connections might also expand memory (or cache) capacity or
provide communication routes. Connecting to a networking chip
would seem to facilitate lower overhead send DMA for non-dynamic
data (dynamic data would be more reasonably sources from processor
caches, I suspect). With a more generic interface (like PCIe),
bandwidth could be shared between memory accesses and I/O accesses
(like old system's buses sharing the bandwidth between I/O and
memory).

As a latency fanboi, I am disappointed by HBM because it trades
price per bit for bandwidth but no similar product has succeeded
in trading price per bit for lower latency. Fanboi sad.☹