Hacker News

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.

Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.

Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.

Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.

Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.

Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.

Yes, and no.

Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.

PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)

In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.

There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.

that go nowhere

But some do end up in the industry: Mythic AI, Encharge AI, d-Matrix.

"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.

Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.

People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.

At this point processing-in-memory has taken on the aura of fusion power.

> Whilst processing in memory is clearly the future

How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.

you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.

what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.

that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.

Sort of, but the cache architecture is replaced with memory architecture. But you could look at it either way.

Maybe we can connect multiple Samsung RAM sticks to a grid and call it RAMsputer.

I guess you also get very high bandwidth that way? I‘m not sure that would come for free though.

if your working set fits in cache PIM is irrelevant

As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required.

Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.

The idea is this:

You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.

614 GB/s x 96 = 58,944 GB/s.

Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast single digit gigabyte models. Without MoE you could reach 100 tokens per second with an 8B fp8 model on a single channel. With MoE you might break 500 tokens per second.

> That being said, I am not sure what's the killer application for this technology...

Build it, and they will come ;)

Not necessarily. If your weights have to go across 2 banks, you just have to split and transfer the input and output vectors, which are much smaller.

For GEMV you lose nothing.

The reason is quite simple. You can split the matrix along both dimensions so you just tile it into 64x64 or whatever fits into the bank and just fill it up. The biggest problem is load balancing the tiles across all banks for maximum parallelism.

For GEMM I believe there is no point in doing PIM, you are better off with a GPU or NPU.

There's been a ton of optimizations already, it hasn't remotely reduced demand even temporarily. More efficiency just makes the compute have even higher ROI per $ and watt spent.

It's not just inference, some things done in data centers like simulations, testing, are complementary to inference.

And in these types of hardware, the time between a successful prototype and a fully deployed product is pretty long. Maybe they count on that to know when to stop?

Jevons Paradox shows that increasing efficiency can increase demand for a product by making it cost effective for more uses.

The better question is: what is the net gain for the overall system? If PIM reduces the net thermal load and power consumption of the system for the same workload, then it’s a win regardless of where the heat sinks end up. The customers Samsung has in mind for this today are not limited to commodity designs. They’re using novel designs with each new hardware generation, so moving heat sinks around is not a deal-breaker.

Interesting read. Definitely hoping this works out so we can have cheap LLM machines at home

that's not at all comparable, you're still paying memory latency and not getting any extra bandwidth

It doesn't suffer from the inherent security issues of the von Neumann architecture. Memory will only be data.