Cuda memory failing after RAM upgrade

[adamdoc]

Hello,

I am having some issues reconstructing using FBP/FDK after a RAM upgrade. I'm using CIL with Tigre backend, but this issue also seems to happen with Tigre by itself, and doesn't happen when using Astra as CIL backend

Our new config has a massive 1TB of RAM, but recon fails when the projection data is over 2GB (independent of recon volume size) with the following error
../Common/CUDA/TIGRE_common.cpp (7): Texture memory allocation fail
../Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:Atb out of memory
The kernel died, restarting...

However, after lowering to 768GB of RAM (or below), it seems I can recon from projection data of any size without issue (only tested to 150GB). The volume does have to be below 8GB or so which is a bit strange given I have 48GB of VRAM on the GPU but this is manageable with chunked recon.

Any advice for what may be going on appreciated, been confusing me for a couple of days now, Thanks!

Build cuda_12.5.r12.5/compiler.34177558_0
tigre 2.6 np126py312h9bba915_0 ccpi
cil 24.2.0 np126py312_0 ccpi

[AnderBiguri]

Heya,
This is very weird. So, few thouhgts: the reason it happens with TIGRE, but not ASTRA, is because TIGRE uses pinned memory, i.e. uses the RAM (not just the VRAM) to do recon. this is why you can do large scale recons in TIGRE. This is used all the time, not just when the image is too big to fit, as it makes the code faster all of the time. So it makes sense that if this process is failing ("the kernel died") then it fails everytime regardless of the size.

However, I am not sure if TIGRE has any control on this. The way the line that fails works is that it asks the operating system for memory in the RAM, and asks the operating system to "page-lock" (or "pin") the memory, i.e. to lock all the memory and force it to never be moved by other processes. From TIGRE side, this is 1 line of CUDA code with no parameters, so I am thinking that this may be more related to your system, rather than anything that I could fix from my side. Its possible that e.g. your CPU/motherboard does not support 1TB of RAM for page-locking, for example. Or that the OS version does not support it. I would dig a little bit that side, as I suspect the "out of memory" comes from TIGRE asking for the memory to the OS, and the OS saying "I can't give you that", and TIGRE/CUDA interpreting that it must be due to lack of memory. But we know its not that.

So dig a bit more int he capabilities of your machine, I suspect that the issue is from that side. I'll check a bit more if there are issues like this from the CUDA side, but my preemptive search suggest no.

As a side note, you say that the volume has to be below 8GB? Is this always for you?

[AnderBiguri]

@adamdoc couple of notes:
The pycuda function you mention allocates memory, but does not page-lock memory, which is not the same. So not necesarily contradictory.

I still don't understand the 8Gb issue, can you elaborate? TIGRE (with page locked memory, which is default) allows for larger-than-memory reconstruction. I reconstructed 64Gb volumes on a 8Gb VRAM GPU. If you are having issues with this, it could be also a memory allocation issue.

[adamdoc]

My apologies, I was using cuda.pagelocked_empty to allocate large memory array without issue

Yes, I think it's also a memory allocation issue for the volume. I get the same cuda error if either the page locked projection data in the RAM is over 2GB, or if the volume I am trying to reconstruct on the VRAM is over 8GB. The former goes away as long as I don't have the full RAM config, but the 8GB limit on the VRAM is always present. I know I should be able to recon larger volumes, and with Astra I can. It's a bit confusing as if our issue is with page-locked memory allocation, then this shouldn't happen?

[AnderBiguri]

@adamdoc ASTRA uses a more naive way to handle the memory, which is why its slower at large images. Ultimately, it doesn't use page-locked memory. I wonder if you can get help from CIL people to compile CIL-TIGRE with page-locked memory turned off, which is just a standard flag when compiling TIGRE. Maybe that will help verify if this is the issue.

[adamdoc]

Okay we can ignore the 8GB limit, the issue only exists with FBP. I'll be using FDK and just tested on that and I can recon a 60GB dataset no issue

I'm speaking to the CIL people now on discord, thanks for all the advice and I'll report back if I find the issue!

[AnderBiguri]

@adamdoc ah, the parallel beam projectors are not multi-GPU, as you can reconstruct slice-per-slice.