GPU spatial sharing with NVIDIA MPS

How to set MPS in many containers?

When you creat a new container, you should add a parameter –ipc=host to make the new container has the same network privilege as host server.
For example:

docker run -itd --name xxx -ipc=host --privileged=true -p 10001:22 image_xxxx:1.0

Start MPS

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 nvidia-cuda-mps-control -d

Once MPS started, each GPU will has a nvidia-cuda-mps-server process. If you run a resnet50 model. it is a MPS client for this GPU.

Quit MPS

echo quit | nvidia-cuda-mps-control

set SM percentage

export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=percentage

client memory limits

export CUDA_MPS_PINNED_DEVICE_MEM_LIMIT="0=1G"

where 0 is GPU id, 1G is the process GPU memory limitation. For detail, you can refer: https://developer.nvidia.com/blog/revealing-new-features-in-the-cuda-11-5-toolkit/