Besides gaming, frequently used for AI and LLM inference and training.
Hosts that offer Nvidia GPUs
Beyond the big cloud providers there are a number of companies now offering Nvidia GPU hosting
Device Management and Order on Linux
When working with CUDA and PyTorch on a machine that has multiple GPUs we might want to restrict our workload to a specific GPU device.
We use CUDA_DEVICE_ORDER=PCI_BUS_ID to tell pytorch to use the same order for device indexing as the operating system (if you run nvidia-smi). Then we can restrict access with a comma separated list of devices in CUDA_VISIBLE_DEVICES.
export CUDA_DEVICE_ORDER="PCI_BUS_ID"
export CUDA_VISIBLE_DEVICES=1Nvidia and Docker
see Nvidia
GCP
Installing Nvidia cards in GCP is non-trivial but I have collected some information here.
Card Specs
Nvidia L4
More recent version of the T4 that uses Lovelace architecture. Has 24GiB VRAM
Nvidia T4
Notably used by Google for the free tier in colab. Based on the Tesla architecture. Has 16 GiB VRAM