Rent NVIDIA GPUs in the Cloud for AI Workloads

This guide walks you through renting NVIDIA GPUs in the cloud for AI workloads—training models, running inference, or batch processing. You’ll learn how to choose a provider, pick the right GPU, spin up an instance, and connect to it, without needing to buy expensive hardware. Before you start, you’ll need an account with a cloud provider (like AWS, Google Cloud, Azure, or a specialized GPU rental service), a payment method, and a basic familiarity with the command line and SSH.

What You’ll Need

A cloud provider account (AWS, Google Cloud, Azure, Lambda Labs, etc.) with billing enabled.
An SSH key pair (public/private) to securely access the instance.
Basic knowledge of Linux commands (ls, cd, ssh, scp).
Your AI workload code and data, ready to upload.
A budget estimate—GPU instances can be expensive, so know your hourly cost.

Step-by-Step

Choose a provider and sign up. Major clouds (AWS, GCP, Azure) offer NVIDIA GPUs like A100, V100, and T4. Specialized services (Lambda, Vast.ai, CoreWeave) often have lower prices and simpler interfaces. Create an account and verify your identity.
Set up billing. Add a payment method. For free tiers, note that GPU instances are rarely included—you’ll be charged per second or per hour.
Select a region. Choose a region with GPU availability (e.g., us-east-1 on AWS, us-central1 on GCP). Some regions have limited stock, so check availability first.
Choose the GPU type. Match the GPU to your workload:
- V100 (16/32 GB) – older but good for many training tasks.
- A100 (40/80 GB) – best for large models and mixed-precision training.
- T4 (16 GB) – cost-effective for inference and small models.
- RTX 4090 (24 GB) – consumer-grade, often cheaper on specialized services.
Configure the instance. Choose an OS (Ubuntu 20.04 or 22.04 LTS is common), set the storage (at least 100 GB for datasets and models), and attach your SSH key.
Launch the instance. Click “Launch” or “Create.” Wait for it to enter “running” state. Note the public IP address and SSH command provided.
Connect via SSH. Open a terminal and run: ssh -i /path/to/your-key.pem ubuntu@PUBLIC_IP. For GCP, the username is often your_username; for AWS it’s ubuntu or ec2-user.
Install NVIDIA drivers and CUDA. Run nvidia-smi to check if drivers are pre-installed. If not, install them:
```
sudo apt update && sudo apt install -y nvidia-driver-470
```
Then install CUDA toolkit from NVIDIA’s official repository (or use conda).
Set up your AI environment. Install Python, PyTorch/TensorFlow, and other dependencies. For example:
```
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```
Upload your code and data. Use scp or rsync to copy files: scp -i key.pem -r ./my_project ubuntu@IP:/home/ubuntu/. Alternatively, use cloud storage (S3, GCS) and download with aws s3 cp or gsutil cp.
Run your workload. Execute your training script: python train.py. Monitor GPU usage with nvidia-smi in another terminal.
Save results and shut down. Copy results back to your local machine, then terminate the instance (not just stop, to avoid storage costs).

Common Mistakes

Leaving the instance running – Forgetting to terminate leads to huge bills. Set up billing alerts.
Choosing the wrong GPU – A100 for a small model is overkill; T4 might be too slow for large training. Check memory requirements.
Ignoring region availability – Some regions show “insufficient capacity.” Try a different region or provider.
Not using spot/preemptible instances – For fault-tolerant jobs, spot instances can save 60-90%.
Forgetting to install CUDA – PyTorch won’t detect GPU if CUDA is missing; verify with python -c "import torch; print(torch.cuda.is_available())".

Tips

Use pre-configured images – Many providers offer deep learning AMIs (AWS) or Deep Learning VMs (GCP) with drivers and frameworks pre-installed.
Start with a small instance – Test your code on a CPU instance first to debug, then scale to GPU.
Use spot instances for non-critical jobs – They can be interrupted, but are much cheaper.
Monitor costs – Set a budget in the provider console and enable alerts.
Automate setup with scripts – Save your setup commands in a shell script to launch instances quickly.
Use containerization – Docker with NVIDIA Container Toolkit simplifies environment reproducibility.

Troubleshooting

Can’t connect via SSH? Check that your security group/firewall allows inbound port 22 from your IP. Ensure the key file permissions are correct: chmod 400 key.pem.

GPU not detected? Run nvidia-smi. If it fails, reinstall drivers. For Ubuntu, you may need to disable the Nouveau driver: add blacklist nouveau to /etc/modprobe.d/blacklist.conf and run sudo update-initramfs -u.

Out of memory? Reduce batch size, use mixed precision (fp16), or choose a GPU with more VRAM.

Slow data transfer? Use compression (rsync -z) or upload to cloud storage in the same region as the instance.

Instance stopped unexpectedly? Check spot instance termination notices or resource limits. Restart with a new instance if needed.

Billing seems high? Review usage in the billing dashboard. Ensure you terminated all instances and deleted any associated storage volumes.

Frequently Asked Questions

How much does it cost to rent an NVIDIA GPU?

Costs vary widely: a T4 might be $0.35/hour on AWS, while an A100 can be $2-4/hour. Specialized services like Lambda offer A100s around $1.10/hour. Spot instances can be 60-90% cheaper.

Can I rent a GPU for a few hours only?

Yes, most providers bill per second or per hour. You can launch an instance, run your job, and terminate it within minutes.

Do I need to install CUDA myself?

Often not. Many cloud images come with CUDA pre-installed. Check with nvcc --version. If not, you can install it via the NVIDIA repository or use conda.

What’s the difference between on-demand and spot instances?

On-demand is guaranteed but expensive. Spot instances use spare capacity at a discount but can be terminated with little notice. Use spot for fault-tolerant workloads.

How do I transfer large datasets to the cloud?

For large data, upload to cloud storage (S3, GCS) first, then download to the instance using CLI tools. This is faster and more reliable than scp over the internet.

Can I use multiple GPUs in one instance?

Yes, many providers offer instances with 4 or 8 GPUs. You’ll need to configure distributed training (e.g., PyTorch DDP) to use them effectively.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →