Yes, a PyTorch checkpoint can be lost after a Colab disconnect if you only saved it to the local VM storage. The fix is to save checkpoints to Google Drive (or another persistent location) and load from there after reconnecting. This article explains why disconnects erase files, how to save and load checkpoints correctly, and how to automate the process so you don’t lose hours of training again.
Why Colab Disconnects Erase Your Files
Google Colab gives you a temporary virtual machine (VM) with a Linux filesystem. Everything you write to that filesystem—including PyTorch checkpoint files like model.pt—lives on a disk that is destroyed when the VM is recycled. Disconnects happen for several reasons:
- Idle timeout: Colab disconnects after about 90 minutes of inactivity (no interaction with the notebook).
- Session limit: Free tiers have a maximum session length (often 12 hours), after which the VM is shut down.
- Network issues: A brief internet drop can end the session.
- Resource limits: Heavy memory or GPU usage can trigger a crash and restart.
When any of these happen, the VM and its disk are discarded. If your checkpoint only exists on that disk, it is gone. The only reliable way to keep a checkpoint is to store it somewhere persistent—Google Drive, Google Cloud Storage, or GitHub (for small files).
How to Save Checkpoints to Google Drive
Mount your Google Drive at the start of your notebook, then save your checkpoint to a path under /content/drive/. Here’s a minimal example:
- Mount Drive:
from google.colab import drive; drive.mount('/content/drive') - Create a folder:
!mkdir -p /content/drive/MyDrive/checkpoints - Save your model:
torch.save(model.state_dict(), '/content/drive/MyDrive/checkpoints/model.pt')
For a full training loop, save both the model state and the optimizer state, plus the epoch number, so you can resume exactly where you left off:
checkpoint = {
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': loss,
}
torch.save(checkpoint, '/content/drive/MyDrive/checkpoints/ckpt_epoch_{}.pt'.format(epoch))
This way, after a disconnect, you can load the checkpoint and continue training without restarting from scratch.
How to Load a Checkpoint After Reconnecting
After Colab reconnects, you need to recreate your model architecture, then load the saved state. Here’s the typical pattern:
- Mount Drive again.
- Recreate your model and optimizer (same architecture, same hyperparameters).
- Load the checkpoint:
checkpoint = torch.load('/content/drive/MyDrive/checkpoints/ckpt_epoch_10.pt') - Restore states:
model.load_state_dict(checkpoint['model_state_dict'])andoptimizer.load_state_dict(checkpoint['optimizer_state_dict']) - Set the epoch to
checkpoint['epoch'] + 1to continue from the next epoch.
Be careful: if you changed your model class or the order of layers, the state dict keys won’t match and loading will fail. Keep your model definition identical between sessions.
Automating Checkpointing to Avoid Loss
Manually saving every epoch is tedious and error-prone. You can automate it with a few techniques:
- Save every N epochs: Inside your training loop, add
if epoch % 5 == 0: torch.save(...)to save periodically. - Save on improvement: Track the best validation loss and save only when it improves, so you always have the best model.
- Use a background thread: Save to Drive is slow (network I/O). You can use a thread to avoid blocking training, but be careful with concurrent writes.
- Use a library like PyTorch Lightning: It has built-in checkpoint callbacks that save to a directory you specify (e.g., on Drive).
Also consider saving to Google Cloud Storage if your Drive fills up or if you need faster I/O. The gcsfs library lets you write directly to a bucket.
Common Mistakes and Misconceptions
Many users lose checkpoints due to avoidable errors:
- Saving only to local /content: This is the #1 cause. Always use a persistent path.
- Forgetting to mount Drive: If you don’t mount, the path
/content/drivedoesn’t exist and saving fails. - Using a relative path:
torch.save(model, 'model.pt')saves to the current working directory, which is on the VM. Use an absolute path under/content/drive/. - Believing Colab autosaves: Colab autosaves the notebook (code cells), not the files you create. Your checkpoint is not automatically backed up.
- Loading with
torch.loadwithoutmap_location: If you trained on GPU and load on CPU (or vice versa), usetorch.load(path, map_location=torch.device('cpu'))to avoid errors.
Alternatives to Google Drive
Google Drive is the most common choice, but not the only one. Depending on your needs, consider:
- Google Cloud Storage: Faster and more scalable, but requires setting up a bucket and authentication.
- GitHub Releases: Good for small checkpoints (<100 MB) but not practical for large models.
- Hugging Face Hub: If you’re training a model for sharing, you can push checkpoints directly.
- Local download: Download the checkpoint to your own machine before the session ends. You can do this manually or with
files.download()fromgoogle.colab.
Each has trade-offs in ease, speed, and cost. For most users, Google Drive is the simplest and free option.
Frequently Asked Questions
Can I recover a checkpoint after a disconnect if I didn’t save to Drive?
No. Once the VM is recycled, the local disk is destroyed. There is no way to recover files from a previous session unless you saved them externally. Some users try to reconnect to the same VM, but Colab does not guarantee that; you get a new VM with a fresh disk.
Does saving to Drive slow down training?
Yes, saving to Drive involves network I/O and can take a few seconds per checkpoint. To minimize impact, save less frequently (e.g., every 5 or 10 epochs) or use a background thread. The overhead is usually small compared to the cost of losing hours of training.
What is the difference between saving the whole model and saving state_dict?
torch.save(model, ...) saves the entire model object including the architecture, while torch.save(model.state_dict(), ...) saves only the learned parameters. The latter is recommended because it is more portable and less likely to break if the model class changes. You need to recreate the model architecture before loading a state_dict.
Can I use Google Drive with a free Colab account?
Yes, free Colab accounts can mount Google Drive. There are storage limits (15 GB free), but that is usually enough for checkpoints. If you exceed the limit, you can delete old checkpoints or use a different storage solution.
What if my checkpoint is too large for Drive?
If your checkpoint exceeds your Drive quota, consider saving only the state_dict (which is smaller than the full model), using a compression format like torch.save(checkpoint, 'file.pt', _use_new_zipfile_serialization=True), or storing in a cloud bucket with more capacity. You can also split the checkpoint into shards, but that adds complexity.
Does Colab Pro have the same disconnect problem?
Yes, Colab Pro gives longer session limits and priority access, but the VM is still temporary. A disconnect (e.g., due to idle timeout or network) will still erase local files. Always save to persistent storage regardless of your plan.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Skip Ryobi and Milwaukee? Harbor Freight Tools to Buy
- Funny Subreddits That Make You Laugh
- Why Retro Gaming Is Still Worth Playing Today
- Survival Crafting Game Fatigue: Why & How to Fix It
- Sell Your EV Now Before Prices Plummet?
- KDE Connect Vulnerabilities: Are These Versions at Risk?