Stop BLAS Threads Overloading Scheduled Python Jobs

If your scheduled Python jobs are slowing down, crashing, or hogging CPU, the culprit is often BLAS (Basic Linear Algebra Subprograms) libraries like OpenBLAS, MKL, or BLIS. These libraries automatically spawn one thread per CPU core for matrix operations, which can overwhelm a shared server or a job scheduler. This guide shows you how to cap BLAS thread counts so your jobs run predictably and play nicely with other processes.

What You’ll Need

Access to the Python environment where your scheduled jobs run (e.g., a cron job, Airflow, Celery, or a systemd timer).
Basic familiarity with the command line and environment variables.
Knowledge of which BLAS library your NumPy/SciPy uses (check with np.__config__.show()).

Step-by-Step

Identify the BLAS library in use. Run the following in a Python shell:
```
import numpy as np
np.__config__.show()
```
Look for blas_info or openblas_info. If it says openblas, you’ll use OPENBLAS_NUM_THREADS. For MKL, it’s MKL_NUM_THREADS; for BLIS, BLIS_NUM_THREADS.
Decide on a thread count. For a scheduled job, a good starting point is 1 or 2 threads, especially if you run multiple jobs concurrently. If your job is CPU-bound and runs alone, you might use 4. The goal is to avoid oversubscription—when total threads across jobs exceed the number of CPU cores.
Set environment variables in the scheduler. The exact method depends on your scheduler:
- Cron: Edit your crontab with crontab -e and add at the top:
```
OPENBLAS_NUM_THREADS=2
MKL_NUM_THREADS=2
BLIS_NUM_THREADS=2
NUMEXPR_NUM_THREADS=2
OMP_NUM_THREADS=2
```
  Then run your job command as usual.
- Airflow: In your DAG, set environment variables in the task’s env parameter, or set them globally in airflow.cfg under [core] with default_task_env.
- Celery: In your Celery worker command, prefix with the environment variables, e.g., OPENBLAS_NUM_THREADS=2 celery -A proj worker. Or set them in the worker’s systemd unit file.
- Systemd timer: In the service unit file, under [Service], add:
```
Environment=OPENBLAS_NUM_THREADS=2
Environment=MKL_NUM_THREADS=2
```
Verify the settings are applied. In your Python code, check the thread count before heavy computation:
```
import os
print(os.environ.get('OPENBLAS_NUM_THREADS'))
print(os.environ.get('MKL_NUM_THREADS'))
```
Also, you can check the actual number of threads used by timing a matrix multiplication and monitoring CPU usage with top or htop.
Set the variables at the top of your Python script as a fallback. If you can’t control the scheduler environment, set them at the very top of your script, before importing NumPy/SciPy:
```
import os
os.environ['OPENBLAS_NUM_THREADS'] = '2'
os.environ['MKL_NUM_THREADS'] = '2'
os.environ['BLIS_NUM_THREADS'] = '2'
os.environ['OMP_NUM_THREADS'] = '2'
import numpy as np  # now import after setting env
```
This must happen before any import that uses BLAS.
Test your scheduled job. Run the job manually with the environment variables set to confirm it works and that CPU usage stays within expected limits. Monitor with top and check the job’s runtime.
Monitor and adjust. Over time, watch your job’s performance. If it’s too slow, increase the thread count slightly. If other processes are starved, decrease it. Keep an eye on the total CPU usage of all concurrent jobs.

Common Mistakes

Setting the variable after importing NumPy. The environment variable must be set before the BLAS library is loaded. If you import NumPy first, the setting won’t take effect.
Using the wrong variable name. Each BLAS library has its own variable. OpenBLAS ignores MKL_NUM_THREADS and vice versa. Check which one you have.
Forgetting the OpenMP variable. Some operations also use OpenMP threads. Setting OMP_NUM_THREADS can help, but be careful—it may affect other libraries.
Setting the variable in the wrong shell. If you set it in your interactive shell but the scheduler runs in a non-login shell, it won’t inherit it. Always set it in the scheduler’s environment.
Assuming all threads are bad. For a single, long-running CPU-bound job, using multiple threads can be beneficial. The problem is when many jobs run concurrently and oversubscribe the CPU.

Tips

Use a wrapper script. Create a shell script that sets the environment variables and then runs your Python script. This makes it easy to change thread counts without editing scheduler configs.
Consider using threadpoolctl. The threadpoolctl Python package lets you control thread counts at runtime, even after imports. Example:
```
from threadpoolctl import threadpool_limits
with threadpool_limits(limits=2):
    # your BLAS-heavy code here
```
This is great for limiting threads only for specific sections of code.
Set a global limit for all jobs. If you run many scheduled jobs, create a shared environment file that each job sources, ensuring consistent thread usage.
Monitor with psutil. In your Python script, you can log the current thread count of your process to help debug issues.
Document your settings. Put a comment in your scheduler config or script explaining why you set the thread count, so future maintainers understand.

Troubleshooting

Job still uses too many threads. Check if you have multiple BLAS libraries installed. For example, if you set OPENBLAS_NUM_THREADS but SciPy uses MKL, it won’t work. Use np.__config__.show() to confirm. Also, check if your code calls threadpool_limits or set_num_threads elsewhere that overrides the environment variable.
Job crashes with a segmentation fault. This can happen if you set thread counts to 0 or a negative number. Always use positive integers. Also, some BLAS versions have bugs with very low thread counts; try 2 instead of 1.
Environment variable not picked up. Verify the scheduler’s environment by adding a line to your script that prints the variable to a log file. If it’s not set, the scheduler might be stripping environment variables. Use the fallback method of setting it in the Python script before imports.
Performance got worse after limiting threads. For small matrices, threading overhead can dominate. If your job uses small matrices, limiting to 1 thread might be faster. Test with different counts to find the sweet spot.
Other jobs still experience CPU spikes. Maybe the spikes come from a different library like TensorFlow or PyTorch. Those have their own thread controls (tf.config.threading, torch.set_num_threads).

Frequently Asked Questions

What is BLAS and why does it use multiple threads?

BLAS (Basic Linear Algebra Subprograms) is a library that performs matrix and vector operations. To speed up large computations, it automatically uses multiple CPU threads. This is great for interactive use, but in scheduled jobs it can cause resource contention.

Which environment variable should I set?

It depends on the BLAS implementation your NumPy/SciPy is linked to. Check np.__config__.show(). For OpenBLAS, use OPENBLAS_NUM_THREADS; for MKL, MKL_NUM_THREADS; for BLIS, BLIS_NUM_THREADS. Setting all three is a safe bet.

Can I set the thread count inside my Python code?

Yes, but you must set the environment variable before importing NumPy/SciPy. Alternatively, use the threadpoolctl package to change limits at runtime after imports.

What is a good thread count for a scheduled job?

Start with 1 or 2. If your job is the only one running on a machine with many cores, you can use more. The key is to avoid oversubscription—total threads across all concurrent jobs should not exceed the number of CPU cores.

Will limiting threads slow down my job?

Possibly, but often not significantly. Many operations are memory-bound, not CPU-bound, so extra threads don’t help. Test with your actual workload. If it’s too slow, increase the thread count gradually.

Does this affect all Python libraries?

It affects libraries that use BLAS, such as NumPy, SciPy, pandas (for some operations), and scikit-learn. Other libraries like pure Python code or those using their own threading won’t be affected.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →