Cassandra Troubleshooting: Step-by-Step Guide

This guide walks you through diagnosing and fixing common Apache Cassandra issues, from connection failures and high latency to compaction problems and node crashes. You’ll learn a systematic approach to identify the root cause, apply the right fix, and verify that your cluster is healthy again.

What You’ll Need

• Access to the Cassandra nodes via SSH or a terminal
• Cassandra’s bin directory in your PATH (or full paths to nodetool, cqlsh, etc.)
• Basic familiarity with Linux command line and YAML configuration files
• Cluster credentials (username/password) if authentication is enabled
• Monitoring tools (optional but helpful): Prometheus, Grafana, or at least top and iostat

Step-by-Step

1. Identify the symptom and scope. Start by asking: Is the entire cluster down, or just one node? Are clients getting timeouts, connection refused errors, or slow responses? Check the application logs and note any error messages. This will guide your next steps.
2. Check node status with nodetool status. Run this on any node. It shows each node’s IP, state (UN = Up/Normal, DN = Down/Normal, UJ = Up/Joining, UL = Up/Leaving), and load. If any node is down, check its system.log for crashes or OOM kills.
3. Verify gossip and ring health. Use nodetool gossipinfo to see if nodes are exchanging gossip. If a node is unreachable, check network connectivity (firewall, DNS) and Cassandra’s listen_address and broadcast_address in cassandra.yaml.
4. Check for disk space and inode exhaustion. Run df -h and df -i on each node. Cassandra requires free disk for commit logs and compaction. If disk is full, the node will stop accepting writes. Delete old snapshots (nodetool clearsnapshot) or add capacity.
5. Inspect JVM heap usage. Run nodetool status and look at the “Load” and “Owns” columns. Use nodetool info to see heap usage. If heap is above 70–80% for long periods, you may need to tune heap_new_size and max_heap_size in cassandra-env.sh, or optimize queries to reduce memory pressure.
6. Examine latency and timeouts. If clients report timeouts, check nodetool tpstats for dropped mutations and pending tasks. High pending counts indicate a bottleneck. Review your consistency level: if it’s QUORUM or ALL, ensure enough replicas are available. Increase read_request_timeout_in_ms or write_request_timeout_in_ms in cassandra.yaml only after confirming the cause is not a slow disk or overloaded CPU.
7. Look at compaction and SSTable counts. Run nodetool compactionstats and nodetool cfstats (or tablestats). If there are many pending compactions, your disk I/O may be saturated. Check iostat -x for high %util. If needed, adjust compaction_throughput_mb_per_sec or switch to a different compaction strategy (e.g., LeveledCompactionStrategy) for write-heavy workloads.
8. Check for hot partitions or large rows. Use nodetool tablehistograms to see partition size distribution. A single huge partition can cause severe latency. Consider redesigning your data model to avoid unbounded growth, or use clustering columns to split data.
9. Review Cassandra logs. The log file is usually logs/system.log. Look for ERROR, WARN, or exceptions. Common entries include “Unable to complete request,” “GCInspector,” or “OutOfMemoryError.” Search for the first occurrence of the error to find the root cause.
10. Test with cqlsh. Connect locally on the node: cqlsh . Run a simple query. If it fails, the issue may be with authentication, client configuration, or the node itself. If it succeeds, the problem is likely on the client side or network.
11. Restart the node as a last resort. If nothing else works, perform a rolling restart: stop Cassandra (systemctl stop cassandra or kill), wait for it to shut down cleanly, then start it again. Avoid restarting all nodes at once; do one at a time to maintain availability.

Common Mistakes

• Ignoring the gossip state. A node can be up but not part of the ring (e.g., in “Leaving” status forever). Always check nodetool status and gossipinfo.
• Changing timeouts without understanding the cause. Increasing timeouts masks the problem; it does not fix it. Always investigate why the timeout occurs first.
• Running out of disk space on the commit log drive. The commit log is separate from data directories. If it fills, the node will crash. Monitor both locations.
• Using kill -9 to stop Cassandra. This can corrupt data. Always use nodetool stopdaemon or a graceful shutdown.
• Forgetting to check for Java version compatibility. Cassandra requires specific Java versions (usually Java 8 or 11). Using the wrong version can cause cryptic errors.

Tips

• Set up monitoring early. Tools like Prometheus + Grafana can show you trends before they become outages. Watch for increasing pending tasks, heap usage, and disk I/O.
• Use nodetool describecluster to quickly see the cluster name, snitch, and partitioner. Mismatches here can cause nodes to fail to join.
• Keep backups of cassandra.yaml and cassandra-env.sh before making changes. Use a version control system.
• Test changes in a staging environment first. Especially changes to compaction strategies, timeouts, or JVM settings.
• Regularly run nodetool repair to fix anti-entropy. This prevents data inconsistencies that can cause read timeouts.

Troubleshooting

If the steps above don’t resolve the issue, try these additional checks:

• If a node fails to start: Look at the system.log for the exact exception. Common causes include incorrect file permissions on data directories, a full disk, or a port conflict (e.g., 7000, 9042). Use netstat -tulpn to check.
• If clients can’t connect: Verify that rpc_address and broadcast_rpc_address are set correctly. Check firewall rules for port 9042. Test with telnet 9042.
• If performance is terrible: Check for GC pauses by enabling GC logging. Use nodetool gcstats to see recent GC activity. If you see long pauses, reduce heap usage or adjust GC settings.
• If data is inconsistent: Run nodetool repair -pr on each node. If repair fails, check for disk errors or network issues between replicas.
• If all else fails: Consult the Cassandra mailing lists or Stack Overflow, providing your system.log and cassandra.yaml (with secrets redacted).

Frequently Asked Questions

Why is my Cassandra node down?

Check nodetool status to see the state. If it’s DN, look at the system log for crashes, OOM, or disk full errors. Also check if the node was decommissioned accidentally.

How do I fix a node that is stuck in “Joining” state?

This often happens due to a mismatch in the cluster name, snitch, or seed list. Verify these in cassandra.yaml. Also ensure the node can communicate with the seed nodes on port 7000.

What does “Unable to complete request: timeout” mean?

It means the coordinator didn’t receive a response from replicas within the configured timeout. This could be due to slow disks, overloaded nodes, or a large partition. Use nodetool tpstats and tablehistograms to diagnose.

Should I increase timeouts to fix timeouts?

Only as a temporary measure. Increasing timeouts can hide underlying issues like data model problems or insufficient hardware. Fix the root cause first.

How can I tell if a node has disk I/O problems?

Run iostat -x 1 and look at the %util. If it’s consistently above 80%, the disk is a bottleneck. Also check nodetool compactionstats for pending compactions.

Why is my write latency high even with low load?

Check if you’re using a consistency level of ALL or QUORUM with only a few replicas. Also check for large partitions or frequent compactions. Use nodetool proxyhistograms to see where time is spent.

Related

• Galaxy S20 Live Captions: Availability and How to Use It
• Is Gain.gg Safe? Risks, Payments, and Account Security
• Reduce Wasted Tokens in AI Prompts and Outputs