Evaluating Crack Segmentation for Thin or Fragmented Cracks

This guide explains how to evaluate crack segmentation models when the cracks you care about are thin, broken, or barely visible. You’ll learn which metrics to trust, how to set up a fair evaluation, and how to avoid common pitfalls that make thin cracks look worse (or better) than they really are. You’ll need a set of test images with ground truth masks, a trained model, and a scripting environment like Python with common image and metric libraries.

What You’ll Need

• A test dataset of crack images with pixel-level ground truth masks (binary: crack vs. background).
• A trained crack segmentation model that outputs a probability map or binary mask for each image.
• Python with numpy, scikit-image (or OpenCV), and matplotlib for visualization.
• Optional: a library like scipy for morphological operations (skeletonization, dilation).

Step-by-Step

1. Prepare your ground truth masks consistently. Ensure all masks are binary (0 or 1), have the same resolution as the model output, and that the crack pixels are labeled as 1. If your model outputs probabilities, set a threshold (e.g., 0.5) to get a binary mask, but keep the probability map for later analysis.
2. Choose evaluation metrics that reflect thin-structure quality. Standard IoU (Intersection over Union) and Dice are useful, but they are sensitive to small shifts. For thin cracks, also compute the Boundary F1 score (BF score) and the F1 score on the skeleton (also called the crack detection rate). The skeleton F1 compares the centerlines of predicted and ground truth cracks, which is more forgiving to 1-pixel offsets. Implement these using morphological skeletonization.
3. Compute pixel-level metrics first. For each test image, calculate IoU, Dice, precision, recall, and F1. Precision = TP/(TP+FP), Recall = TP/(TP+FN). For thin cracks, recall is often more important than precision because missing a crack is worse than over-predicting a few pixels. Report the mean and standard deviation across all images.
4. Compute the skeleton-based metrics. Skeletonize both the ground truth and the predicted mask (using a thinning algorithm). Then compute the F1 score on the skeleton pixels: treat the skeleton of the ground truth as the reference and the skeleton of the prediction as the detection. This gives you a measure of how well the model captures the crack’s topology, not just its area.
5. Assess connectivity and fragmentation. Count the number of connected components in the ground truth and in the prediction. For each ground truth crack component, check if the prediction overlaps it (any pixel). Compute the detection rate = (number of ground truth components that overlap with the prediction) / (total ground truth components). This tells you how many cracks are completely missed.
6. Visualize failure cases. For each image, overlay the ground truth (green), prediction (red), and overlap (yellow). Look for patterns: are thin cracks systematically under-segmented? Are fragmented cracks broken into many pieces? Create a montage of the worst 10 images by IoU to guide improvements.
7. Run a sensitivity analysis on the threshold. If your model outputs probabilities, vary the threshold from 0.1 to 0.9 and plot IoU and skeleton F1. A good model will have a plateau of high performance. If performance drops sharply, the model is not confident about thin cracks.
8. Report results in a table. For each metric, give the mean and standard deviation. Also report the percentage of images where IoU is below a certain value (e.g., 0.5) to highlight difficult cases.

Common Mistakes

• Using only IoU. IoU is overly harsh on thin cracks because a 1-pixel shift can halve the overlap. Always pair it with skeleton F1 or boundary F1.
• Ignoring the ground truth quality. If the ground truth masks are noisy or have varying thickness, your evaluation will be unreliable. Ensure masks are consistent and ideally have a uniform 1-pixel width for thin cracks.
• Evaluating on downscaled images. Downscaling can make thin cracks disappear entirely. Test at the original resolution or at least at a resolution where the crack width is >2 pixels.
• Not separating fragmented cracks. A model that predicts a solid blob over a fragmented crack may get high IoU, but it fails to capture the fragmentation. Always evaluate connectivity separately.
• Using a fixed threshold without justification. The threshold should be chosen based on the model’s probability distribution, not arbitrarily. Use the sensitivity analysis to pick a threshold that maximizes skeleton F1.

Tips

• Use the skeleton F1 as your primary metric. It correlates better with human perception of crack detection than IoU.
• Dilate the ground truth skeleton by 1-2 pixels before computing the skeleton F1 to allow for small localization errors. This is common in boundary evaluation.
• Report both precision and recall. For thin cracks, recall is usually the bottleneck. If recall is low, the model is missing cracks; if precision is low, it’s over-predicting.
• Consider the crack width. If your dataset has cracks of varying thickness, group results by thickness (e.g., <3 pixels, 3-5 pixels, >5 pixels) to see where the model struggles.
• Use a validation set during development. Don’t evaluate on the test set repeatedly; use a validation split to tune hyperparameters, then evaluate once on the test set.

Troubleshooting

• If the skeleton F1 is very low even when IoU is moderate, your model may be predicting thick blobs that cover the crack but don’t follow its centerline. Try morphological thinning on the prediction before computing the skeleton, or adjust the loss function to encourage thinner predictions.
• If recall is low, the model is missing thin cracks. Lower the threshold, but check precision. If precision drops too much, consider data augmentation (e.g., random erasing) or a loss that penalizes false negatives more (e.g., Tversky loss).
• If the evaluation is inconsistent across runs, ensure you are using the same random seed for any stochastic operations (e.g., dropout) and that the ground truth masks are identical across runs.
• If your model outputs masks with holes (fragmentation), apply a post-processing step like morphological closing with a small kernel (e.g., 3×3) to connect nearby components, but be careful not to over-connect.
• If the ground truth masks are too thick, skeletonize them before evaluation to get a fair comparison. Thick masks inflate IoU because they are easier to overlap.

Frequently Asked Questions

Why is IoU not enough for thin cracks?

IoU measures area overlap. A thin crack has very few pixels, so a small shift in prediction can drastically reduce the overlap, giving a low IoU even if the crack is correctly detected. Skeleton-based metrics are more tolerant to 1-pixel offsets.

What is the best metric for fragmented cracks?

Use a combination of component detection rate (how many ground truth components are detected) and skeleton F1. This tells you both whether the model finds each fragment and how well it matches the topology.

Should I use a fixed threshold for binarization?

No. A fixed threshold may not be optimal for thin cracks. Perform a threshold sweep and choose the threshold that maximizes a combined metric like skeleton F1 or F1 on the validation set.

How do I handle cracks that are only 1 pixel wide?

1-pixel cracks are extremely challenging. Use skeleton F1 and allow a tolerance of 1-2 pixels by dilating the skeleton. Also, ensure your model’s output is not downsampled; use a stride of 1 in the final layers.

What if my ground truth masks are not consistent?

Inconsistent ground truth will make any evaluation unreliable. Re-annotate or at least filter out images where the mask quality is poor. Consistency is more important than having a large dataset.

Can I use the same metrics for 3D crack segmentation?

Yes, but you’ll need to adapt the skeletonization to 3D (e.g., using a 3D thinning algorithm) and compute the metrics on voxels instead of pixels. The principles remain the same.

Related

• GTA 6 vs GTA 5: Open World Changes
• SteelSeries Arctis Nova Pro vs Aeon Pro: Which Fits?
• Fix ‘Origin Could Not Connect to Server