Azure DevOps and AI-Generated Code: What to Measure

In Azure DevOps, you can measure how AI-generated code moves through your delivery process by tracking review, build, test, security, and deployment outcomes—not by treating “AI-written” as a quality score. The useful measures are observable engineering results, such as review time, escaped defects, test failures, and delivery frequency, compared consistently with a clear baseline.

What measurement can—and cannot—tell you

Azure DevOps provides work tracking, source control, pull requests, pipelines, test results, and deployment records. Together, these can show what happened to a change from ticket to production. They do not, by themselves, reliably establish how much of a change was written by an AI assistant, whether a developer accepted a suggestion unchanged, or whether AI caused a particular outcome.

That distinction matters. A pull request may contain code drafted with an assistant, code copied from an existing project, and substantial human edits. Unless your organization deliberately records AI use in a consistent, privacy-conscious way, Azure DevOps data generally describes delivery activity rather than authorship. Even when use is recorded, the relationship between use and outcomes is not proof of causation.

Start with a practical question: did a defined group of changes have better, worse, or unchanged delivery outcomes than comparable work? Avoid vanity measures such as lines of code produced or suggestions accepted. They reward volume and say little about whether software works, is maintainable, or helps users.

Measures available in Azure DevOps

Use measures that are already meaningful for your team, and define each one precisely. Depending on how your project is configured, useful signals include:

Pull-request flow: time from opening to completion, review turnaround, number of review cycles, and how often changes are reopened or abandoned. Separate waiting time from active work where possible.
Build and test outcomes: pipeline success rates, failed jobs, test failures, flaky-test frequency, and time spent diagnosing failures. A failed test is a signal to investigate, not automatically a defect in the new code.
Change size and rework: files or lines changed, follow-up fixes, and revisions after review. Interpret size carefully: generated formatting, dependency updates, and refactoring can distort comparisons.
Defects and reliability: bugs linked to a change, incidents, or regressions found after deployment. These require consistent defect reporting and a defined observation window.
Delivery: deployment frequency, lead time from an agreed starting point to deployment, and recovery time after a service problem. These measures need a stable definition and reliable deployment records.
Security and maintainability checks: findings from configured analysis tools, dependency alerts, and policy failures. A count alone is not a severity-adjusted assessment, and tool coverage may vary between repositories.

Azure DevOps dashboards, Analytics views, and queries can help summarize work-item, repository, pipeline, and test information, subject to the services and permissions enabled in your organization. Some comparisons require joining records or exporting data, and some outcomes—such as customer impact—may live in monitoring or support systems outside Azure DevOps.

Build a fair comparison

A useful comparison begins with a baseline, not a claim that AI code is inherently faster or better. Choose a period before a workflow change, then compare it with a later period using the same definitions. Where possible, compare similar repositories, teams, change types, and levels of experience. A small feature and a large migration should not be treated as equivalent observations.

Choose a narrow question. For example: “Did review-to-merge time change for routine bug fixes after the team adopted an AI coding assistant?”
Set the population and dates. Specify repositories, work-item types, branches, and measurement windows. Exclude or separately report unusual events such as a release freeze.
Define the metric. State exactly when the clock starts and stops, how reopened pull requests count, and whether paused or draft time is included.
Record context. Note changes in staffing, review policy, test coverage, release cadence, or pipeline configuration that could affect results.
Check quality alongside speed. Pair faster completion with defect, rework, test, and security signals. A speed improvement that coincides with more regressions needs closer examination.

If you want to distinguish AI-assisted work, use a clearly explained, voluntary or policy-approved tagging method—such as a work-item field or pull-request label—and define what qualifies. Avoid inferring authorship from commit text or code patterns. Do not collect more personal data than the evaluation requires, and follow your organization’s privacy and retention rules.

Interpreting results without overclaiming

Delivery metrics are affected by many factors. A shorter pull-request cycle could reflect smaller changes, fewer reviewers, reduced scrutiny, or an unusually quiet period. Higher build success could result from infrastructure changes rather than better code. A rise in defects may reflect improved reporting rather than a real decline in quality.

Report distributions, not only averages. Median review time, for example, is less distorted by a few exceptionally long waits. Include the number of changes measured and the time window. If the sample is small, say so: a handful of pull requests cannot establish a dependable trend. Look for consistent movement across several measures and inspect representative changes to understand what the numbers mean.

Do not compare teams as if they had identical work. Differences in architecture, ownership, test maturity, customer exposure, or approval requirements can make raw rankings misleading. Use metrics to prompt investigation and improve a process, not to score individual developers or make unsupported productivity claims.

Common measurement mistakes

Using lines of code as productivity: more code can mean more maintenance, unnecessary complexity, or generated boilerplate.
Counting AI usage as success: adoption or acceptance rates show activity, not correctness or value.
Ignoring the denominator: ten failures among twenty builds do not mean the same thing as ten among a thousand. State counts and rates together.
Changing definitions midstream: a new branch policy, test suite, or timestamp rule can create an apparent trend that is only a measurement change.
Optimizing one number: targets can encourage teams to split pull requests artificially, skip tests, or defer fixes. Balance speed with quality and operational outcomes.

Frequently Asked Questions

Can Azure DevOps tell whether code was written by AI?
Not reliably from ordinary work items, commits, pull requests, or pipeline results. If attribution is needed for a study, agree on a consistent recording method rather than guessing from the code.

What is the best metric for AI-generated code?
There is no single best metric. Pair a delivery measure, such as lead time or review time, with quality signals such as rework, defects, test results, and security findings.

Can I measure whether AI makes developers more productive?
You can compare defined workflow outcomes before and after adoption, but the comparison does not isolate AI’s effect unless other influences are controlled. Treat results as evidence to investigate, not automatic proof of cause.

Does a successful pipeline prove generated code is safe?
No. A green pipeline means only that the configured checks passed. It does not guarantee adequate test coverage, absence of vulnerabilities, or correct behavior in every production situation.

Should AI use be tracked per developer?
That depends on the purpose, organizational policy, and privacy requirements. Team-level or project-level analysis may answer workflow questions with less personal data; avoid using incomplete tracking to rank individuals.

Which Azure DevOps data should I start with?
Start with pull-request timestamps, pipeline and test outcomes, linked work items, and deployment records if consistently maintained. Add defect or incident data only when its definitions and links are dependable.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →