API Tool Discovery: How AI Agents Find Hundreds

This guide explains how AI agents can systematically discover and catalog hundreds of API tools, turning a chaotic sprawl of endpoints into an organized, queryable inventory. You’ll learn a repeatable workflow that combines automated crawling, schema analysis, and semantic clustering—so your agent can find the right tool for any task without manual curation. No special hardware or paid services are required; just a development environment with Python and access to the APIs you want to catalog.

What You’ll Need

• Python 3.9+ installed on your machine or a cloud environment
• An API client library (e.g., requests, httpx) and a JSON parser
• Access to the APIs you want to discover—either public endpoints or your own API gateway
• A text editor or IDE for writing scripts
• Optional: a vector database or simple CSV file for storing results

Step-by-Step

1. Define your discovery scope. Decide which APIs to include—e.g., all endpoints under https://api.example.com/v1/—and what metadata matters (endpoint URL, method, parameters, response schema). Write this down; it guides every later step.
2. Fetch the API specification. Most modern APIs expose an OpenAPI/Swagger document at /swagger.json, /openapi.json, or /api-docs. Write a script that tries these common paths and saves the JSON. If no spec exists, skip to step 4.
3. Parse the specification. Extract every path and operation (GET, POST, etc.) from the OpenAPI document. For each, note the operation ID, summary, description, parameters, and request/response schemas. Store these as structured records—a list of dictionaries works fine.
4. If no spec, crawl the API surface. For undocumented APIs, use a crawler that starts from a known base URL, follows links in responses (e.g., next pagination fields), and records every distinct URL pattern it encounters. Be polite: add a delay between requests and respect robots.txt if it’s a public site.
5. Enrich with schema inference. For each endpoint, make a minimal test request (e.g., GET with no parameters) to capture the response JSON. Infer the field names, types, and nested structure. If the endpoint requires authentication, use a read-only API key.
6. Normalize and deduplicate. Merge records from different sources. Remove duplicate endpoints (same method+path). Standardize field names—e.g., convert user_id and userId to a common form. This is critical for later search.
7. Generate semantic embeddings. For each endpoint, create a text description combining the operation ID, summary, and parameter names. Use a sentence-transformer model (e.g., all-MiniLM-L6-v2) to convert these descriptions into vectors. This step makes semantic search possible.
8. Cluster similar tools. Group the vectors using a clustering algorithm (e.g., k-means or HDBSCAN). Each cluster represents a functional category—like “user management” or “payment processing.” Assign a human-readable label to each cluster by taking the most common keywords.
9. Build a search index. Store the endpoints, their metadata, and embeddings in a searchable format—either a vector database (like Chroma or FAISS) or a simple CSV with a keyword index. Implement a function that takes a natural-language query (e.g., “send an email”) and returns the top-k endpoints by cosine similarity.
10. Test and iterate. Run a set of sample queries against your index. Check if the results are relevant. If not, adjust the text descriptions (add more context), tweak the clustering parameters, or add a reranking step based on endpoint popularity or response time.

Common Mistakes

• Ignoring pagination and nested resources. Many APIs have endpoints that are only discoverable through related resources. For example, /users/{id}/posts won’t appear if you only crawl top-level paths.
• Over-relying on the OpenAPI spec. Specs can be outdated or incomplete. Always verify with live requests.
• Not handling authentication errors. If an endpoint returns 401, you might miss it entirely. Log these and mark them as “requires auth” rather than skipping.
• Using raw URLs as search keys. A URL like /v1/items/123 is not useful for semantic search. You need the textual description, not the path.
• Forgetting to update. APIs change. Your discovery process should be re-run periodically (e.g., weekly) to pick up new endpoints.

Tips

• Start with a small set of 10-20 APIs to validate your pipeline before scaling to hundreds.
• Use a consistent naming convention for operation IDs—many APIs follow verbNoun (e.g., getUser). This makes clustering easier.
• Store the raw OpenAPI JSON alongside your parsed records for auditability.
• When generating embeddings, include example values from the schema—they add context that improves search.
• Consider adding a “health check” field to each endpoint record, so your agent knows which tools are currently working.

Troubleshooting

Problem: The crawler misses endpoints. Check if the API uses JavaScript rendering—in that case, use a headless browser or look for the underlying XHR calls. Also ensure your crawler follows redirects and handles relative URLs correctly.

Problem: Duplicate endpoints appear after merging. Normalize URLs by stripping query parameters and standardizing path variables (e.g., replace /users/123 with /users/{id}). Use a hash of the method+path as the unique key.

Problem: Semantic search returns irrelevant results. Improve your text descriptions—add the endpoint’s purpose, not just its name. Also try a different embedding model (e.g., text-embedding-3-small) or increase the number of clusters.

Problem: Rate limiting blocks your discovery. Add exponential backoff and random delays. If possible, use a test environment or a dedicated API key with higher limits.

Problem: The OpenAPI spec is too large to parse. Stream the JSON and process paths incrementally. If memory is still an issue, split the spec by tag or resource.

Frequently Asked Questions

Can this work for private/internal APIs?

Yes, as long as you have network access and valid credentials. The same process applies—just ensure your crawler respects internal security policies.

Do I need a vector database?

No. For hundreds of endpoints, a simple CSV with a keyword search may suffice. Vector databases become necessary when you have thousands and need fast semantic search.

How do I handle APIs that require OAuth?

Use a service account or a read-only token. If that’s not possible, you can still catalog the endpoints from the OpenAPI spec without making live calls.

What if the API has no OpenAPI spec?

You’ll need to crawl the API manually or use a proxy that logs requests. Some tools like Postman can export collections that serve as a starting point.

How often should I re-run discovery?

It depends on how fast the API changes. For stable public APIs, monthly is fine. For internal APIs under active development, weekly or even daily might be necessary.

Can I use this to discover tools from multiple different API providers?

Absolutely. Just aggregate the records from each provider into a single index. Make sure to include a provider field in each record to avoid confusion.

Related

• Shokz OpenRun vs OpenRun Pro: Which to Choose
• AI SRE Tools Evaluation Checklist for Reliability Teams
• Enable Mojave on Older Mac: Full Compatibility Guide