Composio · AI Product Ops Intern take-home
A research agent crawled, searched and browsed 100 real apps requested by customers to find out whether each one could become an agent toolkit right now — and if not, exactly what's in the way. Below: the patterns first, then the full evidence table, then the honest accounting of what the agent got wrong and how that was caught.
Seven findings that came out of clustering all 100 rows — this is the point of the exercise, not the table itself.
docs.composio.dev/toolkits publicly lists all ~1,561 toolkit slugs Composio already ships, no account required. Matching all 100 apps against it: 67 already exist. The other 33 are the actual net-new opportunity list — see the dedicated section below, ranked by how buildable each one is.
Every single developer-infrastructure app (GitHub, Vercel, Cloudflare, Supabase...) and every productivity/PM app (Notion, Linear, Asana...) is buildable today with zero blockers. These are companies competing for developer mindshare — self-serve API access is table stakes for them.
Half of Finance/Fintech apps are blocked outright. The split is sharp: accounting & crypto tools (Stripe, QuickBooks, Xero, Binance) are wide open; corporate-card fintechs (Brex, Ramp) and data vendors (PitchBook, DealCloud) require you to already be a paying customer before any credentials exist.
Across 146 auth-method mentions (many apps support more than one), plain API key / Bearer token schemes (75) actually edge out OAuth2 (59). OAuth2 dominates mindshare; simple tokens dominate the actual dataset — and are usually faster to build against.
Roughly two in five apps already show some official, community, or vendor-claimed MCP server. Stripe, GitHub, Notion, Airtable, Linear, Apify, Firecrawl and Reducto shipped their own. The "build the first toolkit" opportunity is concentrated in the other 61.
Of the 30 apps that aren't cleanly buildable, 18 have a real API and a real signup flow — they just add an app review (Google/Meta/Pinterest), a use-case review (Plaid), or require you to already be a paying customer (Brex, Ramp, Devin) before it's usable. Only a minority are truly closed-door, contact-sales products.
No independently verifiable public docs could be found for either, even after crawling, searching and manually browsing. Both are flagged honestly as unresolved rather than guessed at — see the Honesty section below.
Green = buildable today, no blockers. Amber = buildable with real friction (a review gate, a paid plan, a non-standard auth scheme). Red = not buildable without a sales conversation or an existing paid relationship.
Cross-referenced all 100 hint URLs against docs.composio.dev/toolkits — a public page listing every toolkit slug Composio currently ships (~1,561 of them), no API key needed. 67 of the 100 researched apps are already there. These 33 aren't, ranked by how buildable each one is today — this is the list worth taking to an engineering backlog.
A four-stage pipeline. The crawler and the structured extraction are automated and re-runnable; two stages needed a human (or a real browser) because the crawler alone got fooled.
fetch_docs.pyPython + requests + BeautifulSoup hits all 100 hint URLs, strips boilerplate, saves cleaned text. Logs every status code and failure — a 403 is a finding, not a null.
Automatedfetch_retry.py91/100 succeeded on the first pass. For the 9 that failed or came back as an empty JS shell, retried against a more specific known-docs URL. Got to 94/100 with usable text.
Automatedcomposio_toolkit_slugs.jsonComposio's authenticated toolkit API (backend.composio.dev/api/v3) needs an account, which wasn't created on the user's behalf. But docs.composio.dev/toolkits is a public page listing all ~1,561 toolkit slugs with no auth at all — scraped and matched against all 100 apps by name/slug. 67 hits, 33 misses.
AutomatedClaude (this session) read every crawled file and, for ~20 apps where the crawl was too thin or ambiguous (niche fintechs, brand-new AI tools, apps behind bot-detection), ran targeted web searches to confirm auth model and access tier before writing the structured verdict.
Human-in-the-loopA real browser session (not the plain-HTTP crawler) re-visited 10 sampled apps' actual docs pages by hand, catching 2 real misses and 3 minor corrections — folded back into the shipped dataset.
Human-in-the-loopWhy extraction wasn't a fully unattended script: this machine had no spare Gemini/Anthropic API key to wire into an unattended extraction step. Rather than skip Composio's own resources entirely, stage 3 uses the one Composio surface that's genuinely public — the toolkit list docs.composio.dev serves to render its own docs site — instead of the authenticated API, which would have needed an account created on the user's behalf (a deliberate line not crossed; see the Honesty section). The reasoning in stage 4 was Claude, directly, inside this research session, reading real crawled text and real search results rather than working from memory. Stages 1, 2 and 3 are fully scripted and re-runnable by anyone with Python; the README explains how to wire in your own LLM key to make stage 4 unattended too.
Every row has an evidence link. Click a row's "details" to see the access note and blocker reasoning in full. Search and filter below.
| # | App | Auth | Access | Buildable? | In Composio? | MCP | Evidence |
|---|
Sampled 10 of the 100 apps — spanning high/medium/low confidence and both easy and hardest cases — and checked every claimed field against the real docs, by hand, in a real browser.
Read literally, "100% after correction" is true by construction — every miss found was fixed. The number that actually matters is the miss rate this sample surfaced: half needed some correction, one-fifth needed a real verdict change. Applied to the other 90 unsampled apps, that suggests a comparable number of undetected errors likely still exist — concentrated, based on this sample, in apps with thin/redirected crawl text (like Gumroad) and very new/small vendors that search doesn't index well (like iPayX).
Said plainly, per the assignment's own rule — an app defeating the research is a correct finding, not a failure to hide.
buildability: no with low/medium confidence rather than guessed at.zoho_cliq, facebook_ads) were slugs invented rather than looked up (they don't exist in the scraped list), and a third (Salesforce Commerce Cloud → the generic salesforce toolkit) matched the wrong product entirely: Composio's salesforce toolkit is documented as Sales/Service Cloud CRM, not the separate Commerce Cloud ecommerce API. Checking each generic-sounding match's actual toolkit description (not just its slug name) moved these 3 from "already covered" to "net-new," and also fixed LinkedIn Ads to point at the correct dedicated linkedin_ads slug instead of the generic linkedin one. Left in as the clearest illustration in this project of why the verification loop exists.