Ask anyone who has wired a large language model into a real product and they will tell you the same thing. The model is rarely the hard part. Getting clean text out of the web is. Fetch a page with a plain HTTP request and you get a wall of navigation menus, cookie banners, inline scripts and tracking pixels. Strip the tags and what is left is technically readable and practically useless.
Firecrawl exists to fix that specific problem. It is a scraping and crawling service, open source at its core with a hosted API on top, that takes a URL and hands back content shaped for LLMs: markdown, structured JSON, or a simple list of links. No headless browser to babysit. No proxy rotation to configure at 2am.
What Firecrawl actually does
Send a URL to the API and you get the page’s main content, cleaned up. The service renders JavaScript, waits for the page to settle, strips boilerplate, and returns one of several output formats depending on what you ask for:
- Markdown that keeps headings, lists and tables intact
- HTML or raw HTML if you need the original markup
- Links found on the page, deduplicated
- Screenshots for visual checks and regression testing
- Structured JSON extracted against a schema you define
- Change tracking that diffs the page against its last scrape
Everything unglamorous happens behind that call: proxy rotation, retries on failed responses, PDF parsing, cookie handling and robots.txt compliance (which you can override, though doing so carries legal implications worth thinking through).
Four endpoints cover most of what you will need
/scrape: one page, right now
The workhorse. One URL in, one document out, usually within a few seconds. If you only ever use one endpoint, use this one. It accepts options like onlyMainContent to drop sidebars and footers, waitFor to give slow JavaScript time to render, and formats to control what comes back.
/crawl: a whole site, politely
Give it a starting URL and it walks the site, respecting a depth limit and a page cap you set. Three parameters do most of the scoping work: includePaths and excludePaths for URL patterns, and limit for the hard ceiling on pages. A crawl of a 400-page documentation site typically finishes in a couple of minutes.
/map: just the URLs
Mapping is the fastest and cheapest call in the toolkit. It returns every URL Firecrawl can discover on a domain without scraping any of them. Run a map first when you are planning a crawl. You will spot the endless tag archives before you pay to scrape them.
/extract: schema-shaped data
This is where things get interesting. You pass a prompt and a JSON schema, and Firecrawl returns typed objects rather than text. Ask for product name, price, currency, stock status and rating across fifty product pages, and you get fifty clean records instead of fifty blobs of prose you have to parse yourself.
Why markdown quietly outperforms raw HTML
A typical news article is around 90KB of HTML and maybe 5KB of actual prose. Feeding raw HTML to a model burns tokens on divs that carry no meaning, and it dilutes the signal in your embeddings. Retrieval quality drops because your chunks are padded with “Skip to content” and newsletter prompts.
Markdown solves this at the source. Headings survive as headings, which gives you natural chunk boundaries. Tables stay tables instead of collapsing into runs of pipe characters. What reaches your vector store is the content a human would actually read.
A pipeline that holds up in production
Map the domain to see what exists. Decide which URL patterns matter. Fan out scrape calls across those pages in parallel. Store the markdown next to the source URL and a content hash, then compare hashes on the next run so you only re-embed pages that genuinely changed. Extraction jobs run separately, against the pages where you need fields rather than text.
Keep the fetch and the embed as distinct steps. When someone asks why a document went missing from the index three weeks later, you want to know whether the scrape failed or the embedding did.
Where it struggles
Firecrawl is not a master key to the entire web. A few honest limits:
- Logged-in content. Anything behind a session needs your own browser automation with stored credentials.
- Interaction-gated pages. Content that appears only after a click, a filter selection or infinite scroll requires Playwright or Puppeteer driving the session.
- Aggressive anti-bot systems. Some sites block datacentre IP ranges outright, and no amount of retrying fixes that.
- Cost at scale. Crawling 100,000 pages is a real bill, not a rounding error. Budget before you start.
- Complex schemas. Extraction is an LLM task, and LLM tasks hallucinate. Validate required fields and add a confidence check.
Credits, pricing and self-hosting
Firecrawl bills in credits. A standard scrape costs one credit, extraction and browser-heavy renders cost more, and mapping is cheap. There is a free allowance to test with, followed by tiered monthly plans that scale with volume. The exact numbers shift, so check the pricing page rather than trusting a blog post written six months ago.
If you would rather not route requests through someone else’s servers, the project is open source and runs in Docker. You supply the browser and the compute. One caveat: the hosted version bundles managed proxies and anti-bot handling that the self-hosted build does not include, so your own deployment will hit more walls on tougher sites.
There is also an MCP server, which lets assistants like Claude and Cursor call Firecrawl directly. It is a surprisingly handy way to pull live documentation into a coding session.
How it compares to rolling your own
Playwright or Puppeteer gives you total control: sessions, clicks, file downloads, custom waits. It is cheaper per page at high volume and roughly ten times more work to maintain. BeautifulSoup is fine for static HTML from simple, well-behaved sites. Jina’s reader endpoint converts a single URL to markdown in one request, which makes it a great quick sanity check.
Firecrawl sits in the middle. You trade a per-page fee for not building and maintaining a scraping stack. For most teams shipping an AI feature, that trade is obvious: the engineering time saved is worth more than the credits spent.
Settings that separate a good crawl from a wasted one
Scope before you spend. Run map first, then write excludePaths for tag archives, author pages, pagination and anything with a query string. Set a limit even when you are sure you do not need one.
Turn on onlyMainContent unless you specifically want navigation for context. Add waitFor values of two to five seconds on sites that render client-side, then check a handful of results by hand before launching a 5,000-page job.
Cache aggressively. Hash the returned markdown and skip re-embedding anything unchanged. On a documentation site that updates twice a week, that alone can cut your costs by an order of magnitude.
Test extraction schemas on twenty pages before you run them across twenty thousand. If the model fumbles a nested field at that sample size, it will fumble it at every larger size, and you will discover it after the invoice arrives.