OpenClaw Web Scraping: Automate Data Collection Without Code

The scraper I built for a competitor monitoring project in 2023 broke four times in six months. Each time, a CSS selector stopped matching because the target site had redesigned its product cards. OpenClaw eliminates that problem — the agent extracts by understanding what the data is, not by where it sits in the DOM tree.

Browser Skill vs Firecrawl: Which to Use When

OpenClaw has two primary scraping tools and choosing the wrong one for your use case costs you either speed or capability. Understanding the distinction is the first decision in any scraping pipeline design.

Firecrawl is a read-only content extraction service. You pass it a URL, it loads the page fully (including JavaScript execution), and returns the content as clean Markdown. It's fast — 2–5 seconds per page — and produces structured output that's easy to parse and store. Use firecrawl when you only need to read what's on a page without any prior interaction.

The browser skill gives the agent full interactive control. Navigate, click, fill forms, scroll, handle authentication. It's slower (5–15 seconds per page) and requires more configuration, but it handles everything firecrawl can't — authenticated sessions, click-to-load content, infinite scroll, and multi-step flows.

Here's where most people stop and pick one arbitrarily. Don't. The best scraping pipelines use both: browser for the interaction layer, firecrawl for the extraction layer once the page is in the correct state.

# Hybrid pipeline: browser to authenticate, firecrawl to extract
system: |
  Step 1: Use browser to navigate to {{LOGIN_URL}} and log in
          with credentials from environment variables.
  Step 2: Navigate to {{TARGET_URL}} and confirm you're authenticated.
  Step 3: Extract the full page URL from the browser.
  Step 4: Pass that URL to firecrawl for clean content extraction.
  Step 5: Write the extracted data to output/results-{{DATE}}.json

skills:
  - browser
  - firecrawl
  - file_write

Target Selection and Data Extraction Patterns

Telling OpenClaw what to extract is the primary configuration challenge. The more specific your extraction instruction, the more consistent your output. Vague instructions produce inconsistent results. Precise field definitions produce structured data you can actually use.

Bad extraction instruction: "Extract the product information from this page."

Good extraction instruction: "Extract the following fields from each product listing on this page and output as a JSON array: product_name (string), price (number, USD), availability (boolean), product_url (string), image_url (string). If a field is missing, use null."

The difference in output reliability between these two instructions is significant. The first produces prose summaries. The second produces a parseable JSON array with consistent field names on every run.

We'll get to rate limiting in a moment — but the extraction instruction is the single highest-leverage configuration decision in any scraping pipeline. Get it right before worrying about anything else.

For pagination — extracting data across multiple pages of results — structure your instruction to loop:

system: |
  Extract all job listings from {{START_URL}}.
  For each page:
  1. Extract listings as JSON with fields: title, company, location, salary, url
  2. Find the "Next page" link
  3. Navigate to the next page and repeat
  4. Stop when there is no next page link or you reach page 20

  Append all results to a single JSON array in output/jobs-{{DATE}}.json

Handling Dynamic JavaScript Sites

JavaScript-rendered content is where traditional scrapers fail and OpenClaw shines. Both firecrawl and the browser skill handle JS execution — the page loads completely before extraction begins.

The patterns that require specific configuration:

Infinite scroll. Content that loads as you scroll down requires the browser skill with explicit scroll instructions. "Scroll to the bottom of the page, wait 2 seconds for content to load, scroll again. Repeat until no new content appears. Then extract all visible listings." Firecrawl won't trigger scroll-loaded content — browser is required here.

Tab-based content. When target data is behind a tab click, use the browser skill to click the tab, wait for content to render, then extract. Firecrawl only sees the default tab state.

Modal and dropdown content. Price breakdowns, size options, or specification details that appear in modals need browser interaction before extraction. Click to open, wait for render, extract, close, proceed to next item.

API-sourced data via XHR. Some sites load data via background API calls that populate the visible UI. For these, consider intercepting the network request directly via the code execution skill rather than extracting from the rendered DOM — the raw JSON response is cleaner and more reliable than parsing rendered HTML.

Rate Limiting Your Scraping Pipeline

OpenClaw has no built-in rate limiting for scraping operations. Left unconstrained, the agent makes requests as fast as the model loop allows — which will get your IP blocked within minutes on most commercial sites.

Rate limiting lives in your system prompt. This is not optional for any production scraping pipeline.

system: |
  RATE LIMITING — follow these rules strictly:
  - Wait 3-5 seconds between each page request (randomize within this range)
  - Never make more than 100 requests per run
  - If you receive a 429 (rate limited) response, wait 60 seconds before retrying
  - If you receive a 403 (forbidden), log the URL and skip — do not retry
  - Stop the run entirely if you receive 3 consecutive 429 responses

  Log every request with: timestamp, URL, response status, items extracted

The randomized delay between requests (3–5 seconds rather than a fixed 3 seconds) is important. Fixed delays are easier to detect as automated behavior. Randomized delays within a human-plausible range reduce detection risk.

For high-volume scraping operations, add a request budget to your instructions. "Do not exceed 500 requests per day across all runs." This prevents scheduled pipelines from accumulating massive request volumes that attract attention and trigger IP bans.

Storing Scraped Results Reliably

Data you can't reliably access isn't data — it's noise. Output configuration for scraped data deserves as much attention as extraction configuration.

JSON with append mode is the recommended pattern for most scraping pipelines:

# Reliable output pattern
system: |
  Output format: JSON array with these exact fields per item:
  {
    "url": string,
    "title": string,
    "price": number or null,
    "extracted_at": ISO 8601 timestamp
  }

  File: output/data-{{DATE}}.json
  Mode: Append to existing file if it exists. Never overwrite.
  On completion: Write a summary line to output/run-log.txt:
  "{{TIMESTAMP}} — Run complete: {{COUNT}} items extracted, {{ERRORS}} errors"

The append mode instruction prevents overwriting previous results on subsequent runs. The run log gives you a record of pipeline health over time without having to open every output file.

For pipeline integration — feeding scraped data directly into a database or downstream system — use the api_call skill to POST results to a webhook endpoint after each page extraction rather than batching everything at the end. This reduces data loss risk if the pipeline fails mid-run.

Common Web Scraping Configuration Mistakes

• No rate limiting in the system prompt. Uncapped scraping gets blocked within minutes. Define delays and request budgets before the first run.
• Vague extraction instructions. "Get the product info" produces inconsistent results. Specify exact field names, types, and null handling behavior.
• Skipping robots.txt review. Always check before building — not after the first IP ban.
• Using browser when firecrawl suffices. Browser automation is 3–5x slower than firecrawl for read-only extraction. Default to firecrawl and escalate to browser only when interaction is required.
• Overwriting output files on each run. Use append mode or date-stamped filenames. Overwriting means a failed run destroys all previous good data.
• No error handling for missing fields. Sites don't always have all the data you expect. Specify null handling for missing fields or the agent will either error or produce incomplete JSON that breaks downstream parsing.

Frequently Asked Questions

What is the best way to scrape websites with OpenClaw?

Use firecrawl for fast, read-only extraction from accessible pages — it handles JavaScript rendering and returns clean markdown. Use the browser skill when you need to interact with the page first (login, click, scroll to load content). Combining both in one pipeline gives you the fastest and most reliable extraction.

Can OpenClaw scrape JavaScript-rendered websites?

Yes. Both the browser skill (via Playwright) and firecrawl handle JavaScript execution fully. The page loads completely — including fetch requests and dynamic rendering — before content is extracted. Static HTML scrapers miss this content; OpenClaw does not.

How do I avoid getting blocked when scraping with OpenClaw?

Add rate limiting instructions to your system prompt (2-5 seconds between requests), randomize request timing, and stay within reasonable request volumes. For sites with aggressive bot protection, add a residential proxy to your browser skill config. Respecting robots.txt also reduces blocking risk significantly.

How does OpenClaw store scraped data?

OpenClaw writes results using the file_write skill to JSON, CSV, or Markdown files. For structured data, instruct the agent to output JSON with defined field names. For pipeline integration, use the api_call skill to POST results directly to a database endpoint or webhook after each extraction run.

Is web scraping with OpenClaw legal?

Legality depends on the target site's terms of service, the jurisdiction, and how you use the data. Always review robots.txt and ToS before scraping. Publicly available data scraped for personal analysis is generally low-risk. Scraping behind authentication or for commercial resale carries higher legal risk — consult legal counsel for your specific use case.

What output formats does OpenClaw web scraping support?

OpenClaw can output scraped data as JSON, CSV, Markdown, plain text, or any custom format you define in your output instructions. JSON is recommended for structured data with multiple fields. Firecrawl natively returns Markdown, which is ideal for content-focused extraction like article text and documentation.

You now have the full scraping toolkit: skill selection logic, extraction instruction patterns, rate limiting configuration, and reliable output setup. Start with a firecrawl extraction against a single page you control. Verify the output matches your field specification. Then expand to pagination and rate limiting. A working single-page extractor is ready in under an hour — the production pipeline is just that pattern repeated at scale.