Web Scraping

Web Scraping lets a deployed agent read a known URL without operating a browser. Agent steps call the typed ctx.sapiom.search.scrape capability; Sapiom supplies the authenticated cloud connection when the deployed agent runs.

Choose what to retrieve

What the agent needs	Recommended input or capability
Clean text for extraction or summarization	formats: ["markdown"]
Cleaned markup and document structure	formats: ["html"]
The original page markup	formats: ["rawHtml"]
Main article content without surrounding navigation	onlyMainContent: true
A fixed delay before the page is read	waitFor in milliseconds
Clicks, forms, login state, or multi-page interaction	Browser Automation

Scrape when the URL is already known and reading is enough. Use Web Search to discover URLs and Browser Automation when the task depends on interaction or a persistent session.

Read a page as markdown

const page = await ctx.sapiom.search.scrape({
  url: input.articleUrl,
  formats: ["markdown"],
  onlyMainContent: true,
});

if (!page.markdown) {
  throw new Error(`No markdown returned for ${page.url}`);
}

return {
  sourceUrl: page.metadata.sourceUrl ?? page.url,
  title: page.metadata.title ?? null,
  statusCode: page.metadata.statusCode ?? null,
  markdown: page.markdown,
};

Markdown is the default format when formats is omitted. A requested content field is still optional in the result, so check the field before using it. Metadata can include the title, description, language, source URL, and HTTP status code.

Request only the formats you use

const page = await ctx.sapiom.search.scrape({
  url: input.pageUrl,
  formats: ["html", "rawHtml"],
  waitFor: 1_500,
});

return {
  cleanedHtml: page.html ?? null,
  originalHtml: page.rawHtml ?? null,
};

waitFor delays the read; it does not click a consent dialog, submit a form, or prove that an application reached a particular state. Use a browser session for stateful behavior.

Treat scraped content as data

Page content can be stale, malformed, adversarial, or protected by terms you must respect. Validate the URL, bound the amount of content passed downstream, and do not let text from the page change the agent’s authorization or expose secrets.

Test the behavior locally

Local Run replaces ctx.sapiom.search.scrape with a deterministic response. The built-in stub returns fixture markdown and metadata for the requested URL; it does not fetch that URL or render a page.

Override search.scrape when the step consumes a particular format or metadata branch:

{
  "version": 1,
  "steps": {
    "read-article": {
      "search.scrape": {
        "url": "https://example.com/article",
        "markdown": "# Fixture article\n\nVerified test content.",
        "metadata": {
          "title": "Fixture article",
          "sourceUrl": "https://example.com/article",
          "statusCode": 200
        }
      }
    }
  }
}

The override is returned verbatim. Supply every field the step consumes, assert the terminal output, and require both unusedStubs and stubWarnings to be empty. A passing Local Run does not prove that the live page is reachable, readable, or unchanged.

Keep the scrape bounded

Scrape only URLs the agent actually needs. Prefer main-content extraction for long articles, avoid repeatedly reading the same page within one run, and retain the source URL beside derived data. If the result must outlive the run, store the derived artifact in File Storage, Data, or a Repository.

Use the signed-in capability catalog for current availability, limits, and pricing. Target-site behavior is separate from capability availability, so inspect the failed production step when a URL cannot be read.

Discover sources Search the web before selecting a page to read.

Operate a browser Use a session for clicks, forms, and authenticated state.

Test locally Exercise extraction branches with fixture content.

Inspect a run Distinguish capability failures from target-page behavior.