urlpipe.dev

URLpipe turns a URL into clean data — Markdown, rendered HTML, a screenshot, page metadata, an AI summary or keywords, console output, or a Lighthouse audit. Pages are fetched with a real headless browser, so JavaScript-rendered sites work. These tools are the URLpipe HTTP API — same arguments, same responses. The one addition is project_id, because this credential reaches the whole organization rather than one project. Start with list_projects to get a project_id. ASYNC IS THE DEFAULT. A call returns a token straight away and the work continues in the background; pass sync: true when you want to wait and be handed the result. Either way, get_result turns a token into the result. An async call also delivers to the project's webhook, if it has one configured. COST. Every call spends credits from one monthly allowance, and the operations are not priced alike: fetch_markdown, fetch_html, capture_screenshot and capture_console_errors cost 1, run_lighthouse_audit 2, extract_metadata 5, extract_keywords 15, summarize_page 17. get_usage has the live numbers. So: to READ a page, use fetch_markdown. It is the cheapest operation and the most compact input for a model — reach for summarize_page or extract_keywords only when the summary or the keyword list is the artefact you are producing, rather than a step towards reasoning you would do yourself. A repeated request is free. Results are stored for 30 days and reused when they are fresher than max_age (7 days by default), so re-fetching a page costs nothing unless you ask for a shorter window. A repeat of a request that is still running waits for it, also for free. list_requests shows what this project has already fetched. To retry a call safely, pass the same idempotency_key: within 24 hours it returns the first call's token and result rather than starting new work. scrape_url runs several operations off one page visit. That is not a discount — each operation is billed as usual — except when residential is on, where the exit surcharge is charged per visit rather than per operation. Every fetching tool takes labels: your own keys for the request, e.g. {"client": "acme"}. They come back with the result, and list_requests filters by them, so work done for one client or project can be found again.