RAG Web Browser: 3 Practical Use Cases for Clean Markdown Extraction

Direct answer

Search the web or fetch direct URLs and return clean markdown for LLM/RAG pipelines. filters: domainAllowlist/Blocklist, minTextLength, keywordsAnyOf. No login, no cookies. To apply this capability effectively, pick a documented pattern, establish your core acceptance standard first, and configure a focused run to produce reviewable output for your team.

Define your operational parameters early

Before launching any collection batch, write down the specific objective and the exact criteria a record must meet to enter your pipeline. Distinguish between mandatory filters that govern inclusion and secondary attributes that simply provide context. Establish a clear exclusion rule and a borderline category to prevent ambiguous items from skewing your dataset.

Practical use cases

These use cases come from RAG Web Browser's published documentation. Each is expanded into an operating pattern so the RAG Web Browser output has a purpose beyond collection.

Use case 1: RAG ingestion

Outcome: pull fresh top-N Google results for a topic, hand markdown to your embedder.

Question to answer: Applying the documented rule as written, which records clearly pass, which clearly fail, and which need a human call?

Configure: Start with maxResults (When query is a search term, how many top organic results to fetch.), startUrls (Direct URLs to fetch. When set, the actor skips Google search and fetches each URL directly.), scrapingTool (raw-http is fast (curlcffi with chrome131 impersonation). browser-playwright runs a headless browser for JS-heavy pages.). Use the narrowest RAG Web Browser values that represent the real task, keep the first result set small, and record why each selected input matters to the RAG Web Browser outcome.

Working method: Write the acceptance rule down before the first record is reviewed, apply it consistently across the batch, and change only one rule or input between batches so any shift in the result has a clear cause.

Deliverable: Create a reviewed queue where every record keeps its raw form alongside the rule that accepted, excluded, or flagged it as uncertain. Include the RAG Web Browser source identifier and the collected fields behind every RAG Web Browser decision.

Stop condition: Pause when the acceptance rule cannot be applied consistently, required fields are frequently missing, or two reviewers reach different conclusions on the same record. Fix the RAG Web Browser question, comparison rule, or configuration before expanding the RAG Web Browser run.

Use case 2: Competitive monitoring

Outcome: domainAllowlist of competitor domains, scrape their blogs weekly.

Question to answer: Between this run and the last, what actually moved, and does the shift clear the bar for action?

Configure: Start with domainAllowlist (Only emit pages whose host contains one of these substrings (case-insensitive).), domainBlocklist (Drop pages whose host contains one of these substrings.), minTextLength (Drop pages with fewer than N characters of extracted text. Filters out thin / parking-page results.). Use the narrowest RAG Web Browser values that represent the real task, keep the first result set small, and record why each selected input matters to the RAG Web Browser outcome.

Working method: Store every run under its collection date, key records by a durable identifier rather than position or display text, and only escalate a difference once it has been confirmed against two consecutive runs.

Deliverable: Create a change log entry per run listing what appeared, what disappeared, and what changed enough to matter. Include the RAG Web Browser source identifier and the collected fields behind every RAG Web Browser decision.

Stop condition: Pause when the identifier used to match records is unstable, the collection window shifts between runs, or a formatting change is being read as a real change. Fix the RAG Web Browser question, comparison rule, or configuration before expanding the RAG Web Browser run.

Use case 3: Reference enrichment

Outcome: feed each citation URL from a paper into the actor for clean text extraction.

Question to answer: Once near-duplicates are grouped, which stories actually add something new for the audience?

Configure: Start with startUrls (Direct URLs to fetch. When set, the actor skips Google search and fetches each URL directly.), query (Free-text search query OR a single URL. When startUrls is also set, both are processed.), htmlTransformer (none keeps full HTML; readable-text extracts the main article text (best for blog posts, news).). Use the narrowest RAG Web Browser values that represent the real task, keep the first result set small, and record why each selected input matters to the RAG Web Browser outcome.

Working method: Cluster near-duplicate coverage first so the queue is not dominated by one story, then score what remains on audience fit and freshness before making any selection.

Deliverable: Create a ranked shortlist of items with the angle, source, and reason each one earned a place, plus the duplicates it was chosen over. Include the RAG Web Browser source identifier and the collected fields behind every RAG Web Browser decision.

Stop condition: Pause when the same handful of sources dominate every batch, novelty cannot be judged against recent history, or duplicate stories are being counted as separate items. Fix the RAG Web Browser question, comparison rule, or configuration before expanding the RAG Web Browser run.

Build one workflow from start to finish

Open RAG Web Browser and select your target operating pattern.
Document your primary objective, audience, inclusion rules, and expected deliverables before touching any configuration setting.
Choose the minimum required input parameters needed to address your specific operational question.
Execute a preliminary small batch and categorize every returned item as accepted, excluded, or uncertain.
Inspect missing mandatory fields and identify duplicates prior to calculating final dataset metrics.
Adjust a single parameter or rule, then compare the new output against your previous sample.
Store your validated configuration settings and an expected output fixture for future reference.
Integrate downstream automation only after your dataset can be consistently reviewed by team members.

Configure the available input parameters

The documented input schema includes the following settings:

query (string) - Free-text search query OR a single URL. When startUrls is also set, both are processed. Default: "what is retrieval augmented generation".
startUrls (array) - Direct URLs to fetch. When set, the actor skips Google search and fetches each URL directly. Default: [].
maxResults (integer) - When query is a search term, how many top organic results to fetch. Default: 3.
outputFormats (array) - Which textual representations to include per page. Combine any of markdown, text, html. Default: ["markdown"].
requestTimeoutSecs (integer) - Per-page HTTP timeout. Default: 40.
scrapingTool (string) - raw-http is fast (curl_cffi with chrome131 impersonation). browser-playwright runs a headless browser for JS-heavy pages. Default: "raw-http".
removeElementsCssSelector (string) - CSS selector matching elements to remove before extraction (e.g. nav, footer, aside, .ad). One comma-separated string. Default: "nav, footer, aside, script, style, .ad, .advertisement, [aria-hidden='true']".
htmlTransformer (string) - none keeps full HTML; readable-text extracts the main article text (best for blog posts, news). Default: "readable-text".
desiredConcurrency (integer) - How many URLs to fetch in parallel. Set 0 for auto. Default: 5.
maxRequestRetries (integer) - Retries on transient HTTP failures (429/5xx). Default: 2.
dynamicContentWaitSecs (integer) - When scrapingTool=browser-playwright, how long to wait for JS-loaded content. Default: 5.
removeCookieWarnings (boolean) - When true, dismisses common cookie/consent dialogs in browser mode. Default: true.
useApifyProxy (boolean) - Route requests through Apify proxy. Recommended for Google search and some target sites. Default: true.
domainAllowlist (array) - Only emit pages whose host contains one of these substrings (case-insensitive). Default: [].
domainBlocklist (array) - Drop pages whose host contains one of these substrings. Default: [].
minTextLength (integer) - Drop pages with fewer than N characters of extracted text. Filters out thin / parking-page results.
excludeContentSelectors (array) - Additional CSS selectors to strip - extends removeElementsCssSelector. Default: [].
keywordsAnyOf (array) - Only emit pages whose extracted text contains at least one of these keywords (case-insensitive). Default: [].

Quality controls before deployment

Maintain a small initial dataset size to allow thorough manual inspection.
Define relevance, exclusion, and uncertainty criteria before running scheduled jobs.
Archive raw records so transformation errors can be rectified later.
Deduplicate outputs based on durable identifiers rather than display text.
Leave missing optional fields unpopulated rather than inventing placeholder values.
Set up alerts for unexpected empty datasets or run failures.
Verify actor documentation when upstream target structures change.
Keep editorial ratings separate from collected source data.
Require documented justifications for top-ranked and bottom-ranked records.
Restrict customer-facing communications to verified data points.

Frequently asked questions

How should I validate my initial run?

Execute a single representative input query and inspect every returned record against your defined decision criteria before scaling up batch volume.

How are missing fields handled during extraction?

Unpopulated optional values are omitted in adherence to clean output rules, preventing artificial defaults from contaminating your dataset.

When is it appropriate to expand run scope?

Scale up volume only after small sample sets successfully pass relevance, completeness, and duplicate checks while maintaining consistent configurations.

What conditions require a complete workflow review?

Initiate a review whenever target site structures, input parameters, or failure rates shift in a way that could compromise downstream decision quality.

Resources

RAG Web Browser on Apify