Building a Multi-Portal Real Estate Data Pipeline with Apify

Building an automated multi-portal real estate data pipeline across geographic sectors requires orchestrating specialized web scrapers tailored to each target platform. By deploying Realtor Scraper for US residential data, Idealista Scraper for Southern European markets, and Crexi Real Estate Scraper for US commercial properties, data teams can ingest standardized listing records into a unified warehouse. This multi-portal approach normalizes disparate attributes such as US square footage, European square meters, tax histories, and commercial net operating income into clean schemas for investment analysis and market research.

Each scraper in this pipeline solves specific access mechanisms and extraction patterns. Realtor Scraper accesses Realtor.com residential data using automatically managed residential proxies. Idealista Scraper retrieves property records across Spain, Italy, and Portugal directly from search cards while bypassing blocking via proxy fallback paths. Crexi Real Estate Scraper extracts US commercial listings directly from the Crexi public search API without requiring proxies or browser instances. Together, these tools form an ingestion engine for global real estate intelligence.

Practical use cases

Use case 1: US residential tax and risk assessment

Outcome: A structured dataset comparing asking prices, tax assessments, historical sale prices, and environmental risk factors across target US residential markets. Question to answer: Which ZIP codes in a target state offer the best ratio between asking price, tax valuation, and low environmental risk ratings? Configure: In Realtor Scraper, set search location to the target market (for example, Houston, TX), select BUY for search mode, enable full details, and cap maximum properties at 500. Working method: Run the scraper to query Realtor.com search pages and fetch individual listing detail pages. Extract core property features alongside tax history, flood factor scores, wildfire risk ratings, and nearby school details. Parse assessment values and tax amounts into year-over-year series for financial comparison. Deliverable: A cleaned tabular dataset or relational database table mapping property IDs to listing price, tax assessment trends, school ratings, and environmental risk scores. Stop condition: The run reaches 500 scraped properties or exhausts all 20 search pages for the specified location.

Use case 2: Southern European rental price monitoring

Outcome: A multi-country market database tracking price per square meter and property features across major cities in Spain, Italy, and Portugal. Question to answer: How do rental price distributions and price-per-square-meter metrics compare across central neighborhoods in Madrid, Rome, and Lisbon? Configure: Set up Idealista Scraper with direct search URLs or location slugs for target cities (madrid-madrid, roma-roma, lisboa), set operation to rent, set property type to homes, and cap total items at 1,800 per run. Working method: Ingest property search-result cards sequentially across Spain, Italy, and Portugal. Extract monthly rent, property size in square meters, price by area, floor level, elevator access, and neighborhood boundaries directly from card payloads. Deliverable: A consolidated dataset exported to CSV or JSON format containing standardized European rental listings, location identifiers, and unit amenities. Stop condition: The process completes when 1,800 property cards are collected or all 60 result pages for the active search query are retrieved.

Use case 3: US commercial inventory and yield screening

Outcome: A catalog of active US commercial assets filtered by property type and price threshold, complete with lease terms and net operating income metrics. Question to answer: Where are commercial office and retail assets priced between $1,000,000 and $10,000,000 currently available in major commercial states? Configure: Configure Crexi Real Estate Scraper with property types set to Office and Retail, locations set to target state codes (such as TX and FL), minimum price set to 1000000, maximum price set to 10000000, and detail enrichment enabled. Working method: Query Crexi public search API to identify matching assets across target states. Trigger Crexi public detail API for each asset ID to pull net operating income, lease terms, zoning classifications, and year built. Apply post-scraping filtering to organize listings by city and calculated yield metrics. Deliverable: A structured JSON or database pipeline payload containing commercial asset identifiers, location coordinates, property square footage, asking price, and full financial detail maps. Stop condition: The run halts when all matching commercial properties in the specified state queries are extracted or the maximum property threshold is reached.

Five-step pipeline implementation workflow

Scope target markets and portal routing: Map your analytical requirements to the correct source scraper. Direct US residential queries to Realtor Scraper, European residential or commercial queries to Idealista Scraper, and US commercial investment queries to Crexi Real Estate Scraper.
Configure search boundaries and performance parameters: Set location parameters, price filters, and pagination limits. Decide whether detail-page enrichment is required based on speed and bandwidth requirements. For quick residential price scans, disable detail enrichment; for deep underwriting, enable detail page execution.
Execute ingestion and collect automated outputs: Run the scrapers on a scheduled basis using the Apify platform. Scrapers process direct search URLs or location inputs, handling internal pagination, API calls, and proxy rotation automatically.
Clean and normalize attribute schemas: Normalize incoming fields across platforms. Standardize property sizes into unified metrics, handle unpriced commercial listings marked as negotiable, and map listing status codes into standardized pipeline statuses.
Store data and feed downstream systems: Export clean output datasets into your primary database. Downstream analytical tools, business intelligence dashboards, and search engine indexes can then process fresh real estate data for internal reporting or AI referral traffic discovery.

Actor roles across property sectors

Understanding the role of each tool ensures proper pipeline construction:

Realtor Scraper serves as the primary US residential ingestion engine. It extracts property records from Realtor.com, capturing listing prices, bedroom and bathroom counts, property photos, agent contacts, nearby school ratings, historical tax records, and environmental risk scores for flood, wildfire, and noise.

Idealista Scraper handles European property data extraction covering Spain, Italy, and Portugal. It retrieves listing records from search result cards, providing prices in EUR, price per square meter, floor level, room counts, outdoor features, and agency contact information.

Crexi Real Estate Scraper covers commercial real estate across all 51 US states and 14 commercial asset classes, including Office, Retail, Industrial, Multifamily, Land, Hotel, and Self Storage. It queries public endpoints directly to collect pricing, property square footage, geocoordinates, brokerages, and optional full detail key-value maps including net operating income and zoning.

Operational limits and data quality boundaries

Each portal enforces specific operational ceilings and structural constraints that dictate pipeline design:

Search pagination ceilings limit the total items retrievable per single search query. Realtor.com exposes up to 42 properties per search page across a maximum of 20 pages, placing a hard cap of 840 properties per search location. Idealista caps search results at approximately 1,800 listings across 60 pages per query. Crexi Real Estate Scraper allows querying up to 100,000 properties per run, though filtering is bounded at the state level.

Detail enrichment performance tradeoffs govern execution speed. Disabling detail enrichment in Realtor Scraper yields a significantly faster run by taking listing summaries directly from search pages at approximately 42 properties every 5 to 8 seconds, whereas full detail enrichment fetches 5 to 10 properties per minute to capture tax and risk histories. In Idealista Scraper, extraction focuses on search-result cards rather than individual detail pages, keeping runs under a minute for typical queries. In Crexi Real Estate Scraper, detail enrichment adds one public API request per asset to retrieve descriptions, subtypes, and net operating income.

Attribute disclosure varies by listing. Missing fields occur when sellers or portals omit data. For instance, Crexi listings without numeric prices omit the price field unless price filters are applied. Idealista search cards expose core fields such as price, size, and features, while detailed fields like exact geocoordinates or full property history are omitted when not present in card payloads.

Production reliability strategies

To maintain clean data flow in production, follow these key practices:

Geographic partitioning: Because portal search results are capped, divide large geographic regions into smaller sub-markets. For Idealista, split large metropolitan areas into district or municipality search slugs. For Crexi, execute state-level queries and post-filter by city in your data pipeline. For Realtor.com, target specific ZIP codes or neighborhood strings.

Proxy management: Match proxy settings to scraper requirements. Realtor Scraper automatically utilizes an internal residential proxy to handle site access restrictions. Idealista Scraper uses a mirror-first fetch path with Apify Proxy enabled by default for fallback when cloud traffic is restricted. Crexi Real Estate Scraper calls the Crexi public search API directly and operates efficiently from standard datacenter IPs without requiring proxies.

Downstream integration and indexing: Store ingested records in structured relational tables or document stores. When publishing market research or property insights derived from these pipelines, ensure clear canonical metadata so search crawlers and answer engines indexing your platform can route AI referral traffic directly to your reporting portals.

Frequently asked questions

How does Realtor Scraper handle access restrictions on Realtor.com?

Realtor Scraper automatically uses a residential proxy to access Realtor.com. This access handling is built directly into the actor, requiring no manual proxy configuration from the user.

Can Idealista Scraper extract full property detail pages?

Idealista Scraper extracts data from search-result cards rather than individual property detail pages. It reliably exposes card-level fields including price, size, price per area, location labels, floor level, and features, while individual detail page URLs provided in start URLs are currently skipped.

Does Crexi Real Estate Scraper require residential proxies?

No. The primary data path for Crexi Real Estate Scraper calls the Crexi public search API directly, which works from datacenter IPs without requiring proxies, cookies, or browser instances. Optional residential proxy configuration is only needed if the browser fallback path is explicitly triggered.

What file formats can be used to export European real estate listings?

Data collected by Idealista Scraper can be exported directly from the Apify platform into JSON, CSV, Excel, or XML formats.

What asset classes are covered by Crexi Real Estate Scraper?

Crexi Real Estate Scraper covers 14 commercial asset classes across all 51 US states, including Office, Retail, Industrial, Multifamily, Land, Hotel, Hospitality, Mixed Use, Special Purpose, Healthcare, Self Storage, Mobile Home Park, Senior Housing, and Student Housing.