Instagram Data Extraction Guide: Profiles, Hashtags, and Comments

Summary of Instagram data extraction

Extracting structured data from Instagram requires matching specific data needs to dedicated automation tools rather than using a single generic web crawler. By isolating endpoint logic, engineering teams can gather account metrics, track hashtag trends, and reconstruct nested comment threads efficiently. The Instagram Profile Scraper extracts profile metadata and up to 500 recent post records per run. The Instagram Hashtag Scraper collects post feeds across up to 50 target tags with optional page enrichment. The Instagram Comment Scraper parses discussion threads up to 10000 comments per post, expanding nested reply chains and handling media placeholders.

Specialized scrapers and distinct extraction roles

Instagram separates data across dedicated endpoints for profile headers, hashtag discovery feeds, and post comment trees. Utilizing individual scrapers tailored to these endpoints prevents request blocks and ensures clean, structured data collection.

Account metadata and post archives

The Instagram Profile Scraper targets individual user handles to collect core account metrics. It extracts bio text, follower counts, following counts, verification status, business classification, and external bio links. When set to inspect posts, it captures captions, hashtags, user mentions, tagged accounts, co-authors, location tags, and individual carousel media links for up to 500 posts per run. To collect account metrics without fetching post feeds, users can set maxPosts to 0 to trigger profile-only mode.

Topic feeds and hashtag discovery

The Instagram Hashtag Scraper gathers post metadata associated with specific topic tags. It accepts up to 50 hashtags per run and supports recent, top, or combined feed modes. When deeper inspection is required, enabling post enrichment causes the scraper to visit individual post pages, retrieving full caption text, detailed engagement counts, location tags, tagged users, and video metadata.

Comment threads and discussion trees

The Instagram Comment Scraper extracts public discussion threads from post and reel links, supporting standard post URLs and reel links. It collects up to 10000 comments per post, capturing commenter handles, user IDs, timestamps, like counts, and reply counts. When reply scraping is enabled, setting maxRepliesPerComment to 0 fetches all available nested replies in each thread, linking every reply to its top-level parent comment ID and parent author username.

Practical use cases

Use case 1: Creator evaluation and account metadata audit

Outcome: A structured dataset containing creator profile metrics and recent content archives for account verification and partnership evaluation.

Question to answer: Does a target creator maintain consistent posting activity and accurate bio information aligned with campaign requirements?

Configure: Set username to the target handle (such as target_creator). Set maxPosts to 50 to evaluate recent post history. Leave cookies blank to utilize managed session pools, or paste custom session cookies in JSON format.

Working method: The scraper connects to Instagram's profile endpoint to extract account fields including followers_count, biography, and external_urls. It then iterates through the 50 most recent posts, extracting media_type, caption, like_count, and comment_count for each item. If Instagram's profile API encounters an internal error for the account, the actor falls back to public profile web parsing, preserving core metrics like follower count, bio text, and verification status while omitting optional business fields.

Deliverable: A JSON dataset containing one profile record with full account metadata followed by 50 post records containing individual media links and engagement statistics.

Stop condition: The actor completes after parsing the account header and 50 post items, or halts immediately with an error message if the specified handle does not exist.

Use case 2: Hashtag trend benchmarking and discovery

Outcome: A comparative multi-tag post dataset mapping recent content volume and visual media types across target topics.

Question to answer: Which media formats and posting schedules yield the highest engagement across specific industry hashtags?

Configure: Pass a list of up to 50 topic tags in hashtags. Set maxPostsPerHashtag to 100. Select recent or top in scrapeMode. Enable enrichPosts to true and set maxPostsToEnrich to 20 for deep page metadata. Set minDelayBetweenRequests to 2 and maxDelayBetweenRequests to 5.

Working method: The actor processes each hashtag string independently. It executes GraphQL queries to paginate through hashtag feeds, using HTML parsing as a fallback if API queries encounter restrictions. For the top 20 items per tag, the scraper visits individual post pages to pull full captions, tagged user handles, location details, and exact like counts.

Deliverable: A dataset grouped by search_hashtag where each item contains post shortcodes, media URLs, caption text, comment counts, and extracted hashtag lists.

Stop condition: The run finishes when 100 posts per configured hashtag have been collected or when available pagination tokens for the hashtag topic are exhausted.

Use case 3: Product feedback and comment reply analysis

Outcome: A nested discussion record capturing user responses, product queries, and community feedback from key promotional posts or video reels.

Question to answer: What specific questions and feedback do community members express in response to promotional video reels?

Configure: Set postUrls to the target post or reel links. Set maxCommentsPerPost to 1000. Set includeReplies to true and maxRepliesPerComment to 0 to collect all nested replies.

Working method: The scraper opens each post URL and paginating through top-level comments until the maximum limit is reached. For every comment with a positive reply count, it expands the reply thread to extract nested replies. It captures comment IDs, author handles, timestamps, and text. When media attachments like images or videos lack direct CDN links in Instagram's web response, the scraper writes placeholder text like [Photo unavailable] or [Image] while marking commentType accordingly. For GIF comments, direct media URLs are recorded in mediaUrl.

Deliverable: A structured table where each row represents a top-level comment or reply, linked via parentCommentId and parentCommentAuthor, including like counts and commenter handles.

Stop condition: The run halts when 1000 comments per post URL have been parsed or when all comment pages for the given posts are processed.

Step-by-step engineering workflow

Select the target scraper based on data requirements: Identify whether the project requires account metrics, topic discovery feeds, or discussion threads. Choose Instagram Profile Scraper for accounts, Instagram Hashtag Scraper for tag feeds, or Instagram Comment Scraper for comment threads.
Configure authentication and session options: Decide whether to use built-in managed session pools or export active session cookies from a logged-in browser using tools like Cookie-Editor. Paste JSON cookie strings into the cookies parameter when accessing restricted endpoints or private follower feeds.
Establish input parameters and extraction bounds: Define target handles, hashtag lists, or post URLs. Set strict limits on maxPosts, maxPostsPerHashtag, or maxCommentsPerPost to maintain predictable execution timing. Configure request delays using minDelayBetweenRequests and maxDelayBetweenRequests to control request frequency.
Execute extraction and handle invalid target responses: Launch the scraper execution. Ensure downstream error handling captures missing profile errors gracefully, as nonexistent handles cause runs to halt with explicit error messages without writing partial profile records.
Normalize dataset structures for downstream storage: Parse output JSON based on record types. Separate profile summary records from individual post records, and map nested comment parent IDs to establish clear comment hierarchy trees.
Integrate structured data into analytics pipelines: Export cleaned dataset files to central data lakes or storage Buckets. Process extracted fields for audience sentiment analysis, competitive benchmarking, or tracking AI referral traffic across digital platforms.

Data quality, platform limits, and operational considerations

Data retrieval from Instagram is influenced by platform access rules and API response behaviors.

Private account restrictions: Post feeds and comments on private profiles are inaccessible to unauthenticated requests or sessions that do not follow the target account. While profile header metadata remains partially accessible, post collection returns status indicators showing content is unavailable.
Omission of optional business contact fields: Business contact details such as email, phone, and category are only included in profile records when Instagram explicitly displays them on the profile page. If an account owner omits business contact info, or if Instagram's internal profile API returns an error requiring public web parsing fallback, these fields are omitted from output JSON rather than returned as null or empty strings.
Comment media placeholders: Instagram's web API withholds direct CDN links for static image and video attachments within comments. In these instances, the Instagram Comment Scraper assigns commentType as image or video and inserts descriptive text placeholders such as [Photo unavailable] or [Image]. GIF comments represent an exception where direct CDN URLs remain available in mediaUrl.
Maximum extraction caps per run: Scrapers enforce upper execution limits to ensure stability. Profile post extraction caps at 500 posts per run, comment extraction caps at 10000 comments per post URL, and hashtag post extraction supports up to 1000 posts per tag.

Frequently asked questions

Do I need an Instagram account to extract public profile data?

No dedicated account or API key is required for basic public extraction. The scrapers include managed session pools for automatic session management. Users can also supply custom browser session cookies in JSON format if preferred.

Can I extract post archives from private Instagram accounts?

No. Posts and comments on private profiles cannot be extracted unless the session used belongs to an account that follows the target profile. For unauthenticated runs, private profiles only return basic profile metadata without post items.

How many posts or comments can be extracted in a single run?

The Instagram Profile Scraper extracts up to 500 posts per run. The Instagram Hashtag Scraper extracts up to 1000 posts per hashtag. The Instagram Comment Scraper extracts up to 10000 comments per post URL.

Why are business email, phone, and category fields missing from some outputs?

These fields are only included when Instagram exposes them on the account profile. If an account owner has not listed public business details, or if Instagram's profile API errors and forces fallback to public web parsing, the fields are omitted from output records.

Why do some media comments display text like [Photo unavailable]?

Instagram's web API withholds direct CDN media URLs for static image and video comments. The comment scraper marks commentType accordingly and inserts descriptive placeholders such as [Photo unavailable] or [Image]. GIF comments retain direct media URLs.

How does thread control function in the comment scraper?

Setting includeReplies to true expands comment reply threads. The maxRepliesPerComment parameter caps how many replies are retrieved per individual comment thread. Setting maxRepliesPerComment to 0 fetches all available replies in the thread.