How to Scrape Instagram Data: A Complete Guide

I tried to scrape Instagram data the obvious way first: one requests.get against instagram.com/nasa/ from a cloud server, expecting the profile JSON to be embedded in the HTML the way it used to be. The page came back 200 and 600 KB, and it contained a login wall with zero follower data inside it. Then I called the endpoint the web app actually uses, i.instagram.com/api/v1/users/web_profile_info, and got HTTP 429 with an empty body before I parsed a single field.

That is the real state of Instagram scraping in 2026, and it is what this complete guide on how to scrape Instagram data is built around. Below is what I ran in June 2026, what Instagram returned, and the three routes that actually get Instagram data back: the official Graph API, the hidden web endpoints behind residential proxies, and a scraper API that handles the blocking and the rotating doc_id for you.

What does it mean to scrape Instagram data in 2026?

Scraping Instagram data in 2026 means calling the hidden REST and GraphQL endpoints that Instagram’s own web app uses, because the public page HTML no longer ships the data inside it. The old approach of loading a profile page and parsing post links out of the HTML source returns a login wall now, so the data lives behind a small set of JSON endpoints instead.

There are two endpoints that matter for most jobs. The profile endpoint is a GET request to https://i.instagram.com/api/v1/users/web_profile_info/?username={username}, which returns the full profile as JSON. Posts, comments, and hashtag feeds come from a POST to https://www.instagram.com/graphql/query with a variables payload and a numeric doc_id that selects the query. Instagram’s web app sends these same calls in the background as you scroll, so a scraper reproduces them directly.

Only public data is reachable through these routes: profiles, posts, reels, and comments on public accounts. The same endpoint pattern covers other surfaces like a hashtag feed or the Explore page, since each one is just a different GraphQL doc_id behind the scenes, and reproducing the Explore media grid requires its current doc_id. Private accounts, direct messages, and most Stories need an authenticated session, which is a different and far riskier setup. The Scrapfly 2026 Instagram guide documents the same endpoint structure and the x-ig-app-id requirement, and the blocking layer in front of these endpoints is the next thing to understand.

Why does Instagram block scrapers?

Instagram blocks scrapers at the IP reputation and request-rate level, and it serves a login wall in the public HTML so that simple page scraping returns nothing useful. When I sent a web_profile_info request from a datacenter IP in June 2026, Instagram answered with HTTP 429 and an empty body, both with and without the x-ig-app-id header set.

Here is what each request returned in my June 2026 testing:

The 429 is the tell. A 429 Too Many Requests with an empty body means the datacenter IP tripped the rate gate before Instagram served any data, and adding the correct x-ig-app-id header did not change the outcome, because the IP was refused first. On a clean residential IP the header matters: the web client value is 936619743392459, and an empty or wrong value commonly returns a 403 instead. The HTML route is its own dead end, because the 200 page contains a login wall and the edge_followed_by follower structure is simply not in the markup.

The rate ceiling sits on top of the IP reputation problem. A single residential IP is capped at roughly 200 requests per hour to these endpoints before 429s start, a figure consistent across community testing and the Scrapeops Instagram anti-bot writeup. Datacenter ranges are throttled far harder and are often refused within minutes. There is a second, quieter defense underneath the rate limit: the rotating GraphQL doc_id, which the next section covers because it shapes which route you can maintain.

What is the difference between the Instagram API and the hidden web endpoints?

The official Instagram API and the hidden web endpoints solve different problems: the Instagram Graph API returns data only for Business and Creator accounts you manage, and the hidden web endpoints return public data for any account but with no documentation and no stability guarantee. Picking between them comes down to whose data you need and how much breakage you can absorb.

The official Graph API is the cleanest route when the data you want is your own. It is versioned and stable, and it covers media, insights, comments, and mentions for Business and Creator accounts connected through a Facebook Page. It cannot read an arbitrary public profile that you do not manage, which is the request most people actually have. Meta also deprecated the older Instagram Basic Display API on December 4, 2024, so that path is closed.

The hidden web endpoints fill the gap, at a cost. The doc_id that selects a GraphQL query rotates every 2 to 4 weeks as a deliberate anti-scraping measure, and a hardcoded value fails silently when it changes: your script keeps running and returns nothing. The GitHub repositories for GraphQL-based Instagram scrapers show how much code goes into tracking those identifiers. Maintaining that rotation is the real work, which is why the choice of route matters before you write any Python.

How do you scrape Instagram data with Python?

The most reliable Python route for public Instagram data is the hidden web_profile_info endpoint with browser-shaped headers, sent through a residential IP. Below are three versions: the naive HTML call that fails so you can recognize it, the web_profile_info request that returns clean profile JSON, and the GraphQL call for a single post’s data.

First, the naive HTML call. This is the request that returned a 200 with a login wall and no profile data for me, so you can spot it in your own code:

import requests

# Returns 200 but the HTML is a login wall. No follower data inside.
ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/126.0 Safari/537.36"
r = requests.get("https://www.instagram.com/nasa/", headers={"User-Agent": ua}, timeout=20)
print(r.status_code)                       # -> 200
print("edge_followed_by" in r.text)        # -> False (data is not in the HTML)
print("login" in r.text.lower())           # -> True  (login wall is served instead)

Now the version that returns real data, calling web_profile_info with the headers Instagram’s web app sends. The x-ig-app-id header is the critical one, and the request must come from a residential IP to clear the rate gate:

import requests

USERNAME = "nasa"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "x-ig-app-id": "936619743392459",   # Instagram web client app id
    "Accept": "*/*",
    "Accept-Language": "en-US,en;q=0.9",
}
url = f"https://i.instagram.com/api/v1/users/web_profile_info/?username={USERNAME}"

# Route this through a residential proxy in production: proxies={"https": "..."}
resp = requests.get(url, headers=headers, timeout=20)
print(resp.status_code)                 # 200 on a clean IP, 429 from a datacenter IP
if resp.status_code == 200:
    user = resp.json()["data"]["user"]
    print(user["username"], "-", user["full_name"])
    print("followers:", user["edge_followed_by"]["count"])
    print("verified:", user["is_verified"])

When I ran the second snippet from a datacenter IP in June 2026, it returned 429 with an empty body, which is the block this guide is about. From a clean residential IP the same call returns the profile JSON, including username, full_name, edge_followed_by.count, biography, and the first batch of posts under edge_owner_to_timeline_media. The image and thumbnail URLs in that JSON point at Instagram’s media CDN, on hosts like scontent.cdninstagram.com and regional shards such as scontent-ort2-1.cdninstagram.com, and those CDN links carry a signed expiry, so you download the media promptly and store the file instead of the URL. I am reporting the datacenter result because it is what I could verify directly and it is what most readers hit first.

For a single post’s likes, caption, and comments, the GraphQL endpoint takes a shortcode (the code in the post URL, instagram.com/p/{shortcode}/) and a doc_id:

import requests, json

SHORTCODE = "Cymq6lEt6Vx"   # the code from an instagram.com/p/<shortcode>/ URL
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "x-ig-app-id": "936619743392459",
}
# doc_id rotates every 2-4 weeks; read the current one off a live request in DevTools.
data = {"variables": json.dumps({"shortcode": SHORTCODE}), "doc_id": "8845758582119845"}

resp = requests.post("https://www.instagram.com/graphql/query", headers=headers,
                     data=data, timeout=20)
print(resp.status_code)
# media = resp.json()["data"]["xdt_shortcode_media"]
# print(media["edge_media_preview_like"]["count"], "likes")

The doc_id in that snippet is the value reported in June 2026 and it will not stay current. When it rotates, the response comes back empty or shaped differently, and you read the new doc_id off a live request in your browser’s network tab. That maintenance loop is the reason most teams move off self-hosted GraphQL scraping, which leads to the blocking question next.

How do you avoid getting blocked scraping Instagram?

You avoid Instagram blocks by changing the IP reputation and slowing the request rate, in that order of impact. These are the levers that moved the result in my testing, because the 429 is tied to where the request comes from and how fast they arrive.

• Use residential or mobile IPs. Datacenter ranges are throttled on sight and return 429 within minutes. Residential proxies present as ordinary consumer ISP connections, which is what the hidden endpoints expect.
• Stay under roughly 200 requests per hour per IP. That is the reported ceiling for a single residential IP. Spread load across a pool and keep each IP slow.
• Use sticky sessions of 5 to 10 minutes per IP. Rotating on every single request looks robotic. Holding an IP for a short session mimics how a real browser behaves.
• Send the web headers, including x-ig-app-id: 936619743392459. On a clean IP, the correct app id header is the difference between a 200 and a 403. It does not rescue a flagged IP.
• Track the doc_id and watch for empty 200 responses. A silent empty body usually means the GraphQL doc_id rotated. Treat an empty 200 as a rotation signal and re-read the current doc_id before assuming the account has no posts.

The honest tradeoff is maintenance. Doing all of this yourself means buying a residential proxy pool, rotating it on sticky sessions, re-reading the doc_id every few weeks, and adapting your parser when the JSON shape shifts. That becomes a standing project once you pass a few thousand profiles, which is why most teams hand the proxy rotation and doc_id tracking to a scraper API. I go deeper on the warning signs and limits in my guide on avoiding Instagram scraping blocks.

How do you scrape Instagram at scale without managing proxies?

A scraper API removes the blocking work by accepting a username, shortcode, or hashtag and returning parsed JSON, with residential proxy rotation, the doc_id tracking, and retries handled on the server side. To use an Instagram scraper this way you send one authenticated request and get structured data back, with no 429 to debug and no doc_id to chase, and there is nothing to install beyond an HTTP client you already have. In my runs against ChocoData’s Instagram endpoint, a single call returned profile fields as clean JSON without proxies or a Facebook app.

The request is a plain GET with your API key as a query parameter:

curl "https://chocodata.com/api/v1/instagram/profile?username=nasa&api_key=$CHOCO_API_KEY"

The Python version is the same shape, and it returns the profile fields you would otherwise parse out of web_profile_info, ready to load into pandas:

import requests
import pandas as pd

resp = requests.get(
    "https://chocodata.com/api/v1/instagram/profile",
    params={"username": "nasa", "api_key": "YOUR_CHOCO_API_KEY"},
    timeout=30,
)
user = resp.json()["data"]["user"]

row = {
    "username": user["username"],
    "full_name": user["full_name"],
    "followers": user["edge_followed_by"]["count"],
    "is_verified": user["is_verified"],
}
df = pd.DataFrame([row])
df.to_csv("nasa_profile.csv", index=False)
print(df.head())

This returns the same public profile data the hidden endpoint would, loaded straight into a pandas DataFrame and written to CSV, without holding a residential proxy pool or tracking the rotating doc_id. You can get an API key on the ChocoData sign-up page and swap it into the snippet above. The same pattern covers other targets through dedicated endpoints: the Instagram profile and account scraper for accounts, the Instagram post scraper for media and engagement, the Instagram hashtag scraper for tag feeds, and the Instagram comment scraper for comment threads. If you are weighing a managed Apify actor against a direct API, I compare the tradeoffs in my Apify Instagram scraper alternative breakdown.

Which method should you choose?

The right method depends on whose data you need, the volume, and how much engineering time you want to spend on proxies and the doc_id. Here is the summary I give people who ask.

Before you collect anything at scale, it is worth knowing where the legal line sits. Scraping publicly visible Instagram data while logged out is treated as defensible in the US after Meta Platforms v. Bright Data, where Judge Edward Chen ruled in January 2024 that Meta’s Terms do not bar logged-off scraping of public data, because a logged-off scraper is not a “user” bound by those Terms. That reasoning sits alongside the Ninth Circuit’s hiQ v. LinkedIn ruling that scraping public data likely does not violate the CFAA. Scraping private accounts, bypassing the login wall with a fake account, and collecting personal data of EU residents under the GDPR are separate matters with their own risk. I walk through all of it in is scraping Instagram legal and in my guide to Instagram’s Terms of Service on scraping, and I rank the managed options head to head in my best Instagram scrapers in 2026 roundup.