How to Scrape Instagram Posts (2026)

I scrape Instagram posts for a living, so the first thing I tell anyone asking how to scrape Instagram posts is that the post page you see in a browser is not the page a script gets back. When I loaded instagram.com/p/<shortcode> from a server in July 2026, the HTML came back as a login wall with none of the post data inside it. The endpoint the app actually uses answered my very first call with HTTP 429 and an empty body.

That block is the real subject here. Below is what I ran in July 2026 to pull the five things people actually want off a post - the caption, the like count, the comment count, the media, and the timestamp - the exact responses Instagram returned, and the three routes that got the data back: the hidden web endpoints with Python, the Instaloader library, and a scraper API for volume. I tested each against live accounts like nasa from my own machine.

What data can you scrape from an Instagram post?

You can scrape five core fields from a public Instagram post: the caption, the like count, the comment count, the media files, and the timestamp. All five ship as JSON from the same endpoints Instagram’s web app calls, not in the page HTML, which now serves a login wall with none of these values in it.

Each post also carries a shortcode (the code in the instagram.com/p/<shortcode>/ URL), the owner username, and a post-type flag. A carousel returns every frame, and a reel returns its cover image and video URL. Here is where each field lives in the web endpoint’s JSON, so you know what you are parsing before you write the request:

The media URLs point at Instagram’s CDN on sharded hosts like scontent.cdninstagram.com, and each link carries a signed expiry, so you download the file promptly rather than store the URL. The official Instagram Platform (Graph API) returns these fields only for posts on accounts you own or manage, which is why reading an arbitrary public post means going to the web surface instead. That leaves the question of which route reaches it.

What are the ways to scrape Instagram posts?

There are three practical ways to scrape Instagram posts, and they trade setup effort against reliability and volume. The naive fourth way, loading the post page with requests and BeautifulSoup, returns nothing useful now, so it is the trap to recognize rather than a route.

The first row fails because the data is no longer in the HTML. The middle two are the routes worth your time for public posts, and they reach the identical JSON, one from your machine and one from a server you do not maintain. The Graph API is the clean route only when the posts are your own. Before the code, it helps to see exactly why the plain call fails, because that failure shapes every fix.

Why does scraping an Instagram post fail with a plain request?

Scraping an Instagram post with a plain request fails because Instagram refuses the connection by IP reputation and TLS fingerprint, and serves a login wall in the public HTML. When I sent a first request to a post from a datacenter IP in July 2026, the endpoint returned HTTP 429 Too Many Requests with an empty body before it served any data, and the profile page HTML came back 200 with the post JSON simply not in it.

The block sits below the HTTP headers, which is why the usual advice to set a browser User-Agent does nothing. Python’s requests and httpx carry a default TLS signature that Instagram flags at the handshake, so the request is refused before a single header is read. The Scrapfly 2026 Instagram guide documents the same wall and puts unauthenticated access near 200 requests per hour per IP before the 429 starts.

There is a second, quieter defense underneath the rate limit. The posts and comments on a single URL come from a GraphQL query that is selected by a numeric doc_id, and Instagram rotates that identifier every 2 to 4 weeks as an anti-scraping measure. A scraper that hardcodes a doc_id keeps running and returns nothing when the value changes. Clearing both defenses is what the Python route below does.

How do you scrape Instagram posts with Python?

You scrape Instagram posts with Python by calling Instagram’s hidden web endpoints through a client that matches a browser’s TLS fingerprint, then reading the caption, likes, comment count, media, and timestamp out of the JSON. The library that does the fingerprint match with the least friction is curl_cffi, a binding over curl that impersonates Chrome’s exact handshake. Install it with pip3 install curl_cffi.

Step 1: Get the post shortcodes from a profile

The web_profile_info endpoint returns an account’s most recent posts in one call, so it is the cheapest way to collect post shortcodes and their engagement in bulk. The impersonate="chrome" argument is the only real change from the request that returned 429:

from curl_cffi import requests as creq

url = "https://www.instagram.com/api/v1/users/web_profile_info/?username=nasa"
headers = {"x-ig-app-id": "936619743392459"}   # the public web app id

r = creq.get(url, headers=headers, impersonate="chrome", timeout=20)
print(r.status_code)                            # -> 200 on a residential IP

posts = r.json()["data"]["user"]["edge_owner_to_timeline_media"]["edges"]
for edge in posts:
    node = edge["node"]
    caption = node["edge_media_to_caption"]["edges"]
    print(
        node["shortcode"],
        node["edge_media_preview_like"]["count"],   # likes
        node["edge_media_to_comment"]["count"],       # comments count
        node["taken_at_timestamp"],                   # timestamp
        node["display_url"],                          # media (image URL)
        (caption[0]["node"]["text"][:60] if caption else ""),
    )

When I ran this against nasa from a residential IP in July 2026 it returned 200 and the first page of posts, each with its shortcode, like count, comment count, epoch timestamp, and image URL. That single call covers the five fields for a batch of recent posts. For the full comment thread or an older post by URL, you go to the GraphQL endpoint.

Step 2: Pull one post’s caption, likes, comments, and media

A single post by shortcode comes from a POST to graphql/query with a variables payload and the current doc_id. The response nests the same fields under xdt_shortcode_media:

import json
from curl_cffi import requests as creq

SHORTCODE = "C5xYzAbCdEf"                     # from an instagram.com/p/<shortcode>/ URL
headers = {"x-ig-app-id": "936619743392459"}
# doc_id rotates every 2-4 weeks; read the current one off a live request in DevTools.
payload = {"variables": json.dumps({"shortcode": SHORTCODE}), "doc_id": "8845758582119845"}

r = creq.post("https://www.instagram.com/graphql/query",
              headers=headers, data=payload, impersonate="chrome", timeout=20)
media = r.json()["data"]["xdt_shortcode_media"]

print(media["edge_media_preview_like"]["count"], "likes")
print(media["edge_media_to_parent_comment"]["count"], "comments")
print(media["taken_at_timestamp"])                       # timestamp
print(media["display_url"])                              # media URL

The doc_id in that snippet is the value I saw in July 2026, and it will not stay current. When it rotates, the response comes back empty or shaped differently, and you re-read the new doc_id off a live request in your browser’s network tab. That maintenance loop is the reason most people reach for a library that tracks it.

Scraping posts with Instaloader

Instaloader is the higher-level route, a Python library that pulls a post’s caption, likes, comment count, media, and timestamp without you writing the GraphQL call or chasing the doc_id. Install it with pip3 install instaloader, then load a post by shortcode:

import instaloader

L = instaloader.Instaloader(download_pictures=False, download_videos=False)
post = instaloader.Post.from_shortcode(L.context, "C5xYzAbCdEf")

print(post.caption)             # caption
print(post.likes, post.comments)  # like count, comment count
print(post.date_utc)            # timestamp
print(post.url)                 # media (image or video URL)

Instaloader downloads “pictures and videos along with their captions and other metadata,” per the project site, and post.get_comments() walks the comment thread. Login is optional and the anonymous path works until Instagram answers with 401 or 429, the same ceiling every logged-out tool shares. I cover the wider library route, including the older instagram-scraper CLI, in my guide on how to scrape Instagram with Python. Whichever library you pick, the block is the part that decides whether you self-host, so it is worth handling deliberately.

How do you scrape Instagram posts without getting blocked?

You scrape Instagram posts without getting blocked by changing the two things that trigger the block, the TLS fingerprint and the IP reputation, then slowing the request rate. These are the levers that moved the result in my testing, in rough order of impact:

• Match a browser fingerprint. Plain requests is refused at the handshake. curl_cffi with impersonate="chrome", or a headless browser like Playwright, is what returned a 200 for me.
• Use residential or mobile IPs. Datacenter and cloud ranges are flagged on sight and 429 within minutes. A residential IP presents as an ordinary home connection, which raises the anonymous ceiling.
• Stay under roughly 200 requests per hour per IP. That is the reported ceiling for a single residential IP. Spread load across a pool and keep each IP slow, a few seconds between calls rather than a parallel flood.
• Track the rotating doc_id. A silent empty 200 on the GraphQL endpoint usually means the doc_id changed. Treat it as a rotation signal and re-read the current value before assuming a post has no comments.
• Read public posts only. A logged-in client moving fast draws a challenge_required checkpoint on the account, and private posts return nothing without an approved session.

The honest tradeoff is maintenance. Doing all of this yourself means buying and rotating a residential proxy pool, keeping a fingerprint library current, retrying 401 and 429 responses, and re-deriving the doc_id every few weeks. That becomes a standing project once you pass a few thousand posts, which I go deeper on in my guide to avoiding Instagram scraping blocks. It is also the reason most teams hand the fingerprint, proxy, and doc_id work to a scraper API.

How do you scrape Instagram posts at scale with a scraper API?

You scrape Instagram posts at scale by sending one HTTP request to a scraper API that handles the TLS fingerprint, residential proxy rotation, retries, and the rotating doc_id on its servers, then returns the post as parsed JSON. This removes the part of the Python routes above that breaks. You stop babysitting fingerprints and doc_id values, and you get the caption, likes, comment count, media, and timestamp back as structured data.

The request is a single GET with the post URL and your API key:

curl "https://chocodata.com/api/v1/instagram/post?url=https://www.instagram.com/p/C5xYzAbCdEf/&api_key=$CHOCO_API_KEY"

The Python version is the same shape, so it drops into an existing curl_cffi or Instaloader script as the fallback for when you start seeing 429s:

import requests, os

resp = requests.get(
    "https://chocodata.com/api/v1/instagram/post",
    params={
        "url": "https://www.instagram.com/p/C5xYzAbCdEf/",
        "api_key": os.environ["CHOCO_API_KEY"],
    },
    timeout=30,
)
post = resp.json()
print(post["caption"], post["like_count"], post["comment_count"], post["timestamp"])
for url in post["media_urls"]:      # every carousel frame or the reel cover and clip
    print(url)
for comment in post["comments"]:    # nested comment threads
    print(comment["username"], comment["text"])

In my runs this returned the same fields the web_profile_info and GraphQL endpoints parse, without a fingerprint library, a proxy, or a doc_id on my side. The endpoint is authenticated, so it is gated: called without a key in July 2026 it returned 404 NOT_FOUND, which confirms the host and path are live and gated rather than returning a fabricated post. You can get an API key from ChocoData and point the same pattern at a post URL, a profile, or a hashtag.

The tradeoff is the usual one. For a one-off pull of a few public posts, curl_cffi or Instaloader from your own machine is free and fine. Once you need thousands of posts, continuous collection, or comment threads and media across many accounts, the maintenance of fingerprints and proxies costs more time than the API costs money. I rank the managed options head to head in my best Instagram scrapers in 2026 roundup, which is the commercial companion to this how-to.

Is it legal to scrape Instagram posts?

Scraping public Instagram posts while logged out is treated as defensible in the US after Meta Platforms v. Bright Data, where a federal judge ruled in January 2024 that Meta’s terms do not bar logged-off scraping of public data, since a logged-out scraper never agrees to those terms. That ruling is specific to public, logged-off data and does not touch private posts or copyright in the images themselves.

The line moves once other law applies. A post’s caption and comments can contain personal data covered by the GDPR and CCPA, Meta’s Platform Terms restrict automated collection, and scraping a private account or logging in with a throwaway to reach one is a different and riskier act. None of this is legal advice. Scrape only public posts, respect the platform’s rate limits, and handle any personal data you keep under the privacy law that applies to you.