Swiggy Tracking: Goal, Progress and Learnings

Last updated 2026-10-27. Source of truth for the project: this document. Project folder: swiggyResearch.

1. Goal

Collect restaurants, outlets, menu dishes and ratings from Swiggy delivery pages and store them in a SQLite database.

Restaurant: name, city, cuisines, cost for two, overall rating and rating count.
Outlet: name, area, full address and FSSAI license number. One brand can have several outlets; each Swiggy restaurant ID is one outlet.
Every dish on the full menu: category, veg flag, price, description.
Dish rating (stars) and number of ratings, for dishes that have them. Many dishes have none.
Deterministic code does the extraction. LLMs are used only for setup and checks.

Scope so far: every Hyderabad delivery restaurant URL found in Swiggy's public sitemap (27,418). Only a few have been mined.

2. Status at a glance

Piece	Status
Menu data source	Found. The `/dapi/menu/pl` JSON response contains the full menu, prices, ratings, descriptions, restaurant info, address and license.
Menu completeness	Matches. Sum of section counts equals parsed items for every restaurant mined so far.
Capture and parse	Working. Mined end to end from the viewer (capture, then parse). Browser-side diagnostics are non-fatal.
Database	11 restaurants, 11 outlets, 1,627 dishes, 1,171 dish ratings, all outlets have a license number.
View	`dish_ratings_named` adds dish, restaurant and outlet names to `dish_ratings`.
Viewer	Working. All, Done and Not done filters. Mine button runs one restaurant.
Outlet switcher	Not done. Swiggy marks restaurants as `multiOutlet`; only the URL's outlet is captured.
Address and license in the view	Stored in tables, not yet in the view.

3. How the pipeline works

List: hyderabad_restaurants.txt holds one URL per restaurant (from the sitemap, section 10).
Capture (capture.js): opens one URL in a visible Chrome, clicks View full menu, saves the menu API response and some diagnostics to snapshots\rest<id>\<timestamp>\.
Parse (parse.js, lib/menu-api.js): reads the saved menu API JSON and writes restaurant, outlet, dishes and ratings to swiggy.db.
View: dish_ratings_named in the DB, or the local viewer.

The capture step never writes to the DB. Parsing reads only saved files, so it can be re-run without touching Swiggy again. That split is the most useful design decision so far.

The older mine.js (DOM-based, writes straight to the DB) is still present but no longer used by the viewer. Its failure path writes placeholder rows, so it is kept only for reference.

4. What the Swiggy data looks like

Where the menu comes from

The menu is not in the first page load. Clicking View full menu requests https://www.swiggy.com/dapi/menu/pl. The response is 230 KB to 430 KB and holds everything needed in one response. There was no pagination in any response seen so far (firstOffsetRequest: true is only a marker).

Fields used

Data	Where in the response
Restaurant name, city, cost for two, cuisines, average rating, rating count	`Restaurant.info`: `name`, `city`, `costForTwo` (paise), `cuisines`, `avgRating`, `totalRatingsString`
Outlet name, area, address	`RestaurantAddress`: `name`, `area`, `completeAddress`
License number	`RestaurantLicenseInfo.text`: "License No. ..."
Dish sections	`ItemCategory` cards, and sub-categories inside `NestedItemCategory`, under `groupedCard.cardGroupMap.REGULAR`
Dish name, description, veg flag, price	`itemCards[].card.info`: `name`, `description`, `itemAttribute.vegClassifier` (VEG / NONVEG), `price`
Dish rating	`info.ratings.aggregatedRating`: `rating` (string), `ratingCountV2`. Missing or `"--"` means no rating.

Things that surprised us

Prices are in paise. 45800 means Rs 458. Cost for two is also in paise.
Some dishes have no top-level price. Their price sits in variantsV2.pricingModels; we take the lowest.
Ratings in the response are not always for this restaurant. A "similar restaurants" grid carries ratings for other outlets. Only the dish info objects are used.
Restaurants with no ratings yet show -- and no average. 0 rated dishes is correct for those.
Same brand, different outlets: three outlets are all named "Pista House" in the API. Each has its own area, address and license. Pista House Bakery is a separate brand name.
Shared address: Pista House (Toli Chowki) and Pista House Bakery list the same address on Swiggy's own page. Licenses and dishes differ, so they are kept as separate outlets.

Markup changes

The DOM dish card used to carry an aria-label with "Costs: N rupees". It is now a paragraph with the same text, and each card has data-testid="normal-dish-item". capture.js accepts both. This change was the cause of the earlier DOM timeouts (section 11).

5. Parser decisions

Decision	Why
Use the menu API as the source of truth. The DOM is diagnostic only.	The API has every dish in one response, with exact ratings and descriptions. The DOM depends on rendering and on markup that changes.
Store prices in rupees (paise divided by 100, rounded).	Matches what the page shows ("Costs: 458 rupees").
Items with variants take the lowest variant price.	The page shows a single price for these. Example: Murgh Musallam is stored as 664 (Half), not 825 (Full). Confirm this is the rule you want.
Category = the section title. Nested sub-categories are flattened to their own section.	Matches how the page groups dishes. Section titles have whitespace collapsed.
Duplicate (category, name) pairs: keep the first, report the rest in the run note.	The DB key is (outlet, category, name). A silent overwrite would lose data.
Outlet name from `RestaurantAddress.name`, falling back to the restaurant name.	Outlet name can differ from the brand name (for example "Pista House Bakery"). This was a bug until it was fixed.
Address from `completeAddress`; license number from `RestaurantLicenseInfo`.	Address is the best human key. License is a stable government ID per outlet, so it is the most reliable way to tell outlets apart.
Ratings: missing means null, not zero.	A dish with no rating is different from a dish rated 0.
Rating count "1.8K+" is stored as 1800 (approximate). The original text is kept too.	The API only gives a rounded count for large numbers.
Multiple menu responses in one snapshot: use the one with the most items.	A snapshot can contain error bodies or other API responses.

Success conditions

Done: the latest parse run for the URL saved at least one dish. The viewer uses this rule.
Not done: no dishes saved yet. If the last Mine failed, the reason shows on the row.
Earlier we used a stricter rule (count must match the sum of section counts). That is still recorded per run (dishes_expected, status), but it no longer decides the viewer status, at your request.
Missing ratings is not a failure. The browser (DOM) steps are logged, and never decide the status.

6. Database

File: swiggy.db (SQLite, WAL mode). Schema is created by lib/db.js every time it opens. Tables:

Table	One row per	Notes
`restaurants`	Swiggy restaurant ID (one outlet each)	Name, city, cuisines, cost for two, rating, rating count
`outlets`	Outlet (one per restaurant ID here)	`outlet_name`, `area`, `address`, `license_no` (added later)
`dishes`	Dish on an outlet's menu	Unique on (outlet, category, name). Veg flag, price, description
`dish_ratings`	Dish, per scrape run	Stars and rating count at that time. Keeps history.
`scrape_runs`	Parse run	Status, dishes found, dishes expected, note. Used for viewer status.
`dish_ratings_named` (view)	Same as `dish_ratings`	Adds `dish_name`, `restaurant_name`, `outlet_name`.

Migrations and backfills

outlets.license_no is added by a migration in openDb when missing. Existing outlets were backfilled from the snapshot files (12 updates, 0 skipped at the time). Every outlet now has a license.
The view is dropped and recreated on each open, so its definition always matches the code.
Re-parsing a snapshot upserts outlets and dishes, updates the rating row for the same timestamp, and adds a run row. Backfills were done with a targeted UPDATE to avoid duplicate run rows.

Known data quirks

restaurants.name was briefly the URL for restaurants whose early mine attempts failed. Successful parses overwrite it.
Placeholder outlets with failed@url addresses (from the old mine.js failure path) were deleted.
Old failed runs remain in scrape_runs as history.

7. Capture decisions (browser)

Decision	Why
Visible browser, not headless.	Headless runs were refused by Swiggy with an HTTP error on the navigation. A visible window worked.
Persistent profile in `browser-profile\`, not your personal Chrome.	Keeps cookies and location between runs without touching your own profile or sign-ins. Override with the `PROFILE` environment variable.
Wait for the restaurant header, then up to 15 s for the View full menu button to be visible, then click.	A fixed sleep made the button check run before the button rendered.
Wait for dish cards to be attached (in the DOM), not visible.	A late or off-screen render should not end the scroll. The API response is what the parser uses.
Card selector: `[data-testid="normal-dish-item"]`, with the old `aria-label` as fallback.	The label moved into the text of a paragraph (section 4).
DOM result recorded as `dom_status` in `meta.json`: `cards_seen`, `no_cards`, `no_button`, `error`.	Diagnostic only. It explains a failure but never decides success.
HTTP 202 on navigation is logged as a note, not an error.	The page still loaded and the menu API returned 200.
Only HTTP 400 and above stops a capture.	Matches the observed behaviour of the site.
One browser context for the whole run; one page per URL.	Keeps the profile shared while avoiding leftover tabs.

Rate and blocking

Repeated requests to Swiggy in a short time start failing with ERR_HTTP_RESPONSE_CODE_FAILURE, and sometimes HTTP 403. Pause between runs, and do not loop over many restaurants at once. The viewer runs one job at a time for this reason.

8. Learnings: Claude in Chrome

How the tools behaved

Tabs: each session opened its own tab. Tab IDs from earlier sessions became invalid; re-run tabs_context_mcp with createIfEmpty when that happens.
Reading pages: read_page gives a long accessibility tree with links and card text. get_page_text returned nothing on Swiggy's app-style pages. javascript_tool was the most useful: read the DOM, count elements, check attributes.
Clicking: find returns a ref, and clicking that ref works better than coordinates.
Screenshots are the best check for "does the page look right". They showed the menu rendering when the scripts said it wasn't.
Downloads: the browser can save a file made in page JavaScript (Blob plus anchor click). It needs your approval first. Files land in Downloads and must be copied into the project.
Writing to disk is not possible from the page. The sitemap download from the command line returned 403, so the browser is the only route to Swiggy's sitemap.

Lessons for using it

Test the exact selector the script uses, against the live page, before blaming timing. The timeouts were a selector change, not a rendering problem, and a simple read of the page missed it.
Don't poll Swiggy repeatedly. A few quick retries were enough to get refused.
Close the tabs you open. Leave the rest alone.
Stop and ask after 2 or 3 failed tool calls, as the skill says.

9. Viewer

Run npm run viewer, then open http://127.0.0.1:3000. It listens on localhost only.

Rows: every URL in the list. Green dot and "Done" when dishes are saved; grey "Not done" otherwise.
Filters: All, Done, Not done. Search by name, area or ID.
Mine (or Re-mine): runs capture.js, then parse.js, for that URL. One job at a time. Log panel shows progress.
Security: the server only runs URLs that are in the list file.

The viewer server does not reload when its code changes. After any change to viewer/server.js, restart it. A stale server returns old fields, which caused the Done and Not done filters to show 0 restaurants. The page now computes Done from the dish count, so it works with either version.

10. Restaurant list (27,418 URLs)

Source: Swiggy's public sitemap index (robots.txt points to sitemap.xml.gz), which lists 40 gzipped sitemap files. Filtered to /city/hyderabad/<slug>-rest<id>.
Result: 27,418 URLs, 27,418 distinct IDs, no duplicates. Both original test restaurants are included.
Not included: Dineout (table booking) pages, which use /restaurants/<slug>/dineout.
Caveats: a sitemap entry does not prove the restaurant is open or delivering. Multi-outlet brands may have one entry per outlet.
File: hyderabad_restaurants.txt in the project (2.5 MB).

11. Issues found and how they were fixed

Symptom	Cause	Fix
Capture did not click View full menu	Button checked once after a fixed 3 s sleep, before it rendered.	Wait for the header, then for the button to be visible, then click.
Menu API captured but the run timed out and was logged as failed	The selector `[aria-label*="Costs:"]` no longer matched. The label moved into paragraph text.	Selector switched to `data-testid="normal-dish-item"`, with the old label as fallback. DOM result made non-fatal.
Placeholder restaurant rows with the URL as the name	The failure path of `mine.js` wrote placeholder restaurants and `failed@url` outlets.	Parse upserts overwrote the names. Placeholder outlets deleted. Viewer no longer uses `mine.js`.
Viewer showed "undefined" statuses, then 0 for Done and Not done	Server process started before a code change, so it returned old fields.	Restart the viewer. Page derives Done from the dish count.
Outlet name was always the brand name	Parser used the restaurant name, not `RestaurantAddress.name`.	Parser uses the outlet name, falling back to the restaurant name.
DB would not open in the VS Code SQLite extension	Suspected: WAL mode side files, and a project folder synced by OneDrive. Not confirmed; you reported it worked afterwards, which you put down to a memory issue.	No code change. Note this if it returns.
Test of a recent parse showed "0 with ratings"	Correct. The restaurant had dishes marked `--`. The two ratings found belonged to other outlets in a grid.	None needed. Recorded here so it isn't mistaken for a bug.

12. Files

File	Purpose
`capture.js`	Opens one URL, clicks the menu, saves the API response and diagnostics.
`parse.js`	Parses saved snapshots into the DB. `npm run parse`.
`lib/menu-api.js`	Menu API parser: restaurant, outlet, license, dishes, ratings.
`lib/db.js`	Schema, migrations, view, saves one scrape in one transaction.
`lib/extract.js`	DOM extractor and shared helpers. Used by the old `mine.js`.
`mine.js`	Older DOM-based miner. Not used by the viewer.
`viewer/server.js`, `viewer/index.html`	Local viewer and mining job runner.
`test/menu-api.test.js`	Parser tests against real snapshots. `npm test`.
`hyderabad_restaurants.txt`	27,418 Hyderabad restaurant URLs.
`ref.txt`	The two original test URLs.
`swiggy.db`	The database.
`snapshots\`	Raw captures. Large, not in the repo.
`browser-profile\`	Browser profile used by capture. Not in the repo.

13. Open items

Outlet switcher: Swiggy marks these restaurants as multi-outlet. Find the other outlets from a brand's listing and capture each one.
Brand grouping: each Swiggy ID is its own restaurant row. Group outlets under a brand, for example "Pista House", if that view is needed.
View: add outlet address and license number to dish_ratings_named.
Price rule: confirm the lowest-variant price rule for items with variants.
Scale: mine the other restaurants in the list. Pace it, since repeated requests get blocked.
Clean up: failed snapshot folders and failed run rows can be removed once no longer needed.
Mine.js: retire or fix its failure path.

14. Corrections to earlier notes

The launch arguments of mine.js and capture.js are the same. The earlier note that they differed was wrong.
HTTP 202 is not a block. The menu API returned 200 in the same runs.
The database was not empty: it held failed runs and placeholder rows, now cleaned up.
The earlier timeout diagnosis (visibility, then browser profile) was wrong. The cause was the selector, which was confirmed on the live page.