- 自動のセッション管理
- 全世界195ヵ国の各都市がターゲット対象
- 無制限の同時セッション
Structured Data Extraction
TLDR: Structured data extraction converts page content written for humans into consistent machine readable records. It is the step that turns a fetched page into usable data.
Structured data extraction identifies specific values inside a document and maps them to named fields. A product page becomes a record with a title, price, currency, availability, and rating. The page itself is presentation markup designed for a browser, so extraction is the work of recovering the underlying data the page was built from.
Extraction Methods:
- CSS selectors: Target elements by class, id, or position. Fast and precise, but tied to the current markup.
- XPath: Navigate the document tree with more expressive queries, including selection based on text content or sibling relationships.
- Embedded structured data: Read JSON-LD, Microdata, or RDFa that the site already publishes for search engines.
- Internal API responses: Capture the JSON the page itself requests, which is already structured and usually more stable than markup.
- Regular expressions: Pattern match against raw text. Appropriate for well formed values such as prices or dates, fragile for anything else.
- Model based extraction: Use a language model to identify fields from page text without selectors, trading precision and cost for resilience.
Method Comparison:
| Method | Precision | Resilience to redesign | Cost per page |
|---|---|---|---|
| CSS selectors | High | Low | Negligible |
| XPath | High | Low to medium | Negligible |
| JSON-LD | High | High | Negligible |
| Internal API | Very high | Medium | Negligible |
| Regex | Medium | Medium | Negligible |
| Model based | Medium to high | Very high | Significant |
Prefer Published Structured Data:
Many sites already expose clean structured data because search engines reward it. A product page with Schema.org markup contains the price, currency, availability, and review count in a JSON block, placed there deliberately by the site.
Reading that block is more reliable than parsing the rendered markup around it. The visible layout changes with every redesign, while the structured block is maintained because breaking it damages the site’s search presence. Checking for JSON-LD before writing selectors is one of the highest value habits in extraction work.
The caveats are that published data is sometimes incomplete, occasionally reflects list price rather than the price actually shown, and is absent entirely on many sites.
Extraction with Fallbacks:
import json
from bs4 import BeautifulSoup
def extract_product(html):
soup = BeautifulSoup(html, "lxml")
record = {}
# Preferred source: published structured data
for block in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(block.string)
except (ValueError, TypeError):
continue
if isinstance(data, dict) and data.get("@type") == "Product":
offers = data.get("offers", {})
record["title"] = data.get("name")
record["price"] = offers.get("price")
record["currency"] = offers.get("priceCurrency")
break
# Fallback: selectors against rendered markup
if not record.get("title"):
node = soup.select_one("h1.product-title")
record["title"] = node.get_text(strip=True) if node else None
if not record.get("price"):
node = soup.select_one("[data-price], span.price")
record["price"] = node.get_text(strip=True) if node else None
return record
Structured data is attempted first, selectors cover the gap, and every lookup degrades to None rather than raising. A field that is missing is recoverable. A crash halfway through a batch is not.
Why Extraction Breaks:
Markup changes. A class renamed from price to price-current silently returns nothing. This is the most common cause of extraction failure and it produces empty fields rather than errors.
Inconsistent templates. Sale items, bundles, and out of stock products frequently use different markup from the standard template, so a selector validated on ten pages fails on the eleventh.
Client rendered content. If the value is injected by JavaScript, it is absent from the initial HTML and requires JavaScript rendering to appear.
Localization. The same site can return different markup, currency symbols, decimal separators, and date formats depending on the visitor’s region.
Validate What You Extract:
Silent failure is the defining risk. A broken selector returns an empty string, the pipeline stores it, and nobody notices until a downstream report shows a product that apparently costs nothing.
The defense is to check extraction output rather than trusting it. Confirm required fields are populated, confirm types and ranges make sense, and alert when the null rate for any field moves sharply between runs. A price field that was 99 percent populated yesterday and is 4 percent populated today indicates a site change, not a market event.
Bright Data’s Scraper APIs return parsed records with defined schemas for over 800 sites, and selector maintenance is handled on the provider side, which removes the most common source of pipeline breakage.