Case Study: Transforming E-commerce Data Scraping with Accurate Product Mapping

About the Client

Our client, a leading global price-comparison platform, serves millions of users monthly by aggregating product prices, availability, and deals across multiple e-commerce websites and mobile apps. Their platform required reliable, real-time e-commerce data scraping services to maintain its edge in the competitive market.

Objective

To develop a scalable and robust scraping system that enables:

Challenges

1. Inconsistent Product Information

Products were listed differently on each platform, making it difficult to match identical items:

Example:

Platform A: "Nike Air Max 270 Sneakers (Black/White)"

Platform B: "Air Max 270 Shoes - Men's - Black & White by Nike"

2. Dynamic and JavaScript-Heavy Websites

Modern e-commerce platforms relied heavily on JavaScript and had anti-bot measures such as CAPTCHAs, IP bans, and rate-limiting mechanisms.

3. No Unique Product Identifiers

Many platforms lacked standardized identifiers like SKUs or UPCs, forcing us to rely on secondary product attributes.

4. Large-Scale, Real-Time Requirements

The client needed daily scraping of 25,000+ products across 10+ platforms, with real-time updates for critical products.

5. Handling Edge Cases

Products with minor differences, such as bundled items (e.g., headphones with/without carrying cases), needed careful handling to avoid mismatches.

Our Step-by-Step Solution

Step 1: Custom Web and Mobile Scraping

We designed tailored scraping solutions for each platform:

• Web Scraping
  • Used Scrapy and Selenium for handling JavaScript-heavy websites.
  • Implemented proxy rotation, CAPTCHA-solving tools, and user-agent switching to bypass bot detection.
• Mobile App Scraping
  • Leveraged Appium for Android and iOS apps to extract app-specific data.
  • Reverse-engineered APIs for seamless access to structured data when available.

Example: For a major retailer’s mobile app, where no web data was accessible, we extracted product details like images, pricing, and reviews via app API requests.

Step 2: Data Normalization

We standardized the scraped product data into a uniform format by:

• Extracting consistent fields such as brand, model, size, price, availability, color, etc.
• Cleaning text using NLP algorithms to remove redundant phrases and unify naming conventions.

Example:

• Before: "Adidas Originals Superstar White/Black Size US 10"
• After: Brand: Adidas | Model: Superstar | Color: White/Black | Size: US 10

Step 3: Product Mapping Algorithm

We developed a three-layer product matching system:

1. Fuzzy Text Matching:

• Compared product titles using algorithms like Levenshtein Distance to calculate similarity scores.

2. Attribute-Based Matching:

• Matched attributes like brand, model, specifications, and price to confirm matches.

3. Image Matching with AI:

• Used TensorFlow to analyze product images for visual similarities when text data wasn’t enough.

Example: Two listings had different names but identical images, allowing the system to confirm a match.

Step 4: Handling Edge Cases

For ambiguous cases, we created a manual review workflow via a custom dashboard:

• Flagged items like bundled vs. standalone products for human validation.
• Trained the machine learning model with feedback to continuously reduce manual interventions.

Example:

• Product A: “Sony WH-1000XM5 Headphones”
• Product B: “Sony WH-1000XM5 Noise Cancelling Headphones with Case”

Our system flagged the carrying case difference, enabling manual review for accurate mapping.

Step 5: Real-Time Updates

We built a real-time scraping pipeline using:

• Apache Kafka and RabbitMQ to process updates efficiently.
• Alerts for critical product changes (e.g., price drops) delivered within minutes.

Example: A user tracking the price of a laptop was notified of a $100 price drop within 5 minutes of the retailer updating their website.

Results Delivered

Why Choose ScrapeEngine?

1. Expertise in web and mobile app scraping at scale.

2. Proven track record of solving complex challenges like product mapping and real-time updates.

3. Cutting-edge use of AI and machine learning to enhance accuracy and scalability.

4. Trusted by global leaders for delivering reliable and compliant data solutions.

Let’s Collaborate

Are you struggling with e-commerce data challenges? ScrapeEngine offers web scraping services and API solutions to build the perfect solution tailored to your needs. Contact us today to get started on transforming your data into actionable insights.