Improve Brand Scraping Stability & Add Error Tracking

Several brand scraping jobs are failing due to site structure changes and missing error detection. This causes outdated or incomplete data in PIM and the Public API.

This task aims to stabilize scraping by:

  • Updating scraping logic to track execution time and success/failure status.

  • Running a full scraping batch for all brands to identify broken scrapers.

  • Improving the scheduler to limit parallel scraping jobs.

  • Adding standardized brand helper modules for easier maintenance.

  • Sending Slack alerts when a brand scraper returns empty or fails.

Please authenticate to join the conversation.

Upvoters
Status

Completed

Board
πŸ’‘

Feature Request

Date

3 months ago

Author

Raihan Ayita Putri

Subscribe to post

Get notified by email when there are changes.