About this project
The Autoteile Data Engine is a monorepo containing three interconnected Python desktop applications built for the automotive parts industry. The ecosystem forms a complete pipeline from data collection to pricing intelligence, addressing the challenge of monitoring thousands of competitor prices across German auto parts platforms.
The web scraper (autoteile-markt-Scraping) uses fast HTTP requests to harvest product data from autoteile-markt.de — 4-6x faster than the original Playwright browser automation approach. It processes 1,375+ product references from a CSV input file, searches each reference on the platform, applies brand and MPN filters from JSON configuration, and extracts seller data including name, price, shipping cost, and delivery time. The scraper handles both single-offer products (scraped directly from the product card) and multi-offer products (navigating to the details page to extract all sellers from the 'Alle Anbieter zum Ersatzteil' section). Session tracking via JSON files enables resume capability, so interrupted scraping runs continue where they left off.
The PricePilot analysis engine (autoteile-Daparto-Data-Engine) is a PySide6 desktop application with a 6-tab analytical interface. The Overview tab provides a statistics dashboard with interactive Plotly charts — price distribution histograms, top seller bar charts, competitive product scatter plots, delivery time analysis, and shipping cost pie charts. The Data Table tab offers advanced filtering, search, and CSV/Excel export for exploring raw market data with 1.25M+ row support.
The Seller View tab computes position statistics — how many times a seller appears at each price rank (1st, 2nd, 3rd, etc.) within each brand group, with brand filtering and export capabilities. The Price Gap tab analyzes products where the seller holds position #1, calculating the gap to position #2 and classifying risk levels: Low Risk (safe to increase price), Medium Risk (caution), High Risk (not recommended), and No Competitor (free to increase). The Price Optimizer tab is the most complex — it accepts seller selection, multiple supplier CSV files with interactive column mapping, and pricing parameters (transport cost, minimum profit, platform commission, price gap). It computes recommended prices for every product where the seller isn't already #1, with three statuses: 'Go for #1' (profitable underbid possible), 'Sell at minimum' (floor price), and 'Can't be #1' (floor above market price).
The Market Entry tab evaluates whether new products from supplier catalogs should be listed on the platform, computing minimum profitable prices against existing market data, recommending entry strategies with position and profit analysis. All heavy calculations run in background QThread workers with real-time progress bars, keeping the UI responsive even with massive datasets.
A third variant (Daparto-Data-Engine, branded 'DataTitan-Daparto') represents the original monolithic architecture — same 6-tab feature set but with business logic embedded directly in worker thread methods rather than separated into testable pure-function modules. This served as the predecessor that was refactored into the modular PricePilot architecture.
All three tools share a PySide6 GUI framework with Fusion styling, custom stylesheets, and a modular architecture. A shared HMAC-SHA256 license system protects commercial use, with a keygen tool that generates time-limited keys accepted by both the scraper and analysis engine. The system handles the complexities of the German automotive parts market: European price formats with comma decimals, 'kostenlos' (free) shipping detection, brand name normalization across platforms, seller name deduplication, and delivery time extraction from German-language page elements.