Supercharged E-Commerce Search: How Indium Implemented Elasticsearch for a Response Time <1 second
Client Overview
The client operates an internet-based price comparison website, providing consumers with quick comparison information for e-commerce products to help them find the best deals. Their platform aggregates data from over 1500+ retailer websites, enabling users to have real-time visibility of products and their dynamically changing prices. The client required an efficient data handling system capable of processing real-time data requests and responses to meet this challenge.
The Growing Complexity of Online Price Comparison for a Retail Company
With the retail industry rapidly moving online, the rise of various digital platforms—such as e-commerce websites, second-hand marketplaces, retailer-owned portals, and online catalogs—has made it difficult for consumers to compare prices across multiple sources and secure the best deals. Given these challenges, the client’s need for their platform is:
Key Requirement:
Categorizing and redefining the data gathered from multiple eCommerce sources into a standardized format before storing it to address the product taxonomy challenge.
Achieving a response time of 1 second or less for processing data refresh requests in order to meet the end user's expectation of receiving price updates within 2 seconds.
Addressing the issue of massive data storage loads resulting from collecting data from over 1500+ retailer websites.
The current single-line indexing approach to data storage is inefficient, with the average data storage process taking approximately 25 hours.
Instant Price Discovery: Indium Accelerated High-Speed Data Indexing
The key highlights of the solution include:
Indium Software implemented a powerful Elasticsearch engine in a Big Data environment for efficient indexing.
A machine learning algorithm using Google Taxonomy was implemented to standardize the product data, addressing the product taxonomy challenge.
The Big Data ecosystem was utilized to process billions of data points, resulting in a response time of less than 1 second for any number of query requests.
The solution leveraged the ETL (Extract, Transform, Load) batch processing capabilities of Elasticsearch, enabling the processing of 10GB of data in approximately 1 second
This helped solve the single line indexing problem, improving data storage efficiency.
Optimized an E-Commerce Platform
By leveraging Indium’s expertise in Elasticsearch for data indexing, the client achieved unparalleled speed, precision, and efficiency, enabling real-time insights and seamless data retrieval for enhanced business performance.
Implementing efficient data storage using batch processing reduced the process time from 25 hours to 7 hours, improving data storage efficiency.
The solution demonstrated powerful data load carrying capacity, allowing for the collection and storage of data from over 1500 retailer websites.
The data refresh and response time were optimized to nearly 1 second, providing real-time price information to users without any latency.
Enhanced user experience allowed for quick price comparisons.
About Indium
Indium is an Al-driven digital engineering company that helps enterprises build, scale, and innovate with cutting-edge technology. We specialize in custom solutions, ensuring every engagement is tailored to business needs with a relentless customer-first approach. Our expertise spans Generative Al, Product Engineering, Intelligent Automation, Data & Al, Quality Engineering, and Gaming, delivering high-impact solutions that drive real business impact.
With 5,000+ associates globally, we partner with Fortune 500, Global 2000, and leading technology firms across Financial Services, Healthcare, Manufacturing, Retail, and Technology-driving impact in North America, India, the UK, Singapore, Australia, and Japan to keep businesses ahead in an Al-first world.
