In 2002 Barnes & Noble asked us to help design and implement their new order management system. This was in preparation for the upcoming Holiday season and the concern was not being able to keep up with the ordering demands of their internet customers. The single monolithic Unix server couldn’t keep up, couldn’t scale, and was being manually tweaked on a 24 hour basis to keep the business running.
The replacement system would be geographically distributed and would easily scale horizontally, and allow for quick and agile expansion for the future.
We built this system on top of the “Order Fulfillment System” that we had built prior. It tied together multiple geographic deployments of the “Order Fulfillment System”, around 10 or so, with real-time order status synchronization from each fulfillment system.
We were extremely successful with this project and were able to complete on time and meet the holiday demand. This gave B&N breathing room to continue growing out their online presence which increased sales over time and allowed the customer base to grow.
Dozens of services were used to asynchronously process the orders, a poor man’s micro-service architecture. We had a direct feed from the web site via SQL Server 6.5 replication. We used a stored procedure that would put in bound orders into an MSMQ queue that would kick off the various processes. Email, credit cards, inventory allocations, shopping to geographically distributed fulfillment centers etc.
We developed a framework that would allow several developers to work in parallel and to rapidly build and deploy the various VB6 services. The VB6 services were deployed as generic applications on a Wolfpack cluster, Microsoft’s initial implementation of their application clustering service. The VB6 services interacted with MTS, Microsoft’s initial COM+ offering, components and used DTC to integrate the MSMQ and MTS transaction logic.
4 clustered SQL server deployments were used as the central data repository (2 for reporting purposes, 1 for CRM / order management, and 1 for fail-over). The primary CRM / order management cluster synchronizes data with the 2 reporting clusters.
4 clustered application server deployments were used to orchestrate the order processing. MSMQ was used to manage the workflow via a proprietary ESB (Enterprise Service Bus). The order processing workflow was broken down into several synchronous / asynchronous sub-processes which were then each distributed across several queues to sustain the throughput. The system was extremely scalable in the fact that if any sub-process required more processing or if there were a larger than normal influx of orders for the Christmas season, more queues could be added for the sub-process. When an application server reached its maximum throughput, a new application server could be added with its own set of processing queues, and the existing load could then be split.
Upon completion of this project we were seeing the following transaction statistics
- 150K+ order confirmation e-mails per day
- 500K+ item sourcing and inventory allocation transactions per day
- 1000K+ item / order status change transactions per day
- 10000K+ OLTP to OLAP synchronization transactions per day
- 100K+ CRM transactions per day
- 10000K+ inventory update transactions per day