Checkout fixed within 6 hours of the report
On a Thursday afternoon I received a report from an automotive parts store: there had been no orders since Monday, even though Analytics showed normal traffic. The existing contractor had tested purchases by hand and concluded that the store was working correctly.
Why the manual test didn’t catch the error
I spent the first hour backing up the database and files, and the second comparing logs from three weeks. The problem affected one payment method, chosen by 78% of customers.
On Sunday night the payment module had updated itself automatically. The new version changed the format of the gateway’s response. The store didn’t recognise the confirmation and left the order with the status “awaiting payment”, even though the operator had taken the money.
The client had tested purchases using a traditional bank transfer. That method worked, so the test didn’t reveal the failure. The whole diagnosis took four hours.
The fix and handling orders in progress
Over the next two hours I added support for the new format while keeping compatibility with the old one. This let the store correctly process earlier orders as well. I recovered 214 orders that had been stuck for three days. Customers received a message, including those who had already bought the products in another store.
Why the report only came after three days
The store had monitoring for page availability, response time and disk space. It did not, however, track the number of orders. The site stayed available, so the working alerts didn’t signal any problem with sales.
I added an SMS alert comparing the number of orders over two hours with the average for the same time window over the previous four weeks. The alert fires when orders drop below 30% of that average. Setting up the monitoring took a day of work. The cost of three days of downtime was more than a hundred times higher.
What changed after the intervention
- The checkout process was working correctly six hours after the report.
- I recovered 214 orders held up by the payment error.
- Automatic module updates were replaced with a procedure for checking changes on a copy of the store.
- Monitoring now covers a drop in the number of orders, not just site availability.