Building receipt parsing and product intelligence with Patrick Questembert, Engineering

Our team has always been passionate about the intersection of AI and the real world, sharing a bold vision to bring the benefits of AI to every person on earth. For nearly a decade, we’ve been developing and delivering diverse products that currently impact more than 300M users across 60 countries.

In an effort to showcase more of our team behind the scenes, we sat down with Patrick Questembert from Engineering who has been building our purchase data solutions from the very beginning, to learn more.

How did you first join the Microblink team?

Our CEO, Darren, was looking for people with prior expertise in extracting text from images in order to help him build and implement his vision to cull purchase data from printed receipts. Our VP of Engineering and I had previously worked on “ScanBizCards,” which was the first mobile app to scan and capture business cards to transform them into address book entries. Darren’s enthusiasm and drive were contagious, so I joined a couple of months after that first meeting.

What was the original problem the team was looking to solve, and how did it lead to our receipt scanning technology?

The key technical problem we needed to solve early on in 2015 was to recognize characters on printed receipts, then interpret that text to understand what products were purchased, the prices paid, which merchant, method of payment, etc. That first-party consumer purchase data is very valuable for a number of companies. While we started with physical receipts, as the retail landscape and consumer shopping trends continue to move online, we’ve evolved our technology to handle online/eCommerce purchases as well.

We work with some of the world’s largest consumer shopping apps, some of whom incentivize their users to snap photos of receipts in exchange for rewards. Our tech can be embedded in the form of a receipt scanning SDK for iOS & Android, as well as a web-based API.

Can you explain in layman’s terms how our data enrichment works. What makes it so magical?

With physical receipts, for example, our software analyzes one or more images of a printed receipt [as many images as required to capture the entire receipt] and performs the “magic” required to ultimately return a digital representation of all the information. We further return additional data about the products or the merchant, not found on the receipt. For example, we expand the abbreviated short product descriptions into full product names, then tap into our product catalog to identify the exact product, its UPC code, category, and more. “CPH SENS & EN SHEI” on a CVS receipt gets mapped to “Crest Pro-Health Sensitive & Enamel Shield Toothpaste” in the personal care > oral care > toothpaste category.

Like many AI-powered products, the magic happens through a combination of large machine learning models and human-built code to complement the ML where applicable. In this instance, we’re dealing with models that recognize words or brand abbreviations across receipts and expand them into full names, alongside our own catalog of 15 million products and growing.

How do we handle online purchase data?

Early on, it became clear that our clients also needed the ability to capture their users’ electronic receipts. eCommerce is growing every year, representing an ever larger percentage of consumer purchases. Online purchases are complex; some eReceipts include the actual receipt in an attached PDF [as opposed to listing the products within the HTML in the email itself], while some merchants don’t include any product information at all within the confirmation email, instead providing a link to the customer’s account where purchases exist in their order history. It’s far from trivial to identify products from confirmation emails or extract the various product properties (e.g., price, quantity, product number, order number, shipping fees, etc.) accurately and efficiently.

Microblink takes an innovative, multi-pronged approach to online purchase data collection that lets consumers link their email inbox or connect to supported merchants via a mobile app or through a browser. As opposed to the process of capturing and uploading a physical receipt, this is more of a “set it and forget it” routine for consumers once they’ve entered their credentials. 

One benefit of a direct merchant connection is that purchase data can be collected from as far back as three years, enabling a vast volume of first-party purchase data insights very quickly.

What projects are you currently working on?  

Believe it or not, I am still spending most of my time on our physical receipts and eReceipts scanning technology — eight years later! My role within the technology stack is to figure out things the ML models couldn’t on their own, which means solving new and challenging problems all the time. So although my “playground” is the same every day, the problems and their solutions change all the time. I’ve stopped counting, but I’ve written close to one million lines of code over the years, which, for an engineer, would not be possible had I not enjoyed it.

What are some standout moments over the years?

One highlight was when I realized we had scanned several billion physical receipts — billion with a “b,” not a typo!

Also, very early on, I recall feeling very proud when we got to a point where the accuracy was incredible, no matter the quality or condition of the physical receipt: bright or dim light, crumpled receipts, faulty point of sale printers and all, we were able to capture and read it.

As cliche as it sounds, my time at Microblink is a reminder that it’s okay to realize mistakes, as long as you adjust quickly. It’s essential to surround yourself with talented people who care more about doing good work than looking good, and that helps me love what I do, which is important because business success alone will only motivate you so far.