Mercari is a C2C e-commerce marketplace – think one big garage sale! Their rubric used to focus on accuracy and adherence to internal policies - but their program felt too punitive, and didn’t really highlight exceptional customer connection in the way that they wanted it to. One mistake could lead to a failing grade, and you couldn’t really tell when an agent went above and beyond to create a great experience for the customer. 

This conversation will walk through the changes that they made to their rubric to turn QA into more of a coaching function, and also to get a better sense of when agents are delivering creating special connections with customers.

Challenge: QA scores are very high, but couldn’t tell the difference between an average vs exceptional customer interaction

We had been reporting a 96-98% QA score for as long as anyone could remember. Sure, we could detect and report mistakes from individual agents, especially the newer ones. But overall, this wasn’t telling us anything new about the program as a whole. 

It was like hiring someone to sit in the middle of the desert in Arizona and say “Yup, it’s still hot here”.

We were highly adherent to our policies (which is what our QA rubric was asking about), and that was a good sign, but we weren’t learning and getting anything new out of the QA program because we couldn’t tell the difference between compliant and excellent interactions.

Impact: No way to distinguish between excellent and just-compliant agents, low engagement in QA program

There was only a 4% difference between passing and perfect when your passing grade is 96%. There was hardly any distinction between a good agent who knew all their policies, and had that memorized, and an agent who was going above and beyond to do excellent work for customers. 

One side effect is that that doesn’t incentivise the kind of customer interactions that drive loyalty. Another side effect of this was that no one was opening their QA grades any more. They either got a 100%, or they had an error and failed - there was really no incentive for an agent to engage with the QA program.

Solution: Change rubric to focus on exceptional CX, change scoring expectations

We overhauled everything. As a leadership team, we first agreed on what the issues were, then changed our whole paradigm. 

This goes above and beyond just changing up the rubric. We had to change our mindset and what we considered to be excellent work. 

Our rubric changed from looking for errors, to focusing on highlighting excellent work. The passing score also changed from 96% down to 80%.

If you got everything right procedurally, you’d get an 80% and you would pass, no problem. That’s table stakes.

But the remaining twenty percent was there to incentivize the agent to go above and beyond for the customer.

Impact: Alignment with company values and competitive differentiation

This change boils down to the question “what kind of business do we want to be?” In a crowded space like ours, our sellers choose to use us over eBay, Craigslist or Letgo, not simply because of our low selling fees, but because of the services and seller/customer protections that we provide when things go wrong.

Craigslist would be the extreme opposite example here: no one really expects CX when things go wrong in a Craigslist deal - you’re listing a product on essentially an unregulated site. But with Mercari, our users expect a certain level of assistance when things go wrong, and we need to be there to help.

CX is our main competitive differentiator against the other marketplaces, and it’s essential for us to invest in it.