That Time I Measured the Impact of a Sign-Up Flow Change

An ex-post analysis dilemma

Summer. Many moons ago. The company I was working for changed the sign-up flow by adding a feature that allowed customers to upload their ID during the sign-up process, preventing them from having to send it later by email.

I don't want to dwell on the background too much. Let's just say that this industry requires ID verification before allowing customers to use their accounts. Back in the day, those IDs had to be manually verified by an agent, one by one (!).

It sounds pretty old-school, but hey—nowadays there are plenty of tools that do that automatically. What a time to be alive!

My boss asked me to assess whether or not this product enhancement had an effect on our sites or our customers. As I stood numbly in the open space, she mumbled a list of KPIs while rushing between meetings. Fairly noted—but I decided to spice things up a bit and give my two cents on the matter.(I’ve always been a nosy little boy—photo evidence below😊)

Early '90s. Me and my dad at the beach somewhere in Southern Italy.Damn, how I miss my hair...

First of all, I wanted to prove whether customers were more likely to upload their ID on the spot, since:

• They no longer had to interrupt the sign-up process

• They didn't need to leave the site and open their email provider to send a .pdf to an anonymous address

I could’ve used the new data stream that provided the clean number of customers who uploaded their ID directly in the sign-up form.But that would’ve meant working without any data from before the release.

Instead, I decided to craft my own KPIs (damn, that sounds bolder than it actually is!) using whatever data was available.The goal was to gather as much historical data as possible to have comparable pre/post-release material.

Step 1 consisted in studying the ratio of customers who uploaded an ID on their sign-up day.

As you can see in the graph above, a clear positive change in the trend is visible after the release date (marked by the grey vertical line).We moved from an average of 15–20% to around 35–40%, with spikes up to 50%.

Customers were indeed more than happy to upload their IDs on the spot!

Last but not least, we wanted to check whether having customers' IDs in our hands sooner had an impact on Customer Care’s time-to-validate.First in, first out, right? Let’s see.

Each line represents a cohort of customers who signed up in a specific month, and every point on the line shows the cumulative percentage of customers in the cohort who reached “Active” status after their ID was validated.

Cohorts that signed up after the go-live performed better, reaching a higher cumulative percentage in a shorter time.

Reducing the distance between sign-up and document upload had a positive impact on the time needed to validate IDs.

Long story short: we got customers on board faster and hassle-free!

Even though all of this might sound simple or straightforward to most, it meant a lot to me at the time. I was (relatively) young, just a few weeks into a new job, and this was the first task I was asked to carry out end-to-end by my new boss. (Thanks for the trust, by the way.)

Bye for now!

Marco

DISCLAIMER: Although the story above is entirely true, the data shown has been anonymized as much as possible.