Tools: SQL, Python, Power BI

Dashboard Link: Live Dashboard Here

Python (A/B and Hypothesis Test): Python Notebook

View Data Schema: Data Schema

GitHub Link: Check SQL Queries, and more.

Note: After clicking on Live Dashboard, please select the “Fit to Page” or “Full Screen Mode” option on the live screen so that the dashboard can adjust according to your desktop or laptop system.

Overview:

This is an end-to-end Sign-up Rate Improvement project/solution. It focuses on the entire pipeline, including the problem (low sign-up rate), defining the hypothesis (a newer signup solution improves the sign-up rate), designing the experiment (A/B test), defining the success metric (sign-up rate), running the test, statistical validation, and analysis of results using hypothesis testing.

It includes the creation of the data schema (check above), converting the business problem into KPIs, metrics, and relevant solutions, SQL queries to clean, pre-process, transform, and EDA on data, along with View generation for dashboard building.

It includes a Python notebook (check above) that includes some A/B test information, hypothesis testing, robustness check, and more for the technical team members to refer to. It also involves comprehensive data modeling in a BI tool (Power BI) to build a detailed yet simple dashboard for leadership and stakeholders to make informed data-driven decisions.

Situation

The startup is facing low signup rates. Lower signup rates are creating difficulties for the startup to grow the way they were expecting.

Task

I collaborated with the relevant teams (Product and UI/UX) to simplify the sign-up page (page and form) and run the A/B test on 5% of the traffic.

Action

The success metric for the test was sign-up rate (Sign-up completed / Total users). The variant group saw the new signup solution, and the control group saw the older one. Also, traffic was equally divided. The test ran for around 2 weeks to get statistically significant results.

Result

After getting test results, we conducted a two-proportion z test (hypothesis test), which gave a p-value < 0.05, which confirmed that sign-up rates are statistically better in variant groups. We launched the new sign-up solution to all the traffic, improving sign-up rates significantly.

Some Quantifiable Results:

• Defined success metric (sign-up rate) for A/B test, selected traffic percentage, event type, exposure type, and more to get a relevant, unbiased comparison between control and variant groups.
• Conducted end-to-end hypothesis testing, confirming that variant groups have better sign-up rates (10-19%) statistically.
• Connected SQL with Python using SQL Alchemy and psycopg2, reducing data access time by 40%.
• Improved decision-making by 35% by creating a Power BI dashboard, conveying insights in a simple yet effective manner.
• Performed a robustness check to confirm the statistical significance, increasing confidence in the result.