Tea-tasting package for A/B-testing in Python

Guys! Evgeny I. has just released something really hot!

This is "tea-testing" - convenient and easy-to-kick-off python package for a/b-testing. Here's what it consists of:

1. Computes statistics on the database side. This means there is no need to pull detailed data into Python. To achieve this, wrap an SQL query that returns a table with all the necessary columns in an Ibis Table. A row represents a unit of randomization in the test. Pandas can also be used as input.

2. Elegant, "Pythonic" API. A framework that helps reduce errors. All complex statistics are hidden under the hood.

3. Detailed documentation with examples.

Here's what it can do:

1. T-test and Z-test for means and mean ratios, Bootstrap for arbitrary statistics, and quantile test (via Bootstrap).

2. Delta method for ratio metrics (more accurate than linearization).

3. Sensitivity improvement using CUPED, including in combination with the delta method for ratio metrics.

4. Confidence interval for relative and absolute change.

5. Sample ratio mismatch check.

6. Statistical power analysis.

If you work with GA4 to BigQuery exports, be sure to check out my SQL cheat sheet.