Metadata-Version: 2.4
Name: ab-analysis-kit
Version: 0.1.0
Summary: A/B experiment analysis as declarative YAML + SQL
Author: ab-analysis-kit authors
License: MIT
Project-URL: Homepage, https://abkit.pipelab.dev
Project-URL: Documentation, https://abkit.pipelab.dev
Project-URL: Repository, https://github.com/alexeiveselov92/ab-analysis-kit
Project-URL: Changelog, https://github.com/alexeiveselov92/ab-analysis-kit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/alexeiveselov92/ab-analysis-kit/issues
Keywords: ab-testing,experimentation,statistics,cuped,bootstrap,causal-inference,clickhouse
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: statsmodels>=0.14.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: click>=8.0
Requires-Dist: jinja2>=3.0
Requires-Dist: orjson>=3.0
Requires-Dist: requests>=2.25.0
Provides-Extra: clickhouse
Requires-Dist: clickhouse-driver>=0.2.0; extra == "clickhouse"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Provides-Extra: mysql
Requires-Dist: pymysql>=1.0.0; extra == "mysql"
Provides-Extra: all-db
Requires-Dist: clickhouse-driver>=0.2.0; extra == "all-db"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all-db"
Requires-Dist: pymysql>=1.0.0; extra == "all-db"
Provides-Extra: orchestration
Requires-Dist: prefect>=3.0; extra == "orchestration"
Provides-Extra: all
Requires-Dist: clickhouse-driver>=0.2.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
Requires-Dist: pymysql>=1.0.0; extra == "all"
Requires-Dist: prefect>=3.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-mock>=3.0; extra == "dev"
Requires-Dist: requests-mock>=1.12; extra == "dev"
Requires-Dist: black==24.4.2; extra == "dev"
Requires-Dist: mypy==1.10.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: integration
Requires-Dist: testcontainers[clickhouse,mysql,postgres]>=4.0; extra == "integration"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "integration"
Requires-Dist: pymysql>=1.0.0; extra == "integration"
Dynamic: license-file

# ab-analysis-kit

**A/B experiment analysis as declarative YAML + SQL — with a chart-first cockpit.**

`ab-analysis-kit` (CLI `abk`) is an open-source, declarative
(dbt / [detectkit](https://dtk.pipelab.dev)-style), database-agnostic, numpy-first
Python library for analyzing A/B experiments. You define an **experiment** and its
**metrics** in YAML + SQL; abkit computes per-method effect + confidence interval +
p-value + MDE/power **cumulatively over the experiment's lifetime** (the
stabilization chart), writes them to a clean warehouse table any BI can read, and
gives you a local cockpit to tune the analysis and a harness to prove your method is
actually calibrated.

> **Status: `0.1.0` (Alpha) — the first release, prepared** (milestones **M1–M6**). The
> statistical core, the declarative config / DB / pipeline layer, the explore cockpit +
> self-contained reports, `abk validate`, opt-in sequential analysis + `abk plan`, and the
> DX layer (`abk init-claude`, docs site, Prefect scaffolding) are all shipped. The tagged
> PyPI publish is the maintainer's pending step. Docs: [abkit.pipelab.dev](https://abkit.pipelab.dev).

## Install

Once `0.1.0` is published to PyPI (a maintainer tags `v0.1.0`; until then install from
source — `pip install -e ".[dev]"`):

```bash
pip install ab-analysis-kit          # Python 3.10+; add a DB extra for real data:
pip install "ab-analysis-kit[clickhouse]"   # or [postgres] / [mysql] / [all-db]
```

`abk --version` and `abk --help` work with no database driver; you can even lint a
config (`abk run --steps validate`) with no database at all. See the
[getting-started guide](https://abkit.pipelab.dev) for the full first run.

## What it does

- **Declarative experiments** — `experiments/*.yml` (assignment + variants +
  comparisons) referencing a reusable `metrics/*.yml` library (YAML + SQL).
- **A rigorous statistical engine** — t-test, two-proportion z-test, CUPED, ratio
  (delta-method), and a vectorised bootstrap family (plain/paired/Poisson/
  post-normed), with relative & absolute effects, MDE/power, and multiple-testing
  correction. Ported from a battle-tested legacy engine and improved deliberately.
- **The cumulative stabilization chart** — effect + CI per day from experiment
  start, so you see the estimate converge and call a winner only once it stabilizes.
- **`abk explore`** — a local, chart-first cockpit to turn method knobs (CUPED,
  stratification, alpha…) and watch the result recompute live, with A/A calibration
  always in view. *The priority interface.*
- **`abk validate`** — an A/A false-positive + power matrix that measures your
  method's **real** α (including the honest cumulative-peeking FPR), not the nominal.
- **BI-agnostic** — results land in one clean table; connect Grafana, Lightdash,
  Metabase, or Superset. Orchestrate with **Prefect**.
- **AI-native** — `abk init-claude` sets up assistant context + skills so an
  assistant can scaffold and tune experiments with (or for) you.

## Design at a glance

```
experiment (YAML)  ──▶ load exposures ──▶ SRM gate ──▶ compute (t/z/CUPED/bootstrap) ──▶ readout
  └ references reusable metrics (YAML + SQL)                                          └ _ab_results → your BI
```

abkit is the sibling of detectkit: same DNA (CLI-first, db-agnostic, numpy-first,
self-contained reports, a chart-first cockpit, `init-claude`), with the anomaly
`detect` stage replaced by a statistical `compute` stage and the primary entity
flipped from *metric* to *experiment*.

## Documentation

- **Docs site:** [abkit.pipelab.dev](https://abkit.pipelab.dev) — getting started, guides, reference
- **Roadmap:** [ROADMAP.md](ROADMAP.md) · **Principles:** [PRINCIPLES.md](PRINCIPLES.md)
- **Contributor guide:** [CLAUDE.md](CLAUDE.md) · design contracts in [docs/specs/](docs/specs/)
- **Master plan (RU):** [docs/ru/project-initiation-spec.md](docs/ru/project-initiation-spec.md)

## License

[MIT](LICENSE).
