mirror of
https://github.com/PostHog/posthog.git
synced 2024-11-22 08:40:03 +01:00
5fb811a365
* perf(lifecycle): add benchmark for lifecycle * Add some notes on running benchmarks locally
66 lines
2.3 KiB
Markdown
66 lines
2.3 KiB
Markdown
# Clickhouse query benchmarks
|
|
|
|
This is the benchmark suite for PostHog clickhouse queries. It tracks performance improvements to clickhouse queries over time.
|
|
|
|
The benchmarks are run using [airspeed velocity](https://asv.readthedocs.io/).
|
|
|
|
To get stable results over time, a stable clickhouse node which has been pre-filled with data is used to run against.
|
|
|
|
Historical benchmark results can be found in https://github.com/PostHog/benchmark-results.
|
|
|
|
# FAQ
|
|
|
|
## Benchmarking your PRs
|
|
|
|
Benchmarks run every day against master branch.
|
|
|
|
If your branch contains significant changes to query performance, add `performance` label to your PR.
|
|
|
|
An action will then run and comment on the benchmarks on your PR.
|
|
|
|
## Installation (local)
|
|
|
|
These benchmarks are run using *airspeed velocity* so, you need to have
|
|
``asv`` installed which in turn needs virtualenv (or an anaconda dist),
|
|
|
|
```bash
|
|
pip install asv virtualenv
|
|
```
|
|
|
|
## Running the benchmarks locally
|
|
|
|
These benchmarks are mostly run in CI for:
|
|
- master branch
|
|
- PRs labeled with `performance`
|
|
|
|
To run the all the benchmarks locally, [get access to the clickhouse node](https://github.com/PostHog/vpc/blob/main/client_values/benchmarking/values.yaml) and:
|
|
|
|
```bash
|
|
# Set up machine
|
|
asv machine --machine ci-benchmarks --config ee/benchmarks/asv.conf.json
|
|
# Replace X with appropriate credentials
|
|
CLICKHOUSE_HOST=X CLICKHOUSE_USER=X CLICKHOUSE_PASSWORD=X CLICKHOUSE_DATABASE=posthog asv run --config ee/benchmarks/asv.conf.json
|
|
```
|
|
|
|
You'll probably want to be running one test, with quick iteration. Running e.g.:
|
|
|
|
```
|
|
asv run --config ee/benchmarks/asv.conf.json --bench track_lifecycle --quick
|
|
```
|
|
|
|
will run any benchmark regex-matching `track_lifecycle` only once.
|
|
|
|
See [asv documentation](https://asv.readthedocs.io/en/stable/commands.html#asv-run) for additional information.
|
|
|
|
## Adding new benchmarks
|
|
|
|
Edit the `benchmarks.py` file as needed. Use `@benchmark_clickhouse` decorator to select tests to run
|
|
|
|
## Backfilling benchmarks
|
|
|
|
- Clone `https://github.com/PostHog/benchmark-results` locally under ee/benchmarks/results
|
|
- Run something like `CLICKHOUSE_HOST=X CLICKHOUSE_USER=X CLICKHOUSE_PASSWORD=X CLICKHOUSE_DATABASE=posthog asv run --config ee/benchmarks/asv.conf.json --date-period 4d master~500..`
|
|
- Run `asv publish` and commit the changes to benchmark-results repo
|
|
|
|
If you have questions, use benchmark.yml github action as a guide.
|