- catch almost all errors: They are sent to Sentry anyways. There is no benefit at this point to make the Celery task fail.
- consolidate query status `complete` flag to include complete results as well as errors
- this makes it easier to use, no more separate checking for errors
- make results live longer: if the chain takes some time to complete, previously results expired/vanished while a dashboard was polling for them
- this fixes POSTHOG-1BC6
- other refactorings
* Revert "revert: "chore(clickhouse): Capture final SQL in Sentry errors" (#21479)"
This reverts commit 15627818f6.
* Only upgrade `clickhouse-driver` to 0.2.6
* Add "Run migrations for this PR"
* Upgrade dependencies
* Fix middleware error
Among
https://docs.djangoproject.com/en/4.2/releases/4.0/#features-removed-in-4-0
* Upgrade psycopg
We need to be on >= 3.1.8
Locally there is an additional problem that somehow psycopg2
seemingly overshadows psycopg, making it appear that 3.1 works.
Had to install pip install "psycopg[binary,pool]==3.1.2" to
recreate the problem.
* Go to Django 4.1 because of problems with psycopg3
We use custom SQL that somehow doesn't get formatted in the right way
using server or client side cursors.
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Switch TaggedItem tests to assert ValidationError
Because full_clean validates since Django 4.1, see
https://docs.djangoproject.com/en/4.2/releases/4.1/#validation-of-constraints
* Remove type: ignore comments
Come up as
error: Unused "type: ignore" comment
* Update query snapshots
* Figure out psycopg problem and try Django 4.2 again
* Update query snapshots
* Fix other IN errors
* Fix getting status
* Fix psycopg3 issues
* Fix psycopg issues
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update deps
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Fix more tests
* Adjust baseline
* Remove sqlcommenter (should be PostgresQL only anyways)
* Fix file
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Fix queries
* Fix query
* Revert
* Update requirements.in
* Remove restore-virtualenv
Because not maintained anymore
* Revert "Remove restore-virtualenv"
This reverts commit c2a7ef8a1e.
* mypy
* Adjust num queries
* Adjust num queries
* Adjust num queries
* Update query snapshots
* Add to updated_fields
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* feat(batch-exports): Add HTTP Batch Export destination
To possibly be reused in the future, but for now it only submits
payloads in the PostHog /batch format.
* add geoip_disable, don't toJSONString elements_chain, and mark some HTTP status codes as non-retryable
* Add heartbeating to HTTP batch export
* Update query snapshots
* Update query snapshots
* fix: Re-use client session
* refactor: Rename last_uploaded_part_timestamp to last_uploaded_timestamp
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* Upgrade pydantic and all related
* Upgrade mypy
* Add mypy-baseline
To update baseline when you fix something (only then!) use:
[mypy cmd] | mypy-baseline sync
* just code for migrations
* use all timezones because update would cause common_timezones to be less inclusive
* install new dependencies
* add comment
* revert
* restore
* fix type
* chore(batch-exports): add snowflake export workflow
This workflow uses Snowflake internal stages to load data from
ClickHouse into a Snowflake table. We maintain the existing events table
schema as used in the existing Snowflake App.
Something I haven't done yet is:
1. made sure e.g. we get the `elements` and `person_set` etc. data into
Snowflake.
2. the additional frontend to enable configuring the Snowflake
connection.
* remove unsed var
* include excluded events in test
* feat(batch_exports): add backend API and S3 temporal workflow
This adds the backend API for batch exports, which will handle reverse
ETL exports to e.g. S3, Snowflake etc.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
