* Make DELETE synchronous in clickhouse tests
This avoids some flaky behavior, example:
- https://github.com/PostHog/posthog/pull/2110/checks?check_run_id=1327279229
Documentation:
- https://clickhouse.tech/docs/en/operations/settings/settings/#mutations_sync
Tested it out by running the following test 1000 times - before change
it fails every ~50 runs locally.
```
def test_delete_person(self):
- person = person_factory(
- team=self.team, distinct_ids=["person_1", "anonymous_id"], properties={"$os": "Chrom
e"},
- )
- event_factory(event="test", team=self.team, distinct_id="person_1")
- event_factory(event="test", team=self.team, distinct_id="anonymous_id")
- event_factory(event="test", team=self.team, distinct_id="someone_else")
-
- response = self.client.delete(f"/api/person/{person.pk}/")
- self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
- self.assertEqual(response.data, None)
- self.assertEqual(len(get_people()), 0)
- self.assertEqual(len(get_events()), 1)
-
- def test_filters_by_endpoints_are_deprecated(self):
+ for index in range(1000):
+ print([index])
+ person = person_factory(
+ team=self.team, distinct_ids=["person_1", "anonymous_id"], properties={"$os": "Chrome"},
+ )
+ event_factory(event="test", team=self.team, distinct_id="person_1")
+ event_factory(event="test", team=self.team, distinct_id="anonymous_id")
+ event_factory(event="test", team=self.team, distinct_id="someone_else")
+
+ response = self.client.delete(f"/api/person/{person.pk}/")
+ self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
+ self.assertEqual(response.data, None)
+ self.assertEqual(len(get_people()), 0)
+ self.assertEqual(len(get_events()), 1)
```
* Add a if not TEST clause to deleting from materialized view
This blew up clickhouse when we try to force the delete to happen sync.
Created an upstream issue: https://github.com/ClickHouse/ClickHouse/issues/16548
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* add test runner to ease pycharm dev
* fix broken import
* drop and recreate the clickhouse test db before running tests
* fix person uuid str json serialization issue
* make kafka optional in tests
* fix inits
* remove need for kafka in person.py
* fix a bunch of mypy errors
* fix function and add process_event to pipeline
* fixed missing params and tests
* change uuid and fix types
* types
* optimize for merge prop test
* make ClickhouseProducer to produce to clickhouse one way or another
* annotate types
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
Co-authored-by: James Greenhill <jams@uber.com>
