* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* test(person-overrides): Update tests to match new constraint
* fix(migration): Add drop extension query to reverse migration
* fix(migration): Use correct table name
* refactor(person-overrides): Make team a regular bigint field without FK
* refactor(person-overrides): Rename Helper model to Mapping
* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* wip: add test for concurrent updates to posthog_personoverride table
* refactor
* Update snapshots
* fix(isort): Correctly sort imports
* fix(ee-cohort-test): Delete person after creating it
* fix: person constraint in person overrides table (#14319)
* nits
* fix migration tests
* chore(migration): Bump migration number to 0302
* Update snapshots
* feat: person-overrides writes
* test(person-overrides): Add concurrent tests to person-overrides model
* feat(person-merge): Update merge to use new helper table
* fix(tests): Pass poEEmbraceJoin to updatePersonState in test
* fix(person-state): Format person-overrides message for ClickHouse
* test(api): Add function to reload dictionary for person overrides
* fix: poe final test failure
* refactor(person-state): Make failed attempts a class variable to
This allows us to mock it during testing as some tests require
immediate failures.
* fix(postgres-utils): Apply some magic changes lost to time
* fix(person-state): Join with helper table to return UUIDs
* fix(person-state): Use single quotes for UUID queries
* test(person-overrides): Skip test that doesn't work without a merge command
* test(person-state): Add a very complicated query to get UUIDs in a test
* test(person-state): Try waiting longer, this is flaky
* fix(migrations): Remove unused migration
* fix(person): Undo unneeded person model changes
* chore: Clean-up artifacts from rebase on model branch
* refactor(person-state): Rename mergeAttempts and don't read from ENV
* chore: Better clarify oldest_event usage in comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* refactor(person-state): Update version in queries
* fix(person-state): Use new mapping model instead of helper
* fix(migrations): Re-add constraint deleted on field drop
* revert: Re-enable test that was skipped
Test was originally skipped due to missing a $merge command, but now
$merge_dangerously is available, so the test should pass.
* test: Attempt to refresh dictionary in test
* Update query snapshots
* Update query snapshots
* test: Expect dictionaries to be refreshed before resuming test
* fix(test): Use alias property for $merge_dangerously in test
* fix(test): Expect any string like in all other tests
* test: Fix order of events to determine merged persons
* chore: Update comment regarding overrides mapping query
* chore: Remove out of date comment
* test: Set number of retries to 0 when updating persons
* test: Delete comment
* test: Assert clickhouse state after identify
* chore: Move comment
* chore: Delete TODO regarding oldest_event updates
* test: Assert events are still processed if merge fails
* test: Assert properties are processed if merge fails
* test: Don't expect to throw when failing merges
* fix: Indent RETURNING query clause
Co-authored-by: Harry Waye <harry@posthog.com>
* chore: Remove out of date comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* test: Resume merges after test suite
* test: Run processPersonStep tests for both poEEmbraceJoin modes
---------
Co-authored-by: Harry Waye <harry@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* fix(queries): explicitly prepend table person_id comes from in cohort queries with pdi joins
* Update snapshots
* another test
* Update snapshots
* Update snapshots
* Update snapshots
* another try
* fixes
* Update snapshots
* trigger new test run
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Revert "chore(cohorts): Revert Add acceptance criteria for new cohorts and feature flag functionality (#9623)"
This reverts commit 8e277ee237.
* better defaults for old cohorts that dont quite work
* Revert "chore(cohorts): Add acceptance criteria for new cohorts and feature flag functionality (#9594)"
This reverts commit 2b86c61a6d.
* fix: revert
* add fields to property
* add validatoin
* fix naming'
* fix errors
* example
* example implementations
* remove none
* more typing
* change condition
* add terrible draft of lifecycle query
* move around date query
* one random test to satisfy
* add funnel persons subquery
* basic func
* use key as event
* change condition
* change to countif
* add base query conditions
* add comments
* condition building
* person props
* param cleanup
* basic test
* stub tests
* remove unnecessary funcs
* adjust typing
* wip
* add filters to cohorts
* add migration, start refactoring property types
* proof of concept of property refactor
* add migration too
* clean up property types
* slight more clean up, add comments
* removed optimizer
* add comment
* support performed_event_regularly
* remove unneeded
* sql params on regular query
* more tests
* typing
* some more clean up, enable person property push downs
* add snapshots too
* clean up types and validation
* fix typing
* remove unused snapshot
* fix bug
* Update ee/clickhouse/queries/cohort_query.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* oops
* update snapshots
* use materialized person props
* update fields
* add date value filter and tests
* event sequence join
* more join magic
* typing
* build fields
* see which tests fail
* resolve migrations conflict
* add some more tests
* fix first_time and event_restarted query
* add negation handling
* update snapshots
* add tests for multiple events case
* update stopped performing implementation
* regularly period validation
* old cohort support
* clean up + some tests
* fix some cohort tests issues
* remove migrations from this pr
* clean up
* support negations
* update snapshots
* more tests
* did all tests just pass?
* fix new failing tests
* fix tests, try new format for cohort insertions
* inter-test dependency resolve
* gracefully handle empty cohorts
* add type annotation
* address comments
* remove properties select in cohort removal query
* address comment
* update snapshots
* add freeze time
* fix some tests
* remove the Xs
* more test fixes and clean up
* raise on cyclic dependencies instead
* fixes
* test waters with parallel execution
* fixes
* update tests
* final test
* clean up
* more test fixes
* gahhhhh
Co-authored-by: eric <eeoneric@gmail.com>
Co-authored-by: Rick Marron <rcmarron@gmail.com>
* Remove Event dependency on action api tests
* Remove a dead function
* Remove BaseQuery
* Remove dead imports
* Remove Event creation from posthog/test/test_person_model.py
* Remove Event.earliest_timestamp function
* Remove some unused event model methods
* Remove query_db_by_action + associated migration code
* Remove dead filtering methods from Events model
* Remove a dead test class
* Remove some event model usage
* Remove events model usage from actions test
* Remove session recording related views
* Remove model usage in posthog/queries/session_recordings/session_recording.py
* Remove old pg-session recording code
* Remove dead import
* Re-add missing dependency
* Make lint/tests pass
* Make filter tests uuid-based
* add the fix and let CI say if that breaks any tests
* allow getting team IDS without direct substitution and mark all the places to check with TODOs
* pass team id in from everywhere and see what tests fail in CI
* named params everywhere
* address mypyc
* get tests passing
* fix tests after merge from master
* remove SQL query that's not referenced from anywhere
* check some marked queries are templated correctly now
* remove unused SQL
* check some marked queries are templated correctly now
* no optional team id
* more passing of team id into things
* don't trust autocomplete
* address review
* add test
* update func
* add include and settings
* diff method
* some prelim changes to make backend work
* add tests and logic for handling groups
* setup basic mixin parsing
* add tests
* types and switchover
* adjustment
* update snapshots
* update more snapshots
* update more snapshots
* update more snapshots
* typing
* snapshots
* clean up TODOs and types
* separate to_dict
* another property dict fix
* use operator dependening on condition
* fix tests
* update tests
* remove dummy condition
* update tests
* change name to groups
* another label
* rename filter_group to property_group
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* version insights
* version and lock update
* make sure all tests work
* restore exception
* fix test
* fix test
* add specific id
* update plugin server test utils
* cleanup
* match filtering
* use timestamp comparison
* make tests work
* one more test field
* fix more tests
* more cleanup
* lock frontend when updating and restore refresh
* pass undefined
* add timestamp to background update
* use incrementer
* add field
* snapshot
* types
* more cleanup
* update tests
* remove crumbs
* use expressions
* make nullable
* batch delete
* fill null for static cohorts
* batch_delete
* typing
* remove queryset function
* Run queries against person_distinct_id2 when async migration is done
* Only write to clickhouse_person_unique_id topic if async migration is incomplete
* Update query snapshots
* Update plugin-server
* Adjust caching logic
* Create a new way to get distinct id queries thats gated by team_id
* Update most cases to use the new query
* Convert EVENT_JOIN_PERSON_SQL to new query
* Mostly convert GET_DISTINCT_IDS_BY_PERSON_ID_FILTER
* Mostly convert GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Convert GET_PERSON_IDS_BY_FILTER
* Flag benchmarks
* Resolve circular imports
* Update a snapshot test
* Add a test for the new query logic
* WIP: Create new property types for simplified cohorts
* Add documentation on simplified_cohort_filter_properties
* Handle static-cohort/precalculated-cohort property types
* Handle new property filters properly
* Add casting
* Test cohorts in more cases
* Fix a bug
* Fix benchmark simplifying
* Avoid redoing work every setup for benchmarks
* Update typing;
* Remove unneeded scope
* Add tests for simplifying and cohorts
* Roll more of "do we need to join persons table" behavior into ClickhousePersonQuery class
* Handle precalculated cohort logic in sessions
* Simplify event query
* More tests without any JSONExtract
* Simplify entity properties as well
* Improve docstring
* Add test for breakdown & precalculated cohorts
* Add test for filtering sessions by precalculated cohorts
* Reset unneeded change
* Update cohort
* Solve some typing issues
* Update benchmarking
* Fix cohort filtering tests
* Fix cohort tests
* Fix a caching issue
* Typecheck
* Handle exclusion filters
* Speed up EE tests by not destroying tables
* Fix broken tests
* Speed up tests by not running migrations
* Fix last test
* req file?
* caching
* Parallel
* revert exp
* debug without parallel
* Revert "debug without parallel"
This reverts commit 83b3ad633f.
* Undo parallel tests
* Speed up more tests
* use final
* correct final
* print exception
* more prints
* move test
* add filter
* remove duplicate test
* prints here
* test wait
* set variable in query
* remove mutations_sync
* cleanup
Co-authored-by: eric <eeoneric@gmail.com>
* Add test cases for calculation of cohort-based cohorts
* Add cyclic dependency test
* Add support for cohort-based cohorts
* Reraise error in `calculate_people_ch` to fail loudly
* Use feedback
* Update `test_error_while_calculating`
* Move CH test case to CH suite
* Fix funnels and code formatting
* WIP: Migration to migrate person_distinct_id to CollapsingMergeTree
* Improve migrations
* Another attempt at migration
Using nullable otherwise it defaults without the default 😅
* Linter approved
* Update test code
* Fix testing issues
* Implement workaround for reading person_distinct_ids
Context under https://github.com/PostHog/product-internal/issues/114
Hopefully a temporary workaround!
* Update query
* use print
* match stickiness test
* match stickiness and trends cohort paths
* bring back distinct_id query
* revert unrelated change
* reutrn to original
* reformat
* run again
* comment out test
Co-authored-by: eric <eeoneric@gmail.com>
* Revert "Revert "Add is_deleted column to person_distinct_id (#5151)" (#5193)"
This reverts commit 401268bdba.
* A tweak for docker-compose builds
Co-authored-by: James Greenhill <fuziontech@gmail.com>
