* Improve process_math
* Add test for overlapping group keys
* Improve event query tests
* Add test for filtering by person properties together with groups
* Avoid flaky tests due to cohort_id changing
* Update queries and snapshots
* Add groups stuff
* Rename column from person_id to `target` in retention queries
No behavioral change, preparing for groups work :)
* Remove dead if statement
* WIP: Retention aggregation by groups
* Handle aggregation by groups in retention
Also handles the case where not every event has a property defined
* Test groups validation mixin
* Reformat
* Improve test for aggregation in retention
* Add /api/projects/@current/groups/property_definitions
To load property definitions for groups
* Add routes for fetching group types
* FE: Group analytics feature flag
* FE: Group analytics base types for properties
* FE: Initial implementation of fetching logics
* add group tabs to filter
* make sure group properties populate and refactor things
* prettier
* disable unused vars
* allow indexing objects by number
* mount group properties logic
Co-authored-by: Li Yi Yu <li@posthog.com>
* Extract GroupsJoinQuery
* Add test for breakdown filtering
* Unify breakdown mixins
* Allow passing breakdown_type == 'group' with breakdown_group_type_index
* Allow breakdown by group props in trends
* Add tests for trends breakdown_props function on group breakdowns
* Solve common issues
* Output snapshot diff into console
* Clean up materialized columns after tests
* Add zero protection
* Solve test failure
* Type math in Entity
* Allow passing group_type_index from FE to BE
* Get a initial query running
* Add group value filter if aggregating by groups
* Add snapshot testing for trends queries
* isort
* Update tests
* Add test for column_optimizer
* Update ee/clickhouse/queries/trends/util.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* adds a test to prove that without any changes the API receives a string and returns one see #938
* put a guard in place so that funnel query can be shifted to arrays without affecting existing clients
* make checking for single property breakdowns safer
* sort imports
* sort imports
* reformat files
* Revert "reformat files"
This reverts commit 58530f134f191d53b927abe6fb6f5d2d384cb255.
* expand short variable names
* corrects a typo
* it wasn't a typo it was a logic error
* Add group type, group_type_index
* Raise an error when handling unsupported properties in CH
* Improve repr
* Fix is_superset function
This was previously broken - sorting and zipping doesn't really work for
this intent.
* Add group_type_index to analysis results
* Add `group_types_to_query`
* Minor typing fixes
* Create groups tables in tests
* Simple first filter by groups query
* isort
* Use snapshot testing in event_query tests, add test for groups
* backend fixes and test
* add breakdown value to pie chart
* adjust test
* fix faulty test
* fill param
* fix formula tests
* more date passing
* more cleanup
* all tests working
* make test data explicit and add better checks
* support both ee and postgres
* length checks
* paginate recording compression
* some tests
* more accurate duration calculation
* add tests and types
* tons of decompression fixes
* rename test file to avoid conflict
* move decompression to helper
* add test for helper
* type fix
* rename method
* simplify paginated decomression
* handle case where offset exceeds length
* clean up
* test fixes
* clean up on aisle 12
* Add surrounding object for metadata response
* Add table for group_type_mapping
* Remove materialized columns from events table schema
These are not used and not needed w/ new mat columns work
* WIP: Migration to add group analytics columns
* Remove event table changes temporarily
* initial refactoring
* popup UI
* refactor path cleaning logic
* add nullable
* all ui working
* fix migration
* use regex replacement from team object
* add flag
* add switch
* fix type
* fix type
* UI update
* restore removed arg
* add local path cleaning filters to api
* add test for local path filters
* working new UI
* reduced repeated code
* fix numbering
* minor refactoring
* update copy
* add under advanced features
* address comments, minor cleanup
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Previously we would attempt to generate a response even though there
were no steps. There appears to be some code paths that blow up if this
happens, so instead we return as soon as we can in this case. This
appears to be the behaviour elsewhere also.
This resolves the sentry error found here:
https://sentry.io/organizations/posthog/issues/2718768248/
* paginate session recording events api
* code quality
* don't rerender replayer every time events get loaded
* refactor session recordings, make it backwards compatible, add bunch of tests
* move limit and offset to after decompress
* change limit
* add caching of recording
* add duration
* fix a few tests
* fix api tests
* add partial chunk test
* fix default limit test
* code quality
* typing
* fix backend tests
* mypy fixes and signature
* remove circular dependency
* mypy and sessionsplaylogic tests
* fix migration
* make single migration
* revert mypy typing
* remove require react resolve
* fix loading state to persist across multi chunks and add test'
* cleanup
* duration to ms
* fix duration tests
* remove require react resolve
* fix test
* bump rrweb player
* change cache behavior plus more
* fix frontend tests and make duration calculation more robust
* fix timestamp in ms bug in tests
* fix duration 0 test
* fix compatibility with rrweb-player
* update lockfile
* yarn unlink
* resolve kea-test-utils changes
* another one
* remove caching for now (leave as todo) and change limit to 100
* Refactor column_optimizer to work differently
* WIP: Use counter over set
* Handle person filters in person query
* Remove a dead argument
* Use enum over parameter for determining behavior
* Allow excluding person properties mode when handled in person query
* Fix _get_person_query type
* Use correct table for funnel_event_query
* Remove unneeded override
* Add extra typing
* Filter by entity.properties in person query for trends
* Handle error 184 due to naming clash
* Better default for prop_filter_json_extract
* Update column_optimizer tests for Counter
* Handle person_props as extra_fields
* Handle breakdowns and person property filter pushdown
* Transform values correctly
* Simplify get_entity_filtering_params
* Fix funnel correlations
* Solve caching issues in trend people queries
* Remove @skip test
* Add syrupy tests for parse_prop_clauses
Can update these via --snapshot-update
* Add snapshot tests for person queries
* Add a few notes
* Update test to avoid collision
* Kill dead code
* Handle PR comments
* Update ee/clickhouse/queries/person_query.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Add `slug` fields to `Organization` and `Team`
* Expose slugs to user
* Add slug autogeneration for new orgs/projects
* Improve slug UX
* Remove slug from settings
* Update org/team instance creation plus add tests
* Only require project slug to be unique for org, not globally
* Fix `get_prep_value`
* Test organization slugification
* Deslugify `Team`
* Clean up changes
* Update test_user.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Random 4 letter suffixes for the win
* Fix import
* Ignore `test_migrations_are_null`
* Fix `RunSQL` query being empty
* Fix `generate_random_short_suffix` testing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
This seems to be semi-consistently failing locally for me and
inconsistently on CI. The cause is unknown - debugging locally, this
seems more of a query correctness issue than a test data incomplete
issue.
* fix bug where session recording is split by new distinct_id
* fix query bug to allow distinct_id clause
* Basic query filtering by events on clickhouse
* add duration filtering
* Add multiple action/event filters
* Move to using person_id + optimize query
* add postgres support and querying on person_id
* move back to distinct_ids
* remove lots of duplicated logic
* Add pagination to backend
* Pagination
* Add duration and date filter
* Use person header is list view
* type fixes
* bug fix
* bugs and tests
* test fixes
* style filters
* logic tests
* Add another test for team leakage
* fix flaky test
* UX changes
* UX tweaks
* auto focus filter box on open
* add test for recording viewed change
* move duration filter to using a tested logic
* move to clsx
* add debounce breakpoint
* Small style change
* fix debounce
* fix breakpoint
* move test to defaultAPIMocks (merging changes)
* react to url change
* WIP: Create new property types for simplified cohorts
* Add documentation on simplified_cohort_filter_properties
* Handle static-cohort/precalculated-cohort property types
* Handle new property filters properly
* Add casting
* Test cohorts in more cases
* Fix a bug
* Fix benchmark simplifying
* Avoid redoing work every setup for benchmarks
* Update typing;
* Remove unneeded scope
* Add tests for simplifying and cohorts
* Roll more of "do we need to join persons table" behavior into ClickhousePersonQuery class
* Handle precalculated cohort logic in sessions
* Simplify event query
* More tests without any JSONExtract
* Simplify entity properties as well
* Improve docstring
* Add test for breakdown & precalculated cohorts
* Add test for filtering sessions by precalculated cohorts
* Reset unneeded change
* Update cohort
* Solve some typing issues
* Update benchmarking
* Fix cohort filtering tests
* Fix cohort tests
* Fix a caching issue
* Typecheck
* Handle exclusion filters
* Simplify filters code
* Simplify filters ASAP if filter is created
* Simplify route
* Remove simplification-specific logic from queries
* Remove recursion, update tests
* Pass team in more cases
* Update column optimizer specs
* Test simplify
* Update trends test
* Fix rebase fail
* events dead letter queue CH table
* format
* update schemas
* also store raw payload
* better naming
* make table name more clear
* wip better testing
* remove unused imports
* remove kafka test
* prevent non null test from running on CH migrations
* add kafka testing
* minor tests cleanup
* test naive longer sleep
* make test end-to-end
* address review
* update ttl, format
* refactor delay func, address review
* fix(correlation): ensure correlation partitioned by team
Previously if distinct_id's were the same between two teams, we'd end up
pulling in the event data between the teams.
* dev(tests): clear cache between partition calls
* fix(correlation): add lower bounds for selected events
Previously we would consider all events for correlation calculation. Now
we use the funnel `date_from` as the lower bounds.
* chore(correlation): exclude funnel steps
* chore(correlation): make sure cache is cleared before each test
* Update funnel success comment
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* fix: filter events by team_id
* chore(correlation): remove CTEs from correlation query
There seems to be an issue with the CTEs and production clickhouse, see
https://github.com/ClickHouse/ClickHouse/issues/29748
Instead of risking it, I'm just removing them.
* chore: update entities -> events for funnel step exclusion
* fix team_id = team_id issue
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* chore(diagnose): add a stub endpoint for event diagnosis
This adds an insight endpoint that takes a `target_entity` and returns
a list of events ordered by significance of a person reaching
`target_entity`
Followup PRs will add actual calculations but this should act as a
conversation piece around the structure of request and response, as well
as a stub from which UI development can start.
* play around to get mypy typings right
* Sort out test reponse structurea
* refactor: address CR comments
* feat(diagnose): add diagnose stubs for implementation
* feat(funnel): add event correlation calculation implementation
This adds to the `/api/projects/<team_id>/funnel/correlation` endpoint
an implementation that calculates the odds_ratios for each event that a
user that has been part of a funnel, successful or otherwise.
* chore(correlation): get query working
* refactor(correlation): move functions to Query methods
* Always allow filtering on distinct id
* fix
* fix
* remove accidentally commited test
* include table name
* add table specific because distinct_id doesn't exist on person table
Co-authored-by: eric <eeoneric@gmail.com>
