* Extract common subquery into a variable
* BE: handle group properties in more cases
* Add tests for lifecycle and sessions query changes
* Better docs
* Stable date range
* backend fixes and test
* add breakdown value to pie chart
* adjust test
* fix faulty test
* fill param
* fix formula tests
* more date passing
* more cleanup
* all tests working
* make test data explicit and add better checks
* support both ee and postgres
* length checks
* WIP: Create new property types for simplified cohorts
* Add documentation on simplified_cohort_filter_properties
* Handle static-cohort/precalculated-cohort property types
* Handle new property filters properly
* Add casting
* Test cohorts in more cases
* Fix a bug
* Fix benchmark simplifying
* Avoid redoing work every setup for benchmarks
* Update typing;
* Remove unneeded scope
* Add tests for simplifying and cohorts
* Roll more of "do we need to join persons table" behavior into ClickhousePersonQuery class
* Handle precalculated cohort logic in sessions
* Simplify event query
* More tests without any JSONExtract
* Simplify entity properties as well
* Improve docstring
* Add test for breakdown & precalculated cohorts
* Add test for filtering sessions by precalculated cohorts
* Reset unneeded change
* Update cohort
* Solve some typing issues
* Update benchmarking
* Fix cohort filtering tests
* Fix cohort tests
* Fix a caching issue
* Typecheck
* Handle exclusion filters
* Simplify filters code
* Simplify filters ASAP if filter is created
* Simplify route
* Remove simplification-specific logic from queries
* Remove recursion, update tests
* Pass team in more cases
* Update column optimizer specs
* Test simplify
* Update trends test
* Fix rebase fail
* Remove dead argument
* Make allow_denormalized_props always explicit
* Change prop_clauses default
* Create a testing decorator for checking materialized columns
This makes it easier to have proper test coverage for materialized
columns and make sure no bugs creep up :)
* Fix event_query
* Test more materialized columns in trends
* Add materialized column tests for funnels
* Cleanup path_event_query
* Fix default
* Fix issue with clashing parameter names
* WIP: port process_math to support materialized columns
* Add skipped test showing trend breakdowns dont use materialized columns
* Simplify testing and test&fix math property aggregation w/ materialized columns
* Add (failing) test for filtering with materialized action props
* Add test around materialized property filtering
* Refactor entity.math materialization impl
* Make trends breakdowns work with materialized columns
* Simplify process_math further
* Handle denormalized properties in format_action_filter for step.properties
Note the following files all called this method:
ee/clickhouse/views/events.py
ee/clickhouse/views/actions.py
ee/clickhouse/queries/trends/util.py
ee/clickhouse/queries/trends/lifecycle.py
ee/clickhouse/queries/trends/breakdown.py
ee/clickhouse/queries/funnels/base.py
ee/clickhouse/queries/sessions/util.py
ee/clickhouse/queries/clickhouse_stickiness.py
ee/clickhouse/queries/clickhouse_retention.py
ee/clickhouse/models/cohort.py
I verified all of them are OK since they query events table directly
with the passed filter
* Handle materialized $current_url in action step filtering
* Remove now unneeded clause
* Update test helper
* Allow denormalized props for filtering with breakdowns
* Allow denormalized props for filtering with lifecycle
* Allow denormalized props for some views
* Fix entity math yet again
* Query materialized columns in insights > sessions
* Handle breakdown edge case
* Allow denormalized props for more views
* PR feedback
* reformat
* Use proper interval calculation in the funnel trends query
* Add some comments
* Update `test_filter`
* Rework `NULL_SQL` to use CH `INTERVAL` too
* Fix week-based relative `date_from` support not existing
* Make use of `toInterval*` functions and inject less
* Add fallback for `date_from` in `ClickhouseSessionsAvg`
* Improve cached_property typing
Noticed that e.g. `filter.breakdown` was getting inferred to be `Any`
which is not correct. Added generics to fix it :)
* Proposed fix: make filter_test_accounts return a bool always
* Fix warning in clickhouse_sessions.py
* Cast in session.events
* Cast 2x in funnel queries
* Ignore error in session recording
We know the valid values here
* Add assertions in stickiness filters
* Cast in more funnel queries
* Untyped dict where inferred type is wrong
* Add types to abstract methods
* Type prop_vals
* Add a lot of casts
These are correct. We should really validate while parsing instead
* Add more casts to funnel trends
* Last fixes
* Load session events asynchronously from a separate endpoint
This mirrors the behavior of postgres query
* Simplify backend & query
event_count is unused
don't select unused columns in list query
* Rename filter_by_session_recordings to filter_by_session_recordings
This is more in-line with what the function actually does
* Update types, handle start/end url properly
* start_url / end_url to session result
* Update sessions list builder tests
* Remove some `session.events` references
* Remove unneeded code
* Simplify filteredSessions
* Fix type issue
* Add test for session properties
* Test and fix start_url/end_url
* Add test for the new sessions API endpoint
* Improve types
* Update py types again
* Fix bug
* restructure the code
* Clean up complicated function using namedtuple
* Speed up action/event filtering in clickhouse
For very rare events, filtering by them in sessions previously was slow.
This was because the the distinct_id query contained a lot of users who
hadn't done the action, resulting in more looping.
This commit speeds things up by querying users who have done any events matching the
event/action filter.
* WIP auto filter test accounts
* finish off
* Fix tests
* Non generic emails
* add list of generic emails
* Move location to below property filters
* Fix typescript errors
* as any
* fix tests
* filters
* fix tests
* Featureflag doesn't really makes sense for this feature
* fix tests
* fix test
* Add clickhouse + tests for each insight
* Fix lifecycle and paths
* Fix sessions
* Fix session tests
* fix sesisons test
* fix migrations
* fix migration chain
* refactor path & remove stale console.log
* adjust useAnchor & minor copy
* rename to avoid confusion with inline component
* test account filter tweaks
* fix filters
* hardcode
* Add tests for funnel trends
* Make generic emails super fast
* Fix migrations
* Default to false for now
* Default to false, remember a user's preference
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Fix and test more sessions pagination cases
Pagination previously did not work correctly on CH/postgres due to
LIMIT clauses
Simplifying the sessions query helps on clickhouse, though might
introduce new issues.
* Make sessions list pagination work
The key idea is divorce distinct_id lookups from result lookups.
This now works in the scenario where 101 users match person filter/have done
an event in time range, but only the 101st has a session matching
action/event filter (see tests)
This will perform even on superdaily, though it might slow down for very
specialized queries.
Potential future speedups:
- apply action/event filters on the distinct_id query -
only return those which who have the possibility of matching.
- Make distinct_id LIMIT higher if we know action/event limit is
involved
- Caching the distinct_id query heavily
* Reorganize code
* Make session list tests pass w/ pagination
* Add tests, fix another corner case for postgres sessions list
distinct_ids were not always returned in the right order.
* Include distinct_id in sessions query
This should now solve https://github.com/PostHog/posthog/issues/3055
Fixes https://github.com/PostHog/posthog/issues/3282
The problem was in pagination as expected.
Ran the query against our team in production clickhouse - it now returns
expected data and speed did not seem to change much.
* Add model for session recording viewed
* Save view when querying for session recording data
* Send information to FE about whether session has been viewed
* Allow filtering by "recording unseen"
* Rename property
* Update migration
* Highlight rows from sessions which are matched by the filter
* Send start_time of recording to frontend
This helps us calculate offsets a bit better
* Use timestamp by server for date shown
* Load session events via kea logic
* Update rrweb, rrweb-player
* Highlight events user is filtering for in sessions player
* Handle empty case properly
* Add positive test
* Order session recording events in query
* Fix filtering by multiple events with differing names
previously only the first would have been used due to overlapping
params.
* Return all highlighted times as action_filter_times
* Send back ids not timestamps
This avoids weird rounding errors causing issues
* Show skeleton for longer
* update typing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Make it possible to filter by (multiple) action filters in postgres
Session will now contain "action_filter_times" key which lists when each
action filter occurred for the first time within the session.
This will be used to highlight rows/show special values in sessions
player.
* Clickhouse: support multiple action filters
* Remove dead code
* Remove (apparently) useless person joins from sessions sql
* WIP: Make sessions list query use python iteration
* WIP: Show loader while session data is loading
* Aggregate together sessions
* Calculate start and end points of session separately
* Remove cruft code
* Load session events asynchronously for self-hosted
Note clickhouse behavior is unchanged.
* Update pagination logic for sessions
In addition to offset, postgres now returns a dict containing person_id,
timestamp which is used to make sure we filter events on different pages
correctly
* Add some tests to SessionListBuilder
* Fix typing errors
* Fix pagination limit
* Move tests to right file
* Query less events for sessions list
Since we're ordering by end_time we know events before last end_time are
all processed.
* Add test for current_url behavior
* Make sure old tests remain working
* Remove unused base class
* Move sessions-related queries to separate subfolder
* Extract sessions list code to separate file(s)
* Sort sessions by end time in ch as well
* List end time in sessions table
* Return person email when querying sessions lists on postgres
This gets used by the view
* Return email over all user properties for sessions in clickhouse and view
* Fix an ordering bug
* Fix a pagination bug
* Fix endpoint
* Fix basic sessions tests for pagination
* Sort consistently for sessions list builder
* Roll pagination into filters
* do proper interval rounding on normal trends
* patch inconsistency
* remove microseconds
* conditionall round interval
* adjust how date_from is handled
* add retention test
* edits and split test
* extract code to format entity filter
* make it possible to filter by action
* Hack to make filtering by actions subfilters possible
Example url:
http://localhost:8000/api/event/sessions?date_from=2020-12-10&date_to=2020-12-10&action_filter={%22type%22:%22events%22,%22id%22:%22test-event%22,%22properties%22:[{%22key%22:%22email%22,%22value%22:%22example.com%22,%22operator%22:%22icontains%22,%22type%22:%22person%22}]}
Not sure if the team filtering is legit here
* Use discriminated union in types
* Add kludge support for action_filter on sessions via url
* Reduce code in buildURL
* Add link to sessions page from persons modal
* Add muted overview of the invisible filter
* Add link to cohort sessions from persons page
* drop irrelevant test code
* Test clickhouse filtering by action filter
* put filter behind a cloud-only conditional
* remove dead import
* Add icons/data-attr to sessions links
* Appease linter gods
* Duplicate existing endpoints under /api/event/sessions
* Use different endpoint for sessions table logic in sessions page
* drop irrelevant code for insights endpoint
* update documentation
* extract method
* move code under only relevant path
* Extract clickhouse sessions logic
* Refactor: Separate session list logic from rest of sessions
* move sessions list to separate file
* Move sessions list tests to separate file
* add clickhouse session list tests
* add support for duration filtering on postgres
* move api tests to right place
* kill dead code
* Use return value of add_session_recording_ids
The mutation here is incidental, this is symmetrical with how postgres
behaves.
* Implement sessions_filter
This will hold session-recording specific filters
* WIP: make filtering by session recording length possible on ch
* Kill dead method
* Allow filtering by session length
* Make logic a bit smoother around filtering by recording duration
* More common code for clickhouse_session_recording
* Add filtering tests for ch
* Resolve mypy issues
* Only run duration tests for ch
* Put new filters logic behind a feature flag
* Kill dead const
* Solve operators-related comment
* Review feedback: rename variable
* Fix test failures
* intial working
* intial working with test
* return tests
* fix interval and add monthly test
* add in person_id join
* small restyle
* postgres lifecycle working
* add action handling to postgres query
* add action handling and tests
* fix typing
* visualizing with temp params and added negatives for dormant
* frontend
* fix intervals
* remove unnecessary import
* add person enppoint
* add person endpoint and tests
* add next
* add pagianted test
* fix types
* add frontend logic'
* fixed date range
* disable compare filter on lifecycle
* add diasbled to chartfilter
* return class to personviewset
* added constant
* fix distinct_id and new event within period condition
* replace people queries and fix ch query too
* DRY
* fix null states
* comparefilter nullstate
* add wrapper back to endpoint
* fix datetime formatting
* remove extra stack flag
* reduce filters when it's lifecycle and replace constants
* add default for lifecycle into trendlogic
* clean up reused function
* remove fluff and use constants
* standardizing compare
* segment logic for clickhouse queries that have a lot
* add default to filter.date_to
* mino unwrap
* fix test
