* fix person modal across multiple dates
* test works for pg but shouldn't for ee
* wrong number
* make it cumulative relevant
Co-authored-by: eric <eeoneric@gmail.com>
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
* Make breakdown join with person if needed
* Refactor trends to be purely class-based, don't rely on wonky inheritance
* Extract method
* Improve person join behavior
* Remove unneeded parameter
* Mark a function thats always passed as such
* Add test case demonstrating previous non-join case
* Unify two get_breakdown_prop_values methods
* Add test for materialized columns
* Simplify trends breakdown query
* Unify _breakdown_prop_params for events/person breakdowns
* Use shared column_optimizer
* Typing fix
* Cleanup
* Code style cleanup
* Code style cleanup
* Fix param ordering
Closes https://github.com/PostHog/posthog/issues/5808
I'd love to fix this at the schema level but looked into it for ~20
minutes and it didn't seem like there's a pretty solution that allows to
pass `None` to .objects.create and for the default to be used.
Other solutions would be even more invasive and require manipulating
serializers.
Best leave this to when we refactor all of out `properties` JSONFields
since they all have this issue
* Hotfix: Use materialized columns on cloud
This was broken since default_kind was different on Distributed tables
on cloud
* Improve __init__.py
* Make reverting DEFAULT column async, only ON CLUSTER for events table
* ColumnOptimizer: Add functionality for person materialized columns
* WIP: Use person properties in trends/funnels
* Test person materialized props in trends cohort breakdown query
* Make use of materialized person properties in breakdowns
* Mark some cases working with materialized columns
* Test and fix breakdown by person props without filtering
* Make filtering by entity/person props on a join work better
By not assuming everything is under `properties`
* Add test case around breakdown with person properties
* Add test cases around materialization
* Add another materialization test
* Test cohorts and fix breakdowns with person filter
This exposes a limitation in the current implementation
* Fix some cohort tests
* Fix event query tests
* Get a funnel materialized column test running
* Cover more funnel breakdown tests with materialized
* Handle person property breakdowns in funnels
* Fixup funnel typo
* Add tests, fix an indentation issue
* TestFunnelPersons with materialized columns
* Test funnels against actions with person filters
* Add failing test for entity filtering failing
* Add test case for filtering with entity properties
* Show my 'broken' test is actually doing a subquery
* Resolve linting issues
* Avoid naming collisions when materializing columns
1. Prefix person properties differently. Mixing them up can break
ambigious column issues
2. When name already exists, suffix with random junk :)
* Implement analyze.py
* Add `suggested_columns_to_materialize`
* Add code to backfill a materialized column
* Add tests for backfilling data
* Cleanup
* Add tests for analyze
* WIP: Crontab for materializing columns
* Nooped task for materializing properties
* Use mutations_sync=0 for column tests
* Add comment
Looking at the generated SQL it has bothered me that the list of entity
names is not unique for funnels in queries like `event IN [somearray]`
This tries to fix that.
* Speed up EE tests by not destroying tables
* Fix broken tests
* Speed up tests by not running migrations
* Fix last test
* req file?
* caching
* Parallel
* revert exp
* debug without parallel
* Revert "debug without parallel"
This reverts commit 83b3ad633f.
* Undo parallel tests
* Speed up more tests
* use final
* correct final
* print exception
* more prints
* move test
* add filter
* remove duplicate test
* prints here
* test wait
* set variable in query
* remove mutations_sync
* cleanup
Co-authored-by: eric <eeoneric@gmail.com>
* Remove dead argument
* Make allow_denormalized_props always explicit
* Change prop_clauses default
* Create a testing decorator for checking materialized columns
This makes it easier to have proper test coverage for materialized
columns and make sure no bugs creep up :)
* Fix event_query
* Test more materialized columns in trends
* Add materialized column tests for funnels
* Cleanup path_event_query
* Fix default
* Fix issue with clashing parameter names
* initial implementation
* debug some tests and revert format to original
* some minor cleanup
* add screen handling
* start point handling
* improve indexing so multiple sessions by the same user are handled properly
* all tests passing
* fix type
* fix types
* change types
* more types
* rename base paths
* upgrade start point functionality
* assertEquals, remove inequality
* Revert "assertEquals, remove inequality"
This reverts commit 5eb229fef7.
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Fix fetching event-based breakdown values with `LIMIT`
* Improve test comment
* Limit prop clauses inclusion to breakdown queries
* Address feedback partly
* Fix imports
* Remove non-3.7-compatible `Literal`
* Don't try to import `Literal`
* Fix oversight
* Always include prop filters in `get_entity_filtering_params`
* Revert "Always include prop filters in `get_entity_filtering_params`"
This reverts commit 4f61edd19a.
* Add and test handling of actions in addition to events
* Update test_funnel.py
* Move new tests to `breakdown_cases`
* Fix `icontains` and support `href/`text` element filters
* Test `is_set`/`is_not_set` too
* Require newer `flake8` to support Py 3.8 walrus operator
* Better handle empty mutlivalue operators and support doublequotes
* Polish all element filtering cases
* Roll back general `icontains` fix
* Make minor improvements
* WIP: port process_math to support materialized columns
* Add skipped test showing trend breakdowns dont use materialized columns
* Simplify testing and test&fix math property aggregation w/ materialized columns
* Add (failing) test for filtering with materialized action props
* Add test around materialized property filtering
* Refactor entity.math materialization impl
* Make trends breakdowns work with materialized columns
* Simplify process_math further
* Handle denormalized properties in format_action_filter for step.properties
Note the following files all called this method:
ee/clickhouse/views/events.py
ee/clickhouse/views/actions.py
ee/clickhouse/queries/trends/util.py
ee/clickhouse/queries/trends/lifecycle.py
ee/clickhouse/queries/trends/breakdown.py
ee/clickhouse/queries/funnels/base.py
ee/clickhouse/queries/sessions/util.py
ee/clickhouse/queries/clickhouse_stickiness.py
ee/clickhouse/queries/clickhouse_retention.py
ee/clickhouse/models/cohort.py
I verified all of them are OK since they query events table directly
with the passed filter
* Handle materialized $current_url in action step filtering
* Remove now unneeded clause
* Update test helper
* Allow denormalized props for filtering with breakdowns
* Allow denormalized props for filtering with lifecycle
* Allow denormalized props for some views
* Fix entity math yet again
* Query materialized columns in insights > sessions
* Handle breakdown edge case
* Allow denormalized props for more views
* PR feedback
* reformat
* Add test cases for calculation of cohort-based cohorts
* Add cyclic dependency test
* Add support for cohort-based cohorts
* Reraise error in `calculate_people_ch` to fail loudly
* Use feedback
* Update `test_error_while_calculating`
* Move CH test case to CH suite
* Fix funnels and code formatting
* Cache action lookups on entity
* Avoid mutating filters
* Single-line a long query line
* Calculate columns to select from table dynamically
* Rename class to ColumnOptimizer
* Improve typing
* Set logic
* Rename var
* Fetch funnel top values via materialized columns
* Add a cast
* Simplify, fix
* Move types to more logical place
* Add tests for ColumnOptimizer
* Tidy code
* Test whether elements_chain is needed or not
* No caching in tests
* Use `ColumnOptimizer` in trends
* Prepend commas
* Fix event_query tests
* Handle another edge case around elements_chain
* Handle funnel exclusions
* Add documentation around trickier pieces of logic
* Comment out a broken case for now
* WIP: Migration to migrate person_distinct_id to CollapsingMergeTree
* Improve migrations
* Another attempt at migration
Using nullable otherwise it defaults without the default 😅
* Linter approved
* Update test code
* Fix testing issues
* Extract method from funnels code
* Add failing test case for using denormalized property in a funnel breakdown
* Handle materialized event properties in funnel breakdown
* Refactor property lookups
* Use denormalized properties in paths
* Update validation tests
* Update validation tests
* Fix a broken test
* Move test around
