* fix person modal across multiple dates
* test works for pg but shouldn't for ee
* wrong number
* make it cumulative relevant
Co-authored-by: eric <eeoneric@gmail.com>
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
* Make breakdown join with person if needed
* Refactor trends to be purely class-based, don't rely on wonky inheritance
* Extract method
* Improve person join behavior
* Remove unneeded parameter
* Mark a function thats always passed as such
* Add test case demonstrating previous non-join case
* Unify two get_breakdown_prop_values methods
* Add test for materialized columns
* Simplify trends breakdown query
* Unify _breakdown_prop_params for events/person breakdowns
* Use shared column_optimizer
* Typing fix
* Cleanup
* Code style cleanup
* Code style cleanup
* Fix param ordering
Closes https://github.com/PostHog/posthog/issues/5808
I'd love to fix this at the schema level but looked into it for ~20
minutes and it didn't seem like there's a pretty solution that allows to
pass `None` to .objects.create and for the default to be used.
Other solutions would be even more invasive and require manipulating
serializers.
Best leave this to when we refactor all of out `properties` JSONFields
since they all have this issue
* Hotfix: Use materialized columns on cloud
This was broken since default_kind was different on Distributed tables
on cloud
* Improve __init__.py
* Make reverting DEFAULT column async, only ON CLUSTER for events table
* ColumnOptimizer: Add functionality for person materialized columns
* WIP: Use person properties in trends/funnels
* Test person materialized props in trends cohort breakdown query
* Make use of materialized person properties in breakdowns
* Mark some cases working with materialized columns
* Test and fix breakdown by person props without filtering
* Make filtering by entity/person props on a join work better
By not assuming everything is under `properties`
* Add test case around breakdown with person properties
* Add test cases around materialization
* Add another materialization test
* Test cohorts and fix breakdowns with person filter
This exposes a limitation in the current implementation
* Fix some cohort tests
* Fix event query tests
* Get a funnel materialized column test running
* Cover more funnel breakdown tests with materialized
* Handle person property breakdowns in funnels
* Fixup funnel typo
* Add tests, fix an indentation issue
* TestFunnelPersons with materialized columns
* Test funnels against actions with person filters
* Add failing test for entity filtering failing
* Add test case for filtering with entity properties
* Show my 'broken' test is actually doing a subquery
* Resolve linting issues
* Avoid naming collisions when materializing columns
1. Prefix person properties differently. Mixing them up can break
ambigious column issues
2. When name already exists, suffix with random junk :)
* Implement analyze.py
* Add `suggested_columns_to_materialize`
* Add code to backfill a materialized column
* Add tests for backfilling data
* Cleanup
* Add tests for analyze
* WIP: Crontab for materializing columns
* Nooped task for materializing properties
* Use mutations_sync=0 for column tests
* Add comment
Looking at the generated SQL it has bothered me that the list of entity
names is not unique for funnels in queries like `event IN [somearray]`
This tries to fix that.
* Speed up EE tests by not destroying tables
* Fix broken tests
* Speed up tests by not running migrations
* Fix last test
* req file?
* caching
* Parallel
* revert exp
* debug without parallel
* Revert "debug without parallel"
This reverts commit 83b3ad633f.
* Undo parallel tests
* Speed up more tests
* use final
* correct final
* print exception
* more prints
* move test
* add filter
* remove duplicate test
* prints here
* test wait
* set variable in query
* remove mutations_sync
* cleanup
Co-authored-by: eric <eeoneric@gmail.com>
* Remove dead argument
* Make allow_denormalized_props always explicit
* Change prop_clauses default
* Create a testing decorator for checking materialized columns
This makes it easier to have proper test coverage for materialized
columns and make sure no bugs creep up :)
* Fix event_query
* Test more materialized columns in trends
* Add materialized column tests for funnels
* Cleanup path_event_query
* Fix default
* Fix issue with clashing parameter names
* initial implementation
* debug some tests and revert format to original
* some minor cleanup
* add screen handling
* start point handling
* improve indexing so multiple sessions by the same user are handled properly
* all tests passing
* fix type
* fix types
* change types
* more types
* rename base paths
* upgrade start point functionality
* assertEquals, remove inequality
* Revert "assertEquals, remove inequality"
This reverts commit 5eb229fef7.
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Fix fetching event-based breakdown values with `LIMIT`
* Improve test comment
* Limit prop clauses inclusion to breakdown queries
* Address feedback partly
* Fix imports
* Remove non-3.7-compatible `Literal`
* Don't try to import `Literal`
* Fix oversight
* Always include prop filters in `get_entity_filtering_params`
* Revert "Always include prop filters in `get_entity_filtering_params`"
This reverts commit 4f61edd19a.
* Add and test handling of actions in addition to events
* Update test_funnel.py
* Move new tests to `breakdown_cases`
* Fix `icontains` and support `href/`text` element filters
* Test `is_set`/`is_not_set` too
* Require newer `flake8` to support Py 3.8 walrus operator
* Better handle empty mutlivalue operators and support doublequotes
* Polish all element filtering cases
* Roll back general `icontains` fix
* Make minor improvements
* WIP: port process_math to support materialized columns
* Add skipped test showing trend breakdowns dont use materialized columns
* Simplify testing and test&fix math property aggregation w/ materialized columns
* Add (failing) test for filtering with materialized action props
* Add test around materialized property filtering
* Refactor entity.math materialization impl
* Make trends breakdowns work with materialized columns
* Simplify process_math further
* Handle denormalized properties in format_action_filter for step.properties
Note the following files all called this method:
ee/clickhouse/views/events.py
ee/clickhouse/views/actions.py
ee/clickhouse/queries/trends/util.py
ee/clickhouse/queries/trends/lifecycle.py
ee/clickhouse/queries/trends/breakdown.py
ee/clickhouse/queries/funnels/base.py
ee/clickhouse/queries/sessions/util.py
ee/clickhouse/queries/clickhouse_stickiness.py
ee/clickhouse/queries/clickhouse_retention.py
ee/clickhouse/models/cohort.py
I verified all of them are OK since they query events table directly
with the passed filter
* Handle materialized $current_url in action step filtering
* Remove now unneeded clause
* Update test helper
* Allow denormalized props for filtering with breakdowns
* Allow denormalized props for filtering with lifecycle
* Allow denormalized props for some views
* Fix entity math yet again
* Query materialized columns in insights > sessions
* Handle breakdown edge case
* Allow denormalized props for more views
* PR feedback
* reformat
* Add test cases for calculation of cohort-based cohorts
* Add cyclic dependency test
* Add support for cohort-based cohorts
* Reraise error in `calculate_people_ch` to fail loudly
* Use feedback
* Update `test_error_while_calculating`
* Move CH test case to CH suite
* Fix funnels and code formatting
* Cache action lookups on entity
* Avoid mutating filters
* Single-line a long query line
* Calculate columns to select from table dynamically
* Rename class to ColumnOptimizer
* Improve typing
* Set logic
* Rename var
* Fetch funnel top values via materialized columns
* Add a cast
* Simplify, fix
* Move types to more logical place
* Add tests for ColumnOptimizer
* Tidy code
* Test whether elements_chain is needed or not
* No caching in tests
* Use `ColumnOptimizer` in trends
* Prepend commas
* Fix event_query tests
* Handle another edge case around elements_chain
* Handle funnel exclusions
* Add documentation around trickier pieces of logic
* Comment out a broken case for now
* WIP: Migration to migrate person_distinct_id to CollapsingMergeTree
* Improve migrations
* Another attempt at migration
Using nullable otherwise it defaults without the default 😅
* Linter approved
* Update test code
* Fix testing issues
* Extract method from funnels code
* Add failing test case for using denormalized property in a funnel breakdown
* Handle materialized event properties in funnel breakdown
* Refactor property lookups
* Use denormalized properties in paths
* Update validation tests
* Update validation tests
* Fix a broken test
* Move test around
* Type python Property.type
* Type Filter.breakdown_type in python
* Fix typing issue in actions view
* Type python Filter.insight
* Type python Filter.display
* Type python Filter.interval, fix resulting typing errors
* Type python funnel Filter.layout
* Type python funnel Filter.funnel_window_interval_unit
* Type python Filter.retention_type
* Type python funnel Filter.funnel_window_interval_unit_ch
* Type python path Filter.path_type
* Type python path Filter.target_event
* Type Entity.type
* Type python Property.operator
* Improve insight refreshing UX
* Clean up
* Make sure `ComputationTimeAndRefresh` is on the right in Trends too
* Fix typing and some inconsistencies
* Remove unused const
* Update test_clickhouse_insights.py
* Update decorators.py
* Refactor `ComputationTimeWithRefresh` into own file
* allow cohort breakdown to show
* Sanitize `order` in `Entity.__init__`
* Remove kludge from #5485 by making `entity.order` useful in practice
* Use synthetic `index` instead of user-provided `order`
* Fix `processed_entities` sorting
* Update frontend/src/scenes/funnels/funnelLogic.ts
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* change type guard and add number back
* remove typeguard and add correct types for cohorts
* remove type change
* working cohort breakdowns ui
* one more type fix
* adjust tests
* adjust tests
* more test adjustmenet
* minor syntax change
* use breakdown type
* add comment
* remove leakover
Co-authored-by: Michael Matloka <dev@twixes.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: Sam Winslow <sammywinslow@gmail.com>
* Sanitize `order` in `Entity.__init__`
* Remove kludge from #5485 by making `entity.order` useful in practice
* Use synthetic `index` instead of user-provided `order`
* Fix `processed_entities` sorting
* Add `index` comment
* Add migration marking existing columns as materialized with a comment
* Add method to allow looking up materialized columns
* Add test for fetching and creating materialized columns
* Disable caching in tests
* Remove some callsites to CLICKHOUSE_DENORMALIZED_PROPERTIES
* Fixup caching logic
* Add (currently broken) test for denormalizing person properties
* Update funnel event query to work with new functionality
* Update other tests/queries
* Remove now-unused setting
* Use proper interval calculation in the funnel trends query
* Add some comments
* Update `test_filter`
* Rework `NULL_SQL` to use CH `INTERVAL` too
* Fix week-based relative `date_from` support not existing
* Make use of `toInterval*` functions and inject less
* Add fallback for `date_from` in `ClickhouseSessionsAvg`
* Improve cached_property typing
Noticed that e.g. `filter.breakdown` was getting inferred to be `Any`
which is not correct. Added generics to fix it :)
* Proposed fix: make filter_test_accounts return a bool always
* Fix warning in clickhouse_sessions.py
* Cast in session.events
* Cast 2x in funnel queries
* Ignore error in session recording
We know the valid values here
* Add assertions in stickiness filters
* Cast in more funnel queries
* Untyped dict where inferred type is wrong
* Add types to abstract methods
* Type prop_vals
* Add a lot of casts
These are correct. We should really validate while parsing instead
* Add more casts to funnel trends
* Last fixes
* Use f-strings in trends/formula
* Make formulas and breakdowns work
Closes issue https://github.com/PostHog/posthog/issues/5390
Previously the query returned a 500 for some breakdowns, with the error
`Arrays passed to arrayMap must have equal size: while executing 'FUNCTION arrayMap`
The issue came from the different subqueries having different breakdowns
- some would break down with values A and B and other would have B and C
After the full outer join, arrays for breakdown values A and C would be empty in some cases. Filling them with zeroes fixes the issue.
* Fix "ambigious column distinct_id"
The problem came in a trends query:
1. Which filters by action having person property filters
2. The overall query having a person property filter
See also the regression test
Closes https://github.com/PostHog/posthog/issues/5388
* Add calculate_events call
* Implement workaround for reading person_distinct_ids
Context under https://github.com/PostHog/product-internal/issues/114
Hopefully a temporary workaround!
* Update query
* use print
* match stickiness test
* match stickiness and trends cohort paths
* bring back distinct_id query
* revert unrelated change
* reutrn to original
* reformat
* run again
* comment out test
Co-authored-by: eric <eeoneric@gmail.com>
* add more options for funnel windowing
* tweaks
* add test
* accomodate any case and make to_dict smaller
* remove import
* change name
* store lower
* fix test
* partial implementation
* working event breakdown
* remove hardcoded fill
* add person test
* remove test unneeded
* use original breakdown implementation
* add cohort handling
* add clause lost on merge
* count display update
* do not reload list of people when loading more people
* Revert "do not reload list of people when loading more people"
This reverts commit eb9ef11690.
* fix flag
* separate initial loading state
* update title with "+", fix person/persons
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Add migration for `distinct_id` column on clickhouse
Note that code under ee/clickhouse/models/person.py does not work
anymore since it does not populate the column correctly. That is fine -
will be handled when doing the actual migration, rather than for this
experiment.
* Update plugin server to 1.1.8
Co-authored-by: posthog-bot <posthog-bot@users.noreply.github.com>
* add median time for conversion steps
* patch tests
* fix time to convert funnel query
* fix strict and unordered
* add median conversion time to base
* fix
* more tests
* fix broken funnel pagination
* account for both return results
* add limit param
* move order
* move the check
* change magic number
* adjust more tests
* Trigram fuzzy search for property definitions
* support empty search param
* Only use pg_trgm search if pg extension is installed
* suppress exceptions if migration does not succeed
* checks for index
* troubleshooting
* wip: change migration number; strip out conditionals
* wip: copy code from #4110
* remove is_pg_trgm_installed
* try ngram search on CH
* add AND before ngram
* correct some check failures
* remove ordering
* remove ordering from tests
* fix ordering
* rewrite queries for psql, add test bypass for EE features
* remove ordering filter bc of bugs
* remove test code
* check if isinstance of RawQuerySet
* cleanup sql
* add ordering
* typing
* test for ee prop definition
* adjustments to test_ee_property_definition
* add test_ee_event_definition
* restore this prop
* convert to simple term filter
* use term search in EE
* fix EE tests
* fix non-EE tests
* fix test
* fix filter types
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Revert "Revert "Add is_deleted column to person_distinct_id (#5151)" (#5193)"
This reverts commit 401268bdba.
* A tweak for docker-compose builds
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* add all steps option
* all steps working; add total and mean time to convert
* change display type checks to use enum
* kea types
* dangling console log
* Add average conversion time to time to convert results
* respond to feedabck
* responsive histogram sizes
* merged @Twixes backend changes; adjust data shape on frontend; add responsiveness to histogram
* add tooltip label
* adjust copy and tooltip
* minor tweaks
* respond to general feedback
* kea auto
* better empty state:
* error handling null time bins
* fix tests
Co-authored-by: Michael Matloka <dev@twixes.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Update PERSONS_ACTIVE_USER_SQL query
* Remove dead import
* Update lifecycle queries
* Update BREAKDOWN_ACTIVE_USER_INNER_SQL to use new persons query
* Update STICKINESS_SQL
* Update STICKINESS_PEOPLE_SQL
* Update STICKINESS_ACTIONS_SQL
* Update paths query
* Update events query
* Update CALCULATE_COHORT_PEOPLE_SQL
* Update retention queries
* Update TOP_PERSON_PROPS_ARRAY_OF_KEY_SQL
* Update EVENT_JOIN_PERSON_SQL
* Update GET_PERSON_ID_BY_ENTITY_COUNT_SQL
* Remove remaining references to old get latest person query
* Update GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Fix code style issue
* Update table engine for person_distinct_id table
* don't select team_id
* Make person deletion work
* Use replacingmergetree over collapsing with is_deleted
Replacing an existing engine is hard, let's not do it
* Update query in test
* add migration
* set database on materialized views
* Update plugin server to 1.1.6
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: posthog-bot <posthog-bot@users.noreply.github.com>
* Kill dead code around updating person properties
This caused some confusion around mutations that got queued. Dead code =
bad code
* Remove unused imports
Co-authored-by: Michael Matloka <dev@twixes.com>
As-is this default hurts local dev - one more setting you need to
export. Let's keep the setup simple instead.
Also the new default would have broken helm chart and production
when updating with new migrations - the cluster name there is `posthog`
for both it seems.
Let's force other deploys to be explicit instead.
* Make DDLs more friendly towards running on a cluster
* Use primary CLICKHOUSE host for migrations and DDL
* loose ends on person kafka create
* posthog -> cluster typo
* add cluster to KAFKA create for plugin logs
* Feed the type monster
* clusterfy local clickhouse
* test docker-compose backed github action
* run just clickhouse and postgres from docker-compose
* move option to between up and <services>
* posthog all the things
* suggest tests run on cluster
* posthog cluster for ci
* use deploy path for docker-compose
* fix for a clickhouse bug 🐛
* complete CH bug fixes
* 5439 the github actions pg configs
* remove CLICKHOUSE_DATABASE (handled automatically)
* update DATABASE_URL for code quality checks
* Missed a few DDLs on Person
* 5439 -> 5432 to please the people
* cleanup persons and use f strings <3 f strings
* remove auto parens
* Update requirements to use our fork of infi.clickhouse_orm
* fix person.py formatting
* Include boilerplate macros for a cluster
* JOIN in funnel trends query in a more optimal way
* Reformat funnel trends assert slightly
* Refactor `ClickhouseFunnelTrends` slightly
* Add `ClickhouseFunnelTrendsPersons` code
* Allow parametrization of funnel trends persons query
* Update test_funnel_trends.py
* Clean some funnel analysis code up
* Fix `drop_off` default in `FunnelTrendsPersonsMixin`
* Refactor for `ClickhouseFunnelTrends.get_step_counts_without_aggregation_query`
* Update test_funnel_trends.py
* Add an API test for funnel trends persons
* Use `FUNNEL_PERSONS_BY_STEP_SQL`
* Tests persons some more
* Remove unused imports
* first pass mix and match everything
* cleanup
* small refactoring of get_query
* remove top level filter props
* save the file
* enable person breakdowns querying for regular + strict funnels
* move mixin testing to more appropriate location
* add unordered, testing quality of life improvements
* clean up
* add strict+unordered connectivity test for time to convert
* trim after extracting
* clean up, fix tests, trim everywhere
* Load session events asynchronously from a separate endpoint
This mirrors the behavior of postgres query
* Simplify backend & query
event_count is unused
don't select unused columns in list query
* Rename filter_by_session_recordings to filter_by_session_recordings
This is more in-line with what the function actually does
* Update types, handle start/end url properly
* start_url / end_url to session result
* Update sessions list builder tests
* Remove some `session.events` references
* Remove unneeded code
* Simplify filteredSessions
* Fix type issue
* Add test for session properties
* Test and fix start_url/end_url
* Add test for the new sessions API endpoint
* Improve types
* Update py types again
* Fix bug
* Improve time to convert binning behavior, add autobinning
* Test and fix custom bin count
* Expose `ClickhouseFunnelTimeToConvert` in the API
* Revert a column name change
* Fix `bin_count` clamping
* Add an API test for viz type `time_to_convert`
* Update SQL comments
* Remove some unused imports
* Fix `FunnelTypeMixin.funnel_viz_type`
* Remove `DEBUG=1`
* Load action event count asynchronously
This speeds up loading/editing events quite a bit on clickhouse.
Issue: https://github.com/PostHog/posthog/issues/4802
* Add test for /api/action/{id}/count endpoint
* Update code
* move breakdown logic to separate class
* working breakdown without limits
* add limit logic
* remove print
* arr to dict
* tests
* fix test
* fix tests
* make suggested change
* remove one of the changes
* addjust again
* make funcs pure
* add limit
* Fix funnel trends backwards compatibility
* Don't set funnel_viz_type outside of funnel insight filters
* Reorder conditions for clarity and use null check
* setup funnel option type, and proper API for persons and usual funnel routes
* update tests, fix bugs
* make insights compulsory
* nvm
* remove insights key
* fix person caching
* revert problems, fix typings
* address comments
* dont use defaults in mixins
* typing
* cleanup and fix TODOs
* update test
* adapt api to 4957
Co-authored-by: eric <eeoneric@gmail.com>
* Create funnel_time_to_convert.py
* Create test_funnel_time_to_convert.py
* Remove unused things
* Rework time to convert query to use our own SQL
* Add comments to query
* Fix typo in comment
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Use feedback
* Only override `ClickhouseFunnelTimeToConvert._format_results`
* Switch test users B and C around
* Fix typo
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: eric <eeoneric@gmail.com>
* Raise 400 instead of 500 if filter cohort does not exist
* Actually return empty 200 results instead of 400
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
* add strict option for funnels
* typing
* use new persons pattern
* persons for funnel strict ordering
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* unordered step and test
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* resolve issues with unordered funnel
* oops
* remove breakdown, fix mypy error
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
* from O(2^N) to O(N)
* add query intuition blurb
* rm todo
* wip persons
* wip persons 2
* address comments
* test things, fix bugs
* match result format to funnel.py
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: eric <eeoneric@gmail.com>
* Proposal: Use `ValidationError` over `ValueError` in views
This generates a nicer exception for our users (400 response instead of
500) and we cut down on 500 errors for things that are not expected to
work.
* Capitalization
Co-authored-by: Michael Matloka <dev@twixes.com>
* make funnel persons api call return values consistent with others
* update for regular funnels
* update tests
* black
* fix tests
* test fix
* fix tests
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* unordered step and test
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* resolve issues with unordered funnel
* oops
* remove breakdown, fix mypy error
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
* from O(2^N) to O(N)
* add query intuition blurb
* rm todo
* address comments
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: eric <eeoneric@gmail.com>
* open person link in new tab
* set up filtering for person modal
* leave basic search with fuse
* allow for property filtering
* clean up UI
* linter
* replace fuse with backend querying because of pagination issues
* lint
* cohort test failure fix
* prettier
* fix bug
* do not open new persons tab
* return unique persons
* reuse filter request method for persons modal search
* fix test and linter failures
* isort
* feature flag it
* don't disappoint mypy
* try to separate persons modal logic
* cleanup component and logic code
* try to make types happy
* more type fixing
* fix count bug
* more fixes
* type with session string instead of string
* sanitize url param
* add or session back
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* adjust comment
* add caveat
* Clarify funnel trends description and terminology more
* Add a test to nail an assumption down
* Add `FunnelTrendsMixin` to `Filter` to control funnel trends better
* Fix `TestFunnelTrends
* Fix `FunnelTrendsMixin` method name
* Make mypy comfy
* Update dict keys in `TestFunnelTrends`
* Rework mixin
* Fix `test_to_dict`
* Move defaulting funnel trends steps from Filter mixin to query class
* Delete funnel_trends.py
* Update comment
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
* add ability to specify per step or dropoff persons
* remove defaults
* remove funnel_window parameter unless it's needed
* add param to filters
* test api
* remove print
* fix tests
* change distribution
* add none condition for funnel step
* add order by
* remove funnel window days
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* adjust comment
* add caveat
* Clarify funnel trends description and terminology more
* Add a test to nail an assumption down
* Fix `TestFunnelTrends
Co-authored-by: Michael Matloka <dev@twixes.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* Make `SHELL_PLUS_PRINT_SQL` clearer
* Add ClickhouseFunnelTrendsNew
* Create test_funnel_trends_new.py
* Create test_funnel_trends_v2.py
* move builder functions to funnel base
* add test classe for new funnel
* Inherit from `ClickhouseFunnelNew` and fix intervals
* Add proper formatting of trends results
* Clean tests up a little bit
* Group `FunnelWindowDaysMixin` tests in `test_funnel_persons`
* Rename `ClickhouseFunnelTrendsNew` things for clarity
* Port some original `ClickhouseFunnel` trends tests for the new query
* Only fetch initial page (100) of persons in trends query
* Describe assumptions and rename things
* Finish porting old ClickhouseFunnelTrends tests and add some new ones
* Remove unused imports
* Try to fix `test_period_not_final`
* Try to fix `test_period_not_final` again
* remove persons lists
* rename
* fix test
* add timezone to results
* add funnel trends new to api path
* revert random change
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* wrap css classnames and allow for property definition filtering
* event properties section
* properties stats table and object tag functionality
* property descriptions and save all descriptions button
* fix description editing on the table
* fix test failures
* debouncing input is weird
* split up components
* refactor definition drawer kea file
* drawer should only open for taxonomy feature users
* lint errors
* prettier
* property definition filter backend test
* fix test
* populate test with definitions..
* clean up
* clean up property drawer interactions
* fix everything but scroll top issue
* test scroll issue
* table state wip
* update logic after typegen
* prevent undefined error
* only make the clickable rows clickable, not the entire table
* upgrade typegen and fix error with feature flags
* move to reducers and rename action
* update blacklist
* fix event properties description bug
* update typegen once more
* fix editing for events table
* fix editing for properties table
* add script to clean typegen
* update ts blacklist
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* invoke the funnel or funnel trends class respectively
* mypy corrections and PR feedback
* corrected tests
Co-authored-by: eric <eeoneric@gmail.com>
* make query class inheritable
* swap out class
* move trend event query to separate file and replace date_filter func
* prune imports and fix test
* add type
* remove imports
* Revert "Revert "[Frontend Change] Funnels API GET -> POST|GET (#4769)" (#4777)"
This reverts commit dab41f0f6a.
* Add POST support to clickhouse funnels
* run funnel tests on clickhouse
* Don't fire webhooks and Zapier in case of PLUGIN_SERVER_ACTION_MATCHING
* Add PLUGIN_SERVER_ACTION_MATCHING to settings
* Add PLUGIN_SERVER_ACTION_MATCHING to ECS task defs
* Remove leftover print
* Update `AsyncActionMappingNotice`
* Remove stale import
* change get to post api
* tests
* fixed bug
* isort
* make funnels endpoint backwards compatible, add test to check this
* descriptive comment
* revert funnelLogic to current
* revert code removed in error
Co-authored-by: Alex Kim <alexkim@Alexs-MacBook-Pro-2.local>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add migration
* add fields
* remove comment
* change to is_deleted
* Update migration to work
* Update tests
* initial functions
* fix params
* add migration
* add named param
* working test
* update person prop and cohort tests
* use var
* move sql and fstring
* remove get inside django signal
* just create the table
* add replication logic
* add param
* change replacing name
* refactor and add event based cohorts
* add type
* add to model
* add test for event_id
* move sql
* add sql
* make sure actions work too and add test
* add ability to create cohort by event count
* glob match cypress cache
* fix type
* fix type
* refactor and add datetime support
* fix ordering issue in test
* add timestamp test
* fix type
* fix more types
* use var resolution
* add operators
* add necessary filter and test
* add call for calculating cohort on create
* add triggers to oncreate and onupdate
* breakup test and remove repeated subquery
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* first try
* add entity query
* fix type issues
* used in trends
* fix test
* change select format
* change var
* add cohort filter
* add cohort conditions
* change pdi to distinct_ids
* change get to post api
* tests
* fixed bug
* isort
* make funnels endpoint backwards compatible, add test to check this
* descriptive comment
Co-authored-by: Alex Kim <alexkim@Alexs-MacBook-Pro-2.local>
* split up files and organize code
* set up definition drawer and logic and add tagging
* add change owner selection
* definition description editing working
* definition drawer graph and events table
* remove graph logic for now
* small fixes
* property definition doesn't have an owner
* minor tweaks
* lots of small fixes
* show tags on table, disable editing for posthog events, fix tags autocomplete
* fix font sizes and alignment
* allow event limiting and hide behind feature flag again
* linter things
* test fix
* lint
* clean up events limit
* limitOffsetPagination in events
* ignore type
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* make sure to add table name to prop clauses so that event/person properties column isn't vague
* pass table name when formatting breakdown query
* add test
* add migration
* add fields
* remove comment
* change to is_deleted
* Update migration to work
* Update tests
* initial functions
* fix params
* add migration
* add named param
* working test
* update person prop and cohort tests
* use var
* move sql and fstring
* remove get inside django signal
* just create the table
* add replication logic
* add param
* change replacing name
* make sure actions work too and add test
* add necessary filter and test
* add triggers to oncreate and onupdate
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* checkpoint: refactoring funnel trends so that they work correctly
* wip: refactoring funnel trends query to return the results we actually need
* wip: added in new query for testing
* wip: moved sql into a separate file, converted list to dictionary, and added several tests to check data quality
* wip: with a better understaning of funnel trends I've refactored the query so that I can write a data transformer in python
* moved code into funnel_trends for both logic and tests to isolate the concern
* reordered methods for readability
* wip: refactoring funnel trends to support filters
* wip: added updated SQL which will replace the existing FUNNEL_TREND_SQL
* correct column name so that it's clearer
* added substitution variables to new query
* fixed missing substitution variable
* wip: integrating new query with correct params, added mixins for funnel_window, and working toward working test
* several query corrections
* summarize funnel trends
* moved method down
* removed unused code
* added data quality checks
* corrected cohort size for tests
* test window size and incomplete status
* corrected a few names
* removed unnecessary comment
* commented out old funnel trends tests
* removed print statement
* removed old funnel trend code
* made funnel trends response match existing data structure layout
* removed unused imports
* removed more unused imports
* fixed mypy errors
* Added ClickhouseFunnelBase to extract common methods for both ClickhouseFunnelTrends and ClickhouseFunnel; this also fixes issues with tests;
* removed unused type comment
* corrected test to account for new funnel_window_days mixin
* fixed clickhouse funnel tests
* fixes for automated tests
* changed team_id to use client substitution to avoid sql injection attempts in the future but since it's not user supplied it's not currently an issue
* corrections prompted by PR review
* corrected test to dict test with funnel_window_days
* add description and tag fields to event and property definitions
* set up description and tagging on models
* frontend functionality for description editing
* connect backend and kea logic for description editing
* update event and property definitions model and migration
* delete set null instead of cascade
* migration merge fix
* add owner column
* undo posthog event property taxonomy migrations
* set up definitions on enterprise level
* allow enterprise definitions description editing
* fix licensing conditions and add tests
* proper get and update methods for the multi inheritance table and new column fields for enterprise event model
* check for license to separate routes
* migrate existing definitions to ee definitions tables and render ee vs non-ee definition views based on existing feature conditional
* all the working backend updates
* updated tests
* frontend fixes and linting updates
* feature flag it
* Save Organization.available_features as a DB column
* `update_available_features()` before an organization is created too
* Run the task half past the hour
* Adjust tests and fix available_features sync on start
* Improve action and property TS types
* Use plugin server ActionManager pubsub
* Use better pubsub channel names for plugin server ActionManager
* Fix PubSub's lack of teamId
* Use f-strings for property CH SQL formatting
