* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* add flag to funnel event query
* remove unnecessary logic
* typing
* working actors func
* fix typos
* remove unused func
* use serialized result
* wrong var
* use actors
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
* update typing
* clean up
* use actor func for all funnel persons
* types
* make sure limits are set
* remove functions
* fix limits
* funnel trends
* types
* indexing
* moved basic funnel tests
* unordered test
* remove duplicate test
* check ids
* more test adjustments
* sort
* change class names
* sort
* reformat
* change string to int
* remove duplicate
* update comment
* use actor result to paginate not serialized
* adjust tests after merging
* simplify typing
* don't import removed func
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* don't add leading slash to absolute url with no protocol
* don't add a base uri in only one funnel instantiation - will probably break tests
* remove console logs
* every API verb normalises URLs
* Tiny rename
* Minor refactor
* Test on FeatureFlagMatcher
* Add a field to feature_flag indicating if aggregating by groups
* Use f-strings
* Handle property filter building for group property
* Rename is_person_query
* Add message
* Tests for property_to_Q changes
* Add tests for groups w/ rollout
* Add comment for my_flags
* Hook up decide with groups
* New test helper for snapshotting pg queries
* Add snapshot tests for feature flags
* More is_simple_flag tests
* Add decide endpoint tests
* move conftest for ee tests to work
* Fix typo
* conftest troubles
* query for events with matching session_id if possible
* move test to clickhouse
* move event list to list instead of set for flaky test snapshot
* comment to clarify test
* original behaviour was to set breakdown_values regardless of if it was truthy
* add a test for the bug
* ensmallen the test case
* lift tested interface up a level
* implement multi property breakdown as an array from the spike
* correct type hint on method
* really resolve the conflict
* don't break groups
* refactor test assertions for breakdown cases
* adds a test to prove that funnels can receive a string and not an array
* protect saved dashboards from multi property changeover
* WIP
* multi breakdown working with funnel step breakdown
* prove funnel step person breakdown works with multi property breakdown
* don't need to protect cached dashboards from multi property breakdowns when they can't be set from the UI
* capitalise keywords in SQL
* convert a single test to journey helper
* wip
* account for funnel step breakdown sometimes being an array sent as a string
* safer handling of funnel step breakdown
* convert a test
* revert commits that made things worse
* simpler handling of funnel step breakdown
* no need to change funnel step breakdown type hint
* update imports
* guard against integer properties
* compare funnel step breakdown differently now there are arrays involved
* look for strict intersection for funnel step breakdown
* update test snapshots
* need to set breakdown_values earlier in processing
* remove tests that cover speculative functionality
* update snapshot
* move setup of breakdown values back out of update_filters
* update snapshots
* remove a sql parameter that was never assigned to
* Update ee/clickhouse/models/test/test_property.py
Co-authored-by: Harry Waye <harry@posthog.com>
* Update ee/clickhouse/queries/funnels/base.py
Co-authored-by: Harry Waye <harry@posthog.com>
* address review comment to simplify reading json expressions for breakdown
* clarify why some uses of get_property_string_expr escape params before passing
* add keyword arguments for calls to getting property string expressions in funnels
* switch to keyword arguments in test helper method
* fix parameterised test
* add multi property materialized column tests
* introduce the shim to allow new API for breakdown properties
* can't remove the naive funnel step breakdown list detection
* move funnel step breakdown list handling
* better handling of numeric funnel step breakdown values
* update snapshots
Co-authored-by: Harry Waye <harry@posthog.com>
* set debug, test, and primary_db env variables based on context
* remove unnecessary print
* also use pytest.ini for some tests run straight from pytest at the CLI
* Addtest for people_urls in retention trend response
* Add people_urls to retention trend response
* Run black formater
* sort imports
* Add default value for base_uri
* Let type be missing in EventPattern in tests
TypedDict is so rubbish in not allowing missing attributes 🤷
* fix typing
* refactor creation helper functions
* revert docker compose change
* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* remove unnecessary logic
* typing
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
I built this image a while ago, it's built with `yarn
arm64:build:clickhouse`, then I did a multipart build to copy only one
binary over and symlinking the various incantations (eg. server, client
etc) to keep the size down.
* WIP persons table schema migration
* add version to create_person
* test increased timeout
* Update ee/clickhouse/migrations/0019_person_table_version_column.py
* remove flaky kafka test
* format
* test a new approach - make this a non-large migration?
* Update ee/clickhouse/migrations/0021_person_table_version_column.py
* update
* Refactor stickiness to have its own event_query
This will speed up queries significantly and allow for filtering by
group properties
* Use same event_query for stickiness people
* Minor cleanup
* Add tests (and missing file) to group filtering in stickiness
* Allow aggregating by groups in stickiness
* Show group property filters in FE for stickiness
* convert the smallest of the funnel test files
* convert unorder funnel person test to journey for helper
* convert another file
* convert another file
* convert another file
* convert another file
* convert another file
* undelete snapshot files
* undelete snapshot files
* Revert "convert another file"
This reverts commit ef08511509.
* seperate clickhouse and postgres queries
* move to single query
* remove ununsed imports
* fix cohort + subquery issues
* some cleanup
* fix mypy issue
* fixes to query + tests
* add test for person and cohort properties
* remove cruft
* fix api tests
* Remove cohort + person prop filter from postgres
* move cohort and person property tests to clickhouse
* Filter down events
* typing fix
* fix test
* remove unneeded snapshot
* tests working with snapshots
* switch limit to property
* use materialized columns
* move to new distinct_id query
* add snapshots
* fix materialized column bug
* support cohort and person prop filters
* add person and cohort filtering
* remove cohort filtering from clickhouse
* small tweaks
* Lint fix
* remove cohort comment
* feat(funnels): update frontend to use new people_urls in response
This change updates the `FunnelBarGraph` and `FunnelStepTable`
components to use the new `converted_people_url` and
`dropped_poeple_url` in the funnels response.
I need to check that this covers all the cases for funnels. I've only
added people urls for funnels of type step for ordered/unordered/strict
and haven't yet touched the time bins and conversion time funnel
endpoint variants.
* add /some/people/url/ to tests
* chore: add action back, update tests to reference it
* fix typing
* wip
* get list of groups
* move backend endpoints
* frontend wip
* fix routes for groups
* add groups file
* try to fix tab nav
* fix groups query
* reformat results, add limits and offsets and next url
* move groups pages logic into own file
* fix and clean up tabs
* account for pagination and response reformatting
* lint fixes
* optional prev input
* typescript errors
* typescript fixes
* address feedback comments
* groups list logic test
* render properties expandable row
* fix test and lints
* prettier
* simplify table column title
* Fix a typing issue
* Use enum > boolean parameter
This makes what's going on more explicit
* Turn off pagination at the bottom of the table
* Rename table column
* Make pagination in groups work
Quite bugs fixed:
1. `previous` page was always shown, even if going into negative offsets
2. query_result could never be larger than `limit` with the code
* Cleanup code
* Make sidenav active under persons
* Use the correct query
* Add tests for pagination
* Update wording again
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* cumulative dau for non breakdown
* breakdown case
* clean up test
* add trends request util
* use simpler test util
* types
* insight tests
* add snapshots
* black
* adds a first draft of a test helper for clearer event setup in tests
* one spelling of journeys
* obey the type checker
* convert all of breakdown_cases to the new test helper
* Removes unused helper method
* test(funnel-people): add tests for converted/dropped people urls
As per https://github.com/PostHog/posthog/issues/6935 we are adding urls
to responses to attempt to improve consistency between insight responses
and the people we display in the UI.
This simply adds tests for these urls, just for funnel step response
shape. The other two types of response will be handled separately.
* feat(funnel-people): return converted/dropped people url in funnel
Here I have simply added converted / dropped urls in the base funnel.
This touches both clickhouse and postgres functionality, so I will
probably add in tests for postgres also, or just disable to postgres.
It doesn't pass yet as there is an issue in that the order appears to be
ignored by the people endpoint.
* fix(funnels): actually use strict/unordered persons querying
Previously we were always using the standard `ClickhouseFunnelPersons`
class for retrieving people from the `/api/persons/funnel/` endpoint.
This change selects from the unordered and strict variants based on the
`funnel_order_type` setting.
Refers to https://github.com/PostHog/posthog/issues/7058 although there
is a frontend component to add that will truely resolve the issue.
* feat(funnel-people): add people urls for funnels with breakdown
In this I have also removed any changes from the non-clickhouse code.
Note that there are two places I've added the url generation code, one
in the `ClickhouseFunnelBase` and another in the `ClickhouseFunnel`
I think the former covers the unordered and strict cases, and the later
for the breakdown case. I think there are further test that I'll need to
add to validate e.g. if breakdown + strict work as expected.
* Fix people urls for unordered funnels + breakdown
* test: add test for strict breakdown with people urls
* make funnel response assertions not check people url equality
* fix typing
* fix tests
* remove new line in postgres funnel.py
* clear cache on insight test start
* no really, clear the cache
* remove flakiness from strict funnel test
* correct the unordered test
* use absolute uris
* use step.index not step.order
* remove out of date comment
* use step.index, not step.order
* use step.index, remove unordered funnels comment
* use journeys_for instead of create_events
* add test_helper_methods
* move all the tests where the properties are the same for all events to the journey helper
* compare funnel results without caring about person order
* spell words correcterly
* Revert "spell words correcterly"
This reverts commit befb83b183.
* Revert "compare funnel results without caring about person order"
This reverts commit 268927b8ba.
* correct types for test props
* add groups property values
* update groups property tests
* newline
* fix
* update test
* python black
* clean up tests
* rename to endpoint in operator value select
* add empty property values test
* do not return in an array
* fix
