* Extract common subquery into a variable
* BE: handle group properties in more cases
* Add tests for lifecycle and sessions query changes
* Better docs
* Stable date range
* wip
* normalise to breakdown and breakdowns on the way out of the API
* read from breakdowns or breakdown property of filters
* wip
* wip
* delete a console.log
* remove console.log and speculative multi breakdown changes
* remove empty test
* remove console log
* more tidying
* more tidying
* more tidying
* simplify funnel logic slightly
* a test for the data transformations with no breakdown
* can test data transformation with a single property breakdown
* keep expanding the test to see why multi prop doesn't display
* don't throw key error if there is no breakdown property to pop from the multi property API response
* funnel step table can detect breakdown without relying on filter
* try to highlight the differences in the paths through TaxonomicBreakdownFilter
* better behaviour when adding and removing multi property breakdowns
* displays multi property breakdown in old style steps funnel views
* it is safe to always show breakdown in new vertical layout
* resolve multi property breakdown typehinting
* updates comment
* updates comment
* don't print breakdown value twice
* tooltip for long values and prettier pill
* convert tests to journeys_for
* fix multi breakdown helper
* better behaviour when navigating between trends and funnels
* Refactor the multi property breakdown helper to calm the mypy type checker down
* safer access to dict in multi property helper
* makes clean filters feature flags aware and keeps breakdown values when switching between trends and funnels
* test fix
* remove overly verbose data tests
* add some more specific tests to multi prop breakdown
* obey mypy
* remove the breakdown type from the backend which was causing serialization issues
* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* add flag to funnel event query
* remove unnecessary logic
* typing
* working actors func
* fix typos
* remove unused func
* use serialized result
* wrong var
* use actors
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
* update typing
* clean up
* use actor func for all funnel persons
* types
* make sure limits are set
* remove functions
* fix limits
* funnel trends
* types
* indexing
* moved basic funnel tests
* unordered test
* remove duplicate test
* check ids
* more test adjustments
* sort
* change class names
* sort
* reformat
* change string to int
* remove duplicate
* update comment
* use actor result to paginate not serialized
* adjust tests after merging
* simplify typing
* don't import removed func
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* don't add leading slash to absolute url with no protocol
* don't add a base uri in only one funnel instantiation - will probably break tests
* remove console logs
* every API verb normalises URLs
* Tiny rename
* Minor refactor
* Test on FeatureFlagMatcher
* Add a field to feature_flag indicating if aggregating by groups
* Use f-strings
* Handle property filter building for group property
* Rename is_person_query
* Add message
* Tests for property_to_Q changes
* Add tests for groups w/ rollout
* Add comment for my_flags
* Hook up decide with groups
* New test helper for snapshotting pg queries
* Add snapshot tests for feature flags
* More is_simple_flag tests
* Add decide endpoint tests
* move conftest for ee tests to work
* Fix typo
* conftest troubles
* query for events with matching session_id if possible
* move test to clickhouse
* move event list to list instead of set for flaky test snapshot
* comment to clarify test
* original behaviour was to set breakdown_values regardless of if it was truthy
* add a test for the bug
* ensmallen the test case
* lift tested interface up a level
* implement multi property breakdown as an array from the spike
* correct type hint on method
* really resolve the conflict
* don't break groups
* refactor test assertions for breakdown cases
* adds a test to prove that funnels can receive a string and not an array
* protect saved dashboards from multi property changeover
* WIP
* multi breakdown working with funnel step breakdown
* prove funnel step person breakdown works with multi property breakdown
* don't need to protect cached dashboards from multi property breakdowns when they can't be set from the UI
* capitalise keywords in SQL
* convert a single test to journey helper
* wip
* account for funnel step breakdown sometimes being an array sent as a string
* safer handling of funnel step breakdown
* convert a test
* revert commits that made things worse
* simpler handling of funnel step breakdown
* no need to change funnel step breakdown type hint
* update imports
* guard against integer properties
* compare funnel step breakdown differently now there are arrays involved
* look for strict intersection for funnel step breakdown
* update test snapshots
* need to set breakdown_values earlier in processing
* remove tests that cover speculative functionality
* update snapshot
* move setup of breakdown values back out of update_filters
* update snapshots
* remove a sql parameter that was never assigned to
* Update ee/clickhouse/models/test/test_property.py
Co-authored-by: Harry Waye <harry@posthog.com>
* Update ee/clickhouse/queries/funnels/base.py
Co-authored-by: Harry Waye <harry@posthog.com>
* address review comment to simplify reading json expressions for breakdown
* clarify why some uses of get_property_string_expr escape params before passing
* add keyword arguments for calls to getting property string expressions in funnels
* switch to keyword arguments in test helper method
* fix parameterised test
* add multi property materialized column tests
* introduce the shim to allow new API for breakdown properties
* can't remove the naive funnel step breakdown list detection
* move funnel step breakdown list handling
* better handling of numeric funnel step breakdown values
* update snapshots
Co-authored-by: Harry Waye <harry@posthog.com>
* set debug, test, and primary_db env variables based on context
* remove unnecessary print
* also use pytest.ini for some tests run straight from pytest at the CLI
* Addtest for people_urls in retention trend response
* Add people_urls to retention trend response
* Run black formater
* sort imports
* Add default value for base_uri
* Let type be missing in EventPattern in tests
TypedDict is so rubbish in not allowing missing attributes 🤷
* fix typing
* refactor creation helper functions
* revert docker compose change
* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* remove unnecessary logic
* typing
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
I built this image a while ago, it's built with `yarn
arm64:build:clickhouse`, then I did a multipart build to copy only one
binary over and symlinking the various incantations (eg. server, client
etc) to keep the size down.
* WIP persons table schema migration
* add version to create_person
* test increased timeout
* Update ee/clickhouse/migrations/0019_person_table_version_column.py
* remove flaky kafka test
* format
* test a new approach - make this a non-large migration?
* Update ee/clickhouse/migrations/0021_person_table_version_column.py
* update
* Refactor stickiness to have its own event_query
This will speed up queries significantly and allow for filtering by
group properties
* Use same event_query for stickiness people
* Minor cleanup
* Add tests (and missing file) to group filtering in stickiness
* Allow aggregating by groups in stickiness
* Show group property filters in FE for stickiness
* convert the smallest of the funnel test files
* convert unorder funnel person test to journey for helper
* convert another file
* convert another file
* convert another file
* convert another file
* convert another file
* undelete snapshot files
* undelete snapshot files
* Revert "convert another file"
This reverts commit ef08511509.
* seperate clickhouse and postgres queries
* move to single query
* remove ununsed imports
* fix cohort + subquery issues
* some cleanup
* fix mypy issue
* fixes to query + tests
* add test for person and cohort properties
* remove cruft
* fix api tests
* Remove cohort + person prop filter from postgres
* move cohort and person property tests to clickhouse
* Filter down events
* typing fix
* fix test
* remove unneeded snapshot
* tests working with snapshots
* switch limit to property
* use materialized columns
* move to new distinct_id query
* add snapshots
* fix materialized column bug
* support cohort and person prop filters
* add person and cohort filtering
* remove cohort filtering from clickhouse
* small tweaks
* Lint fix
* remove cohort comment
* feat(funnels): update frontend to use new people_urls in response
This change updates the `FunnelBarGraph` and `FunnelStepTable`
components to use the new `converted_people_url` and
`dropped_poeple_url` in the funnels response.
I need to check that this covers all the cases for funnels. I've only
added people urls for funnels of type step for ordered/unordered/strict
and haven't yet touched the time bins and conversion time funnel
endpoint variants.
* add /some/people/url/ to tests
* chore: add action back, update tests to reference it
* fix typing
* wip
* get list of groups
* move backend endpoints
* frontend wip
* fix routes for groups
* add groups file
* try to fix tab nav
* fix groups query
* reformat results, add limits and offsets and next url
* move groups pages logic into own file
* fix and clean up tabs
* account for pagination and response reformatting
* lint fixes
* optional prev input
* typescript errors
* typescript fixes
* address feedback comments
* groups list logic test
* render properties expandable row
* fix test and lints
* prettier
* simplify table column title
* Fix a typing issue
* Use enum > boolean parameter
This makes what's going on more explicit
* Turn off pagination at the bottom of the table
* Rename table column
* Make pagination in groups work
Quite bugs fixed:
1. `previous` page was always shown, even if going into negative offsets
2. query_result could never be larger than `limit` with the code
* Cleanup code
* Make sidenav active under persons
* Use the correct query
* Add tests for pagination
* Update wording again
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* cumulative dau for non breakdown
* breakdown case
* clean up test
* add trends request util
* use simpler test util
* types
* insight tests
* add snapshots
* black
* adds a first draft of a test helper for clearer event setup in tests
* one spelling of journeys
* obey the type checker
* convert all of breakdown_cases to the new test helper
* Removes unused helper method
* test(funnel-people): add tests for converted/dropped people urls
As per https://github.com/PostHog/posthog/issues/6935 we are adding urls
to responses to attempt to improve consistency between insight responses
and the people we display in the UI.
This simply adds tests for these urls, just for funnel step response
shape. The other two types of response will be handled separately.
* feat(funnel-people): return converted/dropped people url in funnel
Here I have simply added converted / dropped urls in the base funnel.
This touches both clickhouse and postgres functionality, so I will
probably add in tests for postgres also, or just disable to postgres.
It doesn't pass yet as there is an issue in that the order appears to be
ignored by the people endpoint.
* fix(funnels): actually use strict/unordered persons querying
Previously we were always using the standard `ClickhouseFunnelPersons`
class for retrieving people from the `/api/persons/funnel/` endpoint.
This change selects from the unordered and strict variants based on the
`funnel_order_type` setting.
Refers to https://github.com/PostHog/posthog/issues/7058 although there
is a frontend component to add that will truely resolve the issue.
* feat(funnel-people): add people urls for funnels with breakdown
In this I have also removed any changes from the non-clickhouse code.
Note that there are two places I've added the url generation code, one
in the `ClickhouseFunnelBase` and another in the `ClickhouseFunnel`
I think the former covers the unordered and strict cases, and the later
for the breakdown case. I think there are further test that I'll need to
add to validate e.g. if breakdown + strict work as expected.
* Fix people urls for unordered funnels + breakdown
* test: add test for strict breakdown with people urls
* make funnel response assertions not check people url equality
* fix typing
* fix tests
* remove new line in postgres funnel.py
* clear cache on insight test start
* no really, clear the cache
* remove flakiness from strict funnel test
* correct the unordered test
* use absolute uris
* use step.index not step.order
* remove out of date comment
* use step.index, not step.order
* use step.index, remove unordered funnels comment
* use journeys_for instead of create_events
* add test_helper_methods
* move all the tests where the properties are the same for all events to the journey helper
* compare funnel results without caring about person order
* spell words correcterly
* Revert "spell words correcterly"
This reverts commit befb83b183.
* Revert "compare funnel results without caring about person order"
This reverts commit 268927b8ba.
* correct types for test props
* add groups property values
* update groups property tests
* newline
* fix
* update test
* python black
* clean up tests
* rename to endpoint in operator value select
* add empty property values test
* do not return in an array
* fix
* feat(correlation): use people_url to load people modal
This change adds a new method to the `personsModelLogic` to allow for
specifying a url to be used to retrieve people from. I've avoided
overloading the existing `loadPeople` and rather added a new one,
`lostPeopleFromUrl`.
Currenlty this is only being used by correlations, but the intention is
to extend it's usage to anywhere else that is performing drilldowns to
people from, for instance, the trends aggregation.
* add paycard back
* make labels look right
* remove unused code
* fix typing in test
* fix typing in personsModelLogic
* refactor to make eventUsageLogic work
* fix typing
* fix typing again
* remove some funnel stories
* get storybook working with paycard
* cleaned up the interface a little
* chore(correlation): use `api.get` for fetching people
This updates to use api.get, and also converts the returned uri from the
correlation endpoint to an absolut uri, instead of just an absolute
path.
The `api.get` method expects either an abolute uri, or a relative path.
I'd rather not use an relative path as it's not obvious what it should
be relative to.
* fix tests
* make storybook uris absolute
* Paths filtering by groups backend
* update correlation tests, now that CTEs are included in sqls
* use decorator for materialising to ensure clean up happens
* cleanup offending tests
* feat(correlation): add people drill down urls for event correlation
This change simply adds urls in the response of the correlation endpoint
for each correlation returned. Note that this only adds support when
using correlation_type=events, as the
correlation_type=events_with_properties requires further changes that
I'd like the separate out to keep this PR small.
* feat(correlations): add people by property to correlation response
This commit covers the PROPERTIES case for making sure we return a url
that can be used to retrieve the people associated with a correlation,
either success or failure. This case is a completely different url than
for the events people endpoint as we are reusing the funnel people
endpoint, and filtering by "funnel_step_breakdown".
I'm uncertain if this is actually the correct url at this point in time,
as I have just attempted to copy what I have seen in the request from
the live running app.
* fix test
* Remove another param
* fix mypy typing
* Fix tests
* Add url response support for event_with_properties
* remove type assertion
* remove variable hack
* Remove funnel_steps
* use with_data and to_params
* Remove comment regarding complexity, it seems to pass the tests so...
* Add autocapture support for people url
* refactor people url construction a little
* remove properties attr for event people request
* fix if
* no really, fix if not None
* Remove _assert_never, mypy needs updating
* Fix no return mypy error
* Remove json.dumps
* add window-id to ingestion
* move window_id into snapshot data
* Add tests
* add window_id to snapshot_data only if included in the event
* add window_id to session_recording_events table
* revert ph-js downgrade
* fix data generator
* remove partition change
* update window_id defaults for tests
* missed one window_id default
* rename migration for conflict
* Migration to use materialized columns for groups
Workaround for https://github.com/PostHog/posthog/issues/6422
* Use groups materialized columns in queries
* Update mat column creation tests
* Simplify aggregation_target_field
* Fix migration
* Update snapshots
* smallest change to make aggregation work
* address comments
* add snapshot
* move function to groups model
* update funnel snapshot
* rename person_id to aggregation_target
* update snapshots as well
* dont support persons query mods for now
* update snapshot
* make array orders deterministic
* Improve process_math
* Add test for overlapping group keys
* Improve event query tests
* Add test for filtering by person properties together with groups
* Avoid flaky tests due to cohort_id changing
* Update queries and snapshots
* Add groups stuff
* Rename column from person_id to `target` in retention queries
No behavioral change, preparing for groups work :)
* Remove dead if statement
* WIP: Retention aggregation by groups
* Handle aggregation by groups in retention
Also handles the case where not every event has a property defined
* Test groups validation mixin
* Reformat
* Improve test for aggregation in retention
* Add /api/projects/@current/groups/property_definitions
To load property definitions for groups
* Add routes for fetching group types
* FE: Group analytics feature flag
* FE: Group analytics base types for properties
* FE: Initial implementation of fetching logics
* add group tabs to filter
* make sure group properties populate and refactor things
* prettier
* disable unused vars
* allow indexing objects by number
* mount group properties logic
Co-authored-by: Li Yi Yu <li@posthog.com>
* Extract GroupsJoinQuery
* Add test for breakdown filtering
* Unify breakdown mixins
* Allow passing breakdown_type == 'group' with breakdown_group_type_index
* Allow breakdown by group props in trends
* Add tests for trends breakdown_props function on group breakdowns
* Solve common issues
* Output snapshot diff into console
* Clean up materialized columns after tests
* Add zero protection
* Solve test failure
* Type math in Entity
* Allow passing group_type_index from FE to BE
* Get a initial query running
* Add group value filter if aggregating by groups
* Add snapshot testing for trends queries
* isort
* Update tests
* Add test for column_optimizer
* Update ee/clickhouse/queries/trends/util.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* adds a test to prove that without any changes the API receives a string and returns one see #938
* put a guard in place so that funnel query can be shifted to arrays without affecting existing clients
* make checking for single property breakdowns safer
* sort imports
* sort imports
* reformat files
* Revert "reformat files"
This reverts commit 58530f134f191d53b927abe6fb6f5d2d384cb255.
* expand short variable names
* corrects a typo
* it wasn't a typo it was a logic error
* Add group type, group_type_index
* Raise an error when handling unsupported properties in CH
* Improve repr
* Fix is_superset function
This was previously broken - sorting and zipping doesn't really work for
this intent.
* Add group_type_index to analysis results
* Add `group_types_to_query`
* Minor typing fixes
* Create groups tables in tests
* Simple first filter by groups query
* isort
* Use snapshot testing in event_query tests, add test for groups
Co-authored-by: Marcus Hyett (PostHog) <85295485+marcushyett-ph@users.noreply.github.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* feat(correlation): add team wide person property exclusion list
This change just adds the exclusion list to the `Team` model and checks
that it can be updated. Separately we can:
1. add interface for the Project page to display
2. pull and update this list from the funnels correlation page
NOTE: We don't perform any validation on the structure, :fingerscrossed:
this won't be an issue. Alternative would be to use ArrayField, but use
of JSONField is consistent with other fields.
* test(correlation): add test checking exclude properties pulled from team
* feat(correlation): load exclude properties from team settings
Previously we were loading exclude properties from local storage, so we
would not be sharing the exclusion list between users. This change
simply plugs the `excludePropertyNames` into the `teamLogic` for the
purpose of persisting and loading the values.
* use [teamLogic.actionTypes.loadCurrentTeamSuccess]
* get tests passing
* switch frontend to using `correlation_config`
* refactor: use Team.correlation_config for persistence
* add project settings for correlation
* fix lint
* ensure excluded properties are saved to project config
* Add default excluded properties
* format
* make propertyCorrelations not null
* ensure excluded property names config is unique
* rename excludeProperty to excludePropertyFromProject
* update var names
* change to targetProperties
* remove null special casing
* update to filter client side on exclude from project clicked
* update test name to reflect new functionality
* fix tests
Without this, to run plugin-server tests you need to reset all
containers every time since otherwise both test- and non-test clickhouse
would attempt to read from the same topic.
* backend fixes and test
* add breakdown value to pie chart
* adjust test
* fix faulty test
* fill param
* fix formula tests
* more date passing
* more cleanup
* all tests working
* make test data explicit and add better checks
* support both ee and postgres
* length checks
* paginate recording compression
* some tests
* more accurate duration calculation
* add tests and types
* tons of decompression fixes
* rename test file to avoid conflict
* move decompression to helper
* add test for helper
* type fix
* rename method
* simplify paginated decomression
* handle case where offset exceeds length
* clean up
* test fixes
* clean up on aisle 12
* Add surrounding object for metadata response
* Add table for group_type_mapping
* Remove materialized columns from events table schema
These are not used and not needed w/ new mat columns work
* WIP: Migration to add group analytics columns
* Remove event table changes temporarily
* initial refactoring
* popup UI
* refactor path cleaning logic
* add nullable
* all ui working
* fix migration
* use regex replacement from team object
* add flag
* add switch
* fix type
* fix type
* UI update
* restore removed arg
* add local path cleaning filters to api
* add test for local path filters
* working new UI
* reduced repeated code
* fix numbering
* minor refactoring
* update copy
* add under advanced features
* address comments, minor cleanup
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Previously we would attempt to generate a response even though there
were no steps. There appears to be some code paths that blow up if this
happens, so instead we return as soon as we can in this case. This
appears to be the behaviour elsewhere also.
This resolves the sentry error found here:
https://sentry.io/organizations/posthog/issues/2718768248/
* paginate session recording events api
* code quality
* don't rerender replayer every time events get loaded
* refactor session recordings, make it backwards compatible, add bunch of tests
* move limit and offset to after decompress
* change limit
* add caching of recording
* add duration
* fix a few tests
* fix api tests
* add partial chunk test
* fix default limit test
* code quality
* typing
* fix backend tests
* mypy fixes and signature
* remove circular dependency
* mypy and sessionsplaylogic tests
* fix migration
* make single migration
* revert mypy typing
* remove require react resolve
* fix loading state to persist across multi chunks and add test'
* cleanup
* duration to ms
* fix duration tests
* remove require react resolve
* fix test
* bump rrweb player
* change cache behavior plus more
* fix frontend tests and make duration calculation more robust
* fix timestamp in ms bug in tests
* fix duration 0 test
* fix compatibility with rrweb-player
* update lockfile
* yarn unlink
* resolve kea-test-utils changes
* another one
* remove caching for now (leave as todo) and change limit to 100
* Refactor column_optimizer to work differently
* WIP: Use counter over set
* Handle person filters in person query
* Remove a dead argument
* Use enum over parameter for determining behavior
* Allow excluding person properties mode when handled in person query
* Fix _get_person_query type
* Use correct table for funnel_event_query
* Remove unneeded override
* Add extra typing
* Filter by entity.properties in person query for trends
* Handle error 184 due to naming clash
* Better default for prop_filter_json_extract
* Update column_optimizer tests for Counter
* Handle person_props as extra_fields
* Handle breakdowns and person property filter pushdown
* Transform values correctly
* Simplify get_entity_filtering_params
* Fix funnel correlations
* Solve caching issues in trend people queries
* Remove @skip test
* Add syrupy tests for parse_prop_clauses
Can update these via --snapshot-update
* Add snapshot tests for person queries
* Add a few notes
* Update test to avoid collision
* Kill dead code
* Handle PR comments
* Update ee/clickhouse/queries/person_query.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Add `slug` fields to `Organization` and `Team`
* Expose slugs to user
* Add slug autogeneration for new orgs/projects
* Improve slug UX
* Remove slug from settings
* Update org/team instance creation plus add tests
* Only require project slug to be unique for org, not globally
* Fix `get_prep_value`
* Test organization slugification
* Deslugify `Team`
* Clean up changes
* Update test_user.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Random 4 letter suffixes for the win
* Fix import
* Ignore `test_migrations_are_null`
* Fix `RunSQL` query being empty
* Fix `generate_random_short_suffix` testing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
This seems to be semi-consistently failing locally for me and
inconsistently on CI. The cause is unknown - debugging locally, this
seems more of a query correctness issue than a test data incomplete
issue.
* fix bug where session recording is split by new distinct_id
* fix query bug to allow distinct_id clause
* Basic query filtering by events on clickhouse
* add duration filtering
* Add multiple action/event filters
* Move to using person_id + optimize query
* add postgres support and querying on person_id
* move back to distinct_ids
* remove lots of duplicated logic
* Add pagination to backend
* Pagination
* Add duration and date filter
* Use person header is list view
* type fixes
* bug fix
* bugs and tests
* test fixes
* style filters
* logic tests
* Add another test for team leakage
* fix flaky test
* UX changes
* UX tweaks
* auto focus filter box on open
* add test for recording viewed change
* move duration filter to using a tested logic
* move to clsx
* add debounce breakpoint
* Small style change
* fix debounce
* fix breakpoint
* move test to defaultAPIMocks (merging changes)
* react to url change
We'll be using qemu that ships with docker for mac. I'd updated this at
the same time as zookeeper but obviously never removed the kafka
container from before somehow.
* WIP: Create new property types for simplified cohorts
* Add documentation on simplified_cohort_filter_properties
* Handle static-cohort/precalculated-cohort property types
* Handle new property filters properly
* Add casting
* Test cohorts in more cases
* Fix a bug
* Fix benchmark simplifying
* Avoid redoing work every setup for benchmarks
* Update typing;
* Remove unneeded scope
* Add tests for simplifying and cohorts
* Roll more of "do we need to join persons table" behavior into ClickhousePersonQuery class
* Handle precalculated cohort logic in sessions
* Simplify event query
* More tests without any JSONExtract
* Simplify entity properties as well
* Improve docstring
* Add test for breakdown & precalculated cohorts
* Add test for filtering sessions by precalculated cohorts
* Reset unneeded change
* Update cohort
* Solve some typing issues
* Update benchmarking
* Fix cohort filtering tests
* Fix cohort tests
* Fix a caching issue
* Typecheck
* Handle exclusion filters
* Simplify filters code
* Simplify filters ASAP if filter is created
* Simplify route
* Remove simplification-specific logic from queries
* Remove recursion, update tests
* Pass team in more cases
* Update column optimizer specs
* Test simplify
* Update trends test
* Fix rebase fail
* events dead letter queue CH table
* format
* update schemas
* also store raw payload
* better naming
* make table name more clear
* wip better testing
* remove unused imports
* remove kafka test
* prevent non null test from running on CH migrations
* add kafka testing
* minor tests cleanup
* test naive longer sleep
* make test end-to-end
* address review
* update ttl, format
* refactor delay func, address review
I've been getting lot's of errors with the qemu emulated x86 version.
I'm just changing the arm docker-compose as I don't want my issues with
local dev to affect others.
* fix(correlation): ensure correlation partitioned by team
Previously if distinct_id's were the same between two teams, we'd end up
pulling in the event data between the teams.
* dev(tests): clear cache between partition calls
* fix(correlation): add lower bounds for selected events
Previously we would consider all events for correlation calculation. Now
we use the funnel `date_from` as the lower bounds.
* chore(correlation): exclude funnel steps
* chore(correlation): make sure cache is cleared before each test
* Update funnel success comment
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* fix: filter events by team_id
* chore(correlation): remove CTEs from correlation query
There seems to be an issue with the CTEs and production clickhouse, see
https://github.com/ClickHouse/ClickHouse/issues/29748
Instead of risking it, I'm just removing them.
* chore: update entities -> events for funnel step exclusion
* fix team_id = team_id issue
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* chore(diagnose): add a stub endpoint for event diagnosis
This adds an insight endpoint that takes a `target_entity` and returns
a list of events ordered by significance of a person reaching
`target_entity`
Followup PRs will add actual calculations but this should act as a
conversation piece around the structure of request and response, as well
as a stub from which UI development can start.
* play around to get mypy typings right
* Sort out test reponse structurea
* refactor: address CR comments
* feat(diagnose): add diagnose stubs for implementation
* feat(funnel): add event correlation calculation implementation
This adds to the `/api/projects/<team_id>/funnel/correlation` endpoint
an implementation that calculates the odds_ratios for each event that a
user that has been part of a funnel, successful or otherwise.
* chore(correlation): get query working
* refactor(correlation): move functions to Query methods
* Always allow filtering on distinct id
* fix
* fix
* remove accidentally commited test
* include table name
* add table specific because distinct_id doesn't exist on person table
Co-authored-by: eric <eeoneric@gmail.com>
* make sure funnel and path dates are always aligned and add api handling for funnel filter persons
* use post instead of GET for paths
* parse label
* change type
* use post in test
* fix types
* test both get and post
Our queries now contain a lot of comment noise that's not relevant when
developing. This change makes PRINT_SQL env variable usable again by
stripping these comments.
* dev(clickhouse): strip out comments before executing sql
This is so we can easily copy/paste from e.g. Metabase by querying the
system.query_log. In metabase is doesn't display new lines (although you
can download to file for this), but it's not very practical.
* test(clickhouse): use `capture_select_queries` in comment strip test
* test(clickhouse): only sub. params if non-insert query
This parallels `clickhouse_driver` behaviour.
* chore(clickhouse): move sql preparation to dedicated function
* refactor: rearrange func and type definitions
* Add asv code
* Get a benchmark running
* better output folder
* Move benchmark file
* Cleanup of config
* Run a query and benchmark
* Skip benchmarks for isort
* Set up materialized columns before test
* Better skipping logic
* Add first proper benchmark, add some documentation
* Person property filtering
* Add new workflow
* Show stderr in test run
* Continue debugging actions
* Try to save benchmark results to separate repo
* Output to main directory
* Scheduling
* Remove if temporarily
* Int for ms
* Comment on the PR
* Collapsible section
* Update README
* Clarification
* Remove h3
* Remove pyproject.toml, .gitignore changes that are unneeded
* Add project-nested version of all project-dependent endpoints
* Fix hook `perform_create`
* Adjust `plugin-configs` to adhere to style of other endpoints
* Test Kafka
* black format python
* fix imports
* add kafka and zk deps for testing
* Include ZK and Kafka for all tests
* fix signature for kafka helper
* Connect to localhost for kafka
* update kafka host for all test runs
* Wrong env var for kafka
* consolidate env vars for github actions
* set the advertised hostname from the broker to localhost
* add env var to docker-compose for kafka broker advert host
* resort to what we do locally with /etc/hosts
* Remove configs for kafka that won't be used
* restore progress
* working connection no UI visual
* working for events
* edit endpoint to start point
* working with close button
* funnel to path work (no dropoff yet)
* run prettier
* make suggested adjustmenst
* Refactor `AvailableFeature` from strings to an enum everywhere
* Fix circular dependency and type
* Add "Per-project access" feature flag, premium feature, and organization switch
* Rename `OrganizationMembershipLevel` to `OrganizationAccessLevel`
* Create `ExplicitTeamMembership` model
* Show whether projects are restricted in the project switcher
* Update organizations API code
* Fix migrations
* Move organization tests that require EE to `ee`
* Revert `OrganizationMembershipLevel` rename
* Fix organization tests
* Update migration
* Fix schema and add Members to Project Settings
* Build out test memberships API with security tests
* Update `TeamMembers` and `teamMembersLogic`
* Move "Per-project access" description to tooltip
* Add moar tests
* Fix Project Members list logic
* Add additional membership checks
* Update migrations
* Fix typing
* Adjust explicit team memberships API similarly
* Fix typo
* Unify `ExplicitTeamMemberSerializer`
* Remove old changes to `membersLogic` usage
* Use `effective_membership_level` on `TeamBasicSerializer`
* Clean up organization update tests
* Explicitly disallow enabling per-project access for free
* Fix circular import
* Remove `id` from `UserSerializer`
* Fix typing
* Try to fix import
* Fix fatal typing
* Add more tests
* Update permissioning.ts
* Add clarifying comment to migration
* Fix import
* Revert `TopNavigation` changes
* Restore `TopNavigation` changes
This reverts commit 05fd9e4ed6.
* Try to add a story for `TopNavigation`
* minor clarifications
* Revert `TopNavigation` changes
* Restore `TopNavigation` changes
* Make new access control entirely project-based
* Update migrations
* Add `project_based_permissioning` to `TeamBasicSerializer`
* Update test_team.py
* Fix `isRestricted` in `ProjectRow`
* Disable project creation for non-admins
* Make project icon in top nav itself dynamic as well
* Fix story
* Delete 0169_project_based_permissioning.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Update frontend/src/layout/navigation/TopNavigation.tsx
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Address feedback
* Project-based permissioning full data access restrictions (#6068)
* Fix Access Control restriction tooltip
* Add `TeamMemberAccessPermission` and use it in viewsets
* Add `ErrorProjectUnavailable` scene
* Ignore mypy
* Update MainNavigation.tsx
* Update explicit_team_member.py
* Fix frontend detection of unavailable project
* Fix some tests and edge cases
* Fix basic permissions
* Add more tests
* Simplify `ExplicitTeamMemberViewSet` permissions
* Improve restrictions and add moar tests
* Update frontend
* Fix a couple of things
* Fix import
* Fix some edge cases
* Fix typing errors
* Use hedgehog instead of moth
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Address feedback
* Add proper permissioning to dashboard views
* Update ee/api/test/test_dashboard.py
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Run prettier
* Remove debug code
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Move Sessions tab -> Session Recording (behind feature flag)
Note: the /scenes/sessionRecordings folder is a copy of /scenes/sessions. Planning to remove sessions from the new SessionRecordings folder over the next coupld of commits
* Add very basic session recording page
* Add session recordings to persons page and session recs can play
* add endpoints for clickhouse
* fix ordering in existing tests
* add basic test to pg and clickhouse
* small bug fix
* type fix
* add test for data leaking between teams
* fix navigation bugs
* Fix name of logic
* Add basic logic tests
* fix tests
* fix laoding state bug
* revert changes to existing session queries
* move to proper queries for session recordings
* type fixes
* rename feature flag
* type fix
* Fix capitalization
* remove showSessions and clean up ifs
* Style changes and refactor SessionPlayerDrawer
* bug fix
* move to using query params
* remove type from persons logic
* Move sessionRecordingTableLogic key to distinctId
* rename setSessionRecordingId to openSessionPlayer
* improve tests
* Cleanup from review
* move team and limit to parameters
* new path tab querybuilder
* prettier
* clean up
* wip
* format querybuidler
* prepare path type filter component
* dropdown
* formatted popup
* change button for endpoints
* styling the type buttons
* logic for all left side querybuilder components
* exclusion UI and logic
* improve exclusion handling, ensure API calls are made
* API fixes for event types
* fix popup on close issue with exclusions
* hide overflow on buttons
* all ui working
* more formatting adjustments
* type fixes
* leave response ambiguous
* feature flagging and adjust styles
* Add Step limit option
* add tooltip to wildcards
* rewrite tooltip for exclusions
* make Events toggle responsive
* fix types
* responsive at xl instead of lg for no overlaps
* Proof of concept of connecting Funnel to Paths
* protecc wip
* Connect Funnels to Paths
* remove funnel button for now
* remove unused
Co-authored-by: eric <eeoneric@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Export 10k lines in CSV, allow limit= query param
* this should be min
* fix test var
* add limit support to postgres, make tests to check all cases
* update tooltip to say 10k events
* fix 10k event text again
* Refactor `AvailableFeature` from strings to an enum everywhere
* Fix circular dependency and type
* Add "Per-project access" feature flag, premium feature, and organization switch
* Rename `OrganizationMembershipLevel` to `OrganizationAccessLevel`
* Create `ExplicitTeamMembership` model
* Show whether projects are restricted in the project switcher
* Update organizations API code
* Fix migrations
* Move organization tests that require EE to `ee`
* Revert `OrganizationMembershipLevel` rename
* Fix organization tests
* Update migration
* Fix schema and add Members to Project Settings
* Build out test memberships API with security tests
* Update `TeamMembers` and `teamMembersLogic`
* Move "Per-project access" description to tooltip
* Add moar tests
* Fix Project Members list logic
* Add additional membership checks
* Update migrations
* Fix typing
* Adjust explicit team memberships API similarly
* Fix typo
* Unify `ExplicitTeamMemberSerializer`
* Remove old changes to `membersLogic` usage
* Use `effective_membership_level` on `TeamBasicSerializer`
* Clean up organization update tests
* Explicitly disallow enabling per-project access for free
* Fix circular import
* Remove `id` from `UserSerializer`
* Fix typing
* Try to fix import
* Fix fatal typing
* Add more tests
* Update permissioning.ts
* Add clarifying comment to migration
* Fix import
* Revert `TopNavigation` changes
* Add project member addition button+modal
* minor clarifications
* Revert `TopNavigation` changes
* Make new access control entirely project-based
* Update migrations
* Add `project_based_permissioning` to `TeamBasicSerializer`
* Update test_team.py
* Fix Access Control restriction tooltip
* minor improvements
* fix frontend typing
* Fix frontend typing a bit more
* adjust copy & UI a bit
* Address feedback on field comment
* "Privacy settings" to "Access Control"
* Make `FusedTeamMemberType` comment clearer
* Remove useless `export`
* Delete 0169_project_based_permissioning.py
* Clean some code up a bit
* Project-based permissioning member removal (#6067)
* Fix `teamMembersLogic` loaders
* Allow explicit project members to leave
* Add member removal/leaving button to Members with Project Access
* Restore error message
* Fix error message
* Correct things
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* test(breakdown): add test for week interval rounding overshooting
This refers to https://github.com/PostHog/posthog/issues/2675
* fix(breakdown): resolve week rounding interval over shoot for trends
Previously there was a fix for another issue with weekly intervals not
spanning a large enough range, see
69ba0b1d90
for details.
The issue that this was trying to resolve was consider a date range
date_from, date_to and a week interval. The range can cross two weeks,
but prior to the above commit it would only show one tick/week on the
graph. In this case we just added on another week to show.
This doesn't help however in the case where the range is less than a
week, in which case we'd end up showing a week/tick on the graph with
a zero value.
Instead, we remove the additional 1 week and simply ensure that both
lower and upper bounds are included in the clickhouse zero fill.
Closes https://github.com/PostHog/posthog/issues/2675
* fix(lifecycle): update lifecycle to use updated week interval handling
This is due to the change in the way the week interval is handled in the
previous commi, where we removed a +1 we were adding to week intervals
and thus getting too many date axis ticks on trend graphs.
* ci(tests): only run non-ee tests for non ee action step
It seems that we are segregating tests between ee and non-ee via a
folder structure. This is a bit of a hack, but it works for now but
ideally we should converge on one set of tests for which we identify if
they are supported on clickhouse or not via pytest markers. This should
simplify working with the tests somewhat, e.g. making them easier to
locate, update, and e.g. debug in vscode/codespaces.
* test(trends): correct old week interval test. It was wrong before
Now with the change to how we create the interval, it's fixed.
* fix(trends): avoid including one week too many when deciding on bounds
Previously there was a test in here that was saying that if we request a
graph of events, with week interval ticks starting on a Sunday, we'd
actually end up also including the Sunday before due to some arithmetic
mistakes.
* fix(trends): resolve week interval rounding issues for trends query
* test(identify): update test func to accept `properties` argument
* report on self hosted duplicate ids
* satisfy mypy
* satisfy black
* adjust test to run on ch only
* black
* fix stupid test typo
* black, again
* update report
* format
* fix black pre commit hook
* update query
* report on multiple ids per person
* fix foss test
I was having issues with running the clickhouse/ee tests and it was just
hanging. Clickhouse appeared to be up and I could perform queries with
`clickhouse-client`. For some reason it was hanging on querying, and on closer
inspection if looks like for each of the setup queries it was hanging for 6
seconds, failing to find zookeeper, and then continuing to run setup.
It's pretty useless to continue in this case, so it seems more sensible to raise
in this case.
* split out funnel apis and fix histogram paint order bug
* fix broken tests
* simplify some more
* make this fail if no steps returned
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* refactor(retention): add AppearanceRow dataclass to improve readability
This just makes it a little more clear what is being pulled out of the
database. Previously we were just getting a list of lists.
I haven't done anything else here but will follow up with a fix for
https://github.com/PostHog/posthog/issues/5443
* test(retention): reproduce retention marker index error
* fix(retention): make retention forgiving of too much data from query
This just makes the `appearance_to_markers` method not fail if the
underlying clickhouse query returns more data than expected. It should
just be less efficient than it could be and should still give the
correct results. Previously it would silently fail to include any data
in these cases.
Closes https://github.com/PostHog/posthog/issues/5443
Co-authored-by: Harry Waye <harry@scalexp.com>
* m1 clickhouse support
* add clickhouse arm64 compilation dockerfile
* use clang-12 and 2 parallel builds
* working clickhouse docker image
* this needs to be committed out for CH to work... a known issue. WIP
* move things around
* move things around again
* add arm64 build scripts
* comment out broken line in DEBUG mode, still make it break production builds
* update readme
* timeless text
* add issue in comment
* way way way too much white space
* zookeeper restarts sometimes needed
* Revert "Revert "Swap out old paths querying class with new (#5821)" (#5886)"
This reverts commit 82ef50a1e7.
* remove old ref
* test 3.9 general
* test if there's overlapping
* change version back
* make timestamp dynamic
* change funnel path type to str
* change include_timestamp_step name
* type issues
* add argument
* add type
* explicit none check
* change field names
* change constant name
* between step test
* use different arguments for defining funnel steps and add tests
* after step test
* fix types
* test for valueerror
* add test for before dropoff
* custom logic for finding paths up to a dropoff
* fix type
* update test
* add comment
* change event names in test to be clearer
* run again
* test without caching
* return caching
* try not caching again
* try not caching again
* try not caching again
* restore caching
* remove funnel_window_days
* remove unused import
* Raise ValidationError if "math" without "math_property" passed to BE from FE
BE side fix for https://github.com/PostHog/posthog/issues/5756
* Update ee/clickhouse/queries/trends/util.py
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Update typing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* swap out old paths querying class with new
* swap arguments
* swap name
* delete old CH paths file
* dont error out in calculate_by_filter
* write some backwards compatible tests for paths
* remove expanding autocapture element ability
* cleanup
* delete old code
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Move api/action/people SQL code under ee/clickhouse/queries
* Move api/action/people SQL code into a class
* Fix rebase
* Calculate Trends people via TrendsEventQuery
This should significantly speed up the query. In testing, this took one
query from 10s -> 1s.
We can reuse almost all of the plumbing in TrendsEventQuery.
* Update insert_entity_people_into_cohort
* Fix date handling in trends people, add in-depth tests for each interval
* Standardize code style a bit
* Make tests less TZ dependant
* Skip failing tests on-not-ee
* Document
* WIP: flamegraphs
* Get flame graphs generating
* Add files to bin
* Crude frontend for analyzing
* Hack: Use a stable clickhouse host instead of conn pool for flamegraphs
Otherwise no way to get back measurements due to LB.
* Error handling
* Only allow read queries
* Hide query analyze button for non-owners
* Make sure non-admins can't run queries
* Fix query issue
* fix person modal across multiple dates
* test works for pg but shouldn't for ee
* wrong number
* make it cumulative relevant
Co-authored-by: eric <eeoneric@gmail.com>
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
* Make breakdown join with person if needed
* Refactor trends to be purely class-based, don't rely on wonky inheritance
* Extract method
* Improve person join behavior
* Remove unneeded parameter
* Mark a function thats always passed as such
* Add test case demonstrating previous non-join case
* Unify two get_breakdown_prop_values methods
* Add test for materialized columns
* Simplify trends breakdown query
* Unify _breakdown_prop_params for events/person breakdowns
* Use shared column_optimizer
* Typing fix
* Cleanup
* Code style cleanup
* Code style cleanup
* Fix param ordering
Closes https://github.com/PostHog/posthog/issues/5808
I'd love to fix this at the schema level but looked into it for ~20
minutes and it didn't seem like there's a pretty solution that allows to
pass `None` to .objects.create and for the default to be used.
Other solutions would be even more invasive and require manipulating
serializers.
Best leave this to when we refactor all of out `properties` JSONFields
since they all have this issue
* Hotfix: Use materialized columns on cloud
This was broken since default_kind was different on Distributed tables
on cloud
* Improve __init__.py
* Make reverting DEFAULT column async, only ON CLUSTER for events table
* ColumnOptimizer: Add functionality for person materialized columns
* WIP: Use person properties in trends/funnels
* Test person materialized props in trends cohort breakdown query
* Make use of materialized person properties in breakdowns
* Mark some cases working with materialized columns
* Test and fix breakdown by person props without filtering
* Make filtering by entity/person props on a join work better
By not assuming everything is under `properties`
* Add test case around breakdown with person properties
* Add test cases around materialization
* Add another materialization test
* Test cohorts and fix breakdowns with person filter
This exposes a limitation in the current implementation
* Fix some cohort tests
* Fix event query tests
* Get a funnel materialized column test running
* Cover more funnel breakdown tests with materialized
* Handle person property breakdowns in funnels
* Fixup funnel typo
* Add tests, fix an indentation issue
* TestFunnelPersons with materialized columns
* Test funnels against actions with person filters
* Add failing test for entity filtering failing
* Add test case for filtering with entity properties
* Show my 'broken' test is actually doing a subquery
* Resolve linting issues
* Avoid naming collisions when materializing columns
1. Prefix person properties differently. Mixing them up can break
ambigious column issues
2. When name already exists, suffix with random junk :)
* Implement analyze.py
* Add `suggested_columns_to_materialize`
* Add code to backfill a materialized column
* Add tests for backfilling data
* Cleanup
* Add tests for analyze
* WIP: Crontab for materializing columns
* Nooped task for materializing properties
* Use mutations_sync=0 for column tests
* Add comment
Looking at the generated SQL it has bothered me that the list of entity
names is not unique for funnels in queries like `event IN [somearray]`
This tries to fix that.
* Speed up EE tests by not destroying tables
* Fix broken tests
* Speed up tests by not running migrations
* Fix last test
* req file?
* caching
* Parallel
* revert exp
* debug without parallel
* Revert "debug without parallel"
This reverts commit 83b3ad633f.
* Undo parallel tests
* Speed up more tests
* use final
* correct final
* print exception
* more prints
* move test
* add filter
* remove duplicate test
* prints here
* test wait
* set variable in query
* remove mutations_sync
* cleanup
Co-authored-by: eric <eeoneric@gmail.com>
* Remove dead argument
* Make allow_denormalized_props always explicit
* Change prop_clauses default
* Create a testing decorator for checking materialized columns
This makes it easier to have proper test coverage for materialized
columns and make sure no bugs creep up :)
* Fix event_query
* Test more materialized columns in trends
* Add materialized column tests for funnels
* Cleanup path_event_query
* Fix default
* Fix issue with clashing parameter names
* initial implementation
* debug some tests and revert format to original
* some minor cleanup
* add screen handling
* start point handling
* improve indexing so multiple sessions by the same user are handled properly
* all tests passing
* fix type
* fix types
* change types
* more types
* rename base paths
* upgrade start point functionality
* assertEquals, remove inequality
* Revert "assertEquals, remove inequality"
This reverts commit 5eb229fef7.
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Fix fetching event-based breakdown values with `LIMIT`
* Improve test comment
* Limit prop clauses inclusion to breakdown queries
* Address feedback partly
* Fix imports
* Remove non-3.7-compatible `Literal`
* Don't try to import `Literal`
* Fix oversight
* Always include prop filters in `get_entity_filtering_params`
* Revert "Always include prop filters in `get_entity_filtering_params`"
This reverts commit 4f61edd19a.
* Add and test handling of actions in addition to events
* Update test_funnel.py
* Move new tests to `breakdown_cases`
* Fix `icontains` and support `href/`text` element filters
* Test `is_set`/`is_not_set` too
* Require newer `flake8` to support Py 3.8 walrus operator
* Better handle empty mutlivalue operators and support doublequotes
* Polish all element filtering cases
* Roll back general `icontains` fix
* Make minor improvements
