* stores numeric type for property definitions
* writes property definition type for strings
* can write dates for simple string cases as property definition type
* simplify property definition types
* can detect unix timestamps
* rewmove a little more from the test duplication
* some more timestamp thinking
* clearer comment
* rename refactor in tests
* more test correction
* address test failure
* another test fix
* property type as enum
* simplify type detection
* apply typing suggestion
* allow property definition type and format to be set if property is present but they are null
* add more date types for property_type_formats
* add more datetime formats that the backend can accept and filter events with
* delete conflicting migration
* add migration for backend datetime formats
* plugin server can auto detect all the types the backend knows about. And the tests are collapsed from twenty five almost identical tests to two different parameterised blocks
* iso 8601 strings can have fractional seconds
* add link to the RFC 822 regex pattern
* capitalise SQL
* capitalise the SQL the mock looks for too
* remove the under-eager cache change that allowed updating existing property definitions
* convert property type formats to enum
* fix test
* try detecting a property type for existing property definitions but don't keep retrying
* use symbols to make cache state clearer
* fix idiocy
* speculative LRU cache usage with measurement
* tag the statsd guages instead of having team id in the name
* don't pre-emptively optimise the cache
* rename migration
* remove spaces from query strings
* separate property definition code from the team manager
* a little tidying
* correct type definition
* correct test params setup
* wip
* display known unix timestamps as date strings in property filter select box
* show selected unix timestamp property as a datetime string in the property filter
* remove console.log
* update test to construct property definition correctly
* correct another test
* dates have different operators
* format property filter values for display in Select boxes and the PropertyValue component
* fix typescript error
* it _is_ a type, I don't need to say it is
* don't have before and after for unmatched types of operators
* separate the current applicable operators and the operator to display mapping, and add cypress tests for before and after operator visibility
* correct cypress test
* remove unused method
* remove dangling comment
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* log values of operator options, why does this fail in CI but not locally
* why does test fail in CI but not locally
* intercept more decide calls to try and set feature flags from cypress test
* why does test fail in CI but not locally
* Remove cypress install
* add a readme for how to test feature flags so it isn't lost if there are no tests currently testing feature flags
* refactor(lifecycle): simplify clickhouse sql logic
This updates the SQL to be comprised of two queries, one for getting
new, returning, and resurrecting periods of activity, one for getting
dormant periods right after periods of activity.
Refers to https://github.com/PostHog/posthog/issues/7382
* refactor(lifecyle): use `ClickhouseEventQuery` to build event query
* format
* Use bounded_person_activity_by_period for both sides of dormant join
* refactor(lifecycle): reduce pdi2 join by one
This means we're now under the current query memory limit for orgs with
around 20m distinct_ids. It does remove some readability though :(
* update snapshot
* Add further comments to query
* Add further comments to query
* Add further comments to query
* Remove dead variables
* Refactor person_query overriding
* Lifecycle refactoring continued
* Update lifecycle tests (except people ones)
* Make lifecycle people endpoint happy
* Remove django lifecycle tests
* Add some edge case tests
* Add missing type
Co-authored-by: Harry Waye <harry@posthog.com>
* Resolve weird SQL formatting issue
* Use a discriminated union for ActorType
* Use standard response types for related groups
* Update typing
* Always filter related actors by group type index
* Update snapshots & typing
* Migration to add name_singular and name_plural to groups
* Allow modifying group type names
* Use name_plural and name_singular through the app
* Update tab names
* fixup
* Update snapshots
* add trends option to experiment
(cherry picked from commit be9b8911f9)
* connect trends experiments
* clean up some stuff
* fix test
* results chart display for trends
* fix some issues with trends
* add in loading states and different insight types results wording
* move experiment results into loader
* add progress bar and clean up loading state
* fix experiment insight type bug
* add text beneath percent bar
* typing..
* single filter hide duplicate row button
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Don't delete events from Postgres since we don't do it on CH
* Update frontend for accuracy
* Update test_person.py
* Don't display "ID-less user" as a link
* Set time bounds for "all of time" filter
We won't display data points from before 2015 anymore, avoiding
confusion like in https://github.com/PostHog/posthog/issues/7626
* Disable dates before 2015, add tooltip
* Remove flipsort related logic
* Only load events from the last 6 months in events table
* Disable customization in actions tab
* Rename prop
* Dont poll events in actions page
* Show a different header above events table in actions page
* Update tests with frozen time
* Decouple months of actions tab with other months
* Calculate action count over 3 months
* Update tests
* Remove `minute` interval support from frontend
* Remove minute support from backend, default to hour instead
Also adds tests for interval logic
* Remove now-dead minute support from backend
* revert interval_candidate logic
* Move IntervalMixin to separate file
* Run queries against person_distinct_id2 when async migration is done
* Only write to clickhouse_person_unique_id topic if async migration is incomplete
* Update query snapshots
* Update plugin-server
* Adjust caching logic
* chore(pdi): add data migration for pdi to pdi2
This adds an async migration to copy the latest non-deleted
`(team_id, person_id, distinct_id)` tuples from `pdi` to `pdi2`.
Note that this has already be performed for tead_id = 2 on posthog,
cloud so we ensure we're maintaining parity with however this
migration was performed. I've done this by running:
```
SELECT * FROM <old_query>
FULL JOIN <new_query> new
ON old.distinct_id = new.distinct_id
WHERE old.person_id <> new.person_id
```
specifically for team_id = 2.
* Rename migration
* Skip 0003_fill_person_distinct_id2 on fresh installs
* Clarify version requirements
* Run async migrations using a while-loop instead of tail recursion
Python has a stack limit of 1000, which we might easily run into for
0003 migration
* Use built-in progress tracking
* Make description fit into database 400 char limit
* Add correctness test for new async migration
* Migrate person_distinct_id2 team-by-team
* Remove dead code
* Update migration notes
* Fix foss tests
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Keep person_distinct_id2 in sync on split_person
Follow-up to https://github.com/PostHog/posthog/pull/7671, covering a
new corner case.
* Deprecate some methods
* Handle versioning in split_person
* Select for update
* query events by date for properties known to be dates
* can't currently import journeys_for helper outside of ee folder
* actually remove the EE import
* even though no values of the specified property were not parseable account for unparseable values
* move tests into EE folder cos they're clickhouse only
* remove new line to take this file out of the PR diff
* use more forgiving ClickHouse datetime function
* wip
* add more tests and rename filters
* Adds a broken date picker behind a feature flag
* Revert "wip"
This reverts commit 6ec1da8744.
* add working basic UI for date selection in taxonomic property filter
* add a failing test by writing a unix timestamp the way the SDK does
* obey mypy
* add awareness of type and format so we can query our $time property as a DateTime in the events api
* hard code property definition in the eventsTableLogic
* Revert "obey mypy"
This reverts commit f09ae4369a.
* hold taxonomic property filter open - but with possible kludge in cohortsModel
* remove my edits to settings
* put back materialization of columns that test snapshots are expecting
* let the datepicker hold the taxonomic pop up open
* update snapshot
* simpler datepicker visibility in filter row
* simplify property type format for unix timestamps
* remove the datepicker now button
* Actually delete all of the drop down held open tests
* Remove leftover `console.log`s
Co-authored-by: Michael Matloka <dev@twixes.com>
* add property type and format and set them for
* add a failing test
* with passing tests on the ee property definition model
* Add a migration to set as a DateTime
* Clarify in failing test that it's only list lookup failing
* add empty property definition fields to assertions
* fix merge error
* clarify tests
* add more supported types
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* proof of concept
* remove old <PropertyFilter> component
* change import paths
* event properties sorted by event names
* get event names from actions if used
* scope event property filters by event names
* fix eslint
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* remove fields we no longer have
* add some instrumentation to figure out what services we can connect to (helps debug errors locally with kafka)
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* merge duplicate code
* use the right prop for event names
* mute unseen events, add info icon
* use the event-property tracker also for non-EE clients
* simplify duplicates
* add test for event properties
* add test for non-EE event properties
* add flag
* revert the label and unmute the text
* change the flag to UNSEEN_EVENT_PROPERTIES
* sort by query_usage_30_day again
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* use None if no event_names
* fix type
* improve pagination test
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* chore(lifecycle): add comments and CTEs to clickhouse sql
It was really big and confusing, but hopefully this clarifies a little
what is going on. As a followup PR I'll be doing some work to make the
query faster :fingerscrossed: but I think worth at least getting this
in, assuming I haven't broken any tests!
* update snapshots
* remove day references
* update snapshots
* fix(retention): add regression test for not_icontains filter
Previously if using the not_icontains filter, we were trying to double
substitute a query when fetching persons for retention. This adds a
test (and in the next commit a fix) such that we instead apply the
clickhouse substitute only once to each part of the query.
Resolves https://github.com/PostHog/posthog/issues/7747
* avoid double substitution
* Rework solution to not change ActorBaseQuery interface
* Remove unued import
* fix typing
* it works...
* clean up server side
* move logic from query to helpers/api
* typing fixes
* add tests to helpers
* test for metadata queries
* api tests
* util tests
* improve some types and move logic to utils
* Get PlayerMeta working
* tests for sessionRecordingPlayerLogic
* fix some more tests
* analytics fix
* name flip
* player util tests
* Add comments + remove logs
* Clean up some comments
* make is_active a bit cleaner
* remove some log statements
* add comment to metadata function
* some cleanup
* typescript fix
* timestamp plays follows scrubbing
* fix bug with hitting 60 seconds
* fix end position bug
* fix bugs with skip ahead
* fix some bugs
* fix white screen on buffer end
* fix buffer scrubbing bugs
* fix skip bug
* fix circular import
* rename start_window_id to last_active_window_id
* fix tests
* bug fix for no window_id case
* fix bug when scrubbing to the end of the player
* rename setBuffer + setScrub to startBuffer + startScrub
* fix(retention): fix breakdown people urls
This change returns people_url for each breakdown cohort in the
response. We also merge the initial and returning queries together,
as this makes it easier to align the people query also.
Note that I'm talking about person_id as opposed to actor_type etc.
but perhaps that can be a followup.
* clean up clickhouse params
* tidy up a little
* remove import
* remove non-breakdown specific code
* make cohort by initial event date a special breakdown case
* keep date for backwards compat
* Remove unused sql
* make test stable
* wip
* Get most of the tests working
* test(retention): remove graph retention test
We no longer need this, we have all the information we need from the
table response for retention, and can construct this on the frontend.
* revert any changes to posthog/queries/retention.py
* revert any changes to ee/clickhouse/models/person.py
* Revert posthog/queries/retention.py to merge-base
* Ensure actor id is a str
* Add type for actor serialiser for type narrowing
* run black
* sort imports
* Remove retention_actors.py
* fix typings
* format
* reverse str type
* sort imports
* rename
* split out functions
* remove deuplicate logic
* working
* fix type
* don't stringify
* fix test
* ordering doesn't matter
* trigger ci
Co-authored-by: eric <eeoneric@gmail.com>
* Make person hooks only happen in tests
* Remove a data script
* Update split_person code to handle hookless life
* Delete clickhouse data async in the background
* Make settings dynamic for MATERIALIZED_COLUMNS_ENABLED
* migrate MATERIALIZED_COLUMNS_ENABLED to this and test
* allow tests that use materialized columns to hit django model for settings
* nicer installed apps config
* update snapshot from test
* make test_property more deterministic
Co-authored-by: James Greenhill <fuzionech@gmail.com>
Co-authored-by: yakkomajuri <yakko.majuri@gmail.com>
* Resolve another insights N+1 issue
Using `prefetch_related` to avoid loading the same dashboard/user
multiple times if referred to by different insights. More queries _can_
be better! :)
* Improve the test
* Get test stable, add created_by user
* error if unsupported db version
* upgrade mypy
* fix various types for mypy
* you can have it in any color you want, as long as it's black
* fix mypy
* Update `kafka-python`
* Format with Black
* Fix mypy after merge
Co-authored-by: Michael Matloka <dev@twixes.com>
* Order by created_at over id in groups list, similar to persons
* Change table column name to be consistent with persons
* Default to `.name` property if one is passed for groups
This improves a usability issue with groups temporarily
* convert to actor form
* change var name
* remove unused imports
* typing issue
* use subquery
* bad import
* groups for general retention query
* actor in period
* update imports
* update test
* remove comment
* Add group analytics to list of paid features
* Add team-level property for if any group types exist
* WIP: Groups introduction page
* Finish initial empty state page
* Hide upsell if not CH or flag not live
* Extract upsell-related logic into a separate logic to avoid loading data unless needed
* Extract upgradeLink
* Extract announcement logic
* Overcomplicate announcements
* Show initial groups announcement.
* Vary banner content according to groups access status
* Clean up access logic
* Misc announcements improvement
* Add a feature flag for groups announcement given docs are not yet ready
* Keep color for visited button links
* Get groupsList logic tests working
teamLogic <-> userLogic test connection needed to be severed alongside
this
* Migration to add version to person_distinct_id
* Update plugin-server type
* Use queueMessages instead of for loops
* Update distinct id versions in postgres
* Add commented out new query
* Add person_distinct_id2 table setup/migration
This will be used for more efficient person_distinct_id queries
* Avoid sharding person_distinct_id2 on cloud
* Write to new distinct ids topic
* Attempt to use version in tests
* Tests attempt 2
* Fixup version - dont send with all messages
* Flush kafka more frequently
* Actually fix tests
* Add another await
* Add partition to person_distinct_id2 table
* Add a comment to keep topics in sync
* Clean up code relating to table engines
* Add snapshots for table creation queries
* Remove optional import
* Add snapshot tests for CLICKHOUSE_REPLICATION schemas
Note that these are out of sync with cloud in most cases
* Add another warning comment
* Improve naming
* Create a new way to get distinct id queries thats gated by team_id
* Update most cases to use the new query
* Convert EVENT_JOIN_PERSON_SQL to new query
* Mostly convert GET_DISTINCT_IDS_BY_PERSON_ID_FILTER
* Mostly convert GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Convert GET_PERSON_IDS_BY_FILTER
* Flag benchmarks
* Resolve circular imports
* Update a snapshot test
* Add a test for the new query logic
* Add logging to all postgresql queries with query context
Uses the exact same pattern as we do currently for clickhouse, just
hooking in there differently
* Support psycopg2.sql.SQL
* Better docs
* update a test
* test(retention): add http api tests for retention breakdowns
This just adds tests for person and event property breakdowns
* Add helper function for substituting clickhouse sql params
* feat(retention): add support for multiple breakdown props
This just reuses the work done for funnel multiple breakdown values. I
haven't tested this with anything other than person or event properties.
Rather than try to get it working for all the other property types.
The change adds a new `breakdowns` parameter to the retention endpoint,
that is the same as for funnels, e.g. it looks like:
```
{
...
"breakdowns": [{"type": "person", "property": "os"}, ...]
...
}
```
The return structure is the same as the non-breakdown version, except we
also include a `breakdown_values` property that is e.g. `["Chrome",
"95"]`, and the `label` attribute for this case would be "Chrome::95".
* fix typing
* update query snapshots
* remove unused imports
* use `self.assertEqual` instead of `assert`
* Remove explicit should_join_persons
* Revert limit by changes, use breakdown_type
* update snapshots
* fix typing
* use json_encode_request_params in retention test requests
* update json_encode_request_params to encode_get_request_params to better reflect purpose
* perf(retention): add benchmark query for retention with breakdown
* no materialize
* Use GroupTypeIndex type in BE
* Consistent ordering for group types
* Better BE type for group type index
* Handle non-url safe keys in groups
Previously this would crash due to never decoding the url and not
handling it in the backend
* Allow fetching related groups from the API
* Solve minor type errors
* Add a tooltip
* data-attr for tracking links
* Add related groups onto person page
* Fix an import
* Kill useEffect
This was pretty much as copy paste job from the stickiness cases. I'm
going to be merging in some changes to retention so I want to get this
in first such that we can see if it affects the perf of these queries at
all. I don't think it touches them, as seen by the snapshot queries
generated by the retention tests not changing in
https://github.com/PostHog/posthog/pull/7431 but better safe than sorry.
Addresses [this comment](https://github.com/PostHog/posthog/pull/7431#pullrequestreview-820773405)
Fixes
> ee/clickhouse/queries/paths/paths_actors.py:30: error: Incompatible types in assignment (expression has type "PathFilter", base class "ActorBaseQuery" defined the type as "Filter")
Caused by parallel merges & weird typing-related issues.
* query for events with matching session_id if possible
* move test to clickhouse
* move event list to list instead of set for flaky test snapshot
* join events only when needed
* trigger benchmarking
* typo
* refactor(stickiness): refactor one stickiness test to use api
This change demonstrates how to migrate a `Query` object level test to
an api based test. It purely focuses on the method of action invocation
and not on any of the e.g. setup or assertions. The StickinessQuery
object is only used by the REST API (and benchmarking), where as the
REST stickiness API is used by external users including our own frontend
developers, so makes sense to test at this level.
* Migrate stickiness query tests to api
This doesn't touch the stickiness people API however
* Migrate clickhouse specific stickiness tests
* Migrate stickiness people query tests to http api level
NOTE! This isn't just a straight migration, but also makes one important
change to application code that would otherwise result in a test
failure. Specifically, when trying to find an action based on the
`entity_id` query param, we need to consider that the entity_id is a
string. This is fine for when trying to find events, as we are comparing
event ids which are strings, but for actions the id is an int, so we
need to ensure we cast the action id to a string before comparison.
* Move stickiness query tests to api tests location
* make stickiness tests stable across postgres/clickhouse
* Add comment regarding casting action ids to strings
* Update wording
* Update view to use postgres tables
We keep these in sync now
* Add retrieve endpoint
* Link to a (blank) groups page
* Show events table and properties under group
* Update pagination for groups
* update testing utils
* Update tests
* Patch tests relying on returned models
* Remove duplicated line
* Extract common subquery into a variable
* BE: handle group properties in more cases
* Add tests for lifecycle and sessions query changes
* Better docs
* Stable date range
* wip
* normalise to breakdown and breakdowns on the way out of the API
* read from breakdowns or breakdown property of filters
* wip
* wip
* delete a console.log
* remove console.log and speculative multi breakdown changes
* remove empty test
* remove console log
* more tidying
* more tidying
* more tidying
* simplify funnel logic slightly
* a test for the data transformations with no breakdown
* can test data transformation with a single property breakdown
* keep expanding the test to see why multi prop doesn't display
* don't throw key error if there is no breakdown property to pop from the multi property API response
* funnel step table can detect breakdown without relying on filter
* try to highlight the differences in the paths through TaxonomicBreakdownFilter
* better behaviour when adding and removing multi property breakdowns
* displays multi property breakdown in old style steps funnel views
* it is safe to always show breakdown in new vertical layout
* resolve multi property breakdown typehinting
* updates comment
* updates comment
* don't print breakdown value twice
* tooltip for long values and prettier pill
* convert tests to journeys_for
* fix multi breakdown helper
* better behaviour when navigating between trends and funnels
* Refactor the multi property breakdown helper to calm the mypy type checker down
* safer access to dict in multi property helper
* makes clean filters feature flags aware and keeps breakdown values when switching between trends and funnels
* test fix
* remove overly verbose data tests
* add some more specific tests to multi prop breakdown
* obey mypy
* remove the breakdown type from the backend which was causing serialization issues
* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* add flag to funnel event query
* remove unnecessary logic
* typing
* working actors func
* fix typos
* remove unused func
* use serialized result
* wrong var
* use actors
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
* update typing
* clean up
* use actor func for all funnel persons
* types
* make sure limits are set
* remove functions
* fix limits
* funnel trends
* types
* indexing
* moved basic funnel tests
* unordered test
* remove duplicate test
* check ids
* more test adjustments
* sort
* change class names
* sort
* reformat
* change string to int
* remove duplicate
* update comment
* use actor result to paginate not serialized
* adjust tests after merging
* simplify typing
* don't import removed func
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* don't add leading slash to absolute url with no protocol
* don't add a base uri in only one funnel instantiation - will probably break tests
* remove console logs
* every API verb normalises URLs
* Tiny rename
* Minor refactor
* Test on FeatureFlagMatcher
* Add a field to feature_flag indicating if aggregating by groups
* Use f-strings
* Handle property filter building for group property
* Rename is_person_query
* Add message
* Tests for property_to_Q changes
* Add tests for groups w/ rollout
* Add comment for my_flags
* Hook up decide with groups
* New test helper for snapshotting pg queries
* Add snapshot tests for feature flags
* More is_simple_flag tests
* Add decide endpoint tests
* move conftest for ee tests to work
* Fix typo
* conftest troubles
* query for events with matching session_id if possible
* move test to clickhouse
* move event list to list instead of set for flaky test snapshot
* comment to clarify test
* original behaviour was to set breakdown_values regardless of if it was truthy
* add a test for the bug
* ensmallen the test case
* lift tested interface up a level
* implement multi property breakdown as an array from the spike
* correct type hint on method
* really resolve the conflict
* don't break groups
* refactor test assertions for breakdown cases
* adds a test to prove that funnels can receive a string and not an array
* protect saved dashboards from multi property changeover
* WIP
* multi breakdown working with funnel step breakdown
* prove funnel step person breakdown works with multi property breakdown
* don't need to protect cached dashboards from multi property breakdowns when they can't be set from the UI
* capitalise keywords in SQL
* convert a single test to journey helper
* wip
* account for funnel step breakdown sometimes being an array sent as a string
* safer handling of funnel step breakdown
* convert a test
* revert commits that made things worse
* simpler handling of funnel step breakdown
* no need to change funnel step breakdown type hint
* update imports
* guard against integer properties
* compare funnel step breakdown differently now there are arrays involved
* look for strict intersection for funnel step breakdown
* update test snapshots
* need to set breakdown_values earlier in processing
* remove tests that cover speculative functionality
* update snapshot
* move setup of breakdown values back out of update_filters
* update snapshots
* remove a sql parameter that was never assigned to
* Update ee/clickhouse/models/test/test_property.py
Co-authored-by: Harry Waye <harry@posthog.com>
* Update ee/clickhouse/queries/funnels/base.py
Co-authored-by: Harry Waye <harry@posthog.com>
* address review comment to simplify reading json expressions for breakdown
* clarify why some uses of get_property_string_expr escape params before passing
* add keyword arguments for calls to getting property string expressions in funnels
* switch to keyword arguments in test helper method
* fix parameterised test
* add multi property materialized column tests
* introduce the shim to allow new API for breakdown properties
* can't remove the naive funnel step breakdown list detection
* move funnel step breakdown list handling
* better handling of numeric funnel step breakdown values
* update snapshots
Co-authored-by: Harry Waye <harry@posthog.com>
* set debug, test, and primary_db env variables based on context
* remove unnecessary print
* also use pytest.ini for some tests run straight from pytest at the CLI
* Addtest for people_urls in retention trend response
* Add people_urls to retention trend response
* Run black formater
* sort imports
* Add default value for base_uri
* Let type be missing in EventPattern in tests
TypedDict is so rubbish in not allowing missing attributes 🤷
* fix typing
* refactor creation helper functions
* revert docker compose change
* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* remove unnecessary logic
* typing
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
I built this image a while ago, it's built with `yarn
arm64:build:clickhouse`, then I did a multipart build to copy only one
binary over and symlinking the various incantations (eg. server, client
etc) to keep the size down.
* WIP persons table schema migration
* add version to create_person
* test increased timeout
* Update ee/clickhouse/migrations/0019_person_table_version_column.py
* remove flaky kafka test
* format
* test a new approach - make this a non-large migration?
* Update ee/clickhouse/migrations/0021_person_table_version_column.py
* update
* Refactor stickiness to have its own event_query
This will speed up queries significantly and allow for filtering by
group properties
* Use same event_query for stickiness people
* Minor cleanup
* Add tests (and missing file) to group filtering in stickiness
* Allow aggregating by groups in stickiness
* Show group property filters in FE for stickiness
* convert the smallest of the funnel test files
* convert unorder funnel person test to journey for helper
* convert another file
* convert another file
* convert another file
* convert another file
* convert another file
* undelete snapshot files
* undelete snapshot files
* Revert "convert another file"
This reverts commit ef08511509.
* seperate clickhouse and postgres queries
* move to single query
* remove ununsed imports
* fix cohort + subquery issues
* some cleanup
* fix mypy issue
* fixes to query + tests
* add test for person and cohort properties
* remove cruft
* fix api tests
* Remove cohort + person prop filter from postgres
* move cohort and person property tests to clickhouse
* Filter down events
* typing fix
* fix test
* remove unneeded snapshot
* tests working with snapshots
* switch limit to property
* use materialized columns
* move to new distinct_id query
* add snapshots
* fix materialized column bug
* support cohort and person prop filters
* add person and cohort filtering
* remove cohort filtering from clickhouse
* small tweaks
* Lint fix
* remove cohort comment
* feat(funnels): update frontend to use new people_urls in response
This change updates the `FunnelBarGraph` and `FunnelStepTable`
components to use the new `converted_people_url` and
`dropped_poeple_url` in the funnels response.
I need to check that this covers all the cases for funnels. I've only
added people urls for funnels of type step for ordered/unordered/strict
and haven't yet touched the time bins and conversion time funnel
endpoint variants.
* add /some/people/url/ to tests
* chore: add action back, update tests to reference it
* fix typing
* wip
* get list of groups
* move backend endpoints
* frontend wip
* fix routes for groups
* add groups file
* try to fix tab nav
* fix groups query
* reformat results, add limits and offsets and next url
* move groups pages logic into own file
* fix and clean up tabs
* account for pagination and response reformatting
* lint fixes
* optional prev input
* typescript errors
* typescript fixes
* address feedback comments
* groups list logic test
* render properties expandable row
* fix test and lints
* prettier
* simplify table column title
* Fix a typing issue
* Use enum > boolean parameter
This makes what's going on more explicit
* Turn off pagination at the bottom of the table
* Rename table column
* Make pagination in groups work
Quite bugs fixed:
1. `previous` page was always shown, even if going into negative offsets
2. query_result could never be larger than `limit` with the code
* Cleanup code
* Make sidenav active under persons
* Use the correct query
* Add tests for pagination
* Update wording again
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* cumulative dau for non breakdown
* breakdown case
* clean up test
* add trends request util
* use simpler test util
* types
* insight tests
* add snapshots
* black
* adds a first draft of a test helper for clearer event setup in tests
* one spelling of journeys
* obey the type checker
* convert all of breakdown_cases to the new test helper
* Removes unused helper method
* test(funnel-people): add tests for converted/dropped people urls
As per https://github.com/PostHog/posthog/issues/6935 we are adding urls
to responses to attempt to improve consistency between insight responses
and the people we display in the UI.
This simply adds tests for these urls, just for funnel step response
shape. The other two types of response will be handled separately.
* feat(funnel-people): return converted/dropped people url in funnel
Here I have simply added converted / dropped urls in the base funnel.
This touches both clickhouse and postgres functionality, so I will
probably add in tests for postgres also, or just disable to postgres.
It doesn't pass yet as there is an issue in that the order appears to be
ignored by the people endpoint.
* fix(funnels): actually use strict/unordered persons querying
Previously we were always using the standard `ClickhouseFunnelPersons`
class for retrieving people from the `/api/persons/funnel/` endpoint.
This change selects from the unordered and strict variants based on the
`funnel_order_type` setting.
Refers to https://github.com/PostHog/posthog/issues/7058 although there
is a frontend component to add that will truely resolve the issue.
* feat(funnel-people): add people urls for funnels with breakdown
In this I have also removed any changes from the non-clickhouse code.
Note that there are two places I've added the url generation code, one
in the `ClickhouseFunnelBase` and another in the `ClickhouseFunnel`
I think the former covers the unordered and strict cases, and the later
for the breakdown case. I think there are further test that I'll need to
add to validate e.g. if breakdown + strict work as expected.
* Fix people urls for unordered funnels + breakdown
* test: add test for strict breakdown with people urls
* make funnel response assertions not check people url equality
* fix typing
* fix tests
* remove new line in postgres funnel.py
* clear cache on insight test start
* no really, clear the cache
* remove flakiness from strict funnel test
* correct the unordered test
* use absolute uris
* use step.index not step.order
* remove out of date comment
* use step.index, not step.order
* use step.index, remove unordered funnels comment
* use journeys_for instead of create_events
* add test_helper_methods
* move all the tests where the properties are the same for all events to the journey helper
* compare funnel results without caring about person order
* spell words correcterly
* Revert "spell words correcterly"
This reverts commit befb83b183.
* Revert "compare funnel results without caring about person order"
This reverts commit 268927b8ba.
* correct types for test props
