* add recordings to path query
* uncomment cache
* add clarifying comment
* works for start/end paths
* move to extra fields/properties
* add tests
* cleanup
* update ff name
* fix flaky test
* test and handle path_dropoff_key case
* stores numeric type for property definitions
* writes property definition type for strings
* can write dates for simple string cases as property definition type
* simplify property definition types
* can detect unix timestamps
* rewmove a little more from the test duplication
* some more timestamp thinking
* clearer comment
* rename refactor in tests
* more test correction
* address test failure
* another test fix
* property type as enum
* simplify type detection
* apply typing suggestion
* allow property definition type and format to be set if property is present but they are null
* add more date types for property_type_formats
* add more datetime formats that the backend can accept and filter events with
* delete conflicting migration
* add migration for backend datetime formats
* plugin server can auto detect all the types the backend knows about. And the tests are collapsed from twenty five almost identical tests to two different parameterised blocks
* iso 8601 strings can have fractional seconds
* add link to the RFC 822 regex pattern
* capitalise SQL
* capitalise the SQL the mock looks for too
* remove the under-eager cache change that allowed updating existing property definitions
* convert property type formats to enum
* fix test
* try detecting a property type for existing property definitions but don't keep retrying
* use symbols to make cache state clearer
* fix idiocy
* speculative LRU cache usage with measurement
* tag the statsd guages instead of having team id in the name
* don't pre-emptively optimise the cache
* rename migration
* remove spaces from query strings
* separate property definition code from the team manager
* a little tidying
* correct type definition
* correct test params setup
* wip
* display known unix timestamps as date strings in property filter select box
* show selected unix timestamp property as a datetime string in the property filter
* remove console.log
* update test to construct property definition correctly
* correct another test
* dates have different operators
* format property filter values for display in Select boxes and the PropertyValue component
* fix typescript error
* it _is_ a type, I don't need to say it is
* don't have before and after for unmatched types of operators
* separate the current applicable operators and the operator to display mapping, and add cypress tests for before and after operator visibility
* correct cypress test
* remove unused method
* remove dangling comment
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* log values of operator options, why does this fail in CI but not locally
* why does test fail in CI but not locally
* intercept more decide calls to try and set feature flags from cypress test
* why does test fail in CI but not locally
* Remove cypress install
* add a readme for how to test feature flags so it isn't lost if there are no tests currently testing feature flags
* refactor(lifecycle): simplify clickhouse sql logic
This updates the SQL to be comprised of two queries, one for getting
new, returning, and resurrecting periods of activity, one for getting
dormant periods right after periods of activity.
Refers to https://github.com/PostHog/posthog/issues/7382
* refactor(lifecyle): use `ClickhouseEventQuery` to build event query
* format
* Use bounded_person_activity_by_period for both sides of dormant join
* refactor(lifecycle): reduce pdi2 join by one
This means we're now under the current query memory limit for orgs with
around 20m distinct_ids. It does remove some readability though :(
* update snapshot
* Add further comments to query
* Add further comments to query
* Add further comments to query
* Remove dead variables
* Refactor person_query overriding
* Lifecycle refactoring continued
* Update lifecycle tests (except people ones)
* Make lifecycle people endpoint happy
* Remove django lifecycle tests
* Add some edge case tests
* Add missing type
Co-authored-by: Harry Waye <harry@posthog.com>
* Resolve weird SQL formatting issue
* Use a discriminated union for ActorType
* Use standard response types for related groups
* Update typing
* Always filter related actors by group type index
* Update snapshots & typing
* Migration to add name_singular and name_plural to groups
* Allow modifying group type names
* Use name_plural and name_singular through the app
* Update tab names
* fixup
* Update snapshots
* add trends option to experiment
(cherry picked from commit be9b8911f9)
* connect trends experiments
* clean up some stuff
* fix test
* results chart display for trends
* fix some issues with trends
* add in loading states and different insight types results wording
* move experiment results into loader
* add progress bar and clean up loading state
* fix experiment insight type bug
* add text beneath percent bar
* typing..
* single filter hide duplicate row button
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Don't delete events from Postgres since we don't do it on CH
* Update frontend for accuracy
* Update test_person.py
* Don't display "ID-less user" as a link
* Set time bounds for "all of time" filter
We won't display data points from before 2015 anymore, avoiding
confusion like in https://github.com/PostHog/posthog/issues/7626
* Disable dates before 2015, add tooltip
* Remove flipsort related logic
* Only load events from the last 6 months in events table
* Disable customization in actions tab
* Rename prop
* Dont poll events in actions page
* Show a different header above events table in actions page
* Update tests with frozen time
* Decouple months of actions tab with other months
* Calculate action count over 3 months
* Update tests
* Remove `minute` interval support from frontend
* Remove minute support from backend, default to hour instead
Also adds tests for interval logic
* Remove now-dead minute support from backend
* revert interval_candidate logic
* Move IntervalMixin to separate file
* Run queries against person_distinct_id2 when async migration is done
* Only write to clickhouse_person_unique_id topic if async migration is incomplete
* Update query snapshots
* Update plugin-server
* Adjust caching logic
* chore(pdi): add data migration for pdi to pdi2
This adds an async migration to copy the latest non-deleted
`(team_id, person_id, distinct_id)` tuples from `pdi` to `pdi2`.
Note that this has already be performed for tead_id = 2 on posthog,
cloud so we ensure we're maintaining parity with however this
migration was performed. I've done this by running:
```
SELECT * FROM <old_query>
FULL JOIN <new_query> new
ON old.distinct_id = new.distinct_id
WHERE old.person_id <> new.person_id
```
specifically for team_id = 2.
* Rename migration
* Skip 0003_fill_person_distinct_id2 on fresh installs
* Clarify version requirements
* Run async migrations using a while-loop instead of tail recursion
Python has a stack limit of 1000, which we might easily run into for
0003 migration
* Use built-in progress tracking
* Make description fit into database 400 char limit
* Add correctness test for new async migration
* Migrate person_distinct_id2 team-by-team
* Remove dead code
* Update migration notes
* Fix foss tests
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Keep person_distinct_id2 in sync on split_person
Follow-up to https://github.com/PostHog/posthog/pull/7671, covering a
new corner case.
* Deprecate some methods
* Handle versioning in split_person
* Select for update
* query events by date for properties known to be dates
* can't currently import journeys_for helper outside of ee folder
* actually remove the EE import
* even though no values of the specified property were not parseable account for unparseable values
* move tests into EE folder cos they're clickhouse only
* remove new line to take this file out of the PR diff
* use more forgiving ClickHouse datetime function
* wip
* add more tests and rename filters
* Adds a broken date picker behind a feature flag
* Revert "wip"
This reverts commit 6ec1da8744.
* add working basic UI for date selection in taxonomic property filter
* add a failing test by writing a unix timestamp the way the SDK does
* obey mypy
* add awareness of type and format so we can query our $time property as a DateTime in the events api
* hard code property definition in the eventsTableLogic
* Revert "obey mypy"
This reverts commit f09ae4369a.
* hold taxonomic property filter open - but with possible kludge in cohortsModel
* remove my edits to settings
* put back materialization of columns that test snapshots are expecting
* let the datepicker hold the taxonomic pop up open
* update snapshot
* simpler datepicker visibility in filter row
* simplify property type format for unix timestamps
* remove the datepicker now button
* Actually delete all of the drop down held open tests
* Remove leftover `console.log`s
Co-authored-by: Michael Matloka <dev@twixes.com>
* add property type and format and set them for
* add a failing test
* with passing tests on the ee property definition model
* Add a migration to set as a DateTime
* Clarify in failing test that it's only list lookup failing
* add empty property definition fields to assertions
* fix merge error
* clarify tests
* add more supported types
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* proof of concept
* remove old <PropertyFilter> component
* change import paths
* event properties sorted by event names
* get event names from actions if used
* scope event property filters by event names
* fix eslint
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* remove fields we no longer have
* add some instrumentation to figure out what services we can connect to (helps debug errors locally with kafka)
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* merge duplicate code
* use the right prop for event names
* mute unseen events, add info icon
* use the event-property tracker also for non-EE clients
* simplify duplicates
* add test for event properties
* add test for non-EE event properties
* add flag
* revert the label and unmute the text
* change the flag to UNSEEN_EVENT_PROPERTIES
* sort by query_usage_30_day again
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* use None if no event_names
* fix type
* improve pagination test
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* chore(lifecycle): add comments and CTEs to clickhouse sql
It was really big and confusing, but hopefully this clarifies a little
what is going on. As a followup PR I'll be doing some work to make the
query faster :fingerscrossed: but I think worth at least getting this
in, assuming I haven't broken any tests!
* update snapshots
* remove day references
* update snapshots
* fix(retention): add regression test for not_icontains filter
Previously if using the not_icontains filter, we were trying to double
substitute a query when fetching persons for retention. This adds a
test (and in the next commit a fix) such that we instead apply the
clickhouse substitute only once to each part of the query.
Resolves https://github.com/PostHog/posthog/issues/7747
* avoid double substitution
* Rework solution to not change ActorBaseQuery interface
* Remove unued import
* fix typing
* it works...
* clean up server side
* move logic from query to helpers/api
* typing fixes
* add tests to helpers
* test for metadata queries
* api tests
* util tests
* improve some types and move logic to utils
* Get PlayerMeta working
* tests for sessionRecordingPlayerLogic
* fix some more tests
* analytics fix
* name flip
* player util tests
* Add comments + remove logs
* Clean up some comments
* make is_active a bit cleaner
* remove some log statements
* add comment to metadata function
* some cleanup
* typescript fix
* timestamp plays follows scrubbing
* fix bug with hitting 60 seconds
* fix end position bug
* fix bugs with skip ahead
* fix some bugs
* fix white screen on buffer end
* fix buffer scrubbing bugs
* fix skip bug
* fix circular import
* rename start_window_id to last_active_window_id
* fix tests
* bug fix for no window_id case
* fix bug when scrubbing to the end of the player
* rename setBuffer + setScrub to startBuffer + startScrub
* fix(retention): fix breakdown people urls
This change returns people_url for each breakdown cohort in the
response. We also merge the initial and returning queries together,
as this makes it easier to align the people query also.
Note that I'm talking about person_id as opposed to actor_type etc.
but perhaps that can be a followup.
* clean up clickhouse params
* tidy up a little
* remove import
* remove non-breakdown specific code
* make cohort by initial event date a special breakdown case
* keep date for backwards compat
* Remove unused sql
* make test stable
* wip
* Get most of the tests working
* test(retention): remove graph retention test
We no longer need this, we have all the information we need from the
table response for retention, and can construct this on the frontend.
* revert any changes to posthog/queries/retention.py
* revert any changes to ee/clickhouse/models/person.py
* Revert posthog/queries/retention.py to merge-base
* Ensure actor id is a str
* Add type for actor serialiser for type narrowing
* run black
* sort imports
* Remove retention_actors.py
* fix typings
* format
* reverse str type
* sort imports
* rename
* split out functions
* remove deuplicate logic
* working
* fix type
* don't stringify
* fix test
* ordering doesn't matter
* trigger ci
Co-authored-by: eric <eeoneric@gmail.com>
* Make person hooks only happen in tests
* Remove a data script
* Update split_person code to handle hookless life
* Delete clickhouse data async in the background
* Make settings dynamic for MATERIALIZED_COLUMNS_ENABLED
* migrate MATERIALIZED_COLUMNS_ENABLED to this and test
* allow tests that use materialized columns to hit django model for settings
* nicer installed apps config
* update snapshot from test
* make test_property more deterministic
Co-authored-by: James Greenhill <fuzionech@gmail.com>
Co-authored-by: yakkomajuri <yakko.majuri@gmail.com>
* Resolve another insights N+1 issue
Using `prefetch_related` to avoid loading the same dashboard/user
multiple times if referred to by different insights. More queries _can_
be better! :)
* Improve the test
* Get test stable, add created_by user
* error if unsupported db version
* upgrade mypy
* fix various types for mypy
* you can have it in any color you want, as long as it's black
* fix mypy
* Update `kafka-python`
* Format with Black
* Fix mypy after merge
Co-authored-by: Michael Matloka <dev@twixes.com>
* Order by created_at over id in groups list, similar to persons
* Change table column name to be consistent with persons
* Default to `.name` property if one is passed for groups
This improves a usability issue with groups temporarily
* convert to actor form
* change var name
* remove unused imports
* typing issue
* use subquery
* bad import
* groups for general retention query
* actor in period
* update imports
* update test
* remove comment
* Add group analytics to list of paid features
* Add team-level property for if any group types exist
* WIP: Groups introduction page
* Finish initial empty state page
* Hide upsell if not CH or flag not live
* Extract upsell-related logic into a separate logic to avoid loading data unless needed
* Extract upgradeLink
* Extract announcement logic
* Overcomplicate announcements
* Show initial groups announcement.
* Vary banner content according to groups access status
* Clean up access logic
* Misc announcements improvement
* Add a feature flag for groups announcement given docs are not yet ready
* Keep color for visited button links
* Get groupsList logic tests working
teamLogic <-> userLogic test connection needed to be severed alongside
this
* Migration to add version to person_distinct_id
* Update plugin-server type
* Use queueMessages instead of for loops
* Update distinct id versions in postgres
* Add commented out new query
* Add person_distinct_id2 table setup/migration
This will be used for more efficient person_distinct_id queries
* Avoid sharding person_distinct_id2 on cloud
* Write to new distinct ids topic
* Attempt to use version in tests
* Tests attempt 2
* Fixup version - dont send with all messages
* Flush kafka more frequently
* Actually fix tests
* Add another await
* Add partition to person_distinct_id2 table
* Add a comment to keep topics in sync
* Clean up code relating to table engines
* Add snapshots for table creation queries
* Remove optional import
* Add snapshot tests for CLICKHOUSE_REPLICATION schemas
Note that these are out of sync with cloud in most cases
* Add another warning comment
* Improve naming
* Create a new way to get distinct id queries thats gated by team_id
* Update most cases to use the new query
* Convert EVENT_JOIN_PERSON_SQL to new query
* Mostly convert GET_DISTINCT_IDS_BY_PERSON_ID_FILTER
* Mostly convert GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Convert GET_PERSON_IDS_BY_FILTER
* Flag benchmarks
* Resolve circular imports
* Update a snapshot test
* Add a test for the new query logic
