* improve tooltip response, add FF at the end
* add experiments test
* add confusing tooltip
* remove FF banner for now
* Update frontend/src/scenes/experiments/Experiment.tsx
* Update frontend/src/scenes/experiments/experimentLogic.tsx
Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com>
* feat(k8s): add kafka connection status to health check endpoint
This change adds a kafka check to the existing health check that uses
the underlying kafka python libs `bootstrap_connected` to check that we
are connected to kafka.
To accommodate the extra check in the response, I have updated to return
a JSON response with a lookup of `"check_name"` to it's status. We
return if any of the checks return `False`.
Something I didn't do was allow for checking each check in isolation,
e.g. we could expose for instance just the kafka_connected check at
`/_health/kafka`
* sort imports
* Remove unused requests arg, maybe useful later but not now.
* Add readyz and livez endpoint
* Add some docs
* link to k8s
* be specific about postgres
* some tests don't need postgres
* kubernetes -> process orchestration system
* update how we check kafka connection
* remove return
* add description to action
* update tests
* no limit to description and use pageheader
* add ee_action
* add description as global relationship:
* fix tests
* revert to using descriptions
* revert to using descriptions
* allow description to be null
* fix tests
* persist mode
* fix e2e
* Address review points, clean up some margins
* Clean up the way editing state is determined
* Restore default description value and improve typing
* Fix typing better
* Update sql.ts
Co-authored-by: Michael Matloka <dev@twixes.com>
* perf(events): limit select clause when getting ingest requirements
Previously we were pulling in all table columns for every event posted
to the `/e/` column. We actually only need the id, such that we can
associate an event with a `Team`, and a setting which says if we should
anonymize ip addresses.
There are still other optimizations to make, and some of the code paths
do several queries to the database, but I have limited the changes here
to just the most straight forward of them.
Refers to https://github.com/PostHog/posthog/issues/8378
* Add a little comment
* Filter by team_id
* Avoid changing error messages
* Update import get_team -> get_ingest_context
* Fix typing
* fix types
* fix mocking
* fix type
* Rename InjestContext -> EventIngestContext
* Update ingestion context fetch methods
* Further rename fetch functions
* add test for async migrations not being required on fresh installs
* fix
* fix testing
* Update .flake8
* fix test?
* run tests
* fix test for good
* fixes
* attempt
* remove ee tag
* system status number formatting
* Update ee/clickhouse/system_status.py
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* simplifications
* simplify more
* simplify
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* allow datepicker in date only mode
* allow exact date matching in property filters
* marks is_date_after as a date operator
* the date picker needs to work with null/undefined as well as string
* remove unused relative date parsing and fix date only is_date_exact property filters
* default to dates without times matching exactly
* make datetime filters by second work on date values that include milliseconds
* fix TS error
* correct insight url cypress test
* replace useMemo with useEffect
* be more specific in regex date match
* spacing under the quick choices
* when checking after a date for a date only filter check against that day at 23:59:59
* remove redundant text
* add test case and fix is_date_after query
* Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env var
This makes integrating with external clickhouse providers (like altinity
cloud) much easier as users don't need to use undocumented APIs to
upload the schemas.
* Keep underscores in protobuf. This allows using this payload for json ingestion
* Timestamp format
* remove property_type_format concept
* remove nulls from tests
* deprecate not remove (and see what tests fail)
* plugin server tests care about property_type_format
* fix tests
* Allow overriding kafka host for clickhouse via KAFKA_URL_CLICKHOUSE env var
This is needed when using an external clickhouse which doesn't have the
same access to kafka as in-cluster traffic does.
Note that long-term we might need to also provide better auth mechanisms
here as well.
* Rename env variable
* wip
* wip
* wip
* can filter events by reserved words - distinct_id, created_at, and timestamp - in the backend
* add reserved properties when showing property definitions
* capitalise name of property
* properties not reserved words
* get cypress working and make search work with reserved properties
* update event/values endpoint to use property_string_expr and so work with reserved properties
* remove created at from reserved words
* test fix
* fix test
* loosen cypress assertion
* exclude specific properties from the taxonomic property filter
* remove FE property definition fangling
* exclude $time and $timestamp property definitions from the backend
* add reserved properties in the API not the FE using a CTE
* use SQL comments in SQL strings
* remove mypy error
* fix enterprise property definitions
* fix tests and improve comment
* exclude materialised column rows with no values when getting event property definitions
* fix cypress test setup
* clean up dead code
* un-remove some not actually dead code
* correct cypress assertion
* try and fix a cypress test
* Don't allow user to set current team that they don't have access to
* Warn when dashboard is available in different project
* update exceptionshog
* fix tests
* fix failing
* fix typing
* tests
* remove insights and move tests
* update snapshots
* stickiness snapshots
* bring back ee available for ee specific features
* remove endpoint from posthog insightviewset
* fix snapshot
* remove repeat
* switch backend datetime filtering so it does not need property types and formats
* stop populating and using the property_type_format field (without yet removing it)
* add recordings to path query
* uncomment cache
* add clarifying comment
* works for start/end paths
* move to extra fields/properties
* add tests
* cleanup
* update ff name
* fix flaky test
* test and handle path_dropoff_key case
* stores numeric type for property definitions
* writes property definition type for strings
* can write dates for simple string cases as property definition type
* simplify property definition types
* can detect unix timestamps
* rewmove a little more from the test duplication
* some more timestamp thinking
* clearer comment
* rename refactor in tests
* more test correction
* address test failure
* another test fix
* property type as enum
* simplify type detection
* apply typing suggestion
* allow property definition type and format to be set if property is present but they are null
* add more date types for property_type_formats
* add more datetime formats that the backend can accept and filter events with
* delete conflicting migration
* add migration for backend datetime formats
* plugin server can auto detect all the types the backend knows about. And the tests are collapsed from twenty five almost identical tests to two different parameterised blocks
* iso 8601 strings can have fractional seconds
* add link to the RFC 822 regex pattern
* capitalise SQL
* capitalise the SQL the mock looks for too
* remove the under-eager cache change that allowed updating existing property definitions
* convert property type formats to enum
* fix test
* try detecting a property type for existing property definitions but don't keep retrying
* use symbols to make cache state clearer
* fix idiocy
* speculative LRU cache usage with measurement
* tag the statsd guages instead of having team id in the name
* don't pre-emptively optimise the cache
* rename migration
* remove spaces from query strings
* separate property definition code from the team manager
* a little tidying
* correct type definition
* correct test params setup
* wip
* display known unix timestamps as date strings in property filter select box
* show selected unix timestamp property as a datetime string in the property filter
* remove console.log
* update test to construct property definition correctly
* correct another test
* dates have different operators
* format property filter values for display in Select boxes and the PropertyValue component
* fix typescript error
* it _is_ a type, I don't need to say it is
* don't have before and after for unmatched types of operators
* separate the current applicable operators and the operator to display mapping, and add cypress tests for before and after operator visibility
* correct cypress test
* remove unused method
* remove dangling comment
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* why does test fail in CI but not locally
* log values of operator options, why does this fail in CI but not locally
* why does test fail in CI but not locally
* intercept more decide calls to try and set feature flags from cypress test
* why does test fail in CI but not locally
* Remove cypress install
* add a readme for how to test feature flags so it isn't lost if there are no tests currently testing feature flags
* refactor(lifecycle): simplify clickhouse sql logic
This updates the SQL to be comprised of two queries, one for getting
new, returning, and resurrecting periods of activity, one for getting
dormant periods right after periods of activity.
Refers to https://github.com/PostHog/posthog/issues/7382
* refactor(lifecyle): use `ClickhouseEventQuery` to build event query
* format
* Use bounded_person_activity_by_period for both sides of dormant join
* refactor(lifecycle): reduce pdi2 join by one
This means we're now under the current query memory limit for orgs with
around 20m distinct_ids. It does remove some readability though :(
* update snapshot
* Add further comments to query
* Add further comments to query
* Add further comments to query
* Remove dead variables
* Refactor person_query overriding
* Lifecycle refactoring continued
* Update lifecycle tests (except people ones)
* Make lifecycle people endpoint happy
* Remove django lifecycle tests
* Add some edge case tests
* Add missing type
Co-authored-by: Harry Waye <harry@posthog.com>
* Resolve weird SQL formatting issue
* Use a discriminated union for ActorType
* Use standard response types for related groups
* Update typing
* Always filter related actors by group type index
* Update snapshots & typing
* Migration to add name_singular and name_plural to groups
* Allow modifying group type names
* Use name_plural and name_singular through the app
* Update tab names
* fixup
* Update snapshots
* add trends option to experiment
(cherry picked from commit be9b8911f9)
* connect trends experiments
* clean up some stuff
* fix test
* results chart display for trends
* fix some issues with trends
* add in loading states and different insight types results wording
* move experiment results into loader
* add progress bar and clean up loading state
* fix experiment insight type bug
* add text beneath percent bar
* typing..
* single filter hide duplicate row button
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Don't delete events from Postgres since we don't do it on CH
* Update frontend for accuracy
* Update test_person.py
* Don't display "ID-less user" as a link
* Set time bounds for "all of time" filter
We won't display data points from before 2015 anymore, avoiding
confusion like in https://github.com/PostHog/posthog/issues/7626
* Disable dates before 2015, add tooltip
* Remove flipsort related logic
* Only load events from the last 6 months in events table
* Disable customization in actions tab
* Rename prop
* Dont poll events in actions page
* Show a different header above events table in actions page
* Update tests with frozen time
* Decouple months of actions tab with other months
* Calculate action count over 3 months
* Update tests
* Remove `minute` interval support from frontend
* Remove minute support from backend, default to hour instead
Also adds tests for interval logic
* Remove now-dead minute support from backend
* revert interval_candidate logic
* Move IntervalMixin to separate file
* Run queries against person_distinct_id2 when async migration is done
* Only write to clickhouse_person_unique_id topic if async migration is incomplete
* Update query snapshots
* Update plugin-server
* Adjust caching logic
* chore(pdi): add data migration for pdi to pdi2
This adds an async migration to copy the latest non-deleted
`(team_id, person_id, distinct_id)` tuples from `pdi` to `pdi2`.
Note that this has already be performed for tead_id = 2 on posthog,
cloud so we ensure we're maintaining parity with however this
migration was performed. I've done this by running:
```
SELECT * FROM <old_query>
FULL JOIN <new_query> new
ON old.distinct_id = new.distinct_id
WHERE old.person_id <> new.person_id
```
specifically for team_id = 2.
* Rename migration
* Skip 0003_fill_person_distinct_id2 on fresh installs
* Clarify version requirements
* Run async migrations using a while-loop instead of tail recursion
Python has a stack limit of 1000, which we might easily run into for
0003 migration
* Use built-in progress tracking
* Make description fit into database 400 char limit
* Add correctness test for new async migration
* Migrate person_distinct_id2 team-by-team
* Remove dead code
* Update migration notes
* Fix foss tests
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Keep person_distinct_id2 in sync on split_person
Follow-up to https://github.com/PostHog/posthog/pull/7671, covering a
new corner case.
* Deprecate some methods
* Handle versioning in split_person
* Select for update
* query events by date for properties known to be dates
* can't currently import journeys_for helper outside of ee folder
* actually remove the EE import
* even though no values of the specified property were not parseable account for unparseable values
* move tests into EE folder cos they're clickhouse only
* remove new line to take this file out of the PR diff
* use more forgiving ClickHouse datetime function
* wip
* add more tests and rename filters
* Adds a broken date picker behind a feature flag
* Revert "wip"
This reverts commit 6ec1da8744.
* add working basic UI for date selection in taxonomic property filter
* add a failing test by writing a unix timestamp the way the SDK does
* obey mypy
* add awareness of type and format so we can query our $time property as a DateTime in the events api
* hard code property definition in the eventsTableLogic
* Revert "obey mypy"
This reverts commit f09ae4369a.
* hold taxonomic property filter open - but with possible kludge in cohortsModel
* remove my edits to settings
* put back materialization of columns that test snapshots are expecting
* let the datepicker hold the taxonomic pop up open
* update snapshot
* simpler datepicker visibility in filter row
* simplify property type format for unix timestamps
* remove the datepicker now button
* Actually delete all of the drop down held open tests
* Remove leftover `console.log`s
Co-authored-by: Michael Matloka <dev@twixes.com>
* add property type and format and set them for
* add a failing test
* with passing tests on the ee property definition model
* Add a migration to set as a DateTime
* Clarify in failing test that it's only list lookup failing
* add empty property definition fields to assertions
* fix merge error
* clarify tests
* add more supported types
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* proof of concept
* remove old <PropertyFilter> component
* change import paths
* event properties sorted by event names
* get event names from actions if used
* scope event property filters by event names
* fix eslint
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* remove fields we no longer have
* add some instrumentation to figure out what services we can connect to (helps debug errors locally with kafka)
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* merge duplicate code
* use the right prop for event names
* mute unseen events, add info icon
* use the event-property tracker also for non-EE clients
* simplify duplicates
* add test for event properties
* add test for non-EE event properties
* add flag
* revert the label and unmute the text
* change the flag to UNSEEN_EVENT_PROPERTIES
* sort by query_usage_30_day again
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* use None if no event_names
* fix type
* improve pagination test
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* chore(lifecycle): add comments and CTEs to clickhouse sql
It was really big and confusing, but hopefully this clarifies a little
what is going on. As a followup PR I'll be doing some work to make the
query faster :fingerscrossed: but I think worth at least getting this
in, assuming I haven't broken any tests!
* update snapshots
* remove day references
* update snapshots
* fix(retention): add regression test for not_icontains filter
Previously if using the not_icontains filter, we were trying to double
substitute a query when fetching persons for retention. This adds a
test (and in the next commit a fix) such that we instead apply the
clickhouse substitute only once to each part of the query.
Resolves https://github.com/PostHog/posthog/issues/7747
* avoid double substitution
* Rework solution to not change ActorBaseQuery interface
* Remove unued import
* fix typing
