* fill postgres only if cohort is used in feature flag
* update relevant cohorts when feature flag is updated or created
* move calls
* call count test ff
* add ff test
* snapshots
* update types
* get person cohorts
* api all working
* fix tests
* typing and frontend
* capitalize constant
* perf(cohorts): reduce select batch size, and sleep 1 between
Previously we were hammering the database, see
https://github.com/PostHog/product-internal/issues/256 for details. Now
we:
1. reduce the batch by 5 times
2. have a little sleep(1) between batches
We also add a little bit of logging for instance in failure cases.
* include stack trace info in warning
* just reraise so we don't lose stack trace
* use monotonic for timings, not time
* update snapshots
* add test
* update func
* add include and settings
* diff method
* some prelim changes to make backend work
* add tests and logic for handling groups
* setup basic mixin parsing
* add tests
* types and switchover
* adjustment
* update snapshots
* update more snapshots
* update more snapshots
* update more snapshots
* typing
* snapshots
* clean up TODOs and types
* separate to_dict
* another property dict fix
* use operator dependening on condition
* fix tests
* update tests
* remove dummy condition
* update tests
* change name to groups
* another label
* rename filter_group to property_group
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* feat(healthchecks): add clickhouse as a dependency
* feat(healthchecks): add celery broker healthcheck
* feat(healthchecks): add cache healthcheck, plus some misc updates
* add missing typing
* add missing typing
* add role cast
* Add logging for kafka connectivity check
* actually add healthcheck as middleware
* fix postgres failure check
* simplify postgres migration return value
* verboise -> verbose
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* put interface validation at top of view
* remove unused imports
* show resp.content on status failures
* test(healthchecks): set log level to debug to pytest displays it
* restore logger level
* Bring up redis for backend ci tests
* up redis in non-cloud tests
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* test(snapshots): update clickhouse snapshot comments
It looks like Clickhouse has been removed from clickhouse sql comments,
but the snapshots haven't been updated. Not sure how this got past CI.
* missed a couple
* version insights
* version and lock update
* make sure all tests work
* restore exception
* fix test
* fix test
* add specific id
* update plugin server test utils
* cleanup
* match filtering
* use timestamp comparison
* make tests work
* one more test field
* fix more tests
* more cleanup
* lock frontend when updating and restore refresh
* pass undefined
* add timestamp to background update
* use incrementer
* add field
* snapshot
* types
* more cleanup
* update tests
* remove crumbs
* use expressions
* make nullable
* batch delete
* fill null for static cohorts
* batch_delete
* typing
* remove queryset function
* fetch person properties via clickhouse
* cleaner queries
* fix some types
* parse up to 100k persons with this prop, add back counts
* ignore something that's bound to error anyway
* no point making it too flat now
* revert flattening of lists
* add statsd
* ignore type
* add back conversion
* mypy
* better mypy
* refactor
* add support for Experiment groups
* update styling
* incl groups in property selection
* set participant type and respect FF group type on backend
* remove aggregation select
* Revert "Revert "Adds reserved properties for filtering property definitions (#8291)" (#8340)"
This reverts commit cb6839fe21.
* limit reserved properties to events table, include this in unit tests
* add api level tests for datetime and reserved word filtering of trends
* remove unused constant
* make the cypress test even looser
* delete file that is no longer in master
* extract a query object
* implement reserved attribute querying for date filters
* update datetime cypress test
* separate and clarify tests and ensure date queries only use reserved attributes on event filters
* delete file that isn't in master
* improve tooltip response, add FF at the end
* add experiments test
* add confusing tooltip
* remove FF banner for now
* Update frontend/src/scenes/experiments/Experiment.tsx
* Update frontend/src/scenes/experiments/experimentLogic.tsx
Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com>
* feat(k8s): add kafka connection status to health check endpoint
This change adds a kafka check to the existing health check that uses
the underlying kafka python libs `bootstrap_connected` to check that we
are connected to kafka.
To accommodate the extra check in the response, I have updated to return
a JSON response with a lookup of `"check_name"` to it's status. We
return if any of the checks return `False`.
Something I didn't do was allow for checking each check in isolation,
e.g. we could expose for instance just the kafka_connected check at
`/_health/kafka`
* sort imports
* Remove unused requests arg, maybe useful later but not now.
* Add readyz and livez endpoint
* Add some docs
* link to k8s
* be specific about postgres
* some tests don't need postgres
* kubernetes -> process orchestration system
* update how we check kafka connection
* remove return
* add description to action
* update tests
* no limit to description and use pageheader
* add ee_action
* add description as global relationship:
* fix tests
* revert to using descriptions
* revert to using descriptions
* allow description to be null
* fix tests
* persist mode
* fix e2e
* Address review points, clean up some margins
* Clean up the way editing state is determined
* Restore default description value and improve typing
* Fix typing better
* Update sql.ts
Co-authored-by: Michael Matloka <dev@twixes.com>
* perf(events): limit select clause when getting ingest requirements
Previously we were pulling in all table columns for every event posted
to the `/e/` column. We actually only need the id, such that we can
associate an event with a `Team`, and a setting which says if we should
anonymize ip addresses.
There are still other optimizations to make, and some of the code paths
do several queries to the database, but I have limited the changes here
to just the most straight forward of them.
Refers to https://github.com/PostHog/posthog/issues/8378
* Add a little comment
* Filter by team_id
* Avoid changing error messages
* Update import get_team -> get_ingest_context
* Fix typing
* fix types
* fix mocking
* fix type
* Rename InjestContext -> EventIngestContext
* Update ingestion context fetch methods
* Further rename fetch functions
* add test for async migrations not being required on fresh installs
* fix
* fix testing
* Update .flake8
* fix test?
* run tests
* fix test for good
* fixes
* attempt
* remove ee tag
* system status number formatting
* Update ee/clickhouse/system_status.py
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* simplifications
* simplify more
* simplify
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* allow datepicker in date only mode
* allow exact date matching in property filters
* marks is_date_after as a date operator
* the date picker needs to work with null/undefined as well as string
* remove unused relative date parsing and fix date only is_date_exact property filters
* default to dates without times matching exactly
* make datetime filters by second work on date values that include milliseconds
* fix TS error
* correct insight url cypress test
* replace useMemo with useEffect
* be more specific in regex date match
* spacing under the quick choices
* when checking after a date for a date only filter check against that day at 23:59:59
* remove redundant text
* add test case and fix is_date_after query
* Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env var
This makes integrating with external clickhouse providers (like altinity
cloud) much easier as users don't need to use undocumented APIs to
upload the schemas.
* Keep underscores in protobuf. This allows using this payload for json ingestion
* Timestamp format
* remove property_type_format concept
* remove nulls from tests
* deprecate not remove (and see what tests fail)
* plugin server tests care about property_type_format
* fix tests
* Allow overriding kafka host for clickhouse via KAFKA_URL_CLICKHOUSE env var
This is needed when using an external clickhouse which doesn't have the
same access to kafka as in-cluster traffic does.
Note that long-term we might need to also provide better auth mechanisms
here as well.
* Rename env variable
* wip
* wip
* wip
* can filter events by reserved words - distinct_id, created_at, and timestamp - in the backend
* add reserved properties when showing property definitions
* capitalise name of property
* properties not reserved words
* get cypress working and make search work with reserved properties
* update event/values endpoint to use property_string_expr and so work with reserved properties
* remove created at from reserved words
* test fix
* fix test
* loosen cypress assertion
* exclude specific properties from the taxonomic property filter
* remove FE property definition fangling
* exclude $time and $timestamp property definitions from the backend
* add reserved properties in the API not the FE using a CTE
* use SQL comments in SQL strings
* remove mypy error
* fix enterprise property definitions
* fix tests and improve comment
* exclude materialised column rows with no values when getting event property definitions
* fix cypress test setup
* clean up dead code
* un-remove some not actually dead code
* correct cypress assertion
* try and fix a cypress test
* Don't allow user to set current team that they don't have access to
* Warn when dashboard is available in different project
* update exceptionshog
* fix tests
* fix failing
* fix typing
* tests
