* Swapped to use KAFKA_HOSTS everywhere
* Fixed up type of kafka config options and setup separate kafka hosts for blob consumer
* allow session recordings to have its own kafka security protocol
* remove slash commands from this pr
* syntax must be obeyed
* Update UI snapshots for `chromium` (2)
* Update UI snapshots for `chromium` (2)
* fix
* Update query snapshots
* no empty strings in kafka hosts
* fix snapshot
* fix test
---------
Co-authored-by: Paul D'Ambra <paul@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* ci(cdp): push image as latest on master
* chore: add SECRET_KEY env var to codespaces
Now that I'm sharing the key between the Django app and the CDP work, we
want it to be set as a global env var.
* fix: run prettier formatting on .devcontainer.json
* fix format
* ci(cdp): push image as latest on master
* chore: add SECRET_KEY env var to codespaces
Now that I'm sharing the key between the Django app and the CDP work, we
want it to be set as a global env var.
Was updated elsewhere, but not here. Some people are reporting issues
with not being able to run the app with Python3.10 so I wanted to make
sure I could!
* chore(web): add django-prometheus exposed on /_metrics
This exposes a number of metrics, see
97d5748664/documentation/exports.md
for details. It includes histogram of timings by viewname before and
after middleware.
I'm not particularly interested in these right now, but rather would
like to expose Kafka Producer metrics as per
https://github.com/PostHog/posthog/pull/10997
* Refactor to use gunicorn server hooks
* also add expose to dockerfile
* wip
* chore(codespaces): add env vars for running codespaces dev
These will be used by vscode and any terminals spawned
* Use default python interpreter
* chore(codespaces): add env vars for running codespaces dev
These will be used by vscode and any terminals spawned
* Use default python interpreter
* add kafka to hosts
* fix hosts
* add to hosts on docker run
* keep python path
* wip
* chore(dev): use network mode host for docker-compose services
This removes the need to add kafka to /etc/hosts.
As far as I can tell this should be fine for peoples local dev except
they will be required to reset and re-migrate ClickHouse tables as they
will be trying to pull from `kafka` instead of `localhost`.
* remove ports from redis
* Update a few more references
* chore: bump defaults for clickhouse to 22.3 across the board
* remove docker build instructions for clickhouse
* remove 21.11.11.1 from matrix
* prettier it
* chore: Remove ClickHouse version 21.6 and add 22.3 for testing and docker-compose (21.6 is not in clickhouse dockerhub)
* update service requirements
* update arm64 compose as well
* Default to 21.11.11.1 and support multiple dockerhub repositories
* prettier .github/workflows/ci-backend.yml
* dev(codespaces): use Dockerfile not docker-compose
I wasn't so confident around the updating of the docker-compose file to
include minio, and the workflow around it. I'd rather just drop into
something with docker-in-docker and run docker-compose myself.
Also finally added some extensions I'm using so I don't need to enable
them all the time.
TODO: update docker-compose file to work with docker-in-docker. At the
moment I don't think the networking will be correct.
* Merge remote-tracking branch 'origin/master' into hackaton_session_rec
* add new line
* remove yarn install, it will be overwritten by the codespaces mount
* revert compose formatting
* format with prettier
* feat(trends): add moving average `smoothing_intervals` param
This will average over the preceding `smoothing_intervals` intervals.
The driver for this is to remove either weekly or monthly trends.
Limitations at the moment are that it only works for total volume
queries, as it relies on the results from clickhouse being from a single
timeline series.
* WIP UI for selecting smoothing
* WIP: Fix typing and appearce logic
* Made smoothing extensible beyond day
* Test now considers a non-trivial case
* chore(smoothing): use url params for state storage of smoothing value
This should allow for url navigation to work, although I haven't quite
figured out how we then use this to make the insights/trends POST
request
* Added further tests for smoothing intervals
* fix test_to_dict
* fix trend tests
* Added test to validate preceeding limit
* Fixed test for 2 interval
* clear django redis cache before each test
* add explicit ordering to dashboard items in refresh test
Otherwise the test is not deterministic.
* Revert "clear django redis cache before each test"
This is a good thing to do, but I don't want to open up a discussion on
this PR. It was not fix for the immediate intermittent
test_refresh_cache test
This reverts commit aa52f332f3.
* format
* clickhouse trends: SET allow_experimental_window_functions = 1
* fix setting of allow_experimental_window_functions
* add cache clears for requests
* Delete SmoothingFilterLogic.ts
Removed smoothing filter logic file
* Use `SMOOTHING_INTERVALS` instead of string literal
This should ensure that if we update the url param key somewhere, we don't have to remember to change it here as well.
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* move test_trends to ee
* actually delete test_trends.py
* Transitioned smoothing logic to use Logic
* WIP: Moving away from url based logic
* WIP: Refactored logic on listeners
* fix storybook
* refactor
* set redis url
* remove keastory
* Move interval filter to setFilters insight logic
* update trend snapshots
* run prettier
* Add stickiness test deps that were in test_trends
* Fix storybook
* fix stickiness tests
* update snapshot
* fix cohort test
* only display smoothing on linear
* fix tests
* Add some tests
* added box to make dropdown consistent
* Floor
* fix tests
* fix tests
* fix test
* Add feature flag
Co-authored-by: Marcus Hyett <marcus@posthog.com>
Co-authored-by: Marcus Hyett (PostHog) <85295485+marcushyett-ph@users.noreply.github.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Tim Glaser <tim@glsr.nl>
* feat(ClickHouse): upgrade Docker references to 21.11.11.1 (1/2)
* ci(backend): run tests on docker-compose changes
Co-authored-by: Harry Waye <harry@posthog.com>
* dev(codespaces): add "lean in" docker configuration for codespaces
This changes creates a dedicated `docker-compose.yml` and `Dockerfile`
for codespaces and doesn't attempt to have compatability with the
existing docker config files. This is intentional and is designed to be
a close to what codespaces would consider the ideal, so as to understand
what great would look like.
At the moment it doesn't do, but could:
1. install python and node packages in the image
2. push this image up to dockerhub
3. specify this image to be used for caching. There is a "cacheFrom"
option in the [.devcontainer
reference](https://code.visualstudio.com/docs/remote/devcontainerjson-reference)
although I haven't checked how this works with using
`docker-compose.yml`. Perhaps we just need to add the `image:`
property to the `app` service :fingerscrossed:
4. specify default extensions and settings to be used.
* Use python 3.9, not the latest python 3 codespaces supports
* Add some basic python default settings and extensions
* add github codespaces image build workflow
* chore: update docker-compose cache from :cache tag
* chore: use docker cache-to inline
* add codespaces platform comment
* temp run on branch
* chore: fix flatforms format
* chore: update qemu comment
* chore: codespaces build use master image
* chore: target x86_64
* chore: just use image for codespaces, don't try to build
* add in python, node requirements
* install node
* install node
* uncomment node deps install
* use app network stack for all services
* move .env to .dev.env and us in docker-compose
* set WORKDIR to workspace
* pin apt packages
* pin versions, remove apt-key
* add yarn cache clean
* Fix hadolint warnings
* make codespace build on master, or a PR with codespaces-build label
* add missing &&
* dont bother with clickhouse install, its huge
* dont be cute with the networks, just expose explicitly
* fix network
* create container zsh
* correct .env path
* point app to the clickhouse container
* Also install saml deps
* ignore .env again
* libxml already installed
* correct conditional workflow comment
* Revert "Revert "Add is_deleted column to person_distinct_id (#5151)" (#5193)"
This reverts commit 401268bdba.
* A tweak for docker-compose builds
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Update PERSONS_ACTIVE_USER_SQL query
* Remove dead import
* Update lifecycle queries
* Update BREAKDOWN_ACTIVE_USER_INNER_SQL to use new persons query
* Update STICKINESS_SQL
* Update STICKINESS_PEOPLE_SQL
* Update STICKINESS_ACTIONS_SQL
* Update paths query
* Update events query
* Update CALCULATE_COHORT_PEOPLE_SQL
* Update retention queries
* Update TOP_PERSON_PROPS_ARRAY_OF_KEY_SQL
* Update EVENT_JOIN_PERSON_SQL
* Update GET_PERSON_ID_BY_ENTITY_COUNT_SQL
* Remove remaining references to old get latest person query
* Update GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Fix code style issue
* Update table engine for person_distinct_id table
* don't select team_id
* Make person deletion work
* Use replacingmergetree over collapsing with is_deleted
Replacing an existing engine is hard, let's not do it
* Update query in test
* add migration
* set database on materialized views
* Update plugin server to 1.1.6
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: posthog-bot <posthog-bot@users.noreply.github.com>
