* add persistence to redis and kafka on hobby
This is to address https://github.com/PostHog/posthog/issues/22291, but potentially fixes other redis/kafka related conditions over containers updates / rebuilds
* proper path to kafka data volume
* add volume definitions
---------
Co-authored-by: Frank Hamand <frank@posthog.com>
* added a way to preview the key value without exposing the key
* updated api keys preview
* Make the UI a bit nicer
* Write out the reasoning behind personal API key entropy
* Include underscore in masked value
* updated migrations for API key mask
* updated personal api keys tests
---------
Co-authored-by: Michael Matloka <michal@matloka.com>
* Update docker-compose.hobby.yml
* use named volume instead of volume mount
we don't want to mount /etc from user's computers as 1. it's usually owned by root, 2. not portable to macos/windows
use a named volume instead like we do with e.g. posthogres
---------
Co-authored-by: Frank Hamand <frank@posthog.com>
currently default objectstorage address is http://127.0.0.1:19000, but ports are not exposed from docker-compose. This commit exposes ports AND updates OBJECT_STORAGE_ENDPOINT to direct to MinIO container rather than localhost, but localhost port is useful either way for admin and testing purposes.
* Update base postgres to version 15
* Pin pg12 for hobby installations
* Update docker-compose.hobby.yml
Co-authored-by: Xavier Vello <xavier.vello@gmail.com>
---------
Co-authored-by: Xavier Vello <xavier.vello@gmail.com>
* feat: Add Temporal to the dev and hobby stacks
* disable elastic for hobby because of resources
* checkpoint
* update requirements
* worker is up, but without the sandbox
* ensure temporal does not depend on elastic
* Feedbacked
* pip-compile dev
* mypy fixes
* add a bit of colorful logging
* add django temporal worker to the mix
* checkpoint for dev-full docker
* Working on docker-full, but checkpointing for now
* add migration bits for full
* feat: Have a local docker-compose stack for developing 100% in docker
* lol found a docker-compose bug where you can't have volumes created in root dir
* scale -> deploy.replicas
* don't forget to add image for asyncmigrationscheck
* env vars to each component
* Rename local to dev-full
* feat: remove version from docker compose to support new spec
* feat: simplify the docker-compose setup so we do less version coordinations
* update hobby bin
* bump docker-compose version for hobby for extends compat
* move ci to ubuntu-latest
* Revert "move ci to ubuntu-latest"
This reverts commit a0462adfec.
* use docker compose for github ci
* correct comments on base
* feat(async-migrations): add auto complete trivial migrations option
This change is to ensure that the `run_async_migrations --check` command
option is a light operation such that we can safely use this e.g. in an
init container to gate the starting of a K8s Pod.
Previous to this change, the command was also handling the
auto-completion of migrations that it deamed to not be required, via the
`is_required` migration function. Aside from this heaviness issue, it's
good to avoid having side effects from a check command.
One part of the code that I wasn't sure about is the version checking,
so any comments here would be much appreciated.
Note that `./bin/migrate` is the command we call from both ECS migration
task and the Helm chart migration job.
* update comment re versions
* wip
* wip
* wip
* update hobby
* rename to noop migrations
* Add persistent volumes to docker-compose-hobby
Per the discussion in https://github.com/PostHog/posthog/issues/10792, implemented the "Kessel Fix" in less than a parsec.
* Add warning text to user prompts to avoid data loss
Following discussion with PH team, we wanted to give users the information needed to properly manage the data in their installation and avoid potential data loss.
* refactor: Eliminate the `KAFKA_ENABLED` setting
* Remove dead code
* Consolidate plugin server test scripts and CI
* Fix CI command
* Remove Celery queues
* Rearrange test directories
* Update import paths
* chore(hobby deployments): various fixes
* default do not check versions for current hobby release
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* chore: bump defaults for clickhouse to 22.3 across the board
* remove docker build instructions for clickhouse
* remove 21.11.11.1 from matrix
* prettier it
* feat(object_storage): add unused object storage with health checks
* only prompt debug users if object storage not available at preflight
* safe plugin server health check for unused object storage
* explicit object storage settings
* explicit object storage settings
* explicit object storage settings
* downgrade pip tools
* without spaces?
* like this?
* without updating pip?
* remove object_storage from dev volumes
* named volume on hobby
* lazily init object storage
* simplify conditional check
* reproduced error locally
* reproduced error locally
* object_storage_endpoint not host and port
* log more when checking kafka and clickhouse
* don't filter docker output
* add kafka to hosts before starting stack?
* silly cloud tests (not my brain)
* chore: Remove ClickHouse version 21.6 and add 22.3 for testing and docker-compose (21.6 is not in clickhouse dockerhub)
* update service requirements
* update arm64 compose as well
* Default to 21.11.11.1 and support multiple dockerhub repositories
* prettier .github/workflows/ci-backend.yml
Previously we were not defining volumes in ZooKeeper config for
docker-compose.hobby.yml.
We noticed the issue with the upgrade from 1.34.2 to 1.35.0 due to the
changes to ZooKeeper version. Previously I believe we were just stopping
and starting the ZooKeeper container. With no image changes this is
fine, we'll reuse the existing container. However, as we updated we end
up with a new container, without any of the previous data.
Coupled with the introduction of Replication which relies on ZooKeeper,
this brought down hobby deployments.
It looks like we also do not provide explicit volumes for e.g.
ClickHouse either, but I'm doing one fix at a time...
This change will not try to save anything that is in the existing
container. After running this command, data will need to be reinstated
with ClickHouse SYSTEM RESTORE REPLICA.
* feat(ClickHouse): upgrade Docker references to 21.11.11.1 (1/2)
* ci(backend): run tests on docker-compose changes
Co-authored-by: Harry Waye <harry@posthog.com>
