allow asgi/wsgi to be configurable by env var
This will let us roll out asgi separate across services as we had
issues with our recordings capture pods on asgi
* Remove old dockerfile and only build a single image with unit
We have migrated over to the `unit` image (instead of gunicorn) but we
were still building both. This could lead to confusion - we have no
plans to move back so let's remove the second image build and only
build the unit based image
* Remove unit build id
* chore: log librd kafka features on blob ingester start
* one line to rule them all
* wat
* feat: gzip support for plugin server kafka (#16240)
* feat: gzip support for plugin server kafka
* need to specify earth standard gravity
Set the posthog uid/gid to be hard-coded
Currently the uid and gid are 999/1000 respectively, but this could potentially
change when updating the base image, so hard-code these so that we can
rely on these not changing and breaking volume ownership
* Revert "Revert "perf(recordings): use node-librdkafka for ingester production" (#15069)"
This reverts commit ac5e084f48.
* fix(plugin-server): only set ssl config when defined
Hopefully this means it will use the global CA bundle.
* hack: enable debug logs
* honor KAFKAJS_LOG_LEVEL envvar
* add SegfaultHandler
* disable ssl verification
* debug -> info
* only log brokers
* Revert "add SegfaultHandler"
This reverts commit b22f40b802.
---------
Co-authored-by: Xavier Vello <xavier@posthog.com>
Previously we've been using the KafkaJS Producer with a wrapper around
it to handle batching. There are a number of issues with the batching
implementation e.g. not having a way to provide guarantees on delivery
and rather than fix that, we can simply use the librdkafka Producer
which is a lot more mature and battle-tested.
* Python 3.10
Performance gains go brrr
* Add missing SAML deps
* Add missing dep to dockerfile
* Update mypy to 0.981 for 3.10.7 compatibility
Needed this bug to be fixed: https://github.com/python/mypy/issues/13627
This also incidentally fixed the mypy bug in csv_exporter.py
* bump to 3.10.10
* chore: use pnpm to manage dependencies
* Fix CI errors
* Don't report Docker image size for external PRs
* Fix pnpm-lock.yaml formatting
* Fix module versions
* Ignore pnpm-lock.yaml
* Upgrade Cypress action for pnpm support
* Set up node and pnpm before Cypress
* Fix typescript issues
* Include patches directory in Dockerfile
* Fix Jest tests in CI
* Update lockfile
* Update lockfile
* Clean up Dockerfile
* Update pnpm-lock.yaml to reflect current package.json files
* remove yarn-error.log from .gitignore
* formatting
* update data exploration readme
* type jest.config.ts
* fix @react-hook issues for jest
* fix react-syntax-highlighter issues for jest
* fix jest issues from query-selector-shadow-dom
* fix transform ignore patterns and undo previous fixes
* add missing storybook peer dependencies
* fix nullish coalescing operator for storybook
* reorder storybook plugins
* update editor-update-tsd warning to new npm script
* use legacy ssl for chromatic / node 18 compatibility
* use pnpm for visual regression testing workflow
* use node 16 for chromatic
* add @babel/plugin-proposal-nullish-coalescing-operator as direct dependency
* try fix for plugin-server
* cleanup
* fix comment and warning
* update more comments
* update playwright dockerfile
* update plugin source types
* conditional image size reporting
* revert react-native instructions
* less restrictive pnpm verions
* use ref component name in line with style guide
Co-authored-by: Jacob Gillespie <jacobwgillespie@gmail.com>
* feat(ingestion): remove Graphile worker as initial ingest dependency
At the moment if the Graphile enqueing of an anonymous event fails e.g.
due to the database that it is uing to store scheduling information
fails, then we end up pushing the event to the Dead Letter Queue and to
not do anything with it further.
Here, instead of directly sending the event to the DB, we first push it
to Kafka, an `anonymous_events_buffer`, which is then committed to the
Graphile database. This means that if the Graphile DB is down, but then
comes back up, we will end up with the same results as if it was always
up*
(*) not entirely true as what is ingested also depends on the timings of
other events being ingested
* narrow typing for anonymous event consumer
* fix types import
* chore: add comment re todos for consumer
* wip
* wip
* wip
* wip
* wip
* wip
* fix typing
* Include error message in warning log
* Update plugin-server/jest.setup.fetch-mock.js
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* include warning icon
* fix crash message
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* setup event handlers as KafkaQueue
* chore: instrument buffer consumer
* missing import
* avoid passing hub to buffer consumer
* fix statsd reference.
* pass graphile explicitly
* explicitly cast
* add todo for buffer healthcheck
* set NODE_ENV=production
* Update comment re. failed batches
* fix: call flush on emitting to buffer.
* chore: flush to producer
* accept that we may drop some anonymous events
* Add metrics for enqueue error/enqueued
* fix comment
* chore: add CONVERSION_BUFFER_TOPIC_ENABLED_TEAMS to switch on buffer
topic
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* chore(web): add django-prometheus exposed on /_metrics
This exposes a number of metrics, see
97d5748664/documentation/exports.md
for details. It includes histogram of timings by viewname before and
after middleware.
I'm not particularly interested in these right now, but rather would
like to expose Kafka Producer metrics as per
https://github.com/PostHog/posthog/pull/10997
* Refactor to use gunicorn server hooks
* also add expose to dockerfile
* wip
* feat: add libmaxminddb0 as dependency. C library will speed things up significantly
* pin libmaxminddb to 1.5 for whats available from APK
* get geolite2 db during build
* add settings for geoip2 django contrib library
* black formatting
* consistently use share director
* isort fixes
* remove GeoLite2-City.mmdb from git and add script to ./bin/start to download it if file does not exist
* remove GeoLite2-City.mmdb from git
* add doc for share directory expaining why it exists
* relative path for curl in build
* shared vs share consistency
* feat(feature-flags): Allow GeoIP property overrides for persons in decide
* add geoip class
* Update snapshots
* add geoip to ci
* wrong place
* check where I am
* decompress database
* Update snapshots
* brotli decompress
* ..everywhere
* remove annoying ff snapshot
* make exception surface larger
* add permissions to view folder & read file for everyone
* make exception surface larger
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: neilkakkar <neilkakkar@users.noreply.github.com>
* feat: add libmaxminddb0 as dependency. C library will speed things up significantly
* pin libmaxminddb to 1.5 for whats available from APK
* get geolite2 db during build
* add settings for geoip2 django contrib library
* black formatting
* consistently use share director
* isort fixes
* remove GeoLite2-City.mmdb from git and add script to ./bin/start to download it if file does not exist
* remove GeoLite2-City.mmdb from git
* add doc for share directory expaining why it exists
* relative path for curl in build
* shared vs share consistency
* Update snapshots
* brotli decompress
* ..everywhere
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: neilkakkar <neilkakkar@users.noreply.github.com>
* chore(dockerfile): make docker build multistage
The built image is >4GB uncompressed atm, I'm pretty sure there is a lot
of cruft.
Plan is to split out the django, frontend, plugin-server builds and
hopefully get some gains in there to not include build deps.
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* fix dockerfile lints
* cache from branch
* add load :true
* Update production.Dockerfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Update production.Dockerfile
* update to use compressed size from remote repo
* tag with branch and sha
* add ref on pull_request events
* install python
* be a bit more lax with python version
* fix image size calc
* hardcode lower case image name
* use @
* only add sha on master branch, add master tag on master
* chore: use docker image in e2e tests
This is to try to add some guarantees to the docker image without having
to manually test it, so we can be a bit more aggressive with
improvements without, e.g. having to push to the playground or run
locally.
* wip
* add to summary
* wip
* chore: put cypress tests after docker build
I couldn't figure out a way to get workflow_run to run without merging
in, so I'm just putting after the build.
* wip
* wip
* wip
* remove quotes
* remove separate cypress install
* wip
* wip
* wip
* add gunicorn.config.py
* ci: run docker image build on master as well
This way we get the caching from the master build.
* wip
* wip
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* chore(deps): upgrade node.js version to v16
* update npm version'
* try to update alpine
* Revert "try to update alpine"
This reverts commit 58024968e2.
* don't update alpine, use node-current instead:
* go back to npm 7
* upgrade @google-cloud packages
* handle unhandled rejections :D
* add handling for unhandled promise rejections in the main thread as well
