Co-authored-by: Tiina Turban <tiina303@gmail.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben White <ben@posthog.com>
* chore(batch-exports): add snowflake export workflow
This workflow uses Snowflake internal stages to load data from
ClickHouse into a Snowflake table. We maintain the existing events table
schema as used in the existing Snowflake App.
Something I haven't done yet is:
1. made sure e.g. we get the `elements` and `person_set` etc. data into
Snowflake.
2. the additional frontend to enable configuring the Snowflake
connection.
* remove unsed var
* include excluded events in test
* feat(batch_exports): add backend API and S3 temporal workflow
This adds the backend API for batch exports, which will handle reverse
ETL exports to e.g. S3, Snowflake etc.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* docs(capture): add spec for capture endpoints
This adds an OpenAPI specification for the capture endpoints. This is
useful for documenting the API, and further can be used e.g. in tests to
validate that both clients and the API are behaving as expected.
* feat: Add Temporal to the dev and hobby stacks
* disable elastic for hobby because of resources
* checkpoint
* update requirements
* worker is up, but without the sandbox
* ensure temporal does not depend on elastic
* Feedbacked
* pip-compile dev
* mypy fixes
* add a bit of colorful logging
* add django temporal worker to the mix
* checkpoint for dev-full docker
* Working on docker-full, but checkpointing for now
* add migration bits for full
* fix(auth): Remove 2fa requirement if user is using 2fa
* Remove last few social_2fa survivors
---------
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* feat: Automatically set hot partition keys in local cache
This allows us to override partition keys that have been locally
determined to be over a given threshold.
This commit is inteded as a test to motivate discussion. A final
solution should also consider handling the size of the local cache to
avoid memory overflows, specially for long running deployments.
* fix: Add type hint for counter
* refactor: Use token-bucket to decide to randomly partition
* revert: info window position changes (#13929)
* chore: Update comments
* chore: Add instrumentation to partition key override
* refactor: Rebase on master to bypass dynamic settings
* refactor: Move variables to settings and disable by default
* feat: Also rate limit the logging of override keys
* fix: Set capacity as float
* fix(capture): Make default setting a float
Co-authored-by: Harry Waye <harry@posthog.com>
* fix(capture): Typo
Co-authored-by: Harry Waye <harry@posthog.com>
* fix(capture): Format ingestion settings
* fix(capture): Use patched limiter for testing instead of reloading module
---------
Co-authored-by: Paul D'Ambra <paul@posthog.com>
Co-authored-by: Harry Waye <harry@posthog.com>
so that we upgrade urllib3 which sentry latest needs
Problem
running pip install -r ./requirements.txt worked fine
but running pip install -r ./requirements.txt && pip install -r ./requirements-dev.txt gave an error message
Installing collected packages: urllib3
Attempting uninstall: urllib3
Found existing installation: urllib3 1.26.13
Uninstalling urllib3-1.26.13:
Successfully uninstalled urllib3-1.26.13
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sentry-sdk 1.11.1 requires urllib3>=1.26.11; python_version >= "3.6", but you have urllib3 1.26.5 which is incompatible.
Changes
Upgraded the requests library and regenerated the requirements files
* Revert "chore(tests): add pytest-randomly plugin to randomize test run order (#12307)"
This reverts commit 313de83cb8.
* Update snapshots
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
## Problem
Django validates file content type by reading "magic bytes" from the start of the file. It doesn't then check that file really is the type it claims to be. https://docs.djangoproject.com/en/4.1/topics/security/#user-uploaded-content
That's not good enough for rock and roll. And would allow an attacker to attempt to upload HTML with magic bytes that pretend to be an image file. We would store that and then serve it back to a dashboard. ☠️ See more at https://trailofbits.github.io/ctf/forensics/
## Changes
On upload use the [Pillow image library](https://github.com/python-pillow/Pillow) to open the file and transpose it. The image must be valid to be successfully transposed.
## How did you test this code?
* Adding a valid image file and seeing the developer tests still run
* Adding a file handcrafted to start with gif magic bytes but not actually be a gif and seeing validation fail
* Uploading an image to a dashboard and seeing it still work
