* Update test_schema.py
* Create DLQ kafka table only for relevant tests
* Remove DLQ mv tables from conftest
* Standardize conftest
* Parallel clickhouse setup
* Create django db based on models
Speeds up django test database setup from ~15s to 1s.
* Update imports
* Re-add conditionality, fix typing
* Run plugin-server tests when this file changes
* Simplify imports in `setup_test_environment`
* refactor(ingestion): Unify tests into single `process-event.test.ts`
* Update test scripts for CI
* Fix imports
* Run prettier
* Fix missing import
* Fix the OTHER imports
* Nuke `includeNewPropertiesUpdatesTests`
* feat: wip add script to backfill persons and groups columns on events table
* updates
* cleanup + query id feature
* refactor
* fixes and add person tests
* add groups tests
* fix ci tests?
* test
* keep dictionaries around
* updates + massive sleep test?
* fixes
* updates
* fix for ch 21.6.5
* feat(sharding): add command to sync tables onto new nodes
clickhouse-operator only syncs some tables onto new nodes. This new
command ensures that when adding new shards, they are automatically
synced up on redeploying
Note that there might be timing concerns here as resharding on altinity
cloud does not redeploy automatically. In practice however what this
means is that new nodes just won't ingest any data until another deploy
* Add test to the new command
* Improve non-replicated test
Currently this would throw on a fresh install with an error similar to
```
│ posthog-events Migration would get applied: 0024_materialize_window_and_session_id │
│ posthog-events Traceback (most recent call last): │
│ posthog-events File "manage.py", line 21, in <module> │
│ posthog-events main() │
│ posthog-events File "manage.py", line 17, in main │
│ posthog-events execute_from_command_line(sys.argv) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", li │
│ posthog-events utility.execute() │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", li │
│ posthog-events self.fetch_command(subcommand).run_from_argv(self.argv) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 3 │
│ posthog-events self.execute(*args, **cmd_options) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 3 │
│ posthog-events output = self.handle(*args, **options) │
│ posthog-events File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 41, in │
│ posthog-events self.migrate(CLICKHOUSE_HTTP_URL, options) │
│ posthog-events File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 57, in │
│ posthog-events sql = getattr(op, "_sql") │
│ posthog-events AttributeError: 'RunPython' object has no attribute '_sql'
```
This is due to the command not working with `RunPython` commands.
* error if unsupported db version
* upgrade mypy
* fix various types for mypy
* you can have it in any color you want, as long as it's black
* fix mypy
* Update `kafka-python`
* Format with Black
* Fix mypy after merge
Co-authored-by: Michael Matloka <dev@twixes.com>
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
* Make DDLs more friendly towards running on a cluster
* Use primary CLICKHOUSE host for migrations and DDL
* loose ends on person kafka create
* posthog -> cluster typo
* add cluster to KAFKA create for plugin logs
* Feed the type monster
* clusterfy local clickhouse
* test docker-compose backed github action
* run just clickhouse and postgres from docker-compose
* move option to between up and <services>
* posthog all the things
* suggest tests run on cluster
* posthog cluster for ci
* use deploy path for docker-compose
* fix for a clickhouse bug 🐛
* complete CH bug fixes
* 5439 the github actions pg configs
* remove CLICKHOUSE_DATABASE (handled automatically)
* update DATABASE_URL for code quality checks
* Missed a few DDLs on Person
* 5439 -> 5432 to please the people
* cleanup persons and use f strings <3 f strings
* remove auto parens
* Update requirements to use our fork of infi.clickhouse_orm
* fix person.py formatting
* Include boilerplate macros for a cluster
* Add setting for handing off process_event_ee to plugin server
* Add StatsD settings to KEYS
* bin/plugin-server → start-plugin-server & docker-plugin-server
* Simplify to only add docker-plugin-server
* Bring back original comment
* Turn down verbosity of plugin server install
* Remove redundant if
* Fix comment
* Remove lone newline
* Roll back unsafe script changes
* Simplify dockerized plugins
* Add some depends_on
* Clarify HAND_OFF_INGESTION env var
* Use posthog-plugin-server 1.0.0-alpha.1
* Enhance bin/plugin-server and rm bin/docker-plugin-server
* Move around PLUGIN_SERVER_INGESTION_HANDOFF ifs
* Use posthog-plugin-server@1.0.0-alpha.2
* Support kafka+ssl:// in plugin-server
* Produce to topic events_ingestion_handoff for plugin server
* Use posthog-plugin-server@1.0.0-alpha.3
* Don't import Kafka topics in FOSS
* Use @posthog/plugin-server
* Update yarn.lock
* Add commands for external ClickHouse setup/teardown
* Actually delete test CH teardown command
* ClickhouseTestRunner.setup_test_environment() in setup_test_clickhouse
* Rework test setup script to work with Postgres too
* Restore master plugins dir for merge
* Unset PLUGIN_SERVER_INGESTION_HANDOFF in docker-compose.ch.yml
* Fix unimportant typo
* Build log_event data dict only once
* Make it clear in bin/plugin-server help that it's bin
* Space space
* Add user param for clickhouse if provided
* remove duplicate setting - forgot it was there!
* inconsistent naming between migrations and clickhouse-driver
* Upload static cohort using CSV, closes #2868
* fix tests
* Fix tests
* Fix e2e test
* Avoid double inserts
* Speed up query
* Move to params
* fix tests
* Use JSON instead of protobuf
* unused import
* produce instead of produce_proto
* Insert directly into clickhouse, no need for kafka
* a few quick fixes
* insert in batch to clickhouse
* test SQLi since we can't really trust what we are inserting
* Extra check for duplicates
Co-authored-by: James Greenhill <fuziontech@gmail.com>
