* feat: add the performance events clickhouse schema
* add support for replication being off
* move and refactor performance events sql file
* with the same kafka columns as other tables
* don't account for replication settings in migration
* update snapshots
* kafka table doesn't have explicit kafka columns
Co-authored-by: Ben White <ben@posthog.com>
* refactor(plugin-server): split out plugin server functionality
To get better isolation we want to allow specific functionality to run
in separate pods. We already have the ingestion / async split, but there
are further divides we can make e.g. the cron style scheduler for plugin
server `runEveryMinute` tasks.
* split jobs as well
* Also start Kakfa consumers on processAsyncHandlers
* add status for async
* add runEveryMinute test
* avoid fake timers, just accept slower tests
* make e2e concurrent
* chore: also test ingestion/async split
* increase timeouts
* increase timeouts
* lint
* Add functional tests dir
* fix
* fix
* hack
* hack
* fix
* fix
* fix
* wip
* wip
* wip
* wip
* wip
* fix
* remove concurrency
* remove async-worker mode
* add async-handlers
* wip
* add modes to overrideWithEnv validation
* fix: async-handlers -> exports
* update comment
* chore(plugin-server): Consume from buffer topic
* Refactor `posthog` extension for buffering
* Properly form `bufferEvent` and don't throw error
* Add E2E test
* Test buffer more end-to-end and properly
* Put buffer-enabled test in a separate file
* Update each-batch.test.ts
* Test that the event goes through the buffer topic
* Fix formatting
* Refactor out `spyOnKafka()`
* Ensure reliability batching-wise
* Send heartbeats every so often
* Make test less flaky
* Commit offsets if necessary before sleep too
* Update tests
* Use seek-based mechanism (with KafkaJS 2.0.2)
* Add comment to clarify seeking
* Update each-batch.test.ts
* Make minor improvements
* fix: Make kafka health check not wait 5000ms
When no messages to consume, by default the consumer waits 5000ms,
causing flakiness in tests and slowing other things down.
* Improve timeout test
* Refactor method to do everything it's supposed to do
No tests though?
* Remove unused argument from a function
* Remove dead code
* Run event pipeline on worker threads
* Remove code in way of refactor
* Make metrics happy
* Silence a yeetcode-related failure
* Mock over stopping unneeded services
* Yeetcode a test
* Reset the right table between tests
* Reset healthcheck consumer between tests
* Silence logspam around creating topics
* Significantly speed up resetting clickhouse tables between tests
This takes the function from 13-17s to 4s locally. Not great but not
terrible either.
* Remove try-catch
* Remove a dead test
* Migration to add version to person_distinct_id
* Update plugin-server type
* Use queueMessages instead of for loops
* Update distinct id versions in postgres
* Add commented out new query
* Add person_distinct_id2 table setup/migration
This will be used for more efficient person_distinct_id queries
* Avoid sharding person_distinct_id2 on cloud
* Write to new distinct ids topic
* Attempt to use version in tests
* Tests attempt 2
* Fixup version - dont send with all messages
* Flush kafka more frequently
* Actually fix tests
* Add another await
* Add partition to person_distinct_id2 table
