* Revert "perf(ingestion): use rdkafka consumer for both ingestion queues (#15695)"
This reverts commit fea9e4d77c.
* format
* fix split test
* no really, fix the test
---------
Co-authored-by: Harry Waye <harry@posthog.com>
This _should_ give us better performance and reliability, but it's
hard to tell without a lot of testing. Will monitor closely on rollout.
Note that this will require a delete on the old consumer members as they
are using round eager robin partition strategy, whereas this is using
the cooperative sticky partition strategy. librdkafka doesn't support
mixing the two, unlike the Java Kafka Client.
---------
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* feat(ingestion-slowlane): Add token-bucket utility
* feat(ingestion-slowlane): Re-route overflow events
* fix: Import missing stringToBoolean
* fix(ingestion-slowlane): Flip around kafka topics according to mode
* refactor(ingestion-slowlane): Use dash instead of underscore in filename
* fix(ingestion-slowlane): Do not increase tokens beyond bucket capacity
* feat(ingestion-slowlane): Add ingestion-overflow mode/capability/consumer
* feat(ingestion-slowlane): Add ingestion warning for capacity overflow
* test(ingestion-slowlane): Add test for ingestion of overflow events
* fix(ingestion-slowlane): Rate limit warnings to 1 per hour
* test(ingestion-slowlane): Add a couple more tests for overflow re-route
* fix(slowlane-ingestion): Look at batch topic to determine message topic
* refactor(slowlane-ingestion): Use refactored consumer model
* fix(slowlane-ingestion): Undo topic requirement in eachMessageIngestion
* refactor(slowlane-ingestion): Only produce events if ingestionOverflow is also enabled
* refactor(slowlane-ingestion): Use an env variable to determine if ingestionOverflow is enabled
* chore(slowlane-ingestion): Add a comment explaining env variable
* add support for token field in kafka message
* formPipelineEvent
* rename pipeline files according to new order
* wip team_id and anonymize ips
* conditional handlers and tests
* some plugin server fixes
* fix capture bug
* fix
* more fixes
* fix capture tests
* pipeline update
* fix + investigate database resets
* fix import order
* testing and typing updates
* add test for capture endpoint
* testing
* python typing
* plugin server test
* functional test
* fix test
* another fix
* make sure no team ids clash in tests
* fix
* add more metrics and logs
* cache nulls
* updates
* add more metrics
* refactor(plugin-server): simplify groups caching
* add multi groups test
* remove comments
* fix type, add debug
* fix
* stringify
* add groups created_at to types
* more test fixes
* use the right clickhouse timestampo format
* update created at to ch format in tests
* finally
* more fixes
* Add new app_metrics feature available on scale and enterprise
* chore(ingestion): cache available_features for a longer period in OrganizationManager
This will come in handy for app metrics and is generally a performance win
* Add service to track app metrics
* refactor(historical-exports): Move retry limit handling code to same place as other error handling
* Track app metrics in processEvent/onEvent/exportEvents and historical exports
* Add mising app-metrics file
* Add missing __init__.py module
* Use correct topic + columns for app metrics
* Add a placeholder schema
* Set timestamp correctly
* Fix a typeerror in organization-manager.ts
* Schema fixup
* Add test showing read-own-writes logic
* Remove unneeded TODO
* Add missing constant
* Simplify flushing logic
* Stabilize VM tests
* Use correct sharding key
* Revert hooks changes
* Add a is_system flag to activity logs
* Allow writing activity logs from within plugin-server
* Make changes object non-required
* Render system user information
* Log when export finishes or fails in plugin activity log
* Update activityLogLogic.plugin.test.tsx
Re-ingesting historically exported autocaptured events currently fails: https://sentry.io/organizations/posthog2/issues/3552913543/?query=is%3Aunresolved+level%3Aerror
This is because of attr__class being sometimes there under attributes, other times not, so $elements.attr_class could be either a string or an array when leaving posthog in an export.
Rather than fix this at the export, making the parsing logic in ingestion more flexible
* refactor tests to be more extensible
* Move elements-related code to separate file
* Copy over tests for element chain
* Handle `undefined` match group
As this test from python indicates, the right-most match group may be
empty. In javascript the behavior for this is different from python and
the match group may be `undefined`
* Update import
* Fix http capabilities
