It's also an optimization as we also already do not schedule the job if
the plugin is disabled. It just means that if loads of jobs are
scheduled to graphile before the plugin was disabled then it will take a
while to get through.
We want to be able to make plugins non-global, but we don't want to
break existing plugins. This commit adds a test to ensure that plugins
continue to work when they are made non-global.
Previously they would have been disabled by the `is_global` check.
TODO: what will be the impact of making this change.
* test(plugin-server): add test for attachments
We didn't have one before, now we do!
* test: add tests for plugin secrets
Adds basic test for using configurations in plugins. This is a first
step towards getting lazy loading of plugins in safely:
https://github.com/PostHog/posthog/pull/15704
* feat(jobs): don't execute queued jobs for disabled configs
* add functional test for disabled plugins
---------
Co-authored-by: Harry Waye <harry@posthog.com>
There is a race condition in these tests where the consumer isn't
consuming in time to pick up bad messages, so we ensure that we set the
offsets to the earliest messages.
* chore(plugin-server): use librdkafka producer everywhere
We say some 10x improvements in the throughput for session recordings.
Hopefully there will be more improvements here as well, although it's a
little less clear cut.
I don't try to provide any improvements in guarantees around message
production here.
* we still need to enable snappy for kafkajs
* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
This also adds a test to ensure we are capturing usage of $set/$set_once
at the top level of the event, as posthog-js uses this method.
This was initiated by the issue mentioned
[here](https://github.com/PostHog/posthog-js/issues/615).
* chore(recordings): use cooperative-sticky rebalance strategy
This should make rebalances and lag during deploys a little less
painful. I'm setting this as the globally used strategy, when we e.g.
want to use another strategy for a specific consumer group, we can make
this configurable.
* disable rebalance_callback
* use node-rdkafka-acosom fork instead, for cooperative support
* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* test(person-overrides): Update tests to match new constraint
* fix(migration): Add drop extension query to reverse migration
* fix(migration): Use correct table name
* refactor(person-overrides): Make team a regular bigint field without FK
* refactor(person-overrides): Rename Helper model to Mapping
* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* wip: add test for concurrent updates to posthog_personoverride table
* refactor
* Update snapshots
* fix(isort): Correctly sort imports
* fix(ee-cohort-test): Delete person after creating it
* fix: person constraint in person overrides table (#14319)
* nits
* fix migration tests
* chore(migration): Bump migration number to 0302
* Update snapshots
* feat: person-overrides writes
* test(person-overrides): Add concurrent tests to person-overrides model
* feat(person-merge): Update merge to use new helper table
* fix(tests): Pass poEEmbraceJoin to updatePersonState in test
* fix(person-state): Format person-overrides message for ClickHouse
* test(api): Add function to reload dictionary for person overrides
* fix: poe final test failure
* refactor(person-state): Make failed attempts a class variable to
This allows us to mock it during testing as some tests require
immediate failures.
* fix(postgres-utils): Apply some magic changes lost to time
* fix(person-state): Join with helper table to return UUIDs
* fix(person-state): Use single quotes for UUID queries
* test(person-overrides): Skip test that doesn't work without a merge command
* test(person-state): Add a very complicated query to get UUIDs in a test
* test(person-state): Try waiting longer, this is flaky
* fix(migrations): Remove unused migration
* fix(person): Undo unneeded person model changes
* chore: Clean-up artifacts from rebase on model branch
* refactor(person-state): Rename mergeAttempts and don't read from ENV
* chore: Better clarify oldest_event usage in comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* refactor(person-state): Update version in queries
* fix(person-state): Use new mapping model instead of helper
* fix(migrations): Re-add constraint deleted on field drop
* revert: Re-enable test that was skipped
Test was originally skipped due to missing a $merge command, but now
$merge_dangerously is available, so the test should pass.
* test: Attempt to refresh dictionary in test
* Update query snapshots
* Update query snapshots
* test: Expect dictionaries to be refreshed before resuming test
* fix(test): Use alias property for $merge_dangerously in test
* fix(test): Expect any string like in all other tests
* test: Fix order of events to determine merged persons
* chore: Update comment regarding overrides mapping query
* chore: Remove out of date comment
* test: Set number of retries to 0 when updating persons
* test: Delete comment
* test: Assert clickhouse state after identify
* chore: Move comment
* chore: Delete TODO regarding oldest_event updates
* test: Assert events are still processed if merge fails
* test: Assert properties are processed if merge fails
* test: Don't expect to throw when failing merges
* fix: Indent RETURNING query clause
Co-authored-by: Harry Waye <harry@posthog.com>
* chore: Remove out of date comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* test: Resume merges after test suite
* test: Run processPersonStep tests for both poEEmbraceJoin modes
---------
Co-authored-by: Harry Waye <harry@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* fix(plugin-server): don't DLQ session recordings with invalid token
* chore(session-recordings): add test for not DLQing on no token/team
---------
Co-authored-by: Harry Waye <harry@posthog.com>
* test(plugin-server): use librdkafka for functional tests
While trying to port the session recordings to use node-librdkafka I
found it useful to first implement it in the functional tests.
* use obj destructuring to make calls more self explanatory
