* everything except plugin server and sync_available_features
* sync_available_features_done, some plugin_server done?
* and a tiny bit more
* linting
* try to fix some tests
* more test fixes/
* clean up typos
* weed wacking bugs
* more test shenanigans
* fix plugin server
* actually fix plugin server test?
* still fixing tests
* another attempt
* some pr feedback
* small fix
* fix database query accessor
* fix functional tests
* fix tests
* Update query snapshots
* Update query snapshots
* Update query snapshots
* update some comments and fxn names
* fix plugin server test
* Update query snapshots
* Update query snapshots
* Update query snapshots
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* cleanup: remove unused team arg from registerLastStep
* cleanup: rename promises to ackPromises to make it more clear thats what they are
* cleanup(plugin-server): make waitForAck explicit/required
* add Kafka produce/ack metrics
* Clarify Kafka produce metric/labels
* test(plugin-server): add test for attachments
We didn't have one before, now we do!
* test: add tests for plugin secrets
Adds basic test for using configurations in plugins. This is a first
step towards getting lazy loading of plugins in safely:
https://github.com/PostHog/posthog/pull/15704
* feat(jobs): don't execute queued jobs for disabled configs
* add functional test for disabled plugins
---------
Co-authored-by: Harry Waye <harry@posthog.com>
* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
This also adds a test to ensure we are capturing usage of $set/$set_once
at the top level of the event, as posthog-js uses this method.
This was initiated by the issue mentioned
[here](https://github.com/PostHog/posthog-js/issues/615).
* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* test(person-overrides): Update tests to match new constraint
* fix(migration): Add drop extension query to reverse migration
* fix(migration): Use correct table name
* refactor(person-overrides): Make team a regular bigint field without FK
* refactor(person-overrides): Rename Helper model to Mapping
* feat(person-override): Add a helper model to indirectly reference person overrides
This allows us to use an exclusion constraint on the person overrides table instead of directly using a FK on posthog_person.
* wip: add test for concurrent updates to posthog_personoverride table
* refactor
* Update snapshots
* fix(isort): Correctly sort imports
* fix(ee-cohort-test): Delete person after creating it
* fix: person constraint in person overrides table (#14319)
* nits
* fix migration tests
* chore(migration): Bump migration number to 0302
* Update snapshots
* feat: person-overrides writes
* test(person-overrides): Add concurrent tests to person-overrides model
* feat(person-merge): Update merge to use new helper table
* fix(tests): Pass poEEmbraceJoin to updatePersonState in test
* fix(person-state): Format person-overrides message for ClickHouse
* test(api): Add function to reload dictionary for person overrides
* fix: poe final test failure
* refactor(person-state): Make failed attempts a class variable to
This allows us to mock it during testing as some tests require
immediate failures.
* fix(postgres-utils): Apply some magic changes lost to time
* fix(person-state): Join with helper table to return UUIDs
* fix(person-state): Use single quotes for UUID queries
* test(person-overrides): Skip test that doesn't work without a merge command
* test(person-state): Add a very complicated query to get UUIDs in a test
* test(person-state): Try waiting longer, this is flaky
* fix(migrations): Remove unused migration
* fix(person): Undo unneeded person model changes
* chore: Clean-up artifacts from rebase on model branch
* refactor(person-state): Rename mergeAttempts and don't read from ENV
* chore: Better clarify oldest_event usage in comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* refactor(person-state): Update version in queries
* fix(person-state): Use new mapping model instead of helper
* fix(migrations): Re-add constraint deleted on field drop
* revert: Re-enable test that was skipped
Test was originally skipped due to missing a $merge command, but now
$merge_dangerously is available, so the test should pass.
* test: Attempt to refresh dictionary in test
* Update query snapshots
* Update query snapshots
* test: Expect dictionaries to be refreshed before resuming test
* fix(test): Use alias property for $merge_dangerously in test
* fix(test): Expect any string like in all other tests
* test: Fix order of events to determine merged persons
* chore: Update comment regarding overrides mapping query
* chore: Remove out of date comment
* test: Set number of retries to 0 when updating persons
* test: Delete comment
* test: Assert clickhouse state after identify
* chore: Move comment
* chore: Delete TODO regarding oldest_event updates
* test: Assert events are still processed if merge fails
* test: Assert properties are processed if merge fails
* test: Don't expect to throw when failing merges
* fix: Indent RETURNING query clause
Co-authored-by: Harry Waye <harry@posthog.com>
* chore: Remove out of date comment
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* test: Resume merges after test suite
* test: Run processPersonStep tests for both poEEmbraceJoin modes
---------
Co-authored-by: Harry Waye <harry@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* test(plugin-server): use librdkafka for functional tests
While trying to port the session recordings to use node-librdkafka I
found it useful to first implement it in the functional tests.
* use obj destructuring to make calls more self explanatory
* refactor(recordings): remove session code from event pipeline
We have moved session recrodings to a separate topic and consumer. There
may be session recordings in the old topic, but we divert these to the
new logic for processing them.
* refactor to just send to the new topic!
* fix import
* remove empty line
* fix no team_id test
* implement recordings opt in
* remove old $snapshot unit tests
* remove performance tests
* Update plugin-server/functional_tests/session-recordings.test.ts
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* Update plugin-server/functional_tests/session-recordings.test.ts
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* add back $snapshot format test
* Add comment re functional test assumptions
Co-authored-by: Tiina Turban <tiina303@gmail.com>
* fix(session-recordings): fix missing distinct_id for session recordings
Previously I'd assumed that the distinct_id would be in the event.
That's not true, rather it is at the top level of the Kafka message
value JSON.
This commit fixes that, and also updates all functional tests to not
include the `distinct_id` within the event body.
* Revert "chore(session-recordings): revert to sending events to old topic (#13756)"
This reverts commit 41874de277.
* add test for session without team_id, only token
* pull out event names as variable
* Change info -> debug otherwise its very noisy
* chore(session-recordings): separate topics for events as recordings
WIP
* fix tests
* Use simpler consumer for session recordings
* wip
* still batch things by batchSize
* add tests, improve comments
* rename topic var
* push performance_events to session recordings topic also
* Add completely separate consumer for session-recordings
* wip
* use session_id for partition key
* fix test
* handle team_id/token null
* wip
* fix tests
* wip
* use kafka_topic var in logs
* use logger
* fix test
* Fix $performance_event topic usage
* fix tests
* fix check for null/undefined
* Update posthog/api/capture.py
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* Add test for kafka error handling
* Remove falsy teamId check
* fix statsd error
* kick ci
* Use existing getTeamByToken
* remove partition key from recordings
* Make sure producer is connected !
* fix session id kafka key test
* add back throws!
* set producer on each test
* skip flaky test
* add flush error logs
* wait for persons to be ingested
* fix skip
Co-authored-by: Tomás Farías Santana <tomas@tomasfarias.dev>
* fix(historical-exports): preserve sent_at in exports
It seems that when exporting historical events (and likely all events),
the sent_at field is not preserved, rather we were using the
`created_at` date which by the looks is what we set to `now()` as it
goes through the event processing pipeline. This means timestamps on
exported events will be slightly different, depending how much lag there
was between sending the event and processing the event. It also means,
e.g. that if you were to export events twice, then we'd likely end up
with very off timestamps on the final export.
This commit fixes this by using the `timestamp` field that we get from
the ClickHouse events table.
We also add a test to ensure that the capture endpoint allows for
setting all the appropriate data needed for exports to function as
expected.
This is in relation to
https://posthogusers.slack.com/archives/CTLTM70RM/p1670577437942799
* remove sent_at from historical export
* remove comment re sent_at
* update comment
* update comment
* wip
* Update plugin-server/src/worker/ingestion/timestamps.ts
Co-authored-by: Xavier Vello <xavier@posthog.com>
* Update plugin-server/functional_tests/exports-v2.test.ts
Co-authored-by: Xavier Vello <xavier@posthog.com>
Co-authored-by: Xavier Vello <xavier@posthog.com>
* chore(plugin-server): Add metrics for time of last processed message
Previously we have been alerting on Kafka consumer group offset lag.
However, really we care about the delay between messages being written
to Kafka and being processed by the plugin server.
By adding the last processed timestamp, as a gauge, we can then alert on
if that time and now is greater than a threshold.
This alert would not require the plugin-server to be up to trigger, just
that there be some time registered so it handles complete failure also.
For the case that there are no messages past the committed offsets, we
will end up triggering the alert if we do not also take into
consideration the production rate into the topic.
* wip
* wip
* fix imports order
* fix group id
* Add and use waitForExpect instead
* remove yarn.lock
* move comment
* wip
* add support for token field in kafka message
* formPipelineEvent
* rename pipeline files according to new order
* wip team_id and anonymize ips
* conditional handlers and tests
* some plugin server fixes
* fix capture bug
* fix
* more fixes
* fix capture tests
* pipeline update
* fix + investigate database resets
* fix import order
* testing and typing updates
* add test for capture endpoint
* testing
* python typing
* plugin server test
* functional test
* fix test
* another fix
* make sure no team ids clash in tests
* fix
* add more metrics and logs
* cache nulls
* updates
* add more metrics
* refactor(plugin-server): separate api from functional_tests
This just moves the api helpers to a separate file, such that we can
import from other files.
* test(plugin-server): add functional tests for property definitions
I was going to take a stab at
https://github.com/PostHog/posthog/issues/12529 but I wasn't sure how
the definition bits worked, so thought I'd add some tests first.
This doesn't just add tests but also:
1. starts demonstrating how we can split up the tests into
different files, thereby also allowing jest test isolation.
2. removes --runInBand, such that isolated tests can run in parallel
