0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-29 03:04:16 +01:00
Commit Graph

326 Commits

Author SHA1 Message Date
Yakko Majuri
ca8c4d0271
Revert "feat: buffer 3.0 (graphile) (#10735)" (#10802)
This reverts commit 9a2a9046cb.
2022-07-14 18:24:58 +00:00
Yakko Majuri
9a2a9046cb
feat: buffer 3.0 (graphile) (#10735)
* feat: buffer 3.0 (graphile)

* fixes

* test

* address review

* add test for buffer processAt
2022-07-13 11:32:00 +00:00
Yakko Majuri
985148ee7e
feat: buffer 2.0 (#10653)
* feat: buffer 2.0 proposal

* add tests

* prevent infinite retrying

* perf

* updates

* tweaks

* Update latest_migrations.manifest

* Update plugin-server/src/main/ingestion-queues/buffer.ts

* update

* updates

* fix migrations issue

* reliability uopdates

* fix tests

* test fix

* e2e test

* test

* test

* ??

* cleanup
2022-07-08 10:48:25 +00:00
Yakko Majuri
58a1fea111
fix: handle stale batches in buffer (#10643)
* Revert "Revert "fix: handle stale batches in buffer (#10641)" (#10642)"

This reverts commit b564688ad8.

* fix test
2022-07-05 18:16:49 +00:00
Michael Matloka
b04015f25e
chore(plugin-server): Consume from buffer topic (#10475)
* chore(plugin-server): Consume from buffer topic

* Refactor `posthog` extension for buffering

* Properly form `bufferEvent` and don't throw error

* Add E2E test

* Test buffer more end-to-end and properly

* Put buffer-enabled test in a separate file

* Update each-batch.test.ts

* Test that the event goes through the buffer topic

* Fix formatting

* Refactor out `spyOnKafka()`

* Ensure reliability batching-wise

* Send heartbeats every so often

* Make test less flaky

* Commit offsets if necessary before sleep too

* Update tests

* Use seek-based mechanism (with KafkaJS 2.0.2)

* Add comment to clarify seeking

* Update each-batch.test.ts

* Make minor improvements
2022-06-28 13:30:10 +02:00
Yakko Majuri
a598c7b664
feat(persons-on-events): cache + send persons and groups created_at with events (#10457)
* feat(persons-on-events): cache + send persons and groups created_at with events

* more testing

* Update plugin-server/src/utils/db/db.ts

* better naming

* fixes

* testing

* update test
2022-06-27 11:39:58 +00:00
Neil Kakkar
9712fd9bb5
chore(feature-flags): Upsert hash key overrides on people merges (#10418) 2022-06-24 10:58:42 +01:00
Karl-Aksel Puulmann
773f922eef
feat(apps): Remove onAction plugin function (#10414)
* Remove onAction

* Avoid fetching actions that dont deal with REST - 99% reduction

* Plural hooks

* Avoid hook fetching where not needed

* Remove dead code

* Update lazy VM test

* Rename a function

* Update README

* Explicit reload actions in tests

* Only reload actions which are relevant for plugin server

* Remove excessive logging

* Reload actions when hooks are updated

* update action matching tests

* Remove commented code

* Solve naming issues
2022-06-24 12:29:10 +03:00
Karl-Aksel Puulmann
f4668ed855
refactor(plugin-server): move buffer as first step of event pipeline & more (#10360)
* WIP: Move person creation earlier

* WIP: move person updating, handle person property changing

* WIP: leverage person information

* Update `updatePersonDeprecated` signature

* Avoid (and test avoiding) unneeded lookups whether 'creating' person is needed

Note there were two tricky interactions within handleIdentify, which
again got solved by indirect message passing.

* Solve TODO

* Normalize event before updatePersonIfTouchedByPlugins

* Avoid another lookup for person in updatePersonProperties

* Avoid lookup for newPerson in handleIdentifyOrAlias

* Add kludge comments

* Fix runBufferEventPipeline

* Rename upsertPersonsStep => processPersonsStep

* Update emitToBufferStep tests

* Update some event pipeline step tests

* Update prepareEventStep tests

* Test processPersonStep

* Add tests for updatePersonIfTouchedByPlugins step

* Update runner tests

* verify person vesrion in event-pipeline-integration test

* Update process-event test suite

* Argument ordering for person state tests

* Update runner test snapshots

* Cast to UTC

* Fixup person-state tests

* Dont refetch persons needlessly on $identify

* Add missing version assertion

* Cast everything to UTC

* Remove version assertion

* Undo radical change to event pipeline - will re-add it later!

* Resolve comments
2022-06-23 10:27:01 +03:00
Tiina Turban
c659bad2ef
Revert "revert: Rollout ingestion batch breakup by distinctId (#10393)" (#10398)
This reverts commit 744d4ddf84.
2022-06-21 14:34:45 -07:00
Michael Matloka
744d4ddf84
revert: Rollout ingestion batch breakup by distinctId (#10393)
This reverts commit 9a085cb1f6.
2022-06-21 19:06:31 +02:00
Tiina Turban
9a085cb1f6
chore: Rollout ingestion batch breakup by distinctId (#10370)
* chore: Rollout ingetion batch breakup by distinctId

* Update task-definition.plugins-ingestion.json

Co-authored-by: Michael Matloka <dev@twixes.com>
2022-06-21 17:31:53 +02:00
Karl-Aksel Puulmann
dea8c6973a
perf(plugin-server): reduce number of person lookups in the event pipeline (#10324)
* Return person in PreIngestionEvent if possible

* Avoid unneccessarily fetching person in emitToBufferStep

* Avoid unneccessarily fetching person in createEvent

* Use unified type instead of separate type for cached data

* Pass person info forward explicitly in each event-pipeline step

* minor typing improvement

* Remove person from type

* Remove unneeded `undefined`

* Add person check for prepareEventStep test

* Fix hook test

* Update getPersonData tests

* Cast created_at to UTC

* Cast created_at to utc on fetch

* Remove personUuid var - unneeded

* Add unit tests for process-event.ts#createEvent
2022-06-21 09:18:22 +03:00
Michael Matloka
313226838c
revert: revert: Revert person properties updates refactor (#10349)
* Revert "revert: Revert person properties updates refactor (#10348)"

This reverts commit 6b3c4691b3.

* sanitizeEvent -> normalizeEvent

* Ensure we handle property updates from within plugins, test

Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
2022-06-20 09:49:11 +03:00
Neil Kakkar
6b3c4691b3
revert: Revert person properties updates refactor (#10348) 2022-06-17 17:48:20 +02:00
Karl-Aksel Puulmann
d6ec3aedc6
refactor(plugin-server): person state updating (#10321)
* Remove some excessive call signatures

* refactor: move property sanitization outside of .capture

* Move event sanitization into event sanitization logic

* Move person creation/updating logic outside of capture/createSnapshot

* refactor: remove personManager from arguments

* refactor: remove various properties from arguments

* Update `handleIdentifyOrAlias` signature

* refactor: inline timeoutGuard into personStateManager

* refactor: prefix pipeline steps with indexes

* Extract timestamp parsing logic from process-event.ts

* refactor: move timestamp tests over from process-event.ts

* refactor: update process-event.test.ts

* refactor: person-state-manager -> person-state

* Move sanitizeEvent to a more suitable module

* Fix some process-event tests
2022-06-17 09:17:08 +03:00
Karl-Aksel Puulmann
b4fee54222
refactor(plugin-server): extract person creation/handling logic from EventsProcessor (#10271)
* refactor: Start with PersonStateManager

* refactor: move createPerson to new service

* refactor: move team fetching before aliasing

* refactor: move `createPersonIfDistinctIdIsNew`

* refactor: move `updatePersonProperties`

* refactor: move `handleIdentifyOrAlias`

* refactor: `createPerson` to private

* Fix an import

* Remove weird mocking in an e2e integration test
2022-06-14 11:51:58 +03:00
Karl-Aksel Puulmann
c51a2f7bc1
fix(plugin-server): use histogram in metrics properly (#10276) 2022-06-13 16:48:57 +03:00
Karl-Aksel Puulmann
ebfc8251a7
fix(plugin-server): Properly set version in deletePerson (#10207)
* Use correct style for querying postgres

* Add test showing problems with deletePerson logic

* Fix deleting persons from clickhouse

* Fix concurrent tests

* Version + 100

* Fixup FINAL

* Remove console.log
2022-06-13 12:58:44 +03:00
Tiina Turban
569b50b4ec
feat(plugin-server): batching ingestion events to single process per distinct id (#10071) 2022-06-08 19:20:40 +02:00
Yakko Majuri
004ba66349
fix: pass ISO timestamp to onAction/onEvent (#10178)
* fix: pass ISO timestamp to onAction/onEvent

* fix prettier

* fix import

* update timestamps
2022-06-08 11:05:54 +00:00
Karl-Aksel Puulmann
c3c5eaad02
fix(plugin-server): properly unparse event.properties when PLUGIN_SERVER_MODE=async (#10156)
* Handle string properties in plugin-server convertToIngestionEvent

* Update typing

* fix: Add multi-server process event test

This got accidentally yeeted from my previous PR. Shame!

* Improve tests

* Update test to reflect reality
2022-06-07 13:42:07 +03:00
Karl-Aksel Puulmann
59797efce8
refactor(plugin-server): yeet element-group related postgres code (#10161) 2022-06-07 12:23:20 +03:00
Tiina Turban
95c9045cc1
fix: Postgres and CH Person version across other columns to match (#10135) 2022-06-06 17:57:03 +02:00
Tiina Turban
26435cb70d
feat: Add version column to person in CH (#10117) 2022-06-06 13:42:39 +02:00
Michael Matloka
64317238e6
refactor: Eliminate the KAFKA_ENABLED setting (#10059)
* refactor: Eliminate the `KAFKA_ENABLED` setting

* Remove dead code

* Consolidate plugin server test scripts and CI

* Fix CI command

* Remove Celery queues

* Rearrange test directories

* Update import paths
2022-05-30 18:39:33 +00:00