* cleanup: remove unused team arg from registerLastStep
* cleanup: rename promises to ackPromises to make it more clear thats what they are
* cleanup(plugin-server): make waitForAck explicit/required
* add Kafka produce/ack metrics
* Clarify Kafka produce metric/labels
* chore(plugin-server): Validate fetch hostnames
* Only apply Python host check on Cloud
* Update tests to use valid hook URLs
* Only apply plugin server host check in prod
* Update URLs in a couple more tests
* Only check hostnames on Cloud and remove port check
* Fix fetch mocking
* Roll out hostname guard per project
* Fix fetch call assertions
* Make `fetchHostnameGuardTeams` optional
* chore: fix plugin server tests for acitonMatcher
I broke these but there was an issue with CI so they were merges broken
but now maybe they are fixed?
* more fixes
* wip
* wip
* wip
* chore(plugin-server): simplify action matcher deps
Specifically this only depends on postgres, so passing in the `DB`
object is unnecessary. This should make refactors easier to e.g. only
load the required dependencies when they are needed.
* pass only postgres to ActionMatcher
* Use same logic for `[person]` webhook token as person display in the app
* Allow accessing nested properties in webhook message
* Update hooks.test.ts
* Fix team fetching test
* Update query snapshots
* Update query snapshots
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* add support for token field in kafka message
* formPipelineEvent
* rename pipeline files according to new order
* wip team_id and anonymize ips
* conditional handlers and tests
* some plugin server fixes
* fix capture bug
* fix
* more fixes
* fix capture tests
* pipeline update
* fix + investigate database resets
* fix import order
* testing and typing updates
* add test for capture endpoint
* testing
* python typing
* plugin server test
* functional test
* fix test
* another fix
* make sure no team ids clash in tests
* fix
* add more metrics and logs
* cache nulls
* updates
* add more metrics
* fix(person-on-events): Fix groups caching in ingestion
We were seeing some groups-related events never get ingested in
playground. Digging in, it turned out that these events were serialized
with invalid timestamps due to cache containing dates in different
formats.
The bug was introduced in https://github.com/PostHog/posthog/pull/12403
and makes for a good case study for this common class of errors
There were multiple practices that could have indicated the error sooner:
1. Tests for the feature mocked out the DB and used a different data
format than what is used properly
2. Some methods related to caching were not properly updated to test the
caching logic
3. timestamps-as-strings: we deal with both ISO and clickhouse-format
timestamps, and the code didn't differentiate between them properly
4. `getGroupsColumns` signature was very loose, allowing for everything
to pass by
This change fixes the issue as well as updates relevant code to be more
in-line with best practices.
* Solve minor typing related issue
* refactor(plugin-server): simplify groups caching
* add multi groups test
* remove comments
* fix type, add debug
* fix
* stringify
* add groups created_at to types
* more test fixes
* use the right clickhouse timestampo format
* update created at to ch format in tests
* finally
* more fixes
* fix(plugin-server): Remove wild clickhouseQuery in ingestion pipeline
Point queries against clickhouse are slow and we should avoid them.
They're also not instrumented.
The postgres table already used in the method previously contains the
right data. Use that instead.
Reference: https://github.com/PostHog/posthog/blob/master/posthog/models/cohort/cohort.py#L274-L316
* Fixup and test doesPersonBelongToCohort
* Handle NULLs
* fix issues with fetchPerson() and add tests
- fetchPerson() returned extra columns that were not needed
* Add LazyPersonContainer class
* Load person data lazily through the event pipeline
* Make webhooks and action matching lazy
* Update runAsyncHandlersStep
* Return own person properties in process-event.ts
* Remove snapshots that caused pain
* Handle serialization of LazyPersonContainer
* Merge: Handle LHS only existing
.get() would be cached in that case not to do a query, which we can
avoid
* Serialize result args as well
* Make personContainer functional
* Resolve feedback
* Remove onAction
* Avoid fetching actions that dont deal with REST - 99% reduction
* Plural hooks
* Avoid hook fetching where not needed
* Remove dead code
* Update lazy VM test
* Rename a function
* Update README
* Explicit reload actions in tests
* Only reload actions which are relevant for plugin server
* Remove excessive logging
* Reload actions when hooks are updated
* update action matching tests
* Remove commented code
* Solve naming issues
* WIP: Move person creation earlier
* WIP: move person updating, handle person property changing
* WIP: leverage person information
* Update `updatePersonDeprecated` signature
* Avoid (and test avoiding) unneeded lookups whether 'creating' person is needed
Note there were two tricky interactions within handleIdentify, which
again got solved by indirect message passing.
* Solve TODO
* Normalize event before updatePersonIfTouchedByPlugins
* Avoid another lookup for person in updatePersonProperties
* Avoid lookup for newPerson in handleIdentifyOrAlias
* Add kludge comments
* Fix runBufferEventPipeline
* Rename upsertPersonsStep => processPersonsStep
* Update emitToBufferStep tests
* Update some event pipeline step tests
* Update prepareEventStep tests
* Test processPersonStep
* Add tests for updatePersonIfTouchedByPlugins step
* Update runner tests
* verify person vesrion in event-pipeline-integration test
* Update process-event test suite
* Argument ordering for person state tests
* Update runner test snapshots
* Cast to UTC
* Fixup person-state tests
* Dont refetch persons needlessly on $identify
* Add missing version assertion
* Cast everything to UTC
* Remove version assertion
* Undo radical change to event pipeline - will re-add it later!
* Resolve comments
* Return person in PreIngestionEvent if possible
* Avoid unneccessarily fetching person in emitToBufferStep
* Avoid unneccessarily fetching person in createEvent
* Use unified type instead of separate type for cached data
* Pass person info forward explicitly in each event-pipeline step
* minor typing improvement
* Remove person from type
* Remove unneeded `undefined`
* Add person check for prepareEventStep test
* Fix hook test
* Update getPersonData tests
* Cast created_at to UTC
* Cast created_at to utc on fetch
* Remove personUuid var - unneeded
* Add unit tests for process-event.ts#createEvent
* Use correct style for querying postgres
* Add test showing problems with deletePerson logic
* Fix deleting persons from clickhouse
* Fix concurrent tests
* Version + 100
* Fixup FINAL
* Remove console.log
* refactor: Eliminate the `KAFKA_ENABLED` setting
* Remove dead code
* Consolidate plugin server test scripts and CI
* Fix CI command
* Remove Celery queues
* Rearrange test directories
* Update import paths
