0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-22 08:40:03 +01:00
Commit Graph

42 Commits

Author SHA1 Message Date
Tiina Turban
60ba76d9c4
feat: Part 1 of ability to spin up different pipelines (#23601) 2024-07-17 14:11:49 +02:00
Ben White
47b2882412
chore: Debugging options for V3 ingester (#20569) 2024-02-27 11:44:08 +00:00
Brett Hoerner
e20771ee92
chore(plugin-server): remove unused MAX_PENDING_PROMISES_PER_WORKER (#20097) 2024-02-02 07:21:57 -07:00
Brett Hoerner
9346ae9dd3
chore(plugin-server): README tweaks (#19785) 2024-01-16 15:41:17 +00:00
Brett Hoerner
1f6ce77e66
chore(plugin-server): remove statsd (#19027) 2023-12-05 11:47:12 -07:00
ted kaemming
9299aa09e5
fix(plugin-server): Remove Postgres-based plugin error logging in favor of existing ClickHouse-based approaches (#18764) 2023-11-27 10:41:36 -08:00
Brett Hoerner
d16408784f
fix(plugin-server): fix unicode null byte blowing up the pipeline (#18282) 2023-10-30 16:39:10 -06:00
Brett Hoerner
d68fa14f10
chore(plugin-server): run node-rdkafka with cooperative rebalancing patched in (#17747)
* Remove node-rdkafka-acosom

* Add node-rdkafka

* Replace node-rdkafka-acosom imports with node-rdkafka

* Patch node-rdkafka with changes from https://github.com/PostHog/node-rdkafka/

* Add patch directions to README
2023-10-10 08:13:05 -06:00
Tiina Turban
f5304fc222
chore: nuke unused onSnapshot (#16531) 2023-07-13 12:03:59 +02:00
danielxnj
319708a985
chore: Danielj/split plugins helm chart revert (#16466)
* Revert "chore: Danielj/split plugins helm chart (#16459)"

This reverts commit 27574c00a1.

* Enable workflow dispatch
2023-07-10 12:45:30 -04:00
danielxnj
27574c00a1
chore: Danielj/split plugins helm chart (#16459)
* Trigger plugins deployment when folder is updated

* Update repository dispatch

* Use digest as tag

* Remove empty line

* Create a new workflow

* Change concurrency

* Revert "Remove empty line"

This reverts commit b307c50cb0.

* Revert "Change concurrency"

This reverts commit 458597b80b.

* Revert "Create a new workflow"

This reverts commit 4e1eef73f2.

* Remove empty line

* Dummy commit to trigger deploy

* Have a digest per platform

* Revert "Have a digest per platform"

This reverts commit 8276bd20a0.

* Revert "Revert "Have a digest per platform""

This reverts commit bc6527c48f.

* Have a digest per platform

* Add meta

* Update ci workflow

* Add workflow dispatch

* Dummy commit
2023-07-10 12:18:09 -04:00
danielxnj
74c558d66e
chore: Revert "chore: Trigger plugins deployment when folder is updated deploy " (#16437)
Revert "chore: Trigger plugins deployment when folder is updated deploy  (#16141)"

This reverts commit 53e6b7e413.
2023-07-07 11:13:08 -04:00
danielxnj
53e6b7e413
chore: Trigger plugins deployment when folder is updated deploy (#16141)
* Trigger plugins deployment when folder is updated
2023-07-07 09:19:08 -04:00
Harry Waye
e2da984ff5
chore: remove healthcheck code (#15478)
* chore: remove healthcheck code

It's not used anywhere. We're rather using healthcheck http probes.
2023-05-11 12:20:24 +01:00
Michael Matloka
08eab1f9c0
style: Fix "Posthog" capitalization (#15426)
* style: Fix "Posthog" capitalization

* Update UI snapshots for `chromium` (1)

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-05-09 18:31:34 +02:00
Harry Waye
d0512aec84
feat(recordings): add RECORDING_PARTITIONS_CONSUMED_CONCURRENTLY (#14395)
Allow recordings consumer to be configured to consume multiple
partitions concurrently. This should allow use to use a smaller
deployment for session recordings. It's pretty much mostly IO except for
the JSON marshalling/unmarshalling.
2023-02-24 10:51:30 +00:00
Karl-Aksel Puulmann
4b1ed10ab1
feat(query-performance): Support running some workloads on different nodes (#13493)
* Support creating and using different connection pools for clickhouse in python

* Tag queries with the workload

* Add test_connection

* Mark events list queries to go to offline cluster

* Run app historical exports as an offline workload

* Format README.md

* Fix call

* Add comment
2022-12-29 10:22:56 +02:00
Thomas Obermüller
c15357e3fc
docs(playwright): replace yarn with pnpm in readme (#13278)
* docs(playwright): replace yarn with pnpm in readme

* use quotes for url

* replace for plugin-server
2022-12-13 10:03:50 +01:00
Thomas Obermüller
4a30e78b22
chore: use pnpm to manage dependencies (closes #12635) (#13190)
* chore: use pnpm to manage dependencies

* Fix CI errors

* Don't report Docker image size for external PRs

* Fix pnpm-lock.yaml formatting

* Fix module versions

* Ignore pnpm-lock.yaml

* Upgrade Cypress action for pnpm support

* Set up node and pnpm before Cypress

* Fix typescript issues

* Include patches directory in Dockerfile

* Fix Jest tests in CI

* Update lockfile

* Update lockfile

* Clean up Dockerfile

* Update pnpm-lock.yaml to reflect current package.json files

* remove yarn-error.log from .gitignore

* formatting

* update data exploration readme

* type jest.config.ts

* fix @react-hook issues for jest

* fix react-syntax-highlighter issues for jest

* fix jest issues from query-selector-shadow-dom

* fix transform ignore patterns and undo previous fixes

* add missing storybook peer dependencies

* fix nullish coalescing operator for storybook

* reorder storybook plugins

* update editor-update-tsd warning to new npm script

* use legacy ssl for chromatic / node 18 compatibility

* use pnpm for visual regression testing workflow

* use node 16 for chromatic

* add @babel/plugin-proposal-nullish-coalescing-operator as direct dependency

* try fix for plugin-server

* cleanup

* fix comment and warning

* update more comments

* update playwright dockerfile

* update plugin source types

* conditional image size reporting

* revert react-native instructions

* less restrictive pnpm verions

* use ref component name in line with style guide

Co-authored-by: Jacob Gillespie <jacobwgillespie@gmail.com>
2022-12-12 10:28:06 +01:00
Tiina Turban
4635648eb0
fix: functional tests running instructions (#13135) 2022-12-05 19:30:37 +01:00
Harry Waye
ca1edeec01
chore(plugin-server): add docs for running functional tests (#13029)
To make it more easy to run these tests, I've also added defaults for
certain config variables for when run in dev mode.

These tests were already run in production via
`plugin-server/bin/ci_functional_tests.sh` but it isn't obvious how they
should be run locally. You can of course run `ci_functional_tests.sh`
locally but this is not particularly great developer experience, i.e.
the iteration time is pretty large.
2022-11-30 12:23:03 +00:00
Karl-Aksel Puulmann
0d9b0bfb56
chore: Kill internal metrics (#12943)
* Remove capturing internal metrics

* Remove capturing internal metrics from FE

* Remove internal metrics from plugin-server

* Remove internal metrics from plugin-server

* Remove internal metrics internal dashboard type

* Clean up Dashboard.tsx

* Remove unused vars

* Remove endpoint, update imports

* Update rate limiting tests

* Update calculate event property usage tests
2022-11-28 10:34:27 +02:00
Yakko Majuri
b89a2f8e28
refactor: rename Graphile to Graphile Worker (#12324)
* refactor: rename Graphile to Graphile Worker

* revert env var changes

* one more

* format readme

* add changed files
2022-10-19 11:38:25 -03:00
Jacob Pargin
39e4e59785
chore(yarn): update plugin server dev start instructions (#10706) 2022-07-26 11:15:29 +02:00
Karl-Aksel Puulmann
773f922eef
feat(apps): Remove onAction plugin function (#10414)
* Remove onAction

* Avoid fetching actions that dont deal with REST - 99% reduction

* Plural hooks

* Avoid hook fetching where not needed

* Remove dead code

* Update lazy VM test

* Rename a function

* Update README

* Explicit reload actions in tests

* Only reload actions which are relevant for plugin server

* Remove excessive logging

* Reload actions when hooks are updated

* update action matching tests

* Remove commented code

* Solve naming issues
2022-06-24 12:29:10 +03:00
Michael Matloka
c94e77cee6
refactor(plugin-server): Remove benchmarks (#10065) 2022-05-31 12:37:26 +02:00
Michael Matloka
64317238e6
refactor: Eliminate the KAFKA_ENABLED setting (#10059)
* refactor: Eliminate the `KAFKA_ENABLED` setting

* Remove dead code

* Consolidate plugin server test scripts and CI

* Fix CI command

* Remove Celery queues

* Rearrange test directories

* Update import paths
2022-05-30 18:39:33 +00:00
Tiina Turban
735ceb33a5
refactor(plugin-server): yeet postgres plugin logs (#9881)
* refactor(plugin-server): yeet postgres plugin logs

* forgotten usage

* Remove more Postgres `PluginLogEntry` code

* Modernize tests

* Wait for `KafkaEngine` in `console.test.ts`

* Fix formatting

* Add `resetKafka`

* Update console.test.ts

* Update style

* Use `delayUntilEventIngested` and improve its ergonomics

* Fix tests

* Refactor `test:postgres:2` away

* Fix and optimize `console.test.ts`

* Rearrange things a little bit

* Revert "Fix and optimize `console.test.ts`"

This reverts commit 273dbc1244.

* Rework `console` test to avoid Kafka

* Update `vm.test.ts` to spy on `queueSingleJsonMessage`

* Improve `.queueSingleJsonMessage()` typing

* Try to fix `moveDistinctIds & deletePerson`

Co-authored-by: Michael Matloka <dev@twixes.com>
2022-05-30 13:09:13 +03:00
Karl-Aksel Puulmann
f2ddc532d3
feat(plugin-server): Allow running plugin server in a different mode (#9946)
* Add PLUGIN_SERVER_MODE

* Make capabilities dependent on PLUGIN_SERVER_MODE

* Subscribe to kafka-events topic

* runAsyncHandlersEventPipeline

* Test fixup: fix typing error

* Test fixup: flush right after queueing message

* Parse clickhouse event correctly

* Different consumer group ids for kafka queue based on mode

* Set different prompts for different modes

* Capability for http, disabled in tests

* Elements chain handling in async ingestion

* Test for runner.test.ts

* Update a snapshot

* Update plugin-server/README.md

Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>

* Solve review-related issues

* Fix a test

* Fix imports

* Capabilities test fix

* Update tests

Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
2022-05-25 14:50:36 +03:00
Yakko Majuri
8afe7e859d
refactor(plugin-server): yeet all celery code (#9936)
* refactor(plugin-server): yeet all celery code

* more yeeting
2022-05-24 13:34:24 +02:00
Michael Matloka
9a389847c2
refactor(plugin-server): Remove --idle mode (#9798)
* refactor(plugin-server): Remove `--idle` mode

* Support `PLUGIN_SERVER_IDLE` in `bin/start-worker`

* Don't `export` needlessly in npm script
2022-05-16 16:22:49 +00:00
Karl-Aksel Puulmann
6c637397fc
refactor(plugin-server): extract capabilities for VMs and plugin server (#9714)
* Extract calculation method from vm/lazy.ts

* Add tests for capabilities.ts

* Move capabilities.ts under vm which it's more closely tied to

* Add (blank) capabilities object to hub

This will be used to check whether to start certain jobs

* Don't start plugin scheduled tasks unless have the capabilities

* scheduledTasks => pluginScheduledTasks in capabilities

* Cancel all jobs on schedule exit

* Remove capabilities code from services/schedule.ts

* scheduleControl => pluginScheduleControl

* Flip conditional

* Do less manual cleanup

* Add test for stopping node-schedule at exit

* Move cancelAllScheduledJobs

* Test capabilities check in server.test.ts

* Capabilities check for jobs

* Improve comment

* Rename method

* Rename jobs => processJobs

* Fix failed import
2022-05-11 09:01:54 +03:00
Yakko Majuri
e4a66770c0
perf(kafka): update events_plugin_ingestion partitioning and consumption (#9101)
* perf(kafka): update events_plugin_ingestion partitioning and consumption

* prettier readme

* set partition key on posthog.capture plugin calls, revert task definition concurrency default
2022-03-18 11:27:10 +00:00
Karl-Aksel Puulmann
40a1721d0b
fix: Simplify test setup for plugin-server (#9051)
We don't need multiple commands for setup, also verify TEST env is set.
2022-03-17 10:34:53 +02:00
Tim Glaser
90d30c3e60
Plugins: replace @current with team_id in api calls (#8751)
* Plugins: replace @current with team_id in api calls

* prettier

* Fix test

* reallyf ix test
2022-02-28 17:29:01 +00:00
Yakko Majuri
c7810cbab1
Refactor exportEvents buffer (#8573)
* redo export events buffer

* fix implementation

* don't return promise from trackPromise

* Update plugin-server/src/worker/vm/promise-manager.ts

* Promise.any -> Promise.race

* fix test?

* try to increase timeout?

* add a stupid assertion to silence TypeError
2022-02-16 15:07:05 +00:00
Yakko Majuri
003f0682c4
Plugin server health endpoint (#7983)
* show last event ingested timestamp in system status

* wip plugin server healthcheck endpoint

* plugin server health endpoint

* fix readme prettier issue

* update tests

* move server.listen

* fixes, tests

* longer staleness default

* change endpoint to _health
2022-01-26 09:30:01 -03:00
Karl-Aksel Puulmann
28d1c48959
Update package.json (#6741)
I've been running a custom version of package.json since forever.

Here's my suggestion for cleaning it up
2021-11-03 11:51:06 +02:00
James Greenhill
4976dfb9af
Remove deprecation warning 2021-10-28 15:03:35 -07:00
James Greenhill
434e379f9a Add 'plugin-server/' from commit '01a99a4e26b0b11f068a7073d6b94e53a7214d33'
git-subtree-dir: plugin-server
git-subtree-mainline: 776b056b6d
git-subtree-split: 01a99a4e26
2021-10-28 14:59:19 -07:00
James Greenhill
145937a435
Revert "Monorepo with updated history" 2021-10-28 14:55:17 -07:00
James Greenhill
65512ae16f
Pack up plugin-server for moving 2021-10-12 15:45:42 -07:00