* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
* fix(plugin-server): add shutdown failsave timeout
We were having an issue where a Kafka version upgrade caused the
plugin-server to hang on shutdown. This was because the Kafka producer
was not able to flush its messages to Kafka. This commit adds a timeout
to the shutdown process, so that the plugin-server will exit even if
something hangs.
See https://posthog.slack.com/archives/C046SERP277/p1667878901739969 for
details
I wasn't exactly sure how to make this a little more testable, but there
is clearly a blind spot for certain failure cases.
* Actually call the shutdown sequence
* wip
* Add PLUGIN_SERVER_MODE
* Make capabilities dependent on PLUGIN_SERVER_MODE
* Subscribe to kafka-events topic
* runAsyncHandlersEventPipeline
* Test fixup: fix typing error
* Test fixup: flush right after queueing message
* Parse clickhouse event correctly
* Different consumer group ids for kafka queue based on mode
* Set different prompts for different modes
* Capability for http, disabled in tests
* Elements chain handling in async ingestion
* Test for runner.test.ts
* Update a snapshot
* Update plugin-server/README.md
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* Solve review-related issues
* Fix a test
* Fix imports
* Capabilities test fix
* Update tests
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* show last event ingested timestamp in system status
* wip plugin server healthcheck endpoint
* plugin server health endpoint
* fix readme prettier issue
* update tests
* move server.listen
* fixes, tests
* longer staleness default
* change endpoint to _health
