* chore(plugin-server): Validate fetch hostnames
* Only apply Python host check on Cloud
* Update tests to use valid hook URLs
* Only apply plugin server host check in prod
* Update URLs in a couple more tests
* Only check hostnames on Cloud and remove port check
* Fix fetch mocking
* Roll out hostname guard per project
* Fix fetch call assertions
* Make `fetchHostnameGuardTeams` optional
* chore(plugin-server): remove piscina workers
Using Piscina workers introduces complexity that would rather be
avoided. It does offer the ability to scale work across multiple CPUs,
but we can achieve this via starting multiple processes instead. It may
also provide some protection from deadlocking the worker process, which
I believe Piscina will handle by killing worker processes and
respawning, but we have K8s liveness checks that will also handle this.
This should simplify 1. prom metrics exporting, and 2. using
node-rdkafka.
* remove piscina from package.json
* use createWorker
* wip
* wip
* wip
* wip
* fix export test
* wip
* wip
* fix server stop tests
* wip
* mock process.exit everywhere
* fix health server tests
* Remove collectMetrics
* wip
* feat(ingestion): remove Graphile worker as initial ingest dependency
At the moment if the Graphile enqueing of an anonymous event fails e.g.
due to the database that it is uing to store scheduling information
fails, then we end up pushing the event to the Dead Letter Queue and to
not do anything with it further.
Here, instead of directly sending the event to the DB, we first push it
to Kafka, an `anonymous_events_buffer`, which is then committed to the
Graphile database. This means that if the Graphile DB is down, but then
comes back up, we will end up with the same results as if it was always
up*
(*) not entirely true as what is ingested also depends on the timings of
other events being ingested
* narrow typing for anonymous event consumer
* fix types import
* chore: add comment re todos for consumer
* wip
* wip
* wip
* wip
* wip
* wip
* fix typing
* Include error message in warning log
* Update plugin-server/jest.setup.fetch-mock.js
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* include warning icon
* fix crash message
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
* Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* setup event handlers as KafkaQueue
* chore: instrument buffer consumer
* missing import
* avoid passing hub to buffer consumer
* fix statsd reference.
* pass graphile explicitly
* explicitly cast
* add todo for buffer healthcheck
* set NODE_ENV=production
* Update comment re. failed batches
* fix: call flush on emitting to buffer.
* chore: flush to producer
* accept that we may drop some anonymous events
* Add metrics for enqueue error/enqueued
* fix comment
* chore: add CONVERSION_BUFFER_TOPIC_ENABLED_TEAMS to switch on buffer
topic
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
* Include kafka topic for setup
* Sample runEventPipeline/runBufferEventPipeline less frequently comparatively
This is done by duration - we still want the long transactions, but not
the short ones
* Trace enqueue plugin jobs
* Trace node-fetch
* Trace worker creation
* Various fixes
* Line up query tags properly
* Make fetch mocking work
* Resolve typing-related issues
