* chore(plugin-server): disrtibute scheduled tasks
Changes I've made here from the original PR:
1. add some logging of task run times
2. add concurrency, except only one task of a plugin will run at a time
3. add a timeout to task run times
This reverts commit 23db43a0dc.
* chore: add timings for scheduled tasks runtime
* chore: add timeouts for scheduled tasks
* chore: clarify duration unit
* chore: deduplicate tasks in a batch, add partition concurrency
* chore: add flag to switch between old and new behaviour
This defaults to new, but can be set to old by setting environment
variable `USE_KAFKA_FOR_SCHEDULED_TASKS=false`
* fix tests
* enable USE_KAFKA_FOR_SCHEDULED_TASKS in tests
* ci(plugin-server): fix functional tests running forever
Seems that one of the changes I made resulted in the tests running
forever in GitHub Actions.
* make sure pino transport closed in workers
* chore(plugin-server): split functional tests into feature based files
This is intended to make it more obvious what we are testing, and to try
and identify the major themes of the plugin-server functionality.
As a by product it should make things more parallelizable for jest as
the tests in different files will be isolated, runnable in separate
workers.
* use random api token, avoid db constraints
* make tests silent
* format
* chore: set number of jest workers
These tests should be pretty light given they just hit other APIs and
don't do much themselves. Memory could be an issue on constrained
environments. We shall see.
* ci(plugin-server): fun functional tests in parallel
By running the tests in parallel we should be able to keep test times
pretty low. The only test that is a little awkward is the
`runEveryMinute` 🤔 maybe we can do some something else for this
test.
* run migrations with test = true
* wip
* wip
* wip
* wip
* wip
* wip
* wip
Doesn't try to do any comparison to base yet although that would be
great, but as it stands it offers some useful insights into where we
might be missing coverage.
