0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-12-01 04:12:23 +01:00
posthog/plugin-server/tests/postgres/e2e.timeout.test.ts

65 lines
2.6 KiB
TypeScript
Raw Normal View History

import { startPluginsServer } from '../../src/main/pluginsServer'
import { Hub, LogLevel } from '../../src/types'
import { UUIDT } from '../../src/utils/utils'
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
import { makePiscina } from '../../src/worker/piscina'
import { createPosthog, DummyPostHog } from '../../src/worker/vm/extensions/posthog'
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
import { pluginConfig39 } from '../helpers/plugins'
import { resetTestDatabase } from '../helpers/sql'
import { delayUntilEventIngested } from '../shared/process-event'
jest.setTimeout(60000) // 60 sec timeout
describe('e2e postgres ingestion timeout', () => {
let hub: Hub
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
let stopServer: () => Promise<void>
let posthog: DummyPostHog
beforeEach(async () => {
await resetTestDatabase(`
async function processEvent (event) {
await new Promise(resolve => __jestSetTimeout(() => resolve(), 800))
await new Promise(resolve => __jestSetTimeout(() => resolve(), 800))
await new Promise(resolve => __jestSetTimeout(() => resolve(), 800))
await new Promise(resolve => __jestSetTimeout(() => resolve(), 800))
await new Promise(resolve => __jestSetTimeout(() => resolve(), 800))
event.properties = { passed: true }
return event
}
`)
const startResponse = await startPluginsServer(
{
WORKER_CONCURRENCY: 2,
TASK_TIMEOUT: 2,
PLUGINS_CELERY_QUEUE: 'test-plugins-celery-queue',
CELERY_DEFAULT_QUEUE: 'test-celery-default-queue',
LOG_LEVEL: LogLevel.Log,
KAFKA_ENABLED: false,
},
makePiscina
)
hub = startResponse.hub
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
stopServer = startResponse.stop
const redis = await hub.redisPool.acquire()
await redis.del(hub.PLUGINS_CELERY_QUEUE)
await redis.del(hub.CELERY_DEFAULT_QUEUE)
await hub.redisPool.release(redis)
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
posthog = createPosthog(hub, pluginConfig39)
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
})
afterEach(async () => {
await stopServer()
})
test('event captured, processed, ingested', async () => {
expect((await hub.db.fetchEvents()).length).toBe(0)
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
const uuid = new UUIDT().toString()
await posthog.capture('custom event', { name: 'haha', uuid, randomProperty: 'lololo' })
await delayUntilEventIngested(() => hub.db.fetchEvents())
const events = await hub.db.fetchEvents()
[plugin-server] Babel Loop Timeouts (https://github.com/PostHog/plugin-server/pull/155) * transpile code via babel-standalone * move vm.ts into vm/ * add TASK_TIMEOUT config * add babel transform to timeout while loops * update refactored rename * vm while timeout test * add more unimplemented timeout tests * add transforms for while/for/do-while loops * fix loop timeout ms * remove extra ; * fix import * fix more imports * fix even more imports * fix error messages * simplify errors * small refactor * fix import * add async guard and protect against long promises * async guard around setupPlugin() * add column * slightly safer variables * less noise in vm bench * can not override async guard for the main functions (e.g. processEvent, etc) * reduce how much we do in a benchmark * add types to loop timeout * types for promise timeout * fix line/column numbers * explain some decisions * verify that the process event timeout applies e2e. * managed to get equal, so changing * update message that it's just a warning * add e2e kafka test for bad delay * shorter test in github action * increase test timeout to see if that makes a difference (locally it takes 3min for both tests) * skip the "slow on GH action" test for now * process kafka events in parallel by default * add more metrics to kafka queue * some debug to help fix the test * add debug log in the delayUntilEventIngested function * add "single_event_batch" timing to the event processing steps * fix timer * Improve timeoutGuard default timeout * revert benchmark to the last one that worked * skip bad delay * add back a 1:1 copy of the e2e.kafka test, but with the timeout code. see if GH actions run * remove broken tests, improve logging of working tests * Clan up vm.ts * Improve clarity of loopTimeout * Refactor transforms slightly * Refactor transform call to secureCode func * Add secureCode tests Co-authored-by: Michael Matloka <dev@twixes.com>
2021-02-18 08:52:25 +01:00
expect(events.length).toBe(1)
expect(events[0].properties.name).toEqual('haha')
expect(events[0].properties.passed).not.toEqual(true)
})
})