* feat(object_storage): add unused object storage with health checks
* only prompt debug users if object storage not available at preflight
* safe plugin server health check for unused object storage
* explicit object storage settings
* explicit object storage settings
* explicit object storage settings
* downgrade pip tools
* without spaces?
* like this?
* without updating pip?
* remove object_storage from dev volumes
* named volume on hobby
* lazily init object storage
* simplify conditional check
* reproduced error locally
* reproduced error locally
* object_storage_endpoint not host and port
* log more when checking kafka and clickhouse
* don't filter docker output
* add kafka to hosts before starting stack?
* silly cloud tests (not my brain)
* create plugin source model
* edit source via plugin_source model
* deprecate "source"
* test plugin source updates
* add support for index.ts and index.js
* refactor plugin loading, support plugin sources from db
* fix source code in tests
* remove transpilation code
* reload plugin after saving
* store defaults in the db instead of persisting in form
* remove fields that don't exist
* feat(apps): transpile frontend.tsx
* update timestamp even if no local fields changed
* yarn.lock
* remove unused fields
* commit suggestion
* rename to PluginSourceFile
* rename to PluginSourceFile
* remove dead code
* fix code feedback
* add comments
* make it safer to call
* convert to upsert
* convert to upsert
* comment on the null
* revert plugin server changes
* get source from new model behind the scenes
* simplify
* revert accidental changes
* revert accidental changes
* remove defaults
* sync the old source field
* use the source model in the plugin server
* cache the source from pluginsourcefile
* test that getPluginSource gets data from the database
* safer null check that doesn't override `{}` in the db with `null`
* use enum
* fix error from migration 0233 ([] is falsy in python!)
* fix merge
* error as null
* use the explicit "source__index_ts" field
* cache transpiled code
* remove unused type
* cleaner query
* remove extra line
* fix test
* feat(plugin-server): use swc for running jest tests
Every time we run jest tests, typescript is being compiled. This adds
~13-20s to _starting_ each test suite on my machine.
This change brings it down to 1s at the cost of only _transpiling_ the
code and not doing any typechecking.
The second overhead this fixes is for piscina: every worker thread
compiled its own code every time, speeding this up takes that from 4.4s
to under a second. See https://github.com/PostHog/posthog/pull/9777 for
original context
* Re-add ts-jest
* Simplify imports in `setup_test_environment`
* refactor(ingestion): Unify tests into single `process-event.test.ts`
* Update test scripts for CI
* Fix imports
* Run prettier
* Fix missing import
* Fix the OTHER imports
* Nuke `includeNewPropertiesUpdatesTests`
* (Temporarily) depend on a different version of plugin scaffold
* Change ingestEvent response schema, return PreIngestionEvent
* Refactor onEvent call pattern
* IP on preIngestionEvent
* Make a function generic
This way its return values can be used
* Deal with ProcessedPluginEvents not PluginEvents for onX plugin
functions
* Use beta versions of the package
* Update vm.test.ts
* Update dlq tests
* Continue calling onSnapshot
* Pass forward siteUrl
* yeet some code
* Refactor action matching & webhooks logic to work off of PreIngestionEvent
* Update site_url call pattern
* Remove console.log
* Remove a dead test
* fix hooks tests
* Fix ingest-event test
* update vm.test.ts
* rename a function
* Update plugin-scaffold
* only report failing tests
* move silent reporter into package json so we can add a test:noisy task
* Revert "move silent reporter into package json so we can add a test:noisy task"
This reverts commit 234ae024a0.
* chore(deps): upgrade node.js version to v16
* update npm version'
* try to update alpine
* Revert "try to update alpine"
This reverts commit 58024968e2.
* don't update alpine, use node-current instead:
* go back to npm 7
* upgrade @google-cloud packages
* handle unhandled rejections :D
* add handling for unhandled promise rejections in the main thread as well
* Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env var
This makes integrating with external clickhouse providers (like altinity
cloud) much easier as users don't need to use undocumented APIs to
upload the schemas.
* Keep underscores in protobuf. This allows using this payload for json ingestion
* Timestamp format
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* move ONE_HOUR to constants
* use sane_repr
* fix typo
* use plugin-server in same repo
* remove plugins dir
* update docker files
* clean up docker files:
* visual separation
* run plugin server test stack
* build and push test image
* revert push image
* fix default yarn start
* another fix
* fix some more
* add plugin-server-prod script
* make executable
* update scripts:
* revert prod script
* fixes
* fix tests
* clean
* move build to docker
* build plugin-server in prod dockerfile
* use different entrypoint for e2e test
* fix bugs with ts-node-dev
* restore yarn.lock
* lost a few changes in the rebase
* fix dockerfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
