* Make clickhouse server image version configurable
* Attempt to use a reusable workflow for backend tests
* Try a composite action
* Add missing shell: bash
* Temporarily remove a step
* Cache id parameter
* Include the action when needing to rerun
* Move checking migrations to separate file
This didn't work in composite actions tests and slows down every
parallelized test execution for no reason. This should have been
separate in the first place.
* Rename job
* Improve check-migrations
* Start stack in the new job
* Remove shell
* Solve flaky test
* Remove unused file
* Create a test timings job
* Use checkout@v2 - https://github.com/cds-snc/github-actions/issues/23
* Ignore coverage files
* Save backend test durations
* Install pytest-split
* Clean up ci-backend
* Move actions code around
* wording improvement
* Save backend test durations
* Add a comment
* Make it possible to run tests with a different clickhouse version
* Remove debugging code
* Fix typo
* Boolean inputs please work
Fixes overeager test running as per https://github.com/actions/runner/issues/1483
Co-authored-by: PostHog Bot <hey@posthog.com>
* feat(healthchecks): add clickhouse as a dependency
* feat(healthchecks): add celery broker healthcheck
* feat(healthchecks): add cache healthcheck, plus some misc updates
* add missing typing
* add missing typing
* add role cast
* Add logging for kafka connectivity check
* actually add healthcheck as middleware
* fix postgres failure check
* simplify postgres migration return value
* verboise -> verbose
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* put interface validation at top of view
* remove unused imports
* show resp.content on status failures
* test(healthchecks): set log level to debug to pytest displays it
* restore logger level
* Bring up redis for backend ci tests
* up redis in non-cloud tests
Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com>
* add recordings to path query
* uncomment cache
* add clarifying comment
* works for start/end paths
* move to extra fields/properties
* add tests
* cleanup
* update ff name
* fix flaky test
* test and handle path_dropoff_key case
* Run queries against person_distinct_id2 when async migration is done
* Only write to clickhouse_person_unique_id topic if async migration is incomplete
* Update query snapshots
* Update plugin-server
* Adjust caching logic
* ci(e2e): manually split cypress tests to improve speed
This change simply lists the spec dir for e2e tests, chunks them into
5, and passes these specs to cypress explicitly.
I initially tried to use https://github.com/kamilkisiela/split-tests/blob/master/packages/cypress/README.md
but ran into
```
We invoked the function exported by `/home/runner/work/posthog/posthog/cypress/plugins/index.js`, but it threw an error.
TypeError: Cannot read property 'mochaFile' of null
at loadReports (/home/runner/work/posthog/posthog/node_modules/dist/cypress/src/index.js:14:44)
at module.exports (/home/runner/work/posthog/posthog/node_modules/dist/cypress/src/index.js:44:19)
at module.exports (/home/runner/work/posthog/posthog/cypress/plugins/index.js:27:14)
at /home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:96:12
at tryCatcher (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/method.js:39:29)
at load (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:93:7)
at EventEmitter.<anonymous> (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:204:5)
at EventEmitter.emit (events.js:315:20)
at process.<anonymous> (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/lib/plugins/util.js:19:22)
at process.emit (events.js:315:20)
at process.emit (/home/runner/.cache/Cypress/6.7.0/Cypress/resources/app/packages/server/node_modules/source-map-support/source-map-support.js:495:21)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:85:21)
```
And so decided to go for something "simpler"
See for details https://github.com/PostHog/posthog/runs/4443218344?check_suite_focus=true
* run on master, ensures node_modules cache in scope
* run deps install on master
* increase concurrency
* run yarn install in prep section
* move if to job
* still run yarn install
* refactor
* comments
* change name so we don't have to update requirements
* Update e2e.yml
Co-authored-by: Tim Glaser <tim@posthog.com>
* github action artifact cypress videos
* artifact logs
* Fix cypress video export
* Redirect all output
* wait for initial retention load
* specify retry in ci
* use bin/start instead of bin/docker-*
* capture django app output in gunicorn log
* use docker-worker and docker-server again
* Remove gunicorn capture output settings
* Create a new way to get distinct id queries thats gated by team_id
* Update most cases to use the new query
* Convert EVENT_JOIN_PERSON_SQL to new query
* Mostly convert GET_DISTINCT_IDS_BY_PERSON_ID_FILTER
* Mostly convert GET_DISTINCT_IDS_BY_PROPERTY_SQL
* Convert GET_PERSON_IDS_BY_FILTER
* Flag benchmarks
* Resolve circular imports
* Update a snapshot test
* Add a test for the new query logic
* dev(codespaces): add "lean in" docker configuration for codespaces
This changes creates a dedicated `docker-compose.yml` and `Dockerfile`
for codespaces and doesn't attempt to have compatability with the
existing docker config files. This is intentional and is designed to be
a close to what codespaces would consider the ideal, so as to understand
what great would look like.
At the moment it doesn't do, but could:
1. install python and node packages in the image
2. push this image up to dockerhub
3. specify this image to be used for caching. There is a "cacheFrom"
option in the [.devcontainer
reference](https://code.visualstudio.com/docs/remote/devcontainerjson-reference)
although I haven't checked how this works with using
`docker-compose.yml`. Perhaps we just need to add the `image:`
property to the `app` service :fingerscrossed:
4. specify default extensions and settings to be used.
* Use python 3.9, not the latest python 3 codespaces supports
* Add some basic python default settings and extensions
* add github codespaces image build workflow
* chore: update docker-compose cache from :cache tag
* chore: use docker cache-to inline
* add codespaces platform comment
* temp run on branch
* chore: fix flatforms format
* chore: update qemu comment
* chore: codespaces build use master image
* chore: target x86_64
* chore: just use image for codespaces, don't try to build
* add in python, node requirements
* install node
* install node
* uncomment node deps install
* use app network stack for all services
* move .env to .dev.env and us in docker-compose
* set WORKDIR to workspace
* pin apt packages
* pin versions, remove apt-key
* add yarn cache clean
* Fix hadolint warnings
* make codespace build on master, or a PR with codespaces-build label
* add missing &&
* dont bother with clickhouse install, its huge
* dont be cute with the networks, just expose explicitly
* fix network
* create container zsh
* correct .env path
* point app to the clickhouse container
* Also install saml deps
* ignore .env again
* libxml already installed
* correct conditional workflow comment
* use plugin-server in same repo
* remove plugins dir
* update docker files
* clean up docker files:
* visual separation
* run plugin server test stack
* build and push test image
* revert push image
* fix default yarn start
* another fix
* fix some more
* add plugin-server-prod script
* make executable
* update scripts:
* revert prod script
* fixes
* fix tests
* clean
* move build to docker
* build plugin-server in prod dockerfile
* use different entrypoint for e2e test
* fix bugs with ts-node-dev
* restore yarn.lock
* lost a few changes in the rebase
* fix dockerfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Unify SAML and non-saml tests workflows
* Include matrix name in cache key
* Prettify yaml
* Measure test duration time
* Install pytest-split
* Try out parallelism
* Update name
* Fixup yaml
* Upload artifacts
* Run FOSS tests separately
* Run mark.ee tests only once
* Set up needed files properly
* Run backend tests only on one python version
* Run only cloud tests for cloud repo
* Remove test_durations file for posthog/
* Skip setting up ch on foss tests
* Fixup cloud tests
* Cache .test_durations file
* Fix ternary expression
* Only upload artefacts if needed
* .test_durations change
* Default to 3.8.5 in tests
* Add a note
* Only run `test_migrations_are_null` on new migrations
* Rename `DashboardItem` to `SavedInsight`
* Add explanation to `test_migrations_are_null` script
* Rename `test_dashboard` to `test_saved_insight_model`
* Ditch `DashboardItem{ViewSet,Serializer}` altogether
* Fix refresh via `InsightSerializer`
* "SavedInsight" to just "Insight"
* Update file names
* Update insight.py
* Update dashboardLogic.test.ts
* Update migration
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
