* ci: Run core backend tests with both HogQL and legacy insights
* Double the number of tests
* Fix env var setting
* Add some HOGQL_INSIGHTS_OVERRIDE overrides in tests
* Mark `Insight.last_refresh` as deprecated
* Fix bad merge
* Update query snapshots
* Update query snapshots
* Update test_team.py
* Actually remove legacy backend from matrix
* Update test_fetch_from_cache.py
* Use `update_cached_state` in `calculate_for_query_based_insight()`
* Clean up CI changes and a comment
* Fix `update_cached_state` typing
* Update test_fetch_from_cache.py
* Update test_insight_cache.py
* Update test_insight_cache.py
* Clarify `generate_insight_cache_key` as legacy function
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Revert "chore(CI): revert "chore: syphar is deprecated. fangle python actions…"
This reverts commit 1f2ff4ac32.
* not on depot?
* was missing a syphar removal :/
* was missing a syphar removal :/
* Update query snapshots
* back to depot
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Added suppoer for joined tables and using autocomplete on join constraints
* Added a type for Expression fields
* Fixed mypy
* Fixed mypy
* Update query snapshots
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Remove old dockerfile and only build a single image with unit
We have migrated over to the `unit` image (instead of gunicorn) but we
were still building both. This could lead to confusion - we have no
plans to move back so let's remove the second image build and only
build the unit based image
* Remove unit build id
* fix: Checkout master before checking for hogql changes
* fix: But go back to branch after done
* fix: Instead just diff with origin/master
* fix: Install libantlr, what's the worst that can happen?
* fix: Install antlr the hogql-way
* fix: Let's just checkout everything
* feat(batch-exports): Add backfill model and service support
* test(batch-exports-backfills): Add Workflow test
* chore(batch-exports-backfill): Bump migration
* feat(batch-exports): Add RedshiftBatchExportWorkflow
* feat(batch-exports): Add Redshift to BatchExport destinations
* feat(batch-exports): Support properties_data_type Redshift plugin parameter
* refactor(batch-exports): Insert rows instead of using COPY
* fix: Remove unused migration
* chore: Require aiokafka
* feat: Implement new structlog batch exports logger
* refactor: Use new structlog logger everywhere
* test: Add tests, fix things
* fix: Remove old tests
* chore: Change typing of return logger
* chore: Bump structlog
* chore: Extend docstrings
* fix: Don't use async logging as it's unsupported by temporal runtime
* test: Add logger tests
* fix: Mix pytestmark lists
* fix: Remove unused imports
* fix: Cleanup pytest warnings
* fix: Create and drop dataset for bigquery tests
* fix: Typing issue?
* fix: Let's just checkout everything
* fix: Use global event loop in tests
* fix: Blow-up cache
* fix: Truncate only if table exists
* revert-me: Skip all postgres tests
* fix: Connect to kafka in localhost
* fix: Lazily connect to Kafka
* fix: Resolve conflicts
* fix: Capture temporal context once and bind it to the logger
* fix: Make configure logger sync
* fix: Keep strong reference to background tasks
* fix: Continue consuming from log queue even if we fail to produce
* fix: Also catch the producer not starting
* fix: Remove unused await
* fix: Log kafka producer error after logger is configured
* fix: Replace deprecated set-output GH actions command
* fix: Checkout master before checking for hogql changes
* fix: But go back to branch after done
* fix: Instead just diff with origin/master
* fix: Let's just fetch manually
* fix: Install libantlr, what's the worst that can happen?
* fix: Install antlr the hogql-way
* fix: Amazing hogql-developers already include this in wheel
* Allow placeholders in place of ratio expressions in hogql
* Added C++ changes for RatioExpr
* Bump `hogql_parser` version
* Upgrade ANTLR version to align with brew latest
* Update query snapshots
* Fix parser change message body
The quotes weren't right, so the step failed instead of posting a comment.
* Revert "Update query snapshots"
This reverts commit 1302d3b24f.
* Update MD5 sum
* Bump version again
* Use new hogql-parser version
---------
Co-authored-by: Michael Matloka <michal@matloka.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* Add partial C++ parser of HogQL
* Support all the rules and add proper error handling
* Use `AlignAfterOpenBracket: BlockIndent`
* Reuse all the parser tests for the C++ backend
* Fix relationship between headers and implementations
* Add more error handling and fix minor issues
* Build both Python and C++ parsers in the package.json script
* Update ARRAY JOIN error assertion
* Improve timeit
* Move the C extension to the top level
* Refactor `vector_to_list_string`
* Build the parser on Linux
* Build wheels for the parser
* Simplify Linux build and fix macOS
* Fix Homebrew paths on x86 and don't fail fast in CI
* Set MACOSX_DEPLOYMENT_TARGET for C++20
* Set up QEMU for Linux ARM builds
* Publish the wheels on PyPI
* Avoiding Linux ARM emulation in CI for now
* Build sdist too
* Revert Dockerfile changes
* Fix PyPI publish
* Add README and optimize sdist build
* Use setup.py directly instead of build
* Use PyPI hogql-parser instead of local
* Also revert production-unit.Dockerfile
* Fix sdist upload and add Linux ARM back
* No Linux ARM build in the end
* Fix artifact uploading
* Do try building Linux ARM
We need this for prod.
* Use `npm` in `grammar:build`
`pnpm` is not available in that job.
* Fix formatting of hogql_parser
* Build everything on macOS
* Revert "Build everything on macOS"
Not so fast actually.
* Use hogql-parser=0.1.1
* Fix dylib in macOS wheel
* Bump hogql-parser version
* Fix missing module error
* Delete timeit.py
* Make error handling robust
* Format the C++
* Use `hogql-parser==0.1.1`
* Fix reserved keyword error assertions
* Use HEAD hogql_paresr in CI
* Fix `apt` usage
* Add some sudo in CI
* Ensure package will be releasable before build
* Bump version to 0.1.3
* Cover C++ `unquote_string` with tests
* Use BuildJet ARM runners for ARM builds
* Add some instructions
* Add HogQL version check to backend CI
* Update requirements.txt
* Use `setuptools` instead of the deprecated `distutils`
* Fix working dir in backend CI
* Align ANTLR versions
* Add test for "mismatched input"
This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev
* Add types and bump version
* Comment instead of failing version check
* Automate hogql-release version bump
* Fix checkout token
* Don't build hogql-parser if there were no changes
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Update query snapshots
* Improve documentation
* Use new hogql-parser version
* Fix error start and end initialization
* Note `antlr4-cpp-runtime`
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Also remove NUL chars in C++
* Check ANTLR4 runtime archive checksum for security
* Note more decrefs to add
* Add vector size checks
* Use new hogql-parser version
* Don't support the `start` arg in C++ `parse_expr`
* Use new hogql-parser version
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Add docker image that uses nginx unit instead of gunicorn
🦄🔫
* Add unit build to CI
* Fix duplicate id
* try 3.11
* Only build for amd64
need python3.11 for unit image on arm
* ci-backend runs-on -> buildkite
* Use buildjet runners
* Go away old cache
* Fix deps
* Ensure to update the package registry
* Use bigger runners for django tests
* Use ARM for jobs with less concurrency
* Revert "Use ARM for jobs with less concurrency"
This reverts commit bcbf0d7524.
* Fix typo
* ci: wait for temporal to be up before running backend tests
If we don't wait, then there are some tests that fail because temporal
isn't up yet. These tests ideally shouldn't be using the real temporal
server, but rather the test server that is spun up when e.g. using
`temporalio.testing.WorkflowEnvironment` although for the sake of
getting tests to not be flaky, this is a good enough solution for now.
