0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-30 19:41:46 +01:00
posthog/.github/workflows/build-hogql-parser.yml

132 lines
5.2 KiB
YAML
Raw Normal View History

perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
name: Release hogql-parser
on:
push:
branches:
- master
paths:
- hogql_parser/**
- .github/workflows/build-hogql-parser.yml
pull_request:
paths:
- hogql_parser/**
- .github/workflows/build-hogql-parser.yml
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
jobs:
check-version:
name: Check version legitimacy
if: github.repository == 'PostHog/posthog'
perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
runs-on: ubuntu-22.04
outputs:
parser_any_changed: ${{ steps.changed-files-yaml.outputs.parser_any_changed }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetching all for comparison since last push (not just last commit)
- name: Check if hogql_parser/ has changed
id: changed-files-yaml
uses: tj-actions/changed-files@v39
with:
since_last_remote_commit: true
files_yaml: |
parser:
- hogql_parser/**
- name: Notify about release needed
if: steps.changed-files-yaml.outputs.parser_any_changed == 'true'
shell: bash
run: |
published=$(curl -fSsl https://pypi.org/pypi/hogql-parser/json | jq -r '.info.version')
local=$(python hogql_parser/setup.py --version)
# TODO: Only comment if no comment alraedy exists for $local
if [[ "$published" == "$local" ]]; then
message_body="It looks like the code of \`hogql-parser\` has changed since last push, but its version stayed the same at $local. 👀\nMake sure to resolve this in \`hogql_parser/setup.py\` before merging!"
curl -s -u posthog-bot:${{ secrets.POSTHOG_BOT_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} -X POST -d "{ \"body\": \"$message_body\" }" "https://api.github.com/repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments"
perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
fi
build-wheels:
name: Build wheels on ${{ matrix.os }}
needs: check-version
runs-on: ${{ matrix.os }}
timeout-minutes: 30
if: ${{ needs.check-version.outputs.parser_any_changed == 'true' }}
strategy:
matrix:
# As of October 2023, GitHub doesn't have ARM Actions runners… and ARM emulation is insanely slow
# (20x longer) on the Linux runners (while being reasonable on the macOS runners). Hence, we use
# BuildJet as a provider of ARM runners - this solution saves a lot of time and consequently some money.
os: [ubuntu-22.04, buildjet-2vcpu-ubuntu-2204-arm, macos-12]
steps:
- uses: actions/checkout@v4
- if: ${{ !endsWith(matrix.os, '-arm') }}
uses: actions/setup-python@v4
with:
python-version: '3.11'
- if: ${{ endsWith(matrix.os, '-arm') }}
uses: deadsnakes/action@v3.0.1 # Unfortunately actions/setup-python@v4 just doesn't work on ARM! This does
with:
python-version: '3.11'
- name: Build sdist
if: matrix.os == 'ubuntu-22.04' # Only build the sdist once
run: cd hogql_parser && python setup.py sdist
- name: Install cibuildwheel
run: python -m pip install cibuildwheel==2.16.*
- name: Build wheels
run: cd hogql_parser && python -m cibuildwheel --output-dir dist
env:
MACOSX_DEPLOYMENT_TARGET: '12' # A modern target allows us to use C++20
- uses: actions/upload-artifact@v3
with:
path: |
hogql_parser/dist/*.whl
hogql_parser/dist/*.tar.gz
if-no-files-found: error
publish:
name: Publish on PyPI
needs: build-wheels
environment: pypi-hogql-parser
permissions:
id-token: write
runs-on: ubuntu-22.04
steps:
- name: Fetch wheels
uses: actions/download-artifact@v3
with:
name: artifact
path: dist/
- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
- uses: actions/checkout@v4
with:
token: ${{ secrets.POSTHOG_BOT_GITHUB_TOKEN }}
ref: ${{ github.event.pull_request.head.ref }}
- name: Update hogql-parser in requirements
shell: bash
run: |
local=$(python hogql_parser/setup.py --version)
sed -i "s/hogql-parser==.*/hogql-parser==${local}/g" requirements.in
sed -i "s/hogql-parser==.*/hogql-parser==${local}/g" requirements.txt
- uses: EndBug/add-and-commit@v9
with:
add: '["requirements.in", "requirements.txt"]'
message: 'Use new hogql-parser version'
default_author: github_actions
github_token: ${{ secrets.POSTHOG_BOT_GITHUB_TOKEN }}