0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-22 08:40:03 +01:00
posthog/hogql_parser/HogQLLexer.h

83 lines
3.5 KiB
C
Raw Normal View History

perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
// Generated from HogQLLexer.g4 by ANTLR 4.13.2
perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
#pragma once
#include "antlr4-runtime.h"
class HogQLLexer : public antlr4::Lexer {
public:
enum {
2024-05-28 14:51:45 +02:00
ALL = 1, AND = 2, ANTI = 3, ANY = 4, ARRAY = 5, AS = 6, ASCENDING = 7,
2024-07-25 16:57:28 +02:00
ASOF = 8, BETWEEN = 9, BOTH = 10, BY = 11, CASE = 12, CAST = 13, CATCH = 14,
COHORT = 15, COLLATE = 16, CROSS = 17, CUBE = 18, CURRENT = 19, DATE = 20,
DAY = 21, DESC = 22, DESCENDING = 23, DISTINCT = 24, ELSE = 25, END = 26,
EXTRACT = 27, FINAL = 28, FINALLY = 29, FIRST = 30, FN = 31, FOLLOWING = 32,
2024-08-30 11:51:36 +02:00
FOR = 33, FROM = 34, FULL = 35, FUN = 36, GROUP = 37, HAVING = 38, HOUR = 39,
ID = 40, IF = 41, ILIKE = 42, IN = 43, INF = 44, INNER = 45, INTERVAL = 46,
IS = 47, JOIN = 48, KEY = 49, LAST = 50, LEADING = 51, LEFT = 52, LET = 53,
LIKE = 54, LIMIT = 55, MINUTE = 56, MONTH = 57, NAN_SQL = 58, NOT = 59,
NULL_SQL = 60, NULLS = 61, OFFSET = 62, ON = 63, OR = 64, ORDER = 65,
OUTER = 66, OVER = 67, PARTITION = 68, PRECEDING = 69, PREWHERE = 70,
QUARTER = 71, RANGE = 72, RETURN = 73, RIGHT = 74, ROLLUP = 75, ROW = 76,
ROWS = 77, SAMPLE = 78, SECOND = 79, SELECT = 80, SEMI = 81, SETTINGS = 82,
SUBSTRING = 83, THEN = 84, THROW = 85, TIES = 86, TIMESTAMP = 87, TO = 88,
TOP = 89, TOTALS = 90, TRAILING = 91, TRIM = 92, TRUNCATE = 93, TRY = 94,
UNBOUNDED = 95, UNION = 96, USING = 97, WEEK = 98, WHEN = 99, WHERE = 100,
WHILE = 101, WINDOW = 102, WITH = 103, YEAR = 104, ESCAPE_CHAR_COMMON = 105,
IDENTIFIER = 106, FLOATING_LITERAL = 107, OCTAL_LITERAL = 108, DECIMAL_LITERAL = 109,
HEXADECIMAL_LITERAL = 110, STRING_LITERAL = 111, ARROW = 112, ASTERISK = 113,
BACKQUOTE = 114, BACKSLASH = 115, COLON = 116, COMMA = 117, CONCAT = 118,
DASH = 119, DOLLAR = 120, DOT = 121, EQ_DOUBLE = 122, EQ_SINGLE = 123,
GT_EQ = 124, GT = 125, HASH = 126, IREGEX_SINGLE = 127, IREGEX_DOUBLE = 128,
LBRACE = 129, LBRACKET = 130, LPAREN = 131, LT_EQ = 132, LT = 133, NOT_EQ = 134,
NOT_IREGEX = 135, NOT_REGEX = 136, NULL_PROPERTY = 137, NULLISH = 138,
PERCENT = 139, PLUS = 140, QUERY = 141, QUOTE_DOUBLE = 142, QUOTE_SINGLE_TEMPLATE = 143,
QUOTE_SINGLE_TEMPLATE_FULL = 144, QUOTE_SINGLE = 145, REGEX_SINGLE = 146,
REGEX_DOUBLE = 147, RBRACE = 148, RBRACKET = 149, RPAREN = 150, SEMICOLON = 151,
SLASH = 152, UNDERSCORE = 153, MULTI_LINE_COMMENT = 154, SINGLE_LINE_COMMENT = 155,
WHITESPACE = 156, STRING_TEXT = 157, STRING_ESCAPE_TRIGGER = 158, FULL_STRING_TEXT = 159,
FULL_STRING_ESCAPE_TRIGGER = 160
2024-05-30 20:32:39 +02:00
};
enum {
IN_TEMPLATE_STRING = 1, IN_FULL_TEMPLATE_STRING = 2
perf: Parse HogQL with C++ for a huge speedup (#17659) * Add partial C++ parser of HogQL * Support all the rules and add proper error handling * Use `AlignAfterOpenBracket: BlockIndent` * Reuse all the parser tests for the C++ backend * Fix relationship between headers and implementations * Add more error handling and fix minor issues * Build both Python and C++ parsers in the package.json script * Update ARRAY JOIN error assertion * Improve timeit * Move the C extension to the top level * Refactor `vector_to_list_string` * Build the parser on Linux * Build wheels for the parser * Simplify Linux build and fix macOS * Fix Homebrew paths on x86 and don't fail fast in CI * Set MACOSX_DEPLOYMENT_TARGET for C++20 * Set up QEMU for Linux ARM builds * Publish the wheels on PyPI * Avoiding Linux ARM emulation in CI for now * Build sdist too * Revert Dockerfile changes * Fix PyPI publish * Add README and optimize sdist build * Use setup.py directly instead of build * Use PyPI hogql-parser instead of local * Also revert production-unit.Dockerfile * Fix sdist upload and add Linux ARM back * No Linux ARM build in the end * Fix artifact uploading * Do try building Linux ARM We need this for prod. * Use `npm` in `grammar:build` `pnpm` is not available in that job. * Fix formatting of hogql_parser * Build everything on macOS * Revert "Build everything on macOS" Not so fast actually. * Use hogql-parser=0.1.1 * Fix dylib in macOS wheel * Bump hogql-parser version * Fix missing module error * Delete timeit.py * Make error handling robust * Format the C++ * Use `hogql-parser==0.1.1` * Fix reserved keyword error assertions * Use HEAD hogql_paresr in CI * Fix `apt` usage * Add some sudo in CI * Ensure package will be releasable before build * Bump version to 0.1.3 * Cover C++ `unquote_string` with tests * Use BuildJet ARM runners for ARM builds * Add some instructions * Add HogQL version check to backend CI * Update requirements.txt * Use `setuptools` instead of the deprecated `distutils` * Fix working dir in backend CI * Align ANTLR versions * Add test for "mismatched input" This is thrown differently than other HogQLSyntaxExceptions in C++, so might help reveal what's going on with tests failing only on Linux CI and not macOS dev * Add types and bump version * Comment instead of failing version check * Automate hogql-release version bump * Fix checkout token * Don't build hogql-parser if there were no changes * Update query snapshots * Update query snapshots * Update query snapshots * Update query snapshots * Improve documentation * Use new hogql-parser version * Fix error start and end initialization * Note `antlr4-cpp-runtime` Co-authored-by: Marius Andra <marius.andra@gmail.com> * Also remove NUL chars in C++ * Check ANTLR4 runtime archive checksum for security * Note more decrefs to add * Add vector size checks * Use new hogql-parser version * Don't support the `start` arg in C++ `parse_expr` * Use new hogql-parser version --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Marius Andra <marius.andra@gmail.com>
2023-10-13 15:58:08 +02:00
};
explicit HogQLLexer(antlr4::CharStream *input);
~HogQLLexer() override;
std::string getGrammarFileName() const override;
const std::vector<std::string>& getRuleNames() const override;
const std::vector<std::string>& getChannelNames() const override;
const std::vector<std::string>& getModeNames() const override;
const antlr4::dfa::Vocabulary& getVocabulary() const override;
antlr4::atn::SerializedATNView getSerializedATN() const override;
const antlr4::atn::ATN& getATN() const override;
// By default the static state used to implement the lexer is lazily initialized during the first
// call to the constructor. You can call this function if you wish to initialize the static state
// ahead of time.
static void initialize();
private:
// Individual action functions triggered by action() above.
// Individual semantic predicate functions triggered by sempred() above.
};