ted kaemming
0944412f48
feat: Support marking materialized columns as disabled and dropping them ( #26068 )
2024-11-13 10:57:27 -08:00
timgl
aa036060cb
feat: Improve materialize command ( #22666 )
2024-06-04 16:03:19 +01:00
timgl
68181ad0d6
fix: Fix single property materialization command ( #20798 )
...
* fix: Fix single property materialization command
* default to 0
2024-03-28 11:04:00 +00:00
Tom Owers
5513be7731
chore: upgraded Ruff linter ( #18188 )
...
* Upgraded Ruff linter
* Formatted whole codebase with new ruff rules
* Revert import removal
* Fixed mypi issues or added ignores
* Fixed schema formatting
* Fixed hogvm failing tests
* Remove duplicate key in list
2023-10-26 12:38:15 +02:00
Neil Kakkar
17a25011aa
fix: materialised columns ( #11558 )
2022-08-31 21:02:33 +01:00
Eric Duong
46faa8fab2
refactor(FOSS): foss remove all ee dependencies from /posthog ( #10319 )
...
* refactor: foss move properties to /posthog
* refactor: move replication and materializtaion
* refactor: move file
* refactor: move test and journeys
* refactor: move breakdown props
* refactor: move query imports
* refactor: move more ee dependencies
* refactor: restore groupsjoinquery
* fix: errors
* refactor: no ee.clickhouse dependncies
* try import tasks
* refactor: move materialization
* refactor: change foss split for column optimizer
* run black
* fix: imports
* remove comment
* Update snapshots
* run black
* skip isort
* Update snapshots
* format
* more fixes
* refactor(FOSS): split out paths query code (#10378 )
* refactor: move migrations
* refactor: move idl
* fix: move more imports
* import adjustments
* fix: test import
* fix: test import
* fix: remove restriction
* refactor: split out paths query code
* refactor: more code splitting
* fix: types
* refactor(FOSS): Setup such that FOSS is deployable (#10352 )
* refactor: move migrations
* refactor: move idl
* fix: move more imports
* import adjustments
* fix: test import
* fix: test import
* fix: remove restriction
* fix import
* refactor: add snapshost
* subscription-import
* fix: safe import
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2022-06-22 17:24:03 -04:00
Michael Matloka
2e155609d1
chore: Drop and recreate test CH DB in setup_test_environment ( #10368 )
2022-06-21 11:39:00 +02:00
Karl-Aksel Puulmann
5702e42716
chore(ci): Speed up setup_test_environment ( #10353 )
...
* Update test_schema.py
* Create DLQ kafka table only for relevant tests
* Remove DLQ mv tables from conftest
* Standardize conftest
* Parallel clickhouse setup
* Create django db based on models
Speeds up django test database setup from ~15s to 1s.
* Update imports
* Re-add conditionality, fix typing
* Run plugin-server tests when this file changes
2022-06-20 16:31:25 +03:00
Eric Duong
e8d4d6d49d
refactor(foss): move all /sql queries ( #10230 )
...
* refactor: move prop value and actorbasequery files
* refactor: move test base
* fix: imports
* move middleware
* refactor: move cohort model util
* refactor: move all /sql queries
* fix: imports
* fix imports
* refactor: move cohort model util (#10224 )
2022-06-13 18:05:14 -04:00
Eric Duong
ce7988ecfa
refactor: move prop value and actorbasequery files ( #10258 )
...
* Revert "fix: revert "move prop value and actorbasequery files" due to flaky test (#10257 )"
This reverts commit ac1a485cdf
2022-06-13 16:24:14 -04:00
Eric Duong
ac1a485cdf
fix: revert "move prop value and actorbasequery files" due to flaky test ( #10257 )
...
* Revert "refactor: move prop value and actorbasequery files (#10220 )"
This reverts commit 12bf3a7c3e
2022-06-10 23:49:53 +00:00
Eric Duong
12bf3a7c3e
refactor: move prop value and actorbasequery files ( #10220 )
...
* refactor: move prop value and actorbasequery files
* refactor: move test base
* fix: imports
* move middleware
* refactor: move cohort model util (#10224 )
2022-06-10 18:44:50 -04:00
Eric Duong
c0ccc1fef8
fix: cohort update to original bug 2 ( #10130 )
...
* fix: add test for updating back to original
* fix: add test for updating back to original
* add uuid column to cohortpeople
* Update snapshots
* add migration
* remove random uuid
* Update snapshots
* add recalculation condition
* fix: tests working
* adjust tests
* add swapover test
* wrong var
* wrong param
* remove default
* Update snapshots
* Update snapshots
* refactor: sql
* accepted tables in test
* update tests
* don't alias otherwise condition is misled
* use more inclusive operator
* address comments
* fix: use column solution
* fix: group by version
* fix: add sort key modification
* fix: add comma
* Update snapshots
* Update snapshots
* update snapshots
* remove leftover
* remove stale test
* update snapshots
* add comment
* add test
* add version to count clause
* Update snapshots
* Update snapshots
Co-authored-by: Rick Marron <rcmarron@gmail.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2022-06-07 14:36:28 -04:00
Michael Matloka
1af98a0fea
test(ingestion): Unify tests into single process-event.test.ts ( #9819 )
...
* Simplify imports in `setup_test_environment`
* refactor(ingestion): Unify tests into single `process-event.test.ts`
* Update test scripts for CI
* Fix imports
* Run prettier
* Fix missing import
* Fix the OTHER imports
* Nuke `includeNewPropertiesUpdatesTests`
2022-05-17 21:07:13 +02:00
Yakko Majuri
e7ad8ddc6b
fix: set CLICKHOUSE_PASSWORD on backfill script ( #9816 )
2022-05-17 12:07:30 +00:00
Yakko Majuri
8254fd67d7
feat: add script to backfill persons and groups columns on events table ( #9722 )
...
* feat: wip add script to backfill persons and groups columns on events table
* updates
* cleanup + query id feature
* refactor
* fixes and add person tests
* add groups tests
* fix ci tests?
* test
* keep dictionaries around
* updates + massive sleep test?
* fixes
* updates
* fix for ch 21.6.5
2022-05-17 09:32:31 +00:00
Yakko Majuri
801c2a5f7e
feat(persons-on-events): add groups and persons columns to events schema ( #9510 )
...
* feat(persons-on-events): add persons and group cols to events
* snapshots
* Update ee/clickhouse/models/cohort.py
* update snapshots
* update kafka settings
* final modification to prevent temporary errors
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
2022-04-29 11:31:32 +00:00
Tim Glaser
5887bd2143
test: Speed up backend tests ( #9289 )
...
* Durations
* patch sleep
* BULK
* SPEED UP EVERYTHING
* fiiix
* more tests
* morrr
* fix test
* Don't create demo data every time
* fix
* fix
* Affix
* fix
* fix
* fix
* fix
* fix
* More speed
* fix
2022-04-27 13:38:39 +01:00
Yakko Majuri
6bce3869f1
fix: check that no async migrations are running when upgrading posthog ( #9466 )
...
* fix: add check for no running async migrations before trying to upgrade
* refactor
* also add check for errored migrations
2022-04-20 13:42:58 +00:00
Yakko Majuri
ef543af5c9
fix: make sync_replicated_schema safe to run ( #9465 )
2022-04-20 14:59:14 +02:00
Eric Duong
5eb98ef624
refactor: move event and person query to /posthog ( #8970 )
...
* refactor: move event and person query to /posthog
* move imports
* another import
* refactor: move actions
* refactor: move actions
* enterprise
* trigger tests
* test
* make dummy
2022-03-22 20:27:11 +00:00
Karl-Aksel Puulmann
602378ba7c
feat(sharding): Improve syncing schema ( #9129 )
...
* Revert "Revert "Revert "Revert "feat(sharding): Improve syncing schema (#9077 )" (#9102 )" (#9108 )" (#9111 )"
This reverts commit 326efcc807
2022-03-21 15:06:20 +02:00
Harry Waye
326efcc807
Revert "Revert "Revert "feat(sharding): Improve syncing schema ( #9077 )" ( #9102 )" ( #9108 )" ( #9111 )
...
This reverts commit f0eaf13dbe
2022-03-18 14:00:34 +00:00
Karl-Aksel Puulmann
f0eaf13dbe
Revert "Revert "feat(sharding): Improve syncing schema ( #9077 )" ( #9102 )" ( #9108 )
...
This reverts commit 7ef60231c9
2022-03-18 15:23:39 +02:00
Guido Iaquinti
7ef60231c9
Revert "feat(sharding): Improve syncing schema ( #9077 )" ( #9102 )
...
This reverts commit 6d2b809893
2022-03-18 12:28:54 +02:00
Karl-Aksel Puulmann
6d2b809893
feat(sharding): Improve syncing schema ( #9077 )
...
* Update infi.orm
* Update table syncing logic
* Reformat
2022-03-18 11:38:52 +02:00
Karl-Aksel Puulmann
40a1721d0b
fix: Simplify test setup for plugin-server ( #9051 )
...
We don't need multiple commands for setup, also verify TEST env is set.
2022-03-17 10:34:53 +02:00
Tiina Turban
72042ee844
fix(async-migrations): bin/migrate to check not run async migrations ( #8872 )
2022-03-09 13:55:23 +01:00
Karl-Aksel Puulmann
c8d6b2225f
feat(sharding): add command to sync tables onto new nodes ( #8912 )
...
* feat(sharding): add command to sync tables onto new nodes
clickhouse-operator only syncs some tables onto new nodes. This new
command ensures that when adding new shards, they are automatically
synced up on redeploying
Note that there might be timing concerns here as resharding on altinity
cloud does not redeploy automatically. In practice however what this
means is that new nodes just won't ingest any data until another deploy
* Add test to the new command
* Improve non-replicated test
2022-03-08 12:50:49 +02:00
Guido Iaquinti
0ef7f6a493
ci(flake8): enable unused import check ( #8714 )
2022-03-08 09:53:54 +01:00
Tiina Turban
ac33ce4b00
feat(async-migrations): migration complete early exit ( #8841 )
2022-03-03 17:43:00 +01:00
Karl-Aksel Puulmann
d7d2bcb7a3
Fix python manage.py migrate_clickhouse --plan ( #8741 )
...
Currently this would throw on a fresh install with an error similar to
```
│ posthog-events Migration would get applied: 0024_materialize_window_and_session_id │
│ posthog-events Traceback (most recent call last): │
│ posthog-events File "manage.py", line 21, in <module> │
│ posthog-events main() │
│ posthog-events File "manage.py", line 17, in main │
│ posthog-events execute_from_command_line(sys.argv) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", li │
│ posthog-events utility.execute() │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", li │
│ posthog-events self.fetch_command(subcommand).run_from_argv(self.argv) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 3 │
│ posthog-events self.execute(*args, **cmd_options) │
│ posthog-events File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 3 │
│ posthog-events output = self.handle(*args, **options) │
│ posthog-events File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 41, in │
│ posthog-events self.migrate(CLICKHOUSE_HTTP_URL, options) │
│ posthog-events File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 57, in │
│ posthog-events sql = getattr(op, "_sql") │
│ posthog-events AttributeError: 'RunPython' object has no attribute '_sql'
```
This is due to the command not working with `RunPython` commands.
2022-02-25 16:51:22 +02:00
Tiina Turban
1abe9ef897
async mig check ( #8503 )
2022-02-09 20:45:59 +01:00
Tiina Turban
73e4f0db51
Add --check command to migrate_clickhouse ( #8501 )
2022-02-09 20:44:52 +01:00
Tiina Turban
d5fce14cea
async migrations separate errors table ( #8152 )
2022-01-21 19:54:47 +01:00
Eric Duong
a71e899605
Yeetcode ( #7830 )
...
* remove django query tests
* remove funnel and caching check
* remove ee available var
* remove is_clickhouse_enabled
* remove abstract tests
* change primary db
* missing func
* unnecessary test
* try new e2e ci
* func arg
* remove param
* ci
* remove plugins in docker
* background
* change ur;
* add kafka url
* add step
* update docker
* primary docker file
* mount volumes correctly
* one more
* remove postgres tests
* remove foss
* remove all is_clickhouse_neabled
* remove irrelelvant test
* remove extra arg
* remove var
* arg
* add foss comment
* add foss comment
* plugin server config
* Update posthog/utils.py
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* migrate commands
* comment
* add clickhouse to pg tests
* change script
* change ordering
* deepsource
* restore foss tests
* test remove KAFKA_ENABLED from CI
* always wait
* up proper resources
* use one conftest
* restore
* remove unnecessary tests
* remove more pg
* log event tests
* fix more tests
* more tests
* type
* fix more tests
* last test
* typing
* account for shared class setup
* temp test cloud
* restore cloud master checkout
* adjust contexts
* backwards
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: yakkomajuri <yakko.majuri@gmail.com>
2022-01-18 14:32:28 -05:00
Yakko Majuri
10d8c59146
auto complete non required async migrations at migrate step ( #7942 )
2022-01-10 15:34:07 -03:00
Yakko Majuri
3f6b4ecb76
fix migration details changed but name is the same ( #7783 )
2021-12-17 10:51:25 -08:00
Yakko Majuri
3f85aa4c47
special migration -> async migration ( #7686 )
...
* special migration -> async migration
* rename frontend dir
* format
* rename SPECIAL_MIGRATION
2021-12-14 11:48:51 -03:00
Yakko Majuri
b56b141126
Special Migrations Runner ( #7054 pt. 4) ( #7446 )
...
* add special migration definition and example
* types
* special migrations runner
* fix tests
* fix tests 2
* add clickhouse runner
* add temp fix for tests
* wip
* add special migrations api (#7448 )
* wip new structure
* update example sourcing
* Update .gitignore
* yet another wip structure
* code quality
* cypress
* test docker image build
* implement resumable ops
* code quality
* add comments
* add warning
* add conditional requirements for migration
* add comment on is_required
* add dependency map
* wip dependencies and run migration on startup
* code quality
* fix bugs
* fix more bugs
* format
* types
* remove api from this branch
* types
* types
* update clickhouse script
* add is_migration_in_range util
* fix type
* fix runner
* add AUTO_START_SPECIAL_MIGRATIONS env var
* reset migration on start
* cleanup
* wip per op rollback
* prevent accidental status rollback
* add utils and definition test
* update example with rollback per op
* wip test special migration
* add first runner tests
* add runner tests
* add util for code paths
* fix test
* fix types
* fix types again
* cleanup
* cleanup
* add periodic healthcheck task tests
* remove unused imports
* safer row updates
* fix coalescing none checks
* code quality
* add docstrings
* fix
* fix deploys issue
* update scripts
* add delay
* address reviews
* address review comments
* address review comments
* address final comments
* fix import error
* fix tests
* remove unused imports
* fix tests
* fix task test
* remove unused return value
* remove unused special migrations code from migrate_clickhouse
* tweaks to support fresh deployments
2021-12-13 13:00:27 -03:00
Marius Andra
1696ed5950
Update mypy and typed-ast, fail early if db version not in range ( #7599 )
...
* error if unsupported db version
* upgrade mypy
* fix various types for mypy
* you can have it in any color you want, as long as it's black
* fix mypy
* Update `kafka-python`
* Format with Black
* Fix mypy after merge
Co-authored-by: Michael Matloka <dev@twixes.com>
2021-12-10 09:29:04 +01:00
Michael Matloka
81f7291261
Move is_clickhouse_enabled from posthog.ee to posthog.utils ( #5902 )
...
* Move `is_clickhouse_enabled` from `posthog.ee` to `posthog.utils`
* Remove circular import
2021-09-11 01:43:24 +02:00
Karl-Aksel Puulmann
17836f80f5
Add command for materializing columns ( #5816 )
...
* Add command for materializing columns
Expecting this to get used in both dev as well as when improving
upstream installations
* No clashes in tests
* Solve for feedback
* Comments for clarity
2021-09-07 07:56:29 +00:00
Karl-Aksel Puulmann
96ad4a6d32
Fix build ( #5162 )
...
* Test: Update requirements
* Update version
* Attempt #3
2021-07-16 16:21:36 +03:00
James Greenhill
751a35cd35
Make DDLs more friendly towards running on a cluster and cleanups ( #5091 )
...
* Make DDLs more friendly towards running on a cluster
* Use primary CLICKHOUSE host for migrations and DDL
* loose ends on person kafka create
* posthog -> cluster typo
* add cluster to KAFKA create for plugin logs
* Feed the type monster
* clusterfy local clickhouse
* test docker-compose backed github action
* run just clickhouse and postgres from docker-compose
* move option to between up and <services>
* posthog all the things
* suggest tests run on cluster
* posthog cluster for ci
* use deploy path for docker-compose
* fix for a clickhouse bug 🐛
* complete CH bug fixes
* 5439 the github actions pg configs
* remove CLICKHOUSE_DATABASE (handled automatically)
* update DATABASE_URL for code quality checks
* Missed a few DDLs on Person
* 5439 -> 5432 to please the people
* cleanup persons and use f strings <3 f strings
* remove auto parens
* Update requirements to use our fork of infi.clickhouse_orm
* fix person.py formatting
* Include boilerplate macros for a cluster
2021-07-15 17:20:37 -07:00
Tim Glaser
99cdd640d3
Rename is_ee_enabled to is_clickhouse_enabled ( #4216 )
...
* Rename is_ee_enabled to is_clickhouse_enabled
* Do frontend as well
* Fix import
* fix
2021-05-13 16:56:54 +02:00
Karl-Aksel Puulmann
a50596c3eb
Fix python manage.py migrate_clickhouse --fake ( #4224 )
2021-05-06 09:59:50 +03:00
Karl-Aksel Puulmann
f4009c1e16
Allow migrating on a clickhouse cluster ( #4197 )
...
* Allow migrating clickhouse on cluster
Context under https://github.com/PostHog/posthog/issues/4090
* Add option to print sql
2021-05-05 14:38:39 +03:00
Karl-Aksel Puulmann
c775a5305c
Add --plan and --fake flags to migrate_clickhouse ( #4160 )
...
This can be used to bring a production environment up to date.
Related issue: #4090
2021-04-29 15:40:12 +03:00
Karl-Aksel Puulmann
5a19ed293f
Don't catch all exceptions for migrating clickhouse ( #4016 )
...
This is making it hard to wait until clickhouse has been set up for VPC
deployments.
It was introduced in
https://github.com/PostHog/posthog/pull/2034/files#diff-ea7b79cfe0fd591ead9add4372dab5f31aa4f6155f304bc0d63ef667c9e8ac95L16
for an unknown reason
2021-04-16 15:08:01 +03:00
