0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-12-01 04:12:23 +01:00
Commit Graph

17 Commits

Author SHA1 Message Date
timgl
00ea8c36f2
feat(data-warehouse): Initial data warehouse setup (#16087)
* wip warehouse

* moar progress

* Update UI snapshots for `chromium` (2)

* fix types and bad logic

* make it work

* type

* update migration

* change icon

* Update UI snapshots for `chromium` (2)

* backend errors

* Update UI snapshots for `chromium` (2)

* fix tests

* remove duplicate regexp

* type

* ui

* more tests

* Update UI snapshots for `chromium` (1)

* client error

* Update query snapshots

* Update query snapshots

* Update UI snapshots for `chromium` (2)

* change api call and split out warehouse logic + restore data management tab

* Update query snapshots

* remove

* Update UI snapshots for `chromium` (1)

* Update query snapshots

* typing

* Update UI snapshots for `chromium` (2)

* Update UI snapshots for `chromium` (2)

* Update UI snapshots for `chromium` (2)

* Update UI snapshots for `webkit` (2)

* Update UI snapshots for `chromium` (2)

* Update UI snapshots for `webkit` (2)

* dummy

* remove console log

* Update UI snapshots for `chromium` (2)

* Update UI snapshots for `chromium` (2)

* encrypt

* fix migration

* fix

* tests

* add salt env var

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: eric <eeoneric@gmail.com>
2023-06-20 22:38:58 -04:00
Marius Andra
7e0d22b0fb
feat(hogql): select from s3 (#16038)
* add a few types

* context value adding

* first draft of a system that sort of works

* hacks to get it working

* fix alias

* simplify a lot

* set readonly to 2

* Update docker/clickhouse/config.xml

Co-authored-by: timgl <tim@glsr.nl>

* make it pass our list of urls

* better aliasing

* remove flimsy thing

* different type of input

* dead code

* refactor aapl_stock

* own it

* Update query snapshots

* more cleanup

---------

Co-authored-by: timgl <tim@glsr.nl>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-06-15 10:01:08 +00:00
James Greenhill
0223d7ab86
feat: Add Temporal to the dev and hobby stacks (#14658)
* feat: Add Temporal to the dev and hobby stacks

* disable elastic for hobby because of resources

* checkpoint

* update requirements

* worker is up, but without the sandbox

* ensure temporal does not depend on elastic

* Feedbacked

* pip-compile dev

* mypy fixes

* add a bit of colorful logging

* add django temporal worker to the mix

* checkpoint for dev-full docker

* Working on docker-full, but checkpointing for now

* add migration bits for full
2023-03-20 16:56:14 -06:00
Karl-Aksel Puulmann
87b5eee160
chore(ci): Set stream_poll_timeout_ms in dev (#12122)
This speeds up tests that rely on clickhouse ingestion (plugin-server) significantly. The previous default was 7500, setting this shaves off seconds off of every single test

We don't set this in production due to the extra load it causes.
2022-10-07 10:23:38 +03:00
Harry Waye
d7998cef30
Revert "chore(dev): use network mode host for docker-compose services (#10917)" (#10926)
This reverts commit 225a41db72.
2022-07-22 10:25:59 +01:00
Harry Waye
225a41db72
chore(dev): use network mode host for docker-compose services (#10917)
* chore(dev): use network mode host for docker-compose services

This removes the need to add kafka to /etc/hosts.

As far as I can tell this should be fine for peoples local dev except
they will be required to reset and re-migrate ClickHouse tables as they
will be trying to pull from `kafka` instead of `localhost`.

* remove ports from redis

* Update a few more references
2022-07-21 15:29:31 +01:00
James Greenhill
2e69e0d4bb
chore: bump defaults for clickhouse to 22.3 across the board (#9847)
* chore: bump defaults for clickhouse to 22.3 across the board

* remove docker build instructions for clickhouse

* remove 21.11.11.1 from matrix

* prettier it
2022-05-20 20:32:22 -07:00
Yakko Majuri
8254fd67d7
feat: add script to backfill persons and groups columns on events table (#9722)
* feat: wip add script to backfill persons and groups columns on events table

* updates

* cleanup + query id feature

* refactor

* fixes and add person tests

* add groups tests

* fix ci tests?

* test

* keep dictionaries around

* updates + massive sleep test?

* fixes

* updates

* fix for ch 21.6.5
2022-05-17 09:32:31 +00:00
Guido Iaquinti
98ca2904be
feat(ClickHouse): upgrade Docker references to 21.11.11.1 (1/2) (#8860)
* feat(ClickHouse): upgrade Docker references to 21.11.11.1 (1/2)

* ci(backend): run tests on docker-compose changes

Co-authored-by: Harry Waye <harry@posthog.com>
2022-03-15 11:19:21 +00:00
Guido Iaquinti
7e76387291
Speed improvements for the arm64 ClickHouse build (#6191) 2021-09-30 19:52:59 +00:00
Marius Andra
9da6ee3b5f
ClickHouse M1 Docker (#5215)
* m1 clickhouse support

* add clickhouse arm64 compilation dockerfile

* use clang-12 and 2 parallel builds

* working clickhouse docker image

* this needs to be committed out for CH to work... a known issue. WIP

* move things around

* move things around again

* add arm64 build scripts

* comment out broken line in DEBUG mode, still make it break production builds

* update readme

* timeless text

* add issue in comment

* way way way too much white space

* zookeeper restarts sometimes needed
2021-09-13 13:01:24 +02:00
Karl-Aksel Puulmann
b2fe0868d7
Proposal: Set mutations_sync in client.py instead of docker config (#5748)
This avoids weird divergence with prod that we need in ee tests but don't
want in e.g. plugin server tests or while developing.
2021-08-26 20:32:01 -07:00
James Greenhill
b9dc2d9540
Include users.xml config for dev/test clickhouse instance (#5732) 2021-08-25 01:47:15 +00:00
James Greenhill
dfa1cc4f2e
Set mutations_sync = 1 for testing clickhouse env (#5728) 2021-08-25 00:32:40 +00:00
James Greenhill
751a35cd35
Make DDLs more friendly towards running on a cluster and cleanups (#5091)
* Make DDLs more friendly towards running on a cluster

* Use primary CLICKHOUSE host for migrations and DDL

* loose ends on person kafka create

* posthog -> cluster typo

* add cluster to KAFKA create for plugin logs

* Feed the type monster

* clusterfy local clickhouse

* test docker-compose backed github action

* run just clickhouse and postgres from docker-compose

* move option to between up and <services>

* posthog all the things

* suggest tests run on  cluster

* posthog cluster for ci

* use deploy path for docker-compose

* fix for a clickhouse bug 🐛

* complete CH bug fixes

* 5439 the github actions pg configs

* remove CLICKHOUSE_DATABASE (handled automatically)

* update DATABASE_URL for code quality checks

* Missed a few DDLs on Person

* 5439 -> 5432 to please the people

* cleanup persons and use f strings <3 f strings

* remove auto parens

* Update requirements to use our fork of infi.clickhouse_orm

* fix person.py formatting

* Include boilerplate macros for a cluster
2021-07-15 17:20:37 -07:00
James Greenhill
85a4c1c9d6
Setup logging to system.text_log in dev environment (#3024) 2021-01-20 16:51:11 -08:00
James Greenhill
10f1a52c8e
Update local dev to use proto for events (#2181)
Co-authored-by: James Greenhill <james@jamess-mbp.lan>
2020-11-03 11:37:12 -08:00