0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-28 18:26:15 +01:00
Commit Graph

9 Commits

Author SHA1 Message Date
James Greenhill
751a35cd35
Make DDLs more friendly towards running on a cluster and cleanups (#5091)
* Make DDLs more friendly towards running on a cluster

* Use primary CLICKHOUSE host for migrations and DDL

* loose ends on person kafka create

* posthog -> cluster typo

* add cluster to KAFKA create for plugin logs

* Feed the type monster

* clusterfy local clickhouse

* test docker-compose backed github action

* run just clickhouse and postgres from docker-compose

* move option to between up and <services>

* posthog all the things

* suggest tests run on  cluster

* posthog cluster for ci

* use deploy path for docker-compose

* fix for a clickhouse bug 🐛

* complete CH bug fixes

* 5439 the github actions pg configs

* remove CLICKHOUSE_DATABASE (handled automatically)

* update DATABASE_URL for code quality checks

* Missed a few DDLs on Person

* 5439 -> 5432 to please the people

* cleanup persons and use f strings <3 f strings

* remove auto parens

* Update requirements to use our fork of infi.clickhouse_orm

* fix person.py formatting

* Include boilerplate macros for a cluster
2021-07-15 17:20:37 -07:00
James Greenhill
2935a65dc5
MV -> View for events_with_array_props_view and remove EVENT_PROP_TABLE_SQL (#2766)
* MV -> View for events_with_array_props_view

* sort

* convert another mv into view because mv's are not triggered by views

* pedantic - no mat in naming

* remove unused tables
2020-12-15 10:59:28 -08:00
Paolo D'Amico
066721e3c1
Stability & dev experience improvements (#2152) 2020-11-02 14:55:20 +00:00
Marius Andra
1eeed28751
Fix Master EE code (#1701)
* add test runner to ease pycharm dev

* fix broken import

* drop and recreate the clickhouse test db before running tests

* fix person uuid str json serialization issue

* make kafka optional in tests

* fix inits

* remove need for kafka in person.py

* fix a bunch of mypy errors

* fix function and add process_event to pipeline

* fixed missing params and tests

* change uuid and fix types

* types

* optimize for merge prop test

* make ClickhouseProducer to produce to clickhouse one way or another

* annotate types

Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
2020-09-24 06:14:17 -04:00
James Greenhill
dedf5582c1
first chunk of clickhouse framework (#1613)
* first chunk of clickhouse framework

* prod, not dev docker-compose.yml

* add clickhouse sql files
2020-09-08 16:12:27 -07:00
James Greenhill
c2e03a3a46
Revert this to bring back the working copy of ee / clickhouse (#1588) 2020-09-04 21:12:07 -07:00
James Greenhill
5506135c3c
Bring back clickhouse changes along with queue size metrics (#1579)
* Revert "Revert "Clickhouse setup (#1463)""

This reverts commit 7f2cab4b93.

* add queue backlog to _stats endpoint

* celery queue length into heartbeat -> statsd

* reformat

* type check fixes

* bump expected number of queries to 8

Co-authored-by: Eric <eeoneric@gmail.com>
2020-09-04 13:44:53 -07:00
James Greenhill
7f2cab4b93 Revert "Clickhouse setup (#1463)"
This reverts commit a0327587cb.
Time to process events shot way up and logs are missing.
2020-09-03 19:27:02 -07:00
Eric Duong
a0327587cb
Clickhouse setup (#1463)
* initial

* migration command

* migrations working

* add modelless views for clickhouse

* initial testing structure

* use test factory

* scaffold for all tests

* add insight and person api

* add basic readme

* add client

* change how migrations are run

* add base tables

* ingesting events

* restore delay

* remove print

* updated testing flow

* changed sessions tests

* update tests

* reorganized sql

* parametrize strings

* element list query

* change to seralizer

* add values endpoint

* retrieve with filter

* pruned code to prepare for staged merge

* working ingestion again

* tests for ee

* undo unneeded tests right now

* fix linting

* more typing errors

* fix tests

* add clickhouse image to workflow

* move to right job

* remove django_clickhouse

* return database url

* run super

* remove keepdb

* reordered calls

* fix type

* fractional seconds

* fix type error

* add checks

* remove retention sql

* fix tests

* add property storage and tests

* merge master

* fix tests

* fix tests

* .

* remove keepdb

* format python files

* update CI env vars

* Override defaults and insecure tests

* Update how ClickHouse database gets evaluated

* remove bootstrapping clickhouse database routine

* Don't initialize the clickhouse connection unless we say it's primary

* .

* fixed id generation

* remove dump

* black settings

* empty client

* add param

* move docker-compose for ch to ee dir

* Add _public_ key to repo for verifying self signed cert on server

* update ee compose file for ee dir

* fix a few issues with tls in migrations

* update migrations to be flexible about storage profile and engine

* black settings

* add elements prop tables

Co-authored-by: James Greenhill <jams@uber.com>
2020-09-03 10:27:45 -07:00