0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-28 09:16:49 +01:00
Commit Graph

58 Commits

Author SHA1 Message Date
Michael Matloka
b39dce42cf
Fix webpack-dev-server inside Docker (#2699) 2020-12-08 11:44:39 +01:00
Marius Andra
c24aebe174
Add flags to bin/docker-worker-celery (#2621)
* add flags to docker-worker-celery and rename --with-beat to --with-scheduler

* Clean script up

Co-authored-by: Michael Matloka <dev@twixes.com>
2020-12-03 18:53:06 +01:00
James Greenhill
908cfaf2c8
Don't run preview docker build with debug flags (#2610)
* Don't run web in debug for preview docker build

* worker to not debug
2020-12-01 15:15:30 -08:00
Karl-Aksel Puulmann
df23e3353d
Fix /bin/tests (#2527)
/bin/tests did not work for me anymore - I think this was the original
intent?
2020-11-26 15:32:38 +01:00
Marius Andra
b8d1ba43f4
Disable celery heartbeat, gossip and mingle (#2513)
* disable celery heartbeat, gossip and mingle

* keep the heartbeat

* beat doesn't want these params

* add back "without heartbeat"
2020-11-26 12:44:10 +01:00
Marius Andra
eb8b650aee
Docker nodejs support for plugins (#2452)
* keep npm around in the dockerfiles (needed for plugin-server)

* simplify docker-preview script
2020-11-20 17:04:10 +01:00
Marius Andra
88ad205a3a
Separate Plugins dyno for Heroku (#2213)
* separate plugin worker for heroku

* plugin server for dev with less concurrency

* add back plugins script

* move beat log to beat

* move starting the beat into the celery worker

* add optional process types for plugins and celery

* premium redis for heroku review apps

* fix broken script

* proc/dyno names are alphanumeric

* singularize

* premium-0 redis for all heroku apps, not just review apps

* premium-0 redis also for review apps

* remove heroku redis modifications

* remove out of scope code

* run beat in bg
2020-11-20 10:42:59 +01:00
Tim Glaser
21b195a0c6
Remove unused dependencies (#2416)
* Remove unused dependencies

* Remove more deps

* add reselect back in

* Add cypress terminal back in

* install cypress terminal on the fly

* combine
2020-11-18 20:53:01 +01:00
Karl-Aksel Puulmann
b37c675903
Run E2E tests without celery (#2344)
* Run E2E tests without celery

I spent ~hour today figuring out why a task was not executing,
restarting frontend test runner each time.

By not running celery in test and forcing all things to evaluate eagerly
we can avoid that pain

* Remove a now always-passing test
2020-11-12 16:54:10 +02:00
Karl-Aksel Puulmann
67debacff2
Log web access in docker (#2230)
This is needed to close https://github.com/PostHog/deployment/issues/32
and debug https://github.com/PostHog/posthog/issues/2135
2020-11-05 12:07:39 +02:00
Marius Andra
fce86ef46d
Plugin v8 (#1946)
* plugin progress

* blah

* add posthog config for plugins

* test gitignore

* new functionality for plugins

* support local plugin paths

* also ignore symlinks

* add positional argument

* fixes

* small fixes

* config polish

* config passed to posthog plugin

* ooooooops

* symlink fix

* cleanse dir before loading

* add cache to plugins

* pickle the goods

* unlink symlink

* pass full config

* unlink even if link points to nothing

* fix fix

* return none if value is empty

* plugin model

* plugins scene

* add config schema to plugins

* install plugins

* save descriptions

* show descriptions

* edit plugin

* save plugin config

* plugin modal

* uninstall plugins

* UX cleanup

* add "required" to plugin config

* open plugin modal after install

* split to subcomponents

* install custom plugins

* rework backend for model plugins

* Plugins on models

* simple reload pubsub

* fix apps not installed

* fix master/main issue

* fix reload command

* use the github api to get the default branch

* init plugins only if not running migrate/makemigrations

* store plugins zip archives in postgres

* tag plugins to specific versions

* save plugins in pluginConfig

* update pluginConfigs instead of adding new rows, remove from redux on uninstall

* remove debug

* run plugins from db by team

* reload when deleting

* remove debug

* smarter handling of dynamic plugins, support local plugins again

* improve typings, add some nicer warnings

* yarn lock file after merge

* squash migrations and add "locked" field to plugins

* error if folder not found in zip

* unregister plugins

* skip plugin init in test mode

* basic plugin test

* avoid mutating the same prop hash

* add pip tools to requirements.txt

* fix mypy, fix manage.py script error

* avoid plugins with mypy

* mypy fix

* abstract redis into plugin and add team_id to reload

* refactor and start work on syncing with posthog.json

* start testing plugin loading from json

* test plugin deletion

* test for syncing plugins from config

* complete and then test local json plugin sync

* test converting back and forward between an local and http path

* remove global plugin config from plugins array in posthog.json

* rename configSchema --> config_schema

* fix migration after merge

* rename from_cli to from_json

* mypy

* import pip after plugin loaded

* show error details

* raise exceptions visible to the frontend

* sync plugins on load

* access control to updating plugins from the web

* access control

* remove posthog.json from git

* test config schema from json

* if you can install via the web, you can also configure

* remove separate view access

* title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository

* add self.team to plugin base class

* add instance_init method

* refactor into files

* sync global plugin config from json

* make global plugins work, add test

* global plugins in interface, make them take precedence over local plugins

* add comments to plugin base class

* reload/reset plugins before each test

* add error field to plugins

* add many plugin zips

* add many plugin zips, fix imports

* store errors on plugin object and test them

* fix types

* add null to error

* can be with any team ID in the test

* save problems running plugins in the plugin_config model

* try to create redis connection pool only once

* throw if no redis

* mypy

* get instance inside heartbeat and not top level

* try caching pubsub

* try pip install with -q

* install pip externally

* remove uuid and typing, now in stdlib

* more verbosity

* add pip back

* catch exceptions

* quiet and no input for pip

* check plugin reload every 10sec on new task

* fix type errors

* fix requirements error message

* use repository.json

* only load and reload plugins on workers

* rename task

* support local js plugins via py-mini-racer

* load js plugins from zip files

* extract jsplugin class and convert to syntax that uses global functions instead of initializing a class

* process events via grpc

* process events with the "posthog-plugins" queue to enable plugins

* remove old native python & mini racer plugin code

* default to false

* change env vars

* fix test

* remove grpc tools

* skip plugins in migrate.py scripts

* fix migration

* change output of settings debug banner to STDERR

* start posthog plugin server with worker

* try to fix python 3.7 test

* add fallback for the optional argument

* annoying CI test debug

* try to finally fix python 3.7 test

* here we go again

* move plugins under instance

* move plugins npm start into its own folder

* more console.log debugginf

* and again

* move plugins to separate script

* more prints

* fix test error

* docker config

* small fixes

* dckerfile fix

* reload plugins via pubsub, upgrade version

* plugins that support team setup code

* sync if made changes from config

* move plugins in menu

* require node 14 in heroku for better plugin support (namely ?. support)

* bump node version in dockerfiles

* update node versions for github actions

* update the concurrency for heroku workers

* update the concurrency for heroku workers (add link)

* Fix migrations after merge

* add ignore_result to process_event tasks

* fix: docker-preview run in parallel bug

* change order of commands

* remove separate plugins server conf script

* clarify intent

* revert castaway change

* add context to plugins/sync.py

* change everything to ValidationError

* delegate destroy to super

* no request to repository url if can't install

* make the if cleaner

* add clarifying line

* add clarifying line

* fix url field type

* rename get_redis_instance to get_client, move to posthog.redis

* remove duplicate validation

* flip if around

* simplify api logic

* simplify plugin_config api, fix global_plugin error

* remove unnecessary field rename

* mypy

* Plugins UI (#2090)

* base UI

* more UI

* load plugin image if available

* toggle enabled plugin

* plugin cards for available

* custom plugins

* change plugin configuration to drawer

* asks for confirmation when enabling or disabling a plugin

* loading state

* separation of concerns, leave new styles for separate PR

* general improvements

* remove button when installation is not available

* preemptively avoid merge conflict with #2114

* move papercups widget & hide bottom bar when drawer is open

* allow clicking the entire plugin card

* address all feedback

* move plugins under "project" menu

* Hide "configure" from globally enabled plugins

Co-authored-by: Marius Andra <marius.andra@gmail.com>

* add plugins opt in toggle to project/plugins

* choose pipeline based on team setting

* add "beta"

* plugin opt-in opt-out pages

* adjust install button

* remove tasks that are never called, remove PLUGINS_ENABLED global key

* fix responsive card display

* fix typo and drawer width

* skeleton fixes

* typo

* use "posthog-plugin-server" npm package

* "posthog-plugin-server" doc

* require the plugin server to be online before enabling plugins

* remove a few needeless "?." cases

* add hint for config_schema

* add hint for errors

* show plugin errors

* stop clicks if clicking on error

* show plugin errors

* loading indicators

* reload plugins when opting in/out

* nicer beta tag

* add frontend type

* fix mypy error

* fix test

* disable plugins if MULTI_TENANCY

* upgrade plugin-server version

* save event with plugin error

* upgrade plugin server

* squashed & optimized migrations

* remove unused import

* updates opt-in copy & hides tech details for cloud version

* fix cypress tests

* compare with None

* change plugins url and add redirect

* remove ellipsis

* use code snippet in plugin errors

* change github regex

* fix loading flickering on installing plugins

* add comment to plugin archive

* fix python style

* remove pip-tools (relic from the python plugin era)

* hard pin plugin server version

* remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway)

* update lockfile

Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
2020-11-02 15:08:30 +01:00
Marius Andra
a5d9f98e00
Reduce Heroku worker thread count (#2092)
* update the concurrency for heroku workers

* update the concurrency for heroku workers (add link)
2020-10-30 10:17:55 +01:00
Tim Glaser
30f29cf017
Fix bin/tests too many files watching error (#2078) 2020-10-28 19:03:35 +01:00
Tim Glaser
83ab0bb337
Automatically run CH migrations locally (#1978) 2020-10-22 10:34:30 +02:00
Tim Glaser
88e20220d1
Update test runner (#1952)
* Remove cypress from package.json

* Update test runner

* correct dommand

* Use npx
2020-10-21 11:42:17 +02:00
Tim Glaser
21622d6838
Make log output locally less noisy (#1943) 2020-10-20 16:47:35 +03:00
Yakko Majuri
b48156f37d
fixed webpack on docker-compose.dev (#1921)
Co-authored-by: Yakko <yakko@yakkos-mbp.home>
2020-10-18 12:30:46 +02:00
Tim Glaser
80d20e385b
Clickhouse use elements chain (#1849)
* Clickhouse use elements chain

* Fix stuff

* Add action tests and start regex

* Progress

* Progress part deux

* Fix everything

* Add tag name filtering

* Fix funnels

* Fix tag name regex

* Fix ordering

* Fix type issues

* Fix empty nth-child

* Remove commented code

* Split with semicolon and escaped quotes

* Specify all select columns
2020-10-16 14:07:03 +02:00
Tim Glaser
92e8bbd283
[Clickhouse] Event list (#1787)
* convert sessions table logic to TS

* convert rest of sessions to TS

* sessions table logic refactor, store date in the url

* add back/forward buttons

* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter

* prevent multiple queries

* throw error if failed instead of returning an empty list

* date from filters

* rename offset to nextOffset

* initial limit/offset block

* indent sql

* support limit + offset

* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)

* based offset is always 0

* default limit to 50

* events in clickhouse sessions

* add elements to query results

* add person properties to sessions query response

* show seconds with two digits

* fix pagination, timestamp calculation and ordering on pages 2 and beyond

* mypy

* fix test

* add default time to fix test, fix some any(*) filter issues

* remove reverse

* WIP event list

* Events progress

* Finish off event listing, skip live actions for now

* Fix mypy

* Fix mypy again

* Try fixing mypy

* Fix assertnumqueries

* Fix tests

* Fix tests

* fix test

* Fix tests

* Fix tests

* Fix tests again

Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
2020-10-01 15:47:35 +02:00
James Greenhill
379518e285
"Clickhouse Features V2 (#1565)" (#1750)
* initial

* migration command

* migrations working

* add modelless views for clickhouse

* initial testing structure

* use test factory

* scaffold for all tests

* add insight and person api

* add basic readme

* add client

* change how migrations are run

* add base tables

* ingesting events

* restore delay

* remove print

* updated testing flow

* changed sessions tests

* update tests

* reorganized sql

* parametrize strings

* element list query

* change to seralizer

* add values endpoint

* retrieve with filter

* pruned code to prepare for staged merge

* working ingestion again

* tests for ee

* undo unneeded tests right now

* fix linting

* more typing errors

* fix tests

* add clickhouse image to workflow

* move to right job

* remove django_clickhouse

* return database url

* run super

* remove keepdb

* reordered calls

* fix type

* fractional seconds

* fix type error

* add checks

* remove retention sql

* fix tests

* add property storage and tests

* merge master

* fix tests

* fix tests

* .

* remove keepdb

* format python files

* update CI env vars

* Override defaults and insecure tests

* Update how ClickHouse database gets evaluated

* remove bootstrapping clickhouse database routine

* Don't initialize the clickhouse connection unless we say it's primary

* .

* fixed id generation

* remove dump

* black settings

* empty client

* add param

* move docker-compose for ch to ee dir

* Add _public_ key to repo for verifying self signed cert on server

* update ee compose file for ee dir

* fix a few issues with tls in migrations

* update migrations to be flexible about storage profile and engine

* black settings

* add elements prop tables

* add elements prop tables

* working filter

* refactored

* better url handling

* add mapping table

* add processing to worker task

* working cohort with actions

* add cohort property filtering

* add cohort property filtering

* reformat and add cohort processing

* prop clauses

* add util

* add more util

* add clickhouse modifier

* Clickhouse Sessions (#1623)

* sessions sql

* skeleton

* add endpoint

* better tests

* sessions list

* merge clickhouse-actions

* added session endpoint

* sessions sql working again

* add clickhouse modifier

* session avg with props working

* add dist

* tests working (no list)

* list working

* add formatting

* more formatting

* fix tests

* dummy commit

* fix types

* remove unnecessary improt

* ignore type when importing from ee in task

* fix test running

* Clickhouse Trends Base (#1609)

* initial working

* date param almost working

* fix date range and labels

* fixed monthly math

* handle compare

* change table

* using new event ingestion

* direct query actions working

* remove interface

* fix date range

* properties initial working

* handle operator

* handle operator

* move timestamp parse

* move more to util

* inital breaking down working

* working cohort breakdown

* some tests running

* fix sessions

* cohort tests

* action and interval test

* reorder cohort filtering

* rename retention test

* fix inits

* change multitenancy tests

* fix types

* fix optional types

* replace ch_client.execute with sync_execute

* replace ch_client.execute with sync_execute, part 2

* Clickhouse Stickiness + Process Event (#1654)

* generate clickhouse uuid script

* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG

* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function

* Fix test_process_event_ee.py, remove all calls to Person.objects.*

* add back util

* fix broken imports

* improve process_event test clickhouse queries

* Basic stickiness query

* Clickhouse Stickiness tests

* stickiness test [WIP, actions fail]

* generate clickhouse uuid script

* change default test runner if PRIMARY_DB=clickhouse

* fix stickiness test for actions

* fix merge bug

* remove _create_person stub; cohort person_id is UUID now

* fix typing

* Clickhouse trends process math (#1660)

* most of process math works

* all process math

* fix ordering issue

* unusued imports

* update property comparison for process_event_ee

* indentation wrong missing calls

* demo users and events (#1661)

* finish breakdown filtering tests and reformat label function

* add increment to demo_data

* update demo data populating

* Add people endpoint for ch (#1670)

* add people endpoint for ch

* stickiness people

* fix value padding

* add process math to breakdown and

* add limit

* fix tests

* condensed code

* converted test to factory

* add people tests

* add month handling

* add typing fix

* change people test handling

* fix tests

* Clickhouse funnels 2 (#1668)

* add elements to create_event

* WIP closes #1663 Add funnels to clickhouse

* Make funnels work

* Clean up

* Move filtering around

* Add mypy tests and fix

* Performance improvements

* fix person tests again

* add people for funnel endpoint

* fix prop numbering

Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>

* merge master

* add retention

* update types

* more typing errors

* fix types

* bug with kafka payload, elements insert, and demo data

* Clickhouse Paths (#1657)

* paths clickhouse test (fails)

* add elements to create_event

* make this fail for clickhouse

* hardcoded query that returns good results for $pageviews, no filters yet

* clean up queries

* bound by time, fix 30min new session boundary

* support screen and custom events

* add properties filter

* paths url

* filter by path start

* better path start test

* even better path start test

* start from the first "path start" in a group

* test for person_id in paths

* partition by person_id for POSTGRES paths

* partition by person_id for Clickhouse paths

* clean up order in paths test

* clean up order in paths test

* join elements

* force element order on element group creation

* remove "order" when creating elements in tests and demo

* get list of elements for paths

* add limit to paths query

* use materialized view

* rename "element_hash" to "elements_hash" (no change in db)

* cull rows that are definitely unused

* simplify query

* New highly optimized paths clickhouse query

* start_point for $autocapture paths

* extract event property values from clickhouse

* prevent crash

* select one element sql

* get elements for event

* remove lodash

* remove host from $pageview path elements if same domain as incoming path

* show metadata based on loaded paths filter, not in flight filter

* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph

* add test that makes the Postgres paths query fail

* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points

* create automatic /demo urls that match the real urls (no ending /)

* fix elements queries

* path element joins

* create persons via postgres in paths test

* change serializers back to id

* fix tests with uuid

* fix demo

* more bugs

* fix type

* change now to timezone aware

* [clickhouse] retention filters (#1725)

* implemented target entity and prop filtering

* add insight view override

* fix endpoint and filters

* include tests

* fix tests

* add period filtering

* .

* fix pg param name

* add filtering params to both queries in retention sql

* fix param again

* change to todatetime

* change tz to timezone

* add back timezone in model/event

* [clickhouse] feature flag endpoint requests (#1731)

* add feature flags to endpoints

* add flags to endpoints that check on request

* remove magic strings and fill in missing flags

* fix types

* add missing flag

* change from iso

* fix more timestamps and comparator

* change _people to get_people in actions view

* remove action and cohort populating

* change inheritance

* "Clickhouse Features V2 (#1565)"

This reverts commit 0b371d43ec.

* fix types

* change to super

* change to super x2

Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
2020-09-29 15:17:26 +01:00
James Greenhill
0b371d43ec
Revert "Clickhouse Features (#1565)" (#1748)
This reverts commit 24713b923d.
2020-09-29 12:11:28 +01:00
Eric Duong
24713b923d
Clickhouse Features (#1565)
* initial

* migration command

* migrations working

* add modelless views for clickhouse

* initial testing structure

* use test factory

* scaffold for all tests

* add insight and person api

* add basic readme

* add client

* change how migrations are run

* add base tables

* ingesting events

* restore delay

* remove print

* updated testing flow

* changed sessions tests

* update tests

* reorganized sql

* parametrize strings

* element list query

* change to seralizer

* add values endpoint

* retrieve with filter

* pruned code to prepare for staged merge

* working ingestion again

* tests for ee

* undo unneeded tests right now

* fix linting

* more typing errors

* fix tests

* add clickhouse image to workflow

* move to right job

* remove django_clickhouse

* return database url

* run super

* remove keepdb

* reordered calls

* fix type

* fractional seconds

* fix type error

* add checks

* remove retention sql

* fix tests

* add property storage and tests

* merge master

* fix tests

* fix tests

* .

* remove keepdb

* format python files

* update CI env vars

* Override defaults and insecure tests

* Update how ClickHouse database gets evaluated

* remove bootstrapping clickhouse database routine

* Don't initialize the clickhouse connection unless we say it's primary

* .

* fixed id generation

* remove dump

* black settings

* empty client

* add param

* move docker-compose for ch to ee dir

* Add _public_ key to repo for verifying self signed cert on server

* update ee compose file for ee dir

* fix a few issues with tls in migrations

* update migrations to be flexible about storage profile and engine

* black settings

* add elements prop tables

* add elements prop tables

* working filter

* refactored

* better url handling

* add mapping table

* add processing to worker task

* working cohort with actions

* add cohort property filtering

* add cohort property filtering

* reformat and add cohort processing

* prop clauses

* add util

* add more util

* add clickhouse modifier

* Clickhouse Sessions (#1623)

* sessions sql

* skeleton

* add endpoint

* better tests

* sessions list

* merge clickhouse-actions

* added session endpoint

* sessions sql working again

* add clickhouse modifier

* session avg with props working

* add dist

* tests working (no list)

* list working

* add formatting

* more formatting

* fix tests

* dummy commit

* fix types

* remove unnecessary improt

* ignore type when importing from ee in task

* fix test running

* Clickhouse Trends Base (#1609)

* initial working

* date param almost working

* fix date range and labels

* fixed monthly math

* handle compare

* change table

* using new event ingestion

* direct query actions working

* remove interface

* fix date range

* properties initial working

* handle operator

* handle operator

* move timestamp parse

* move more to util

* inital breaking down working

* working cohort breakdown

* some tests running

* fix sessions

* cohort tests

* action and interval test

* reorder cohort filtering

* rename retention test

* fix inits

* change multitenancy tests

* fix types

* fix optional types

* replace ch_client.execute with sync_execute

* replace ch_client.execute with sync_execute, part 2

* Clickhouse Stickiness + Process Event (#1654)

* generate clickhouse uuid script

* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG

* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function

* Fix test_process_event_ee.py, remove all calls to Person.objects.*

* add back util

* fix broken imports

* improve process_event test clickhouse queries

* Basic stickiness query

* Clickhouse Stickiness tests

* stickiness test [WIP, actions fail]

* generate clickhouse uuid script

* change default test runner if PRIMARY_DB=clickhouse

* fix stickiness test for actions

* fix merge bug

* remove _create_person stub; cohort person_id is UUID now

* fix typing

* Clickhouse trends process math (#1660)

* most of process math works

* all process math

* fix ordering issue

* unusued imports

* update property comparison for process_event_ee

* indentation wrong missing calls

* demo users and events (#1661)

* finish breakdown filtering tests and reformat label function

* add increment to demo_data

* update demo data populating

* Add people endpoint for ch (#1670)

* add people endpoint for ch

* stickiness people

* fix value padding

* add process math to breakdown and

* add limit

* fix tests

* condensed code

* converted test to factory

* add people tests

* add month handling

* add typing fix

* change people test handling

* fix tests

* Clickhouse funnels 2 (#1668)

* add elements to create_event

* WIP closes #1663 Add funnels to clickhouse

* Make funnels work

* Clean up

* Move filtering around

* Add mypy tests and fix

* Performance improvements

* fix person tests again

* add people for funnel endpoint

* fix prop numbering

Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>

* merge master

* add retention

* update types

* more typing errors

* fix types

* bug with kafka payload, elements insert, and demo data

* Clickhouse Paths (#1657)

* paths clickhouse test (fails)

* add elements to create_event

* make this fail for clickhouse

* hardcoded query that returns good results for $pageviews, no filters yet

* clean up queries

* bound by time, fix 30min new session boundary

* support screen and custom events

* add properties filter

* paths url

* filter by path start

* better path start test

* even better path start test

* start from the first "path start" in a group

* test for person_id in paths

* partition by person_id for POSTGRES paths

* partition by person_id for Clickhouse paths

* clean up order in paths test

* clean up order in paths test

* join elements

* force element order on element group creation

* remove "order" when creating elements in tests and demo

* get list of elements for paths

* add limit to paths query

* use materialized view

* rename "element_hash" to "elements_hash" (no change in db)

* cull rows that are definitely unused

* simplify query

* New highly optimized paths clickhouse query

* start_point for $autocapture paths

* extract event property values from clickhouse

* prevent crash

* select one element sql

* get elements for event

* remove lodash

* remove host from $pageview path elements if same domain as incoming path

* show metadata based on loaded paths filter, not in flight filter

* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph

* add test that makes the Postgres paths query fail

* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points

* create automatic /demo urls that match the real urls (no ending /)

* fix elements queries

* path element joins

* create persons via postgres in paths test

* change serializers back to id

* fix tests with uuid

* fix demo

* more bugs

* fix type

* change now to timezone aware

* [clickhouse] retention filters (#1725)

* implemented target entity and prop filtering

* add insight view override

* fix endpoint and filters

* include tests

* fix tests

* add period filtering

* .

* fix pg param name

* add filtering params to both queries in retention sql

* fix param again

* change to todatetime

* change tz to timezone

* add back timezone in model/event

* [clickhouse] feature flag endpoint requests (#1731)

* add feature flags to endpoints

* add flags to endpoints that check on request

* remove magic strings and fill in missing flags

* fix types

* add missing flag

* change from iso

* fix more timestamps and comparator

* change _people to get_people in actions view

* remove action and cohort populating

Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
2020-09-29 06:36:50 -04:00
James Greenhill
5293b4004a
Remove celerybeat.pid before starting docker worker (#1608) 2020-09-08 10:32:41 -07:00
James Greenhill
7714a65108
Use celery defaults, bumping workers only increased latency of event processing (#1584) 2020-09-04 16:21:55 -07:00
James Greenhill
aa7de23aec
Increase number of concurrent celery workers in production (#1583) 2020-09-04 15:56:58 -07:00
Marius Andra
52cdd7e572
New toolbar cookie + Secure session/csrf cookies (#1387)
* - set a custom cookie to tell the toolbar we're logged in
- make other cookies secure

* skip adding new toolbar cookie on API requests

* fix demo page local https

* enable/disable secure cookies as needed

* show django toolbar only for API requests

* update posthog-js and kea versions

* satisfy mypy

* add SECURE_COOKIES=0 to cypress tests, since they run over HTTP

* revert kea version

* add test for toolbar cookie middleware

* try something for cypress

* add debug

* add cypress-terminal-report for better logging

* fix demo api_url

* revert previous change that triggered cypress
2020-08-11 10:52:43 +02:00
Karl-Aksel Puulmann
596635fbcd
Improve ./bin/test command (#1074)
It did not run under `fish` shell at all, added a shebang to make it
work.

Also it assumed nodemon is installed globally. Instead we can add
nodemon as a devDependency and use `npx` to run it.
2020-06-23 18:42:54 +02:00
Tim Glaser
be7a345a61
Https locally (#910)
* Fix temporary token issue

* only pass params once

* HTTPS locally
2020-06-04 10:14:54 +01:00
Tim Glaser
bcf427cb55
Webpack HMR, Add hashes to chunk filenames. (#878)
* Closes #877 chunk loading errors

* Closes #877 chunk loading errors

* add chunks to all webpack files, use webpack-html-plugin to make an index.html with the right names

* change to contenthash for better caching

* add dev server on a different port

* store loaded scenes in reducer

* add react-hot-loader

* add react-hot-loader to all code split points

* fix action pages HMR

* ignore cypress screenshots

* generate django login/signup page layout with webpack html plugin

* move to devDependencies

* expose webpack-dev-server ports

* run tests on the production docker image

* start webpack dev server on a custom host if requested

* revert e2e to dev dockerfile

* add test travis config

* add stages

* add travis conf

* cache node and pip

* node 11

* travis ci bump

* node via nvm

* install v12, disable cypress for a moment

* remove 2 commands, test cached build time

* try different yarn cache, add cypress

* add postgres 12

* migrate before cypress

* remove latest postgres for now

* createdb before cypress

* Try different port

* cull packages

* remove hash from main bundle filenames to not break anything

* cypress port 8000

* cypress tests with production docker

* don't use the /code path in e2e test, use docker image

* remove hash from css to work better with editor

* only one export from actions

* remove travis test

Co-authored-by: Marius Andra <marius.andra@gmail.com>
2020-06-02 10:39:53 +01:00
James Greenhill
2fd1bc6377
Optimize event telemetry pipeline (#833)
* Split out celery worker into its own container for easier debugging

* Refactor creating people from events and serialize Team to celery worker

* type fixes

* remove todo message

* use the correct marshal (unmarshal)

* limit to just removing try except for person creation

* switch from count to exists
2020-05-26 20:18:01 +01:00
Marius Andra
14251dd9bd
Update links to docs.posthog.com (#847)
* update links to docs

* replace all old doc linnks

Co-authored-by: Eric <eeoneric@gmail.com>
2020-05-26 11:20:21 -04:00
Eric Duong
a012aa6723
Cypress tests (#789)
* initial test

* add yaml

* add logs

* without tail

* run again for consistency

* wait longer

* manuall configure cypress

* use latest ubuntu

* try installing all

* yarn install

* yarn install with cypress

* full flow

* run again for consistency

* initial tests for trends

* handle exception

* separate e2e docker-compose

* run on pr

* trends-testing foundation

* run test

* don't pass everything

* handle uncaught exceptions

* fix dashboard tests

* change email domain

* change dataattr to data-attr and boilerplate cleanup

* base url config

* add script for test runner

* change default postgres
2020-05-19 14:48:10 -04:00
Tim Glaser
8e6b4f56b5
Closes #169 break down by cohort (#690)
* Closes #169 break down by cohort

* Fix test

* Prettier breakdown filters

* Rerender each time it's opened

* Fix tests

* Add 'all users' option in breakdown

* Make people work with cohorts

* Precalculate cohorts

* Return last_calculation

* Show precalculation in the frontend

* Closes #675 fix icon alignment

* Frontend fixes, antd and more

* Make sure unsetting breakdown doesn't break things

* Semantic name for migration

* Separate data migration + cohort key bugfix

* Correctly filter people by event

* Typing
2020-05-11 22:06:35 +01:00
Tim Glaser
f198b190b9
Put break down in Graph (#667)
* Put break down in Graph

* Remove unused function

* Order by count

* Order by pt 2

* Fix tests

* Filter people by selected breakdown value

* Fix test
2020-04-28 17:41:05 +01:00
Tim Glaser
853bcfaf57
Closes #636 docker for local dev (#638) 2020-04-22 11:12:37 +01:00
Marius Andra
14f12846ed
Helm redis (#558)
* ignore copying frontend/dist folder - otherwise whatever you build in docker will get overridden by local build artifacts if any exist in frontend/dist

* support configuring redis with POSTHOG_REDIS_HOST and other vars in addition to REDIS_URL

* remove "the next version" in worker requirement modal

* split beat and celery scripts

* remove chart folder

* celery heartbeat every 10sec, reduce distributed beat lock hold time

* remove dockerfile local link

* add localhost redis url for tests
2020-04-13 16:22:06 +01:00
Tim Glaser
1f334d1304
Speed up tests by parallelising them (#543) 2020-04-13 10:06:36 +01:00
Marius Andra
4fd38dd305
kill all start-* scripts on CTRL+C (#502) 2020-04-08 14:38:28 +01:00
Marius Andra
66780ceef7
Make workers optional for now, add a warning if they are offline (#494)
* add warning script to settings and docker-worker in case REDIS_URL is not found

* test celery heartbeat

* save celery heartbeat in redis

* /_stats/ with worker heartbeat

* refactor top content

* show worker stats

* heartbeat delay to 90sec

* update worker error message

* remove line

* add redbeat schdeduler

* add redbeat for local development
2020-04-08 13:52:29 +01:00
Marius Andra
221df7e60a start worker in background for docker 2020-04-07 14:02:09 +02:00
Marius Andra
e075838933 scripts to run background, worker and frontend simultaneously 2020-04-07 14:01:33 +02:00
Marius Andra
8798ad8d4e celery/redis workers in preview.Dockerfile 2020-04-07 14:01:33 +02:00
Marius Andra
afa3617316 add docker gunicorn workers 2020-04-02 12:06:57 +02:00
Marius Andra
93fab5498f run gunicorn in docker-server script 2020-04-02 00:55:37 +02:00
Marius Andra
6a36c304f6 collect static inside docker 2020-04-02 00:55:29 +02:00
Marius Andra
de6ebca84d move package.json to the root folder 2020-03-18 13:41:46 +01:00
Tim Glaser
7dc01a4781 Fix dockerfile 2020-02-27 16:40:56 -08:00
Tim Glaser
f119858bbd Add migrate command onstartup 2020-02-27 14:03:03 -08:00
Tim Glaser
3567523e39 Closes #171 Add Paths screen 2020-02-24 20:52:31 -08:00
Tim Glaser
6a16393ec8 Update array.js to use api_host rather than app_host for editor 2020-02-14 16:14:12 -08:00