* backend fixes and test
* add breakdown value to pie chart
* adjust test
* fix faulty test
* fill param
* fix formula tests
* more date passing
* more cleanup
* all tests working
* make test data explicit and add better checks
* support both ee and postgres
* length checks
* test(breakdown): add test for week interval rounding overshooting
This refers to https://github.com/PostHog/posthog/issues/2675
* fix(breakdown): resolve week rounding interval over shoot for trends
Previously there was a fix for another issue with weekly intervals not
spanning a large enough range, see
69ba0b1d90
for details.
The issue that this was trying to resolve was consider a date range
date_from, date_to and a week interval. The range can cross two weeks,
but prior to the above commit it would only show one tick/week on the
graph. In this case we just added on another week to show.
This doesn't help however in the case where the range is less than a
week, in which case we'd end up showing a week/tick on the graph with
a zero value.
Instead, we remove the additional 1 week and simply ensure that both
lower and upper bounds are included in the clickhouse zero fill.
Closes https://github.com/PostHog/posthog/issues/2675
* fix(lifecycle): update lifecycle to use updated week interval handling
This is due to the change in the way the week interval is handled in the
previous commi, where we removed a +1 we were adding to week intervals
and thus getting too many date axis ticks on trend graphs.
* ci(tests): only run non-ee tests for non ee action step
It seems that we are segregating tests between ee and non-ee via a
folder structure. This is a bit of a hack, but it works for now but
ideally we should converge on one set of tests for which we identify if
they are supported on clickhouse or not via pytest markers. This should
simplify working with the tests somewhat, e.g. making them easier to
locate, update, and e.g. debug in vscode/codespaces.
* test(trends): correct old week interval test. It was wrong before
Now with the change to how we create the interval, it's fixed.
* fix(trends): avoid including one week too many when deciding on bounds
Previously there was a test in here that was saying that if we request a
graph of events, with week interval ticks starting on a Sunday, we'd
actually end up also including the Sunday before due to some arithmetic
mistakes.
* fix(trends): resolve week interval rounding issues for trends query
* test(identify): update test func to accept `properties` argument
* Use proper interval calculation in the funnel trends query
* Add some comments
* Update `test_filter`
* Rework `NULL_SQL` to use CH `INTERVAL` too
* Fix week-based relative `date_from` support not existing
* Make use of `toInterval*` functions and inject less
* Add fallback for `date_from` in `ClickhouseSessionsAvg`
* Load session events asynchronously from a separate endpoint
This mirrors the behavior of postgres query
* Simplify backend & query
event_count is unused
don't select unused columns in list query
* Rename filter_by_session_recordings to filter_by_session_recordings
This is more in-line with what the function actually does
* Update types, handle start/end url properly
* start_url / end_url to session result
* Update sessions list builder tests
* Remove some `session.events` references
* Remove unneeded code
* Simplify filteredSessions
* Fix type issue
* Add test for session properties
* Test and fix start_url/end_url
* Add test for the new sessions API endpoint
* Improve types
* Update py types again
* Fix bug
* Proposal: Use `ValidationError` over `ValueError` in views
This generates a nicer exception for our users (400 response instead of
500) and we cut down on 500 errors for things that are not expected to
work.
* Capitalization
Co-authored-by: Michael Matloka <dev@twixes.com>
* first try
* add entity query
* fix type issues
* used in trends
* fix test
* change select format
* change var
* add cohort filter
* add cohort conditions
* change pdi to distinct_ids
* do proper interval rounding on normal trends
* patch inconsistency
* remove microseconds
* conditionall round interval
* adjust how date_from is handled
* add retention test
* edits and split test
* add abstract test classes
* example implementation
* change retention tests name to implement interval tests
* fix week and month interval test
* only add aggregated when needed
* change labels with new tests
* add timerange tests
* add clickhouse query for people breakdown
* remove log
* add pg implementation
* remove caching and fix timestamp formatting
* modify how people are returned
* reimplement frontend logic for handling new people payload
* remove period check
* change params
* add returndict
* DRY retention param formatting
* remove feature flag for retention
* remove feature flag for retention
* intial working
* intial working with test
* return tests
* fix interval and add monthly test
* add in person_id join
* small restyle
* postgres lifecycle working
* add action handling to postgres query
* add action handling and tests
* fix typing
* visualizing with temp params and added negatives for dormant
* frontend
* fix intervals
* remove unnecessary import
* add person enppoint
* add person endpoint and tests
* add next
* add pagianted test
* fix types
* add frontend logic'
* fixed date range
* disable compare filter on lifecycle
* add diasbled to chartfilter
* return class to personviewset
* added constant
* fix distinct_id and new event within period condition
* replace people queries and fix ch query too
* DRY
* fix null states
* comparefilter nullstate
* add wrapper back to endpoint
* fix datetime formatting
* remove extra stack flag
* reduce filters when it's lifecycle and replace constants
* add default for lifecycle into trendlogic
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* remove unneded comment and import
* split up retention query
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* clean up reused function
* remove fluff and use constants
* standardizing compare
* segment logic for clickhouse queries that have a lot
* add default to filter.date_to
* mino unwrap
* fix test
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
* change inheritance
* "Clickhouse Features V2 (#1565)"
This reverts commit 0b371d43ec.
* fix types
* change to super
* change to super x2
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
