* Improve insight refreshing UX
* Clean up
* Make sure `ComputationTimeAndRefresh` is on the right in Trends too
* Fix typing and some inconsistencies
* Remove unused const
* Update test_clickhouse_insights.py
* Update decorators.py
* Refactor `ComputationTimeWithRefresh` into own file
* Improve time to convert binning behavior, add autobinning
* Test and fix custom bin count
* Expose `ClickhouseFunnelTimeToConvert` in the API
* Revert a column name change
* Fix `bin_count` clamping
* Add an API test for viz type `time_to_convert`
* Update SQL comments
* Remove some unused imports
* Fix `FunnelTypeMixin.funnel_viz_type`
* Remove `DEBUG=1`
* Fix funnel trends backwards compatibility
* Don't set funnel_viz_type outside of funnel insight filters
* Reorder conditions for clarity and use null check
* setup funnel option type, and proper API for persons and usual funnel routes
* update tests, fix bugs
* make insights compulsory
* nvm
* remove insights key
* fix person caching
* revert problems, fix typings
* address comments
* dont use defaults in mixins
* typing
* cleanup and fix TODOs
* update test
* adapt api to 4957
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* Make `SHELL_PLUS_PRINT_SQL` clearer
* Add ClickhouseFunnelTrendsNew
* Create test_funnel_trends_new.py
* Create test_funnel_trends_v2.py
* move builder functions to funnel base
* add test classe for new funnel
* Inherit from `ClickhouseFunnelNew` and fix intervals
* Add proper formatting of trends results
* Clean tests up a little bit
* Group `FunnelWindowDaysMixin` tests in `test_funnel_persons`
* Rename `ClickhouseFunnelTrendsNew` things for clarity
* Port some original `ClickhouseFunnel` trends tests for the new query
* Only fetch initial page (100) of persons in trends query
* Describe assumptions and rename things
* Finish porting old ClickhouseFunnelTrends tests and add some new ones
* Remove unused imports
* Try to fix `test_period_not_final`
* Try to fix `test_period_not_final` again
* remove persons lists
* rename
* fix test
* add timezone to results
* add funnel trends new to api path
* revert random change
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: eric <eeoneric@gmail.com>
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* invoke the funnel or funnel trends class respectively
* mypy corrections and PR feedback
* corrected tests
Co-authored-by: eric <eeoneric@gmail.com>
* Revert "Revert "[Frontend Change] Funnels API GET -> POST|GET (#4769)" (#4777)"
This reverts commit dab41f0f6a.
* Add POST support to clickhouse funnels
* run funnel tests on clickhouse
* change get to post api
* tests
* fixed bug
* isort
* make funnels endpoint backwards compatible, add test to check this
* descriptive comment
* revert funnelLogic to current
* revert code removed in error
Co-authored-by: Alex Kim <alexkim@Alexs-MacBook-Pro-2.local>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* change get to post api
* tests
* fixed bug
* isort
* make funnels endpoint backwards compatible, add test to check this
* descriptive comment
Co-authored-by: Alex Kim <alexkim@Alexs-MacBook-Pro-2.local>
* initial api update
* make api work
* frontend changes
* hello
* adjust trend payload
* prelim working
* fix bugs
* use offset in filters to_dict
* use different pagination check
* use offset in filters to_dict
* test
* extend range for pg
* update test
* remove print
* update test
* match limits
* add api test
* center button
* nits
* UI adjustment
* update comment
* fix type error
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Upload static cohort using CSV, closes #2868
* fix tests
* Fix tests
* Fix e2e test
* Avoid double inserts
* Speed up query
* Move to params
* fix tests
* initial cleanup
* initial scaffold for cohort static logic
* intiial frontend changes
* initial working
* fix error on iterable
* stickiness people func refactor
* Use JSON instead of protobuf
* stickiness working
* clickhouse route
* unused import
* move button
* make interface easier to user
* clickhouse support
* restore cache decorator
* endpoint adjustment
* adjust how clickhouse handles inserting for static cohorts
* split out processing into pg
* use worker for clickhouse querying too
* add pg test
* ee test
* add input for cohort naming
* add stickiness test
* condition for when to show save cohort
* fix types
* minor fix
* remove repeats
* remove hardcoded id
* add date handling to trends cohort func
* remove print
* add link on success creation
* cohort working
* fix backend test
* fix cohort typescript
* remove unnecessary kafka topic
* remove brackets
* shift default
* raise exceptions when debug or test is true
* remove test flag
* move trends insight related components to different folder
* change get_earliest_timestamp
* remove extraneous condition
* fix test call
* put behind ff
* fix test x2
* fix tests
* restore exceptions hog
* fix tests
* restore exceptions hog
* fix test x20
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Deprecate dashboard item type and move to display
* Mypy rerors
* fix test
* fix
* Fix test
* Fix another test
* Remove determineInsightType
* Get rid of RetentionTable display types
* Fix update dashboard
* Duplicate existing endpoints under /api/event/sessions
* Use different endpoint for sessions table logic in sessions page
* drop irrelevant code for insights endpoint
* update documentation
* extract method
* move code under only relevant path
* Extract clickhouse sessions logic
* Refactor: Separate session list logic from rest of sessions
* move sessions list to separate file
* Move sessions list tests to separate file
* add clickhouse session list tests
* add support for duration filtering on postgres
* move api tests to right place
* kill dead code
* Use return value of add_session_recording_ids
The mutation here is incidental, this is symmetrical with how postgres
behaves.
* Implement sessions_filter
This will hold session-recording specific filters
* WIP: make filtering by session recording length possible on ch
* Kill dead method
* Allow filtering by session length
* Make logic a bit smoother around filtering by recording duration
* More common code for clickhouse_session_recording
* Add filtering tests for ch
* Resolve mypy issues
* Only run duration tests for ch
* Put new filters logic behind a feature flag
* Kill dead const
* Solve operators-related comment
* Review feedback: rename variable
* Fix test failures
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* remove unneded comment and import
* split up retention query
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* Nest endpoints under /project/ with StructuredViewSetMixin
* Rewrite URLs
* isort
* Update utils.py
* Fix errors
* Fix almoast all the errors
Last left to do: shared dashboards and permission classes.
* isort
* Adjust for master
* Add compatbility with shared dashboards
* Debug ClickHouse
* Remove some # type: ignores
* Simplify CursorPagination
* Move test base from posthog.api.test to posthog.test
* Improve API structure
* Bring back legacy endpoints
* Fix legacy compatibility
* Fix bugs and typing
* isort
* Fix hooks test
* Try fixing errors
* Fix oversight
* isort
* Fix problems
* isort
* Be more tolerant
* Fix naming and remove redundant code
* Fix imports
* Update deleteWithUndo
* Roll back
* Roll back more
* Update .gitignore
* Rollll back
* Rollllllll
* back
* Betterify
* Address feedback
* clean up reused function
* remove fluff and use constants
* standardizing compare
* segment logic for clickhouse queries that have a lot
* add default to filter.date_to
* mino unwrap
* fix test
* Backend changes to implement #1461
* Added the missing migration files
* Fixes Typecheck errors
* Refactor request.user.team_set.get() to use request.user.team
* Updated user patch method to change current_team on team id instead of name
* Merged migration file
* Changes team property to return first item in queryset
* Fixes failing tests
* Changed User api to return the ids of the teams they are currently part of
* Frontend for changing teams
* Update and simplify migrations
* Improve team and user buttons
* Make team changing backend more logical
* Improve current_team mechanics
* Update test_team.py
* Fix Team.objects.create_with_data
* Update migration
* Update tests
* Make setup_review more convenient
* Add Organization and OrganizationMembership
* Replace is_admin with level
* Extend API
* Update team.py
* Improve modeling
* Improve handling of new mechanics
* Add proper migration
* Remove _ensure_organization_and_team
* Update 0084_org_team_user.py
* Improve user, org and team creation
* Make MembershipLevel more flexible for the future
* Add member deletion
* Fix naive datetime warnings
* Update setup_review.py
* Update API route
* Make PersonalAPIKey changes
* Update models and migrations, fix typing
* Fix typing
* Use MAC-less UUID v1 instead of v4 for better performance
* Add abstract UUIDModel
* Update utils.py
* Update utils.py
* Fix multi/unicast bit
* Update APIs, frontend and tests
* Update pull_request_template.md
* Fix comment
* Fix migration error
* Compress migrations
* Updates with minimal renaming
* More updates
* Make further updates
* Update test_team_user.py
* Fix issues
* Add migration
* Satisfy mypy
* Remove Signup redirect on logged in
* Use uuid1_macless in Person
* Fix typing
* Update tests
* Update /api/team/signup to /api/organization/signup
* Fix mypy issues and update tests
The remaining failures are actually missing functionality (TDD applied), so filling these in.
* Update 0086_org_live
* Make small improvements
* Implement permissions
* Remove now unneccesary membership check
* Update setup_dev.py
* Make small frontend improvements
* Add drf-nested-routers as requirement
* Remove unused import
* Implemented nested routes
* Remove cruft
* Add relevant org/proj/user name to headings
* Fix imports
* Update migration
* Replace unreliable drf-nested-routers with drf-extensions
* Improve unset team handling
* Make org and team creation proper
* Update migration
* Fix migration
* Update TopContent
* Update command palette for new sidebar structure
* Remove deprecated demo data deletion
* Assume that each org has a project and fix typing
* Require paid plan for multiple orgs and projects
* Make HogFlix demo a separate team
* Update migration
* Slightly improve style
* Adjust page layout bottom padding
* Make user dropdown nicer
* Fix base app tests
* Satisfy mypy
* Fix test_leave_organization
* Improve wording
* Possibly fix import
* Remove misplaced None check
* Enhance org and teams APIs and add tests
* Fix /api/projects for particular Team
* Improve invites and demo data
* Address feedback
* Put everything related to billing on Organization
* Fix minor issues
* Simplify invitation creation
* Update team model
* Make orgs and projects premium only on self-hosted
* Improve testing
* Update migration
* Remove extra License import
* Fix minor issues
* Fix Django tests
* Fix Cypress
* Fix yarn build
* Fix TeamSignupViewset
* Fix posthog-production incompatibility
* Remove extraneous insight endpoint registration
* Adjust tests for posthog-production
* Simplify invitations and fix email validation
* Address all feedback
* Satisfy mypy
* Update migration
* Fix constraint removal in migration
* Update tests
* Fix test creation edge case
* Run posthog-production CI tests against this branch and teams-live
* Ensure that js_posthog_api_key is always passed
* Fix preflight check pre-login
* Update cypress tests
* Update instanceStatus.js
* Bring ee tets up to par
* Bring actions-ux-201012 back
* Cypress retry in cypress.json
* Revert "Run posthog-production CI tests against this branch and teams-live"
This reverts commit d79cb844d8.
Co-authored-by: anna <ms.annaphilips@gmail.com>
Co-authored-by: Anna Philips <aphilips@matmacorp.com>
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* fix mypy error, indent SQL
* add pagination test for postgres
* Run insight API tests in Clickhouse
Co-authored-by: Eric <eeoneric@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
* change inheritance
* "Clickhouse Features V2 (#1565)"
This reverts commit 0b371d43ec.
* fix types
* change to super
* change to super x2
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
Co-authored-by: James Greenhill <jams@uber.com>
