* Revert "Revert "Use postgres to grab person (#1957)" (#1963)"
This reverts commit 94f44bdf46.
* Filter by team_id
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Backend changes to implement #1461
* Added the missing migration files
* Fixes Typecheck errors
* Refactor request.user.team_set.get() to use request.user.team
* Updated user patch method to change current_team on team id instead of name
* Merged migration file
* Changes team property to return first item in queryset
* Fixes failing tests
* Changed User api to return the ids of the teams they are currently part of
* Frontend for changing teams
* Update and simplify migrations
* Improve team and user buttons
* Make team changing backend more logical
* Improve current_team mechanics
* Update test_team.py
* Fix Team.objects.create_with_data
* Update migration
* Update tests
* Make setup_review more convenient
* Add Organization and OrganizationMembership
* Replace is_admin with level
* Extend API
* Update team.py
* Improve modeling
* Improve handling of new mechanics
* Add proper migration
* Remove _ensure_organization_and_team
* Update 0084_org_team_user.py
* Improve user, org and team creation
* Make MembershipLevel more flexible for the future
* Add member deletion
* Fix naive datetime warnings
* Update setup_review.py
* Update API route
* Make PersonalAPIKey changes
* Update models and migrations, fix typing
* Fix typing
* Use MAC-less UUID v1 instead of v4 for better performance
* Add abstract UUIDModel
* Update utils.py
* Update utils.py
* Fix multi/unicast bit
* Update APIs, frontend and tests
* Update pull_request_template.md
* Fix comment
* Fix migration error
* Compress migrations
* Updates with minimal renaming
* More updates
* Make further updates
* Update test_team_user.py
* Fix issues
* Add migration
* Satisfy mypy
* Remove Signup redirect on logged in
* Use uuid1_macless in Person
* Fix typing
* Update tests
* Update /api/team/signup to /api/organization/signup
* Fix mypy issues and update tests
The remaining failures are actually missing functionality (TDD applied), so filling these in.
* Update 0086_org_live
* Make small improvements
* Implement permissions
* Remove now unneccesary membership check
* Update setup_dev.py
* Make small frontend improvements
* Add drf-nested-routers as requirement
* Remove unused import
* Implemented nested routes
* Remove cruft
* Add relevant org/proj/user name to headings
* Fix imports
* Update migration
* Replace unreliable drf-nested-routers with drf-extensions
* Improve unset team handling
* Make org and team creation proper
* Update migration
* Fix migration
* Update TopContent
* Update command palette for new sidebar structure
* Remove deprecated demo data deletion
* Assume that each org has a project and fix typing
* Require paid plan for multiple orgs and projects
* Make HogFlix demo a separate team
* Update migration
* Slightly improve style
* Adjust page layout bottom padding
* Make user dropdown nicer
* Fix base app tests
* Satisfy mypy
* Fix test_leave_organization
* Improve wording
* Possibly fix import
* Remove misplaced None check
* Enhance org and teams APIs and add tests
* Fix /api/projects for particular Team
* Improve invites and demo data
* Address feedback
* Put everything related to billing on Organization
* Fix minor issues
* Simplify invitation creation
* Update team model
* Make orgs and projects premium only on self-hosted
* Improve testing
* Update migration
* Remove extra License import
* Fix minor issues
* Fix Django tests
* Fix Cypress
* Fix yarn build
* Fix TeamSignupViewset
* Fix posthog-production incompatibility
* Remove extraneous insight endpoint registration
* Adjust tests for posthog-production
* Simplify invitations and fix email validation
* Address all feedback
* Satisfy mypy
* Update migration
* Fix constraint removal in migration
* Update tests
* Fix test creation edge case
* Run posthog-production CI tests against this branch and teams-live
* Ensure that js_posthog_api_key is always passed
* Fix preflight check pre-login
* Update cypress tests
* Update instanceStatus.js
* Bring ee tets up to par
* Bring actions-ux-201012 back
* Cypress retry in cypress.json
* Revert "Run posthog-production CI tests against this branch and teams-live"
This reverts commit d79cb844d8.
Co-authored-by: anna <ms.annaphilips@gmail.com>
Co-authored-by: Anna Philips <aphilips@matmacorp.com>
* fixed negation operators for person and cohorts
* add person negation test
* fix tests
* add leakeage tests
* add space
* add debug statements
* match distinct_id
* Clickhouse use elements chain
* Fix stuff
* Add action tests and start regex
* Progress
* Progress part deux
* Fix everything
* Add tag name filtering
* Fix funnels
* Fix tag name regex
* Fix ordering
* Fix type issues
* Fix empty nth-child
* Remove commented code
* Split with semicolon and escaped quotes
* Specify all select columns
* Use toDate to speed up ordering
* Add another check
* Try debugging test
* Bla
* aaaaa
* cba
* Only return 100 results
* not even sure
* order by desc both
* Add proper test
Co-authored-by: Eric <eeoneric@gmail.com>
* change parsing to include operators'
* make properties test into factory
* add clickhouse test implementation and fix another test
* add custom test to clickhouse filter tests
* all tests besides json filtering
* add json test
* fix tests
* fix type errors
* Clickhouse use elements chain
* Fix stuff
* Add action tests and start regex
* Progress
* Progress part deux
* Fix everything
* Add tag name filtering
* Fix funnels
* Fix tag name regex
* Fix ordering
* Fix type issues
* Fix empty nth-child
* Remove commented code
* Split with semicolon and escaped quotes
* Specify all select columns
* working date filter
* frontend for period
* remove period
* add params
* add period
* period working
* add test for period
* merge master
* fix test
* add dropdowns for period date filtering
* pass params and handle properly
* fix labels and readjust tests
* test split for retentino period
* make insight retention test just test for response
* add condition
* feature flag the frontend and bring back ff for the ch endpoint
* fix cohort filtering
* fix cohort querying
* remove init
* remove feature flag
* add back featureflag
* block person modal on retention page if clickhouse enabled
* use distinct_id instead of joining for person_id
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
* Fix person querying
* Fix flake
* Fix person stuff
* Fix test
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* fix mypy error, indent SQL
* add pagination test for postgres
* Run insight API tests in Clickhouse
Co-authored-by: Eric <eeoneric@gmail.com>
* Revert "Revert "Convert sync execute to use clickhouse connection pool (#1762)" (#1770)"
This reverts commit cbb8fb01a9.
* remove runtime requirement from posthog, that is convered in posthog-production
* Bump the max number of connections in the pool
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
* change inheritance
* "Clickhouse Features V2 (#1565)"
This reverts commit 0b371d43ec.
* fix types
* change to super
* change to super x2
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* Do not shadow Kafka default columns _timestamp and _offset
* drop the columns only on the kafka_ tables
* include data types
* fields cleanups
* datetime -> datetime64
* double import datetime
* Create Omni-Person model for managing people in Clickhouse
* type fixes
* rebase all the things
* cleanups
* id -> uuid for events in clickhouse
* cleanups and type checks
* Further cleanups and uuid conversions
* kafka fix
* break out serializer across kafka clients
* fix a few bugs w/ datetime types
* basic fix for people kafka table
* fix migration errors (copy pasta errors)
* Use KafkaProducer for Omni Person emitting
* setup mock kafka producer
* undo some work for inserting
* Test TestKafkaProducer
* change if order, obvious mistake
* remove unnecessary function arg
* Fix getters for new column
* Test fixes
* mirror columns across element queries
* firm up handling of timestamps
* only return timestamps for handle_timestamp
* Correct heroku config for Kafka
* Use ReplacingMergeTree for elements, remove element_groups and use elements_hash as a virtual "pk"
* remove unused ELEMENT_GROUP_TABLE_SQL
* merge fixes
* use redis cache to avoid writing duplicate elements to clickhouse
* move fakeredis to requirements.txt
* add team_id to cache key
* remove elements_group kafka table references
* add elements_hash to clickhouse element serializer
* fix cache key
* rename few keys
* add test runner to ease pycharm dev
* fix a some mypy errors
* remove typo
Co-authored-by: Eric <eeoneric@gmail.com>
* add test runner to ease pycharm dev
* fix broken import
* drop and recreate the clickhouse test db before running tests
* fix person uuid str json serialization issue
* make kafka optional in tests
* fix inits
* remove need for kafka in person.py
* fix a bunch of mypy errors
* fix function and add process_event to pipeline
* fixed missing params and tests
* change uuid and fix types
* types
* optimize for merge prop test
* make ClickhouseProducer to produce to clickhouse one way or another
* annotate types
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Query optimizations
* more sql optimizations
* checkpoint
* fix migration
* add UUID field to person
* use django signals to signal that clickhouse needs to be updated
* cleanup person logic
* cleanups
* update migration
* Don't setup the django signals unless we are for sure using ee setup
* expecting back to 30 queries for capturing with new person
* add .venv to .gitignore
* add env back to .gitignore
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-18.ec2.internal>
* Publish events to Kafka for consumption
* Commit avro idl's for event schemas
* convert client to use github.com/dpkp/kafka-python
* events loaded into clickhouse from Kafka
* remove cruft
* Publish events to Kafka for consumption
* convert client to use github.com/dpkp/kafka-python
* remove cruft
* include kafka migrations
* bugfixes for migrations
* use constants for consistency
* wrap up local migrations
* small fixes
* tune ups
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* change property parsing
* indentation wrong missing calls
* uuid4 instead of call to CH
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
Co-authored-by: James Greenhill <jams@uber.com>
* Add missing migration
* Add generate_random_token() model util
* Move PublicTokenAuthentication to utils
* Make use of generate_random_token
* Add User.personal_access_token field
* Add PersonalAccessTokenAuthentication
* Fix PublicTokenAuthentication
* Fix migration and auth import
* Add personal_access_token to user API
* Update Setup.js
* Support trailing slash in API
* Improve PAT auth quality
* Add django-rest-hooks requirement
* Update settings.py for rest_hooks
* Fix django-rest-hooks requirement
* Bring back API routes with no double trailing slash
* Rename posthog.api.team to team_user
* Add API TODO
* Ad PAT auth with X-PAT HTTP header
* Replace User.personal_access_token with PersonalAPIKey model
* Fix PersonalAPIKey max_lengths
* Describe posthog.models.utils.generate_random_token better
* Add personal_api_key to API
* Add authenticate_header to PersonalAPIKeyAuthentication
* Add hook API endpoint
* Use django.utils.timezone in place of datetime.datetime
* Add Personal API Keys to Setup
* Sort personal_api_keys in ORM
* Add Action.on_perform()
* Remove requirements.txt comment
* Add a
* Add REST hook tasks
* Optimize PersonalAPIKeyAuthentication query
* Add a trailing slash version of /e endpoint
* Add team field to PersonalAPIKey model
* Add personal API key support to capture endpoint, get_cached_from_token
* Reject personal API keys from inactive users
* Add extra_properties_json field to /capture
* Improve PAK auth header regex
* Use custom hook model
* Deliver hooks
* Handle action.on_perform
* Consolidate userLogic in userLogic.tsx
* Update PersonalAPIKeys.js
* Make PersonalAPIKey foreign keys read-only
* Update requirements/dev.txt
* Make PersonalAPIKeys TSX
* Fix conflict
* Fix migration
* Fix minor mishaps
* Update and fix tests
* Use CharField of random 32 bits as hook.id
* Fix conflicting migrations
* Fix ValidationError in HookSerializer.validate_event
* Use query param in /api/event/actions ID filtering
* Rename endpoint `hook` to `hooks`
* Satisfy mypy
* Add tests
* Use DRF serialization in action_defined and annotation_created triggers
* Update migration leafs
* Make mypy ignore rest_hooks
* Update Django signal receiver names
* Update TS dependencies
* Revert "Update TS dependencies"
This reverts commit 7fc26fefcd.
* Add field user to Hook model
* Update migration leafs
* Fix circular import
* Fix some code
* Install git before running pip install in Dockerfiles
* Improve personal API keys UI
* Satisfy mypy
* Reword key label placeholder
* Add personal API key support to /api/user/*
Unfortunately these endpoints are still limited by CSRF protections at the moment, so not accessible outside PostHog itself.
* Improve PersonalAPIKeyAuthentication and add CsrfOrKeyViewMiddleware
* Run collectstatic before test
* Don't install dev dependencies in CI
* Update dependency installation order in CI
* Fix bug and describe PersonalAPIKeyAuthentication
* Fix CI issues
* Fix typing issues
* Fix more typing issues
* Use /api/personal_api_keys to list keys
* Move REST hooks (and therefore Zapier) to ee/
* Refactor personal API logic with kea-loaders
* Add "More about API authentication in PostHog docs."
* Update PersonalAPIKeys.tsx
* Use TestMixin
* Fix "Authentication" that should've been "Authorization"
* Add option to skip self.client.force_login in API tests
* Include team_id and user_id in personal API key serialization
* Update test_hooks.py
* Add personal API key tests
* Remove leftover
* Make ee.settings override posthog.settings
* Don't directly import from models
* Remove unused imports
* Fix mypy issues
* Fix HOOK_DELIVERER
* Use decorator for /api/user PAK auth
* Don't fire REST hook if user doesn't have "zapier" feature
* Import Optional
* Reword to "premium Zapier"
* Make mypy happy
* Fix test_delete_personal_api_key
* Fix misclick
* Fix and test /capture with personal API key
* Make mypy happy
* Remove extra_properties_json
* Resolve migrations
* Remove apt-utils
* Optimize and test PAK user.is_active filtering
* Replace DEBUG true with 1
* Remove unused instance_id
* Improve typing
* Fix deletion toast
* Refactor CopyToClipboard and use it in PAKs
* Use toast.success
* Update migrations
* Fix migration
* Fix migrations
* Complete merge
Co-authored-by: Tim Glaser <tim@glsr.nl>
