* Revert "Revert "Convert sync execute to use clickhouse connection pool (#1762)" (#1770)"
This reverts commit cbb8fb01a9.
* remove runtime requirement from posthog, that is convered in posthog-production
* Bump the max number of connections in the pool
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* Use ReplacingMergeTree for elements, remove element_groups and use elements_hash as a virtual "pk"
* remove unused ELEMENT_GROUP_TABLE_SQL
* merge fixes
* use redis cache to avoid writing duplicate elements to clickhouse
* move fakeredis to requirements.txt
* add team_id to cache key
* remove elements_group kafka table references
* add elements_hash to clickhouse element serializer
* fix cache key
* rename few keys
* add test runner to ease pycharm dev
* fix a some mypy errors
* remove typo
Co-authored-by: Eric <eeoneric@gmail.com>
* Publish events to Kafka for consumption
* Commit avro idl's for event schemas
* convert client to use github.com/dpkp/kafka-python
* events loaded into clickhouse from Kafka
* remove cruft
* Publish events to Kafka for consumption
* convert client to use github.com/dpkp/kafka-python
* remove cruft
* include kafka migrations
* bugfixes for migrations
* use constants for consistency
* wrap up local migrations
* small fixes
* tune ups
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
Co-authored-by: James Greenhill <jams@uber.com>
* Add missing migration
* Add generate_random_token() model util
* Move PublicTokenAuthentication to utils
* Make use of generate_random_token
* Add User.personal_access_token field
* Add PersonalAccessTokenAuthentication
* Fix PublicTokenAuthentication
* Fix migration and auth import
* Add personal_access_token to user API
* Update Setup.js
* Support trailing slash in API
* Improve PAT auth quality
* Add django-rest-hooks requirement
* Update settings.py for rest_hooks
* Fix django-rest-hooks requirement
* Bring back API routes with no double trailing slash
* Rename posthog.api.team to team_user
* Add API TODO
* Ad PAT auth with X-PAT HTTP header
* Replace User.personal_access_token with PersonalAPIKey model
* Fix PersonalAPIKey max_lengths
* Describe posthog.models.utils.generate_random_token better
* Add personal_api_key to API
* Add authenticate_header to PersonalAPIKeyAuthentication
* Add hook API endpoint
* Use django.utils.timezone in place of datetime.datetime
* Add Personal API Keys to Setup
* Sort personal_api_keys in ORM
* Add Action.on_perform()
* Remove requirements.txt comment
* Add a
* Add REST hook tasks
* Optimize PersonalAPIKeyAuthentication query
* Add a trailing slash version of /e endpoint
* Add team field to PersonalAPIKey model
* Add personal API key support to capture endpoint, get_cached_from_token
* Reject personal API keys from inactive users
* Add extra_properties_json field to /capture
* Improve PAK auth header regex
* Use custom hook model
* Deliver hooks
* Handle action.on_perform
* Consolidate userLogic in userLogic.tsx
* Update PersonalAPIKeys.js
* Make PersonalAPIKey foreign keys read-only
* Update requirements/dev.txt
* Make PersonalAPIKeys TSX
* Fix conflict
* Fix migration
* Fix minor mishaps
* Update and fix tests
* Use CharField of random 32 bits as hook.id
* Fix conflicting migrations
* Fix ValidationError in HookSerializer.validate_event
* Use query param in /api/event/actions ID filtering
* Rename endpoint `hook` to `hooks`
* Satisfy mypy
* Add tests
* Use DRF serialization in action_defined and annotation_created triggers
* Update migration leafs
* Make mypy ignore rest_hooks
* Update Django signal receiver names
* Update TS dependencies
* Revert "Update TS dependencies"
This reverts commit 7fc26fefcd.
* Add field user to Hook model
* Update migration leafs
* Fix circular import
* Fix some code
* Install git before running pip install in Dockerfiles
* Improve personal API keys UI
* Satisfy mypy
* Reword key label placeholder
* Add personal API key support to /api/user/*
Unfortunately these endpoints are still limited by CSRF protections at the moment, so not accessible outside PostHog itself.
* Improve PersonalAPIKeyAuthentication and add CsrfOrKeyViewMiddleware
* Run collectstatic before test
* Don't install dev dependencies in CI
* Update dependency installation order in CI
* Fix bug and describe PersonalAPIKeyAuthentication
* Fix CI issues
* Fix typing issues
* Fix more typing issues
* Use /api/personal_api_keys to list keys
* Move REST hooks (and therefore Zapier) to ee/
* Refactor personal API logic with kea-loaders
* Add "More about API authentication in PostHog docs."
* Update PersonalAPIKeys.tsx
* Use TestMixin
* Fix "Authentication" that should've been "Authorization"
* Add option to skip self.client.force_login in API tests
* Include team_id and user_id in personal API key serialization
* Update test_hooks.py
* Add personal API key tests
* Remove leftover
* Make ee.settings override posthog.settings
* Don't directly import from models
* Remove unused imports
* Fix mypy issues
* Fix HOOK_DELIVERER
* Use decorator for /api/user PAK auth
* Don't fire REST hook if user doesn't have "zapier" feature
* Import Optional
* Reword to "premium Zapier"
* Make mypy happy
* Fix test_delete_personal_api_key
* Fix misclick
* Fix and test /capture with personal API key
* Make mypy happy
* Remove extra_properties_json
* Resolve migrations
* Remove apt-utils
* Optimize and test PAK user.is_active filtering
* Replace DEBUG true with 1
* Remove unused instance_id
* Improve typing
* Fix deletion toast
* Refactor CopyToClipboard and use it in PAKs
* Use toast.success
* Update migrations
* Fix migration
* Fix migrations
* Complete merge
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Add django-statsd for emitting performance metrics to statsd
* prefix metrics with debug unless overwritten
* Only include statsd middleware if the STATSD_HOST env var is set
* Clean up and have logic in one if block
* lz-string compression
* announce to posthog-js that we support lz-string base64 encoding
* add another gzip test, add lzstring base64 test
* compression string refactor
* ignore types for the lzstring library
* initial working
* small adjustments and make tests run
* working cache
* edited arg handling in actions
* caching dashboard trend items with periodic update
* funnel and trend cache working
* update intervals
* update tests
* updated types and funnel refersh logic
* refresh working
* refresh dashboart item logic
* refresh from dashboard working for trends components
* refresh from dashboard working for funnels and funnel refactored to hook
* dashboard item reload
* let deleted items run stale
* fix tests
* tweak decorator logic
* update datetime format and default expiry
* add team to cache signature
* make sure different workers pick up task
* use tooltip and change dashboard refresh order
* fix redundant calls and cache overwriting
* updated types
* Show tooltip when never refreshed
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* Add Django-debug-toolbar for profiling requests performance
* Only require and import debug-toolbar in dev
* don't bother checking types for debug_toolbar
* mypy configs and try/except for import of debug_toolbar
* include debug toolbar url config
* Add djangorestframework-csv external package
* Export a paginated csv at /api/person.csv
* Do not paginate when a when csv export is requested
* Add export button to download cohort as a csv file
* Fix styling errors caught by deepsource.io
* Suppress checks on an import that mypy is unable to follow
* Dynamically pick person properties including deep nested properties
* initial commit
* initial commit
* initial sessions query working
* with tests and two datapoints
* UI for sessions initial
* reset frontend
* fully working implementation
* updated tests for more cases
* bug fix
* small bug and removing carry over properties
* remove unused line
* missing daterange line
* round seconds
* small fixes
* ignore type
* explicit type conversion
* adjusted sorting and formatting for resulting data
* remove redundant label
* created own friendly time converter
* add warning script to settings and docker-worker in case REDIS_URL is not found
* test celery heartbeat
* save celery heartbeat in redis
* /_stats/ with worker heartbeat
* refactor top content
* show worker stats
* heartbeat delay to 90sec
* update worker error message
* remove line
* add redbeat schdeduler
* add redbeat for local development
