* working for unique_groups math
* fix types
* add null check
* update snapshots
* update payload
* update snapshots
* use constructor
* adjust queries
* introduce base class
* consolidate querying
* shared serializer and typed
* sort imports
* snapshots
* typing
* change name
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* changed implementation to use pg
* unusd
* update type
* update snapshots
* rename and remove inlining
* restore bad merge code
* adjust types
* add flag
* remove var
* misnamed
* change to uuid
* make sure to use string when passing result
* remove from columnoptimizer logic and have group join logic implemented by event query classes per insight
* remove unnecessary logic
* typing
* remove dead imports
* remove verbosity
* update snapshots
* typos
* remove signals
* remove plugin excess
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Extract GroupsJoinQuery
* Add test for breakdown filtering
* Unify breakdown mixins
* Allow passing breakdown_type == 'group' with breakdown_group_type_index
* Allow breakdown by group props in trends
* Add tests for trends breakdown_props function on group breakdowns
* Solve common issues
* Output snapshot diff into console
* Clean up materialized columns after tests
* Add zero protection
* Solve test failure
* Type math in Entity
* Allow passing group_type_index from FE to BE
* Get a initial query running
* Add group value filter if aggregating by groups
* Add snapshot testing for trends queries
* isort
* Update tests
* Add test for column_optimizer
* Update ee/clickhouse/queries/trends/util.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Hotfix: Use materialized columns on cloud
This was broken since default_kind was different on Distributed tables
on cloud
* Improve __init__.py
* Make reverting DEFAULT column async, only ON CLUSTER for events table
* wip: pagination for persons on clickhouse funnels
* wip: added offset support for getting a list of persons; added support for conversion window;
* fixed mypy exception
* helper function to insert data for local testing
* moved generate code into separate class for more functionality later
* corrected person_distinct_id to use the person id from postgres
* minor corrections to generate local class along with addition of data cleanup via destroy() method
* reduce the number of persons who make it to each step
* moved funnel queries to a new folder for better organization; separated funnel_persons and funnel_trends_persons into individual classes;
* funnel persons and tests
* initial implementation
* invoke the funnel or funnel trends class respectively
* add a test
* add breakdown handling and first test
* add test stubs
* remove repeats
* mypy corrections and PR feedback
* run funnel test suite on new query implementation
* remove imports
* corrected tests
* minor test updates
* correct func name
* fix types
* func name change
* move builder functions to funnel base
* add test classe for new funnel
* Handle multiple same events in the funnel (#4863)
* dedup + tests
* deep equality. Tests to come
* write test for entity equality
* finish testing funnels
* clean up comments
* add ability to specify per step or dropoff persons
* remove defaults
* remove funnel_window parameter unless it's needed
* add param to filters
* test api
* remove print
* fix tests
* change distribution
* add none condition for funnel step
* add order by
* remove funnel window days
Co-authored-by: Buddy Williams <buddy@posthog.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* checkpoint: refactoring funnel trends so that they work correctly
* wip: refactoring funnel trends query to return the results we actually need
* wip: added in new query for testing
* wip: moved sql into a separate file, converted list to dictionary, and added several tests to check data quality
* wip: with a better understaning of funnel trends I've refactored the query so that I can write a data transformer in python
* moved code into funnel_trends for both logic and tests to isolate the concern
* reordered methods for readability
* wip: refactoring funnel trends to support filters
* wip: added updated SQL which will replace the existing FUNNEL_TREND_SQL
* correct column name so that it's clearer
* added substitution variables to new query
* fixed missing substitution variable
* wip: integrating new query with correct params, added mixins for funnel_window, and working toward working test
* several query corrections
* summarize funnel trends
* moved method down
* removed unused code
* added data quality checks
* corrected cohort size for tests
* test window size and incomplete status
* corrected a few names
* removed unnecessary comment
* commented out old funnel trends tests
* removed print statement
* removed old funnel trend code
* made funnel trends response match existing data structure layout
* removed unused imports
* removed more unused imports
* fixed mypy errors
* Added ClickhouseFunnelBase to extract common methods for both ClickhouseFunnelTrends and ClickhouseFunnel; this also fixes issues with tests;
* removed unused type comment
* corrected test to account for new funnel_window_days mixin
* fixed clickhouse funnel tests
* fixes for automated tests
* changed team_id to use client substitution to avoid sql injection attempts in the future but since it's not user supplied it's not currently an issue
* corrections prompted by PR review
* corrected test to dict test with funnel_window_days
* Use `statshog` over python-statsd
More support for tags!
* Include custom tags for every query + add annotation to query
After this we can:
- Figure out from query logs where queries are coming from (speeding up debugging)
- Break down query speeds by user queries vs others (e.g. celery) --
better represents overall speed
- Can figure out how fast queries are on average for various teams
* Use tags in more queries over interpolation
This way we can set up more interesting graphs \o/
* Solve mypy error
* Fix a flaky test (due to ordering)
* Upload static cohort using CSV, closes #2868
* fix tests
* Fix tests
* Fix e2e test
* Avoid double inserts
* Speed up query
* Move to params
* fix tests
* initial cleanup
* initial scaffold for cohort static logic
* intiial frontend changes
* initial working
* fix error on iterable
* stickiness people func refactor
* Use JSON instead of protobuf
* stickiness working
* clickhouse route
* unused import
* move button
* make interface easier to user
* clickhouse support
* restore cache decorator
* endpoint adjustment
* adjust how clickhouse handles inserting for static cohorts
* split out processing into pg
* use worker for clickhouse querying too
* add pg test
* ee test
* add input for cohort naming
* add stickiness test
* condition for when to show save cohort
* fix types
* minor fix
* remove repeats
* remove hardcoded id
* add date handling to trends cohort func
* remove print
* add link on success creation
* cohort working
* fix backend test
* fix cohort typescript
* remove unnecessary kafka topic
* remove brackets
* shift default
* raise exceptions when debug or test is true
* remove test flag
* move trends insight related components to different folder
* change get_earliest_timestamp
* remove extraneous condition
* fix test call
* put behind ff
* fix test x2
* fix tests
* restore exceptions hog
* fix tests
* restore exceptions hog
* fix test x20
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Update plugin server to 0.9.24
* Add more hook metrics in Python
Co-authored-by: posthog-bot <posthog-bot@users.noreply.github.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* Fix REST hooks being only fired if project has webhook
* Fix firing more
* Actually still filter but also for Zapier
* Don't .only() Team in EE process_event to optimize REST hook firing
* Fix REST hook duplication in EE
* Update test_process_event.py
* Debug
* Revert "Debug"
This reverts commit 6d4c9f754d.
* Fix organization-less teams partly
* Fix more organization-less teams
* Fix moar organization-less teams
* Explicitly bootstrap org without user in tests
* Fix org bootstrap
* Fix typing
* Fix bootstrap usage
* Update test_process_event.py
* Fix num queries
* Add test_action_on_perform_hook_fired_once
* Fix Action.objects.create call
* Don't test hook firing as it's offloaded to worker
* Nest endpoints under /project/ with StructuredViewSetMixin
* Rewrite URLs
* isort
* Update utils.py
* Fix errors
* Fix almoast all the errors
Last left to do: shared dashboards and permission classes.
* isort
* Adjust for master
* Add compatbility with shared dashboards
* Debug ClickHouse
* Remove some # type: ignores
* Simplify CursorPagination
* Move test base from posthog.api.test to posthog.test
* Improve API structure
* Bring back legacy endpoints
* Fix legacy compatibility
* Fix bugs and typing
* isort
* Fix hooks test
* Try fixing errors
* Fix oversight
* isort
* Fix problems
* isort
* Be more tolerant
* Fix naming and remove redundant code
* Fix imports
* Update deleteWithUndo
* Roll back
* Roll back more
* Update .gitignore
* Rollll back
* Rollllllll
* back
* Betterify
* Address feedback
* Add missing migration
* Add generate_random_token() model util
* Move PublicTokenAuthentication to utils
* Make use of generate_random_token
* Add User.personal_access_token field
* Add PersonalAccessTokenAuthentication
* Fix PublicTokenAuthentication
* Fix migration and auth import
* Add personal_access_token to user API
* Update Setup.js
* Support trailing slash in API
* Improve PAT auth quality
* Add django-rest-hooks requirement
* Update settings.py for rest_hooks
* Fix django-rest-hooks requirement
* Bring back API routes with no double trailing slash
* Rename posthog.api.team to team_user
* Add API TODO
* Ad PAT auth with X-PAT HTTP header
* Replace User.personal_access_token with PersonalAPIKey model
* Fix PersonalAPIKey max_lengths
* Describe posthog.models.utils.generate_random_token better
* Add personal_api_key to API
* Add authenticate_header to PersonalAPIKeyAuthentication
* Add hook API endpoint
* Use django.utils.timezone in place of datetime.datetime
* Add Personal API Keys to Setup
* Sort personal_api_keys in ORM
* Add Action.on_perform()
* Remove requirements.txt comment
* Add a
* Add REST hook tasks
* Optimize PersonalAPIKeyAuthentication query
* Add a trailing slash version of /e endpoint
* Add team field to PersonalAPIKey model
* Add personal API key support to capture endpoint, get_cached_from_token
* Reject personal API keys from inactive users
* Add extra_properties_json field to /capture
* Improve PAK auth header regex
* Use custom hook model
* Deliver hooks
* Handle action.on_perform
* Consolidate userLogic in userLogic.tsx
* Update PersonalAPIKeys.js
* Make PersonalAPIKey foreign keys read-only
* Update requirements/dev.txt
* Make PersonalAPIKeys TSX
* Fix conflict
* Fix migration
* Fix minor mishaps
* Update and fix tests
* Use CharField of random 32 bits as hook.id
* Fix conflicting migrations
* Fix ValidationError in HookSerializer.validate_event
* Use query param in /api/event/actions ID filtering
* Rename endpoint `hook` to `hooks`
* Satisfy mypy
* Add tests
* Use DRF serialization in action_defined and annotation_created triggers
* Update migration leafs
* Make mypy ignore rest_hooks
* Update Django signal receiver names
* Update TS dependencies
* Revert "Update TS dependencies"
This reverts commit 7fc26fefcd.
* Add field user to Hook model
* Update migration leafs
* Fix circular import
* Fix some code
* Install git before running pip install in Dockerfiles
* Improve personal API keys UI
* Satisfy mypy
* Reword key label placeholder
* Add personal API key support to /api/user/*
Unfortunately these endpoints are still limited by CSRF protections at the moment, so not accessible outside PostHog itself.
* Improve PersonalAPIKeyAuthentication and add CsrfOrKeyViewMiddleware
* Run collectstatic before test
* Don't install dev dependencies in CI
* Update dependency installation order in CI
* Fix bug and describe PersonalAPIKeyAuthentication
* Fix CI issues
* Fix typing issues
* Fix more typing issues
* Use /api/personal_api_keys to list keys
* Move REST hooks (and therefore Zapier) to ee/
* Refactor personal API logic with kea-loaders
* Add "More about API authentication in PostHog docs."
* Update PersonalAPIKeys.tsx
* Use TestMixin
* Fix "Authentication" that should've been "Authorization"
* Add option to skip self.client.force_login in API tests
* Include team_id and user_id in personal API key serialization
* Update test_hooks.py
* Add personal API key tests
* Remove leftover
* Make ee.settings override posthog.settings
* Don't directly import from models
* Remove unused imports
* Fix mypy issues
* Fix HOOK_DELIVERER
* Use decorator for /api/user PAK auth
* Don't fire REST hook if user doesn't have "zapier" feature
* Import Optional
* Reword to "premium Zapier"
* Make mypy happy
* Fix test_delete_personal_api_key
* Fix misclick
* Fix and test /capture with personal API key
* Make mypy happy
* Remove extra_properties_json
* Resolve migrations
* Remove apt-utils
* Optimize and test PAK user.is_active filtering
* Replace DEBUG true with 1
* Remove unused instance_id
* Improve typing
* Fix deletion toast
* Refactor CopyToClipboard and use it in PAKs
* Use toast.success
* Update migrations
* Fix migration
* Fix migrations
* Complete merge
Co-authored-by: Tim Glaser <tim@glsr.nl>
