* Raise 400 instead of 500 if filter cohort does not exist
* Actually return empty 200 results instead of 400
Co-authored-by: eric <eeoneric@gmail.com>
* Proposal: Use `ValidationError` over `ValueError` in views
This generates a nicer exception for our users (400 response instead of
500) and we cut down on 500 errors for things that are not expected to
work.
* Capitalization
Co-authored-by: Michael Matloka <dev@twixes.com>
* Don't fire webhooks and Zapier in case of PLUGIN_SERVER_ACTION_MATCHING
* Add PLUGIN_SERVER_ACTION_MATCHING to settings
* Add PLUGIN_SERVER_ACTION_MATCHING to ECS task defs
* Remove leftover print
* Update `AsyncActionMappingNotice`
* Remove stale import
* add migration
* add fields
* remove comment
* change to is_deleted
* Update migration to work
* Update tests
* initial functions
* fix params
* add migration
* add named param
* working test
* update person prop and cohort tests
* use var
* move sql and fstring
* remove get inside django signal
* just create the table
* add replication logic
* add param
* change replacing name
* refactor and add event based cohorts
* add type
* add to model
* add test for event_id
* move sql
* add sql
* make sure actions work too and add test
* add ability to create cohort by event count
* glob match cypress cache
* fix type
* fix type
* refactor and add datetime support
* fix ordering issue in test
* add timestamp test
* fix type
* fix more types
* use var resolution
* add operators
* add necessary filter and test
* add call for calculating cohort on create
* add triggers to oncreate and onupdate
* breakup test and remove repeated subquery
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* first try
* add entity query
* fix type issues
* used in trends
* fix test
* change select format
* change var
* add cohort filter
* add cohort conditions
* change pdi to distinct_ids
* make sure to add table name to prop clauses so that event/person properties column isn't vague
* pass table name when formatting breakdown query
* add test
* add migration
* add fields
* remove comment
* change to is_deleted
* Update migration to work
* Update tests
* initial functions
* fix params
* add migration
* add named param
* working test
* update person prop and cohort tests
* use var
* move sql and fstring
* remove get inside django signal
* just create the table
* add replication logic
* add param
* change replacing name
* make sure actions work too and add test
* add necessary filter and test
* add triggers to oncreate and onupdate
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* Save Organization.available_features as a DB column
* `update_available_features()` before an organization is created too
* Run the task half past the hour
* Adjust tests and fix available_features sync on start
* Improve action and property TS types
* Use plugin server ActionManager pubsub
* Use better pubsub channel names for plugin server ActionManager
* Fix PubSub's lack of teamId
* Use f-strings for property CH SQL formatting
* Use `statshog` over python-statsd
More support for tags!
* Include custom tags for every query + add annotation to query
After this we can:
- Figure out from query logs where queries are coming from (speeding up debugging)
- Break down query speeds by user queries vs others (e.g. celery) --
better represents overall speed
- Can figure out how fast queries are on average for various teams
* Use tags in more queries over interpolation
This way we can set up more interesting graphs \o/
* Solve mypy error
* Fix a flaky test (due to ordering)
* swap out distinct_id table in queries with a subquery that will only consider latest distinct_ids
* wrong import
* fix missin params
* more missing params
* support multiple entries for selector and tag_name filter for EE
* improve support for multiple selector matches
* test for multiple selectors and tag names
* WIP auto filter test accounts
* finish off
* Fix tests
* Non generic emails
* add list of generic emails
* Move location to below property filters
* Fix typescript errors
* as any
* fix tests
* filters
* fix tests
* Featureflag doesn't really makes sense for this feature
* fix tests
* fix test
* Add clickhouse + tests for each insight
* Fix lifecycle and paths
* Fix sessions
* Fix session tests
* fix sesisons test
* fix migrations
* fix migration chain
* refactor path & remove stale console.log
* adjust useAnchor & minor copy
* rename to avoid confusion with inline component
* test account filter tweaks
* fix filters
* hardcode
* Add tests for funnel trends
* Make generic emails super fast
* Fix migrations
* Default to false for now
* Default to false, remember a user's preference
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Don't colorize sql for output
* Handle setting array props as values
* Handle arrays exact/is_not property filters
Also updates tests to use parameterization
* Avoid autoclosing PropertyFilter popup
As key changes, old one got removed from DOM
* Improve filter UX
Don't allow filter to become multi-line and keep inputs the same size.
Previous flex={1} was doing flex: 1 1 auto which was messing things up
* Fix typescript issue
* Handle array lookups in postgres
* Update types
* Fix fixture to recreate tables
* Make replace spaces logic consistent with what was there before
* Handle list filtering better in postgres
* Update frontend tests
* Handle edge case: Switching between multi operator and single operators
* Update plugin server to 0.9.24
* Add more hook metrics in Python
Co-authored-by: posthog-bot <posthog-bot@users.noreply.github.com>
Co-authored-by: Michael Matloka <dev@twixes.com>
* Return no data if invalid regular expression
This currently returns a 500. Fixes 3191 together with frontend
validation.
* Add browser validation
* Typing errors
* Highlight rows from sessions which are matched by the filter
* Send start_time of recording to frontend
This helps us calculate offsets a bit better
* Use timestamp by server for date shown
* Load session events via kea logic
* Update rrweb, rrweb-player
* Highlight events user is filtering for in sessions player
* Handle empty case properly
* Add positive test
* Order session recording events in query
* Fix filtering by multiple events with differing names
previously only the first would have been used due to overlapping
params.
* Return all highlighted times as action_filter_times
* Send back ids not timestamps
This avoids weird rounding errors causing issues
* Show skeleton for longer
* update typing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Make it possible to filter by (multiple) action filters in postgres
Session will now contain "action_filter_times" key which lists when each
action filter occurred for the first time within the session.
This will be used to highlight rows/show special values in sessions
player.
* Clickhouse: support multiple action filters
* Remove dead code
* Upload static cohort using CSV, closes #2868
* fix tests
* Fix tests
* Fix e2e test
* Avoid double inserts
* Speed up query
* Move to params
* fix tests
* Use JSON instead of protobuf
* unused import
* produce instead of produce_proto
* Insert directly into clickhouse, no need for kafka
* a few quick fixes
* insert in batch to clickhouse
* test SQLi since we can't really trust what we are inserting
* Extra check for duplicates
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Fix REST hooks being only fired if project has webhook
* Fix firing more
* Actually still filter but also for Zapier
* Don't .only() Team in EE process_event to optimize REST hook firing
* Fix REST hook duplication in EE
* Update test_process_event.py
* Debug
* Revert "Debug"
This reverts commit 6d4c9f754d.
* Fix organization-less teams partly
* Fix more organization-less teams
* Fix moar organization-less teams
* Explicitly bootstrap org without user in tests
* Fix org bootstrap
* Fix typing
* Fix bootstrap usage
* Update test_process_event.py
* Fix num queries
* Add test_action_on_perform_hook_fired_once
* Fix Action.objects.create call
* Don't test hook firing as it's offloaded to worker
* extract code to format entity filter
* make it possible to filter by action
* Hack to make filtering by actions subfilters possible
Example url:
http://localhost:8000/api/event/sessions?date_from=2020-12-10&date_to=2020-12-10&action_filter={%22type%22:%22events%22,%22id%22:%22test-event%22,%22properties%22:[{%22key%22:%22email%22,%22value%22:%22example.com%22,%22operator%22:%22icontains%22,%22type%22:%22person%22}]}
Not sure if the team filtering is legit here
* Use discriminated union in types
* Add kludge support for action_filter on sessions via url
* Reduce code in buildURL
* Add link to sessions page from persons modal
* Add muted overview of the invisible filter
* Add link to cohort sessions from persons page
* drop irrelevant test code
* Test clickhouse filtering by action filter
* put filter behind a cloud-only conditional
* remove dead import
* Add icons/data-attr to sessions links
* Appease linter gods
* intial working
* intial working with test
* return tests
* fix interval and add monthly test
* add in person_id join
* small restyle
* postgres lifecycle working
* add action handling to postgres query
* add action handling and tests
* fix typing
* visualizing with temp params and added negatives for dormant
* frontend
* fix intervals
* remove unnecessary import
* add person enppoint
* add person endpoint and tests
* add next
* add pagianted test
* fix types
* add frontend logic'
* fixed date range
* disable compare filter on lifecycle
* add diasbled to chartfilter
* return class to personviewset
* added constant
* fix distinct_id and new event within period condition
* replace people queries and fix ch query too
* DRY
* fix null states
* comparefilter nullstate
* add wrapper back to endpoint
* fix datetime formatting
* remove extra stack flag
* reduce filters when it's lifecycle and replace constants
* add default for lifecycle into trendlogic
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* remove unneded comment and import
* split up retention query
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* fix chartfilterlogic
* intiial graph on PG
* initial frontend
* fix filtering and types
* clickhouse working
* change reference date
* fix label and defaults
* fix multiple event retention
* working clickhouse and postgres for multiple events
* remove final
* remove sql parse
* add inital person query
* working clickhouse person query and return testrunner
* implement postgres retrieve person
* add endpoint and pagianted test
* click on datapoint works
* fix params
* fix action handling
* people paginated backend
* finished paginatino frontend logic
* fix variable naming for pg
* working first time persons for clickhouse
* first time people working for pg
* dont replace if recurring
* run again
* fix import
* refactor filter to retention specific and move postgres
* convert clickhouse to use new filter
* move imports
* fix types
* fix types
* fix initial load
* adding to dashboard works
* fixed defaults
* add test for dates
* fix insight panel
* add seert
* fix import
* fix date select for people
* Nest endpoints under /project/ with StructuredViewSetMixin
* Rewrite URLs
* isort
* Update utils.py
* Fix errors
* Fix almoast all the errors
Last left to do: shared dashboards and permission classes.
* isort
* Adjust for master
* Add compatbility with shared dashboards
* Debug ClickHouse
* Remove some # type: ignores
* Simplify CursorPagination
* Move test base from posthog.api.test to posthog.test
* Improve API structure
* Bring back legacy endpoints
* Fix legacy compatibility
* Fix bugs and typing
* isort
* Fix hooks test
* Try fixing errors
* Fix oversight
* isort
* Fix problems
* isort
* Be more tolerant
* Fix naming and remove redundant code
* Fix imports
* Update deleteWithUndo
* Roll back
* Roll back more
* Update .gitignore
* Rollll back
* Rollllllll
* back
* Betterify
* Address feedback
* Cache cohorts in clickhouse
This fixes #2306
* Calculate celery
* Schedule cohorts recalculation once per hour
* Run cohort calculations each in separate task
* Fix max age in cohort recalculation
* Simplify
Co-authored-by: Tim Glaser <tim@glsr.nl>
* remove person properties up to date
* remove person props mv
* move latest person
* prune rest of person materialized
* missing parenth
* add type
* remove migration
* Fix feature flags clickhouse
* Fix feature flags clickhouse
* Fix types
* Fix stuff
* Silly me
Co-authored-by: Eric <eeoneric@gmail.com>
* remove person properties up to date
* remove person props mv
* move latest person
* prune rest of person materialized
* missing parenth
* add type
* remove migration
* Make DELETE synchronous in clickhouse tests
This avoids some flaky behavior, example:
- https://github.com/PostHog/posthog/pull/2110/checks?check_run_id=1327279229
Documentation:
- https://clickhouse.tech/docs/en/operations/settings/settings/#mutations_sync
Tested it out by running the following test 1000 times - before change
it fails every ~50 runs locally.
```
def test_delete_person(self):
- person = person_factory(
- team=self.team, distinct_ids=["person_1", "anonymous_id"], properties={"$os": "Chrom
e"},
- )
- event_factory(event="test", team=self.team, distinct_id="person_1")
- event_factory(event="test", team=self.team, distinct_id="anonymous_id")
- event_factory(event="test", team=self.team, distinct_id="someone_else")
-
- response = self.client.delete(f"/api/person/{person.pk}/")
- self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
- self.assertEqual(response.data, None)
- self.assertEqual(len(get_people()), 0)
- self.assertEqual(len(get_events()), 1)
-
- def test_filters_by_endpoints_are_deprecated(self):
+ for index in range(1000):
+ print([index])
+ person = person_factory(
+ team=self.team, distinct_ids=["person_1", "anonymous_id"], properties={"$os": "Chrome"},
+ )
+ event_factory(event="test", team=self.team, distinct_id="person_1")
+ event_factory(event="test", team=self.team, distinct_id="anonymous_id")
+ event_factory(event="test", team=self.team, distinct_id="someone_else")
+
+ response = self.client.delete(f"/api/person/{person.pk}/")
+ self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
+ self.assertEqual(response.data, None)
+ self.assertEqual(len(get_people()), 0)
+ self.assertEqual(len(get_events()), 1)
```
* Add a if not TEST clause to deleting from materialized view
This blew up clickhouse when we try to force the delete to happen sync.
Created an upstream issue: https://github.com/ClickHouse/ClickHouse/issues/16548
* Protobuf all the things
* oops
* Protobufize events to protect from malformed JSON
* format the generated files (will need to remember this for future)
* format
* clean up kafka produce serializer
* fixes
* Add scheduled task to wipe session recordings
* Create a new table for session recording
* Save snapshot events to different table
* Use SessionRecordingEvent over Events everywhere
We can remove a ton of cruft this way as well
* Add missing signature
* Extract util from models/event
* Attempt to update ingest side of clickhouse session recording events
Note that it's using main kafka topic - not sure if a good idea.
* Get separate table in ch working for session recording events
* WIP: query sessions
* Make both session recording queries work
* Make linter happy
* Rebase migration
* Make tests work
* Apply a TTL to session recordings and other configuration:
- toYYYYMMDD partitioning should be smoother with TTL setup
- TTL achieves not needing to archive the data ourselves
- index_granularity will enable smaller reads per session_id
- ORDER BY clause is to make single session as well as time range query
reasonable
* Convert retention cronjob to new model
* Add tests to process_event changes
* Add test for ee_capture change
* Fixup migration
* Make clickhouse tests drop/create session recording tables
* Make TTL not be there in tests
Otherwise writes get eaten by it during tests when mocking time
* Fix retention task
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Use postgres to grab person
* Fix type
* Fix grabbing by person id
* Fix type
* WIP people from clickhouse
* Fix person delete etc
* fix test
* Fix tests
* add new table migrations and change table names
* include necessaray config for new tables in tests
* fix tests and table
* fix table name param
* add populate clause
* added table for key value person props
* adjust person filtering to use new table
* .
* add ordering on updated_at
* add back all the condition handling on persons filtering endpoint
* fix typgin
* remove print
* re-order sort key for persons_up_to_date
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Revert "Revert "Use postgres to grab person (#1957)" (#1963)"
This reverts commit 94f44bdf46.
* Filter by team_id
Co-authored-by: Tim Glaser <tim@glsr.nl>
* Backend changes to implement #1461
* Added the missing migration files
* Fixes Typecheck errors
* Refactor request.user.team_set.get() to use request.user.team
* Updated user patch method to change current_team on team id instead of name
* Merged migration file
* Changes team property to return first item in queryset
* Fixes failing tests
* Changed User api to return the ids of the teams they are currently part of
* Frontend for changing teams
* Update and simplify migrations
* Improve team and user buttons
* Make team changing backend more logical
* Improve current_team mechanics
* Update test_team.py
* Fix Team.objects.create_with_data
* Update migration
* Update tests
* Make setup_review more convenient
* Add Organization and OrganizationMembership
* Replace is_admin with level
* Extend API
* Update team.py
* Improve modeling
* Improve handling of new mechanics
* Add proper migration
* Remove _ensure_organization_and_team
* Update 0084_org_team_user.py
* Improve user, org and team creation
* Make MembershipLevel more flexible for the future
* Add member deletion
* Fix naive datetime warnings
* Update setup_review.py
* Update API route
* Make PersonalAPIKey changes
* Update models and migrations, fix typing
* Fix typing
* Use MAC-less UUID v1 instead of v4 for better performance
* Add abstract UUIDModel
* Update utils.py
* Update utils.py
* Fix multi/unicast bit
* Update APIs, frontend and tests
* Update pull_request_template.md
* Fix comment
* Fix migration error
* Compress migrations
* Updates with minimal renaming
* More updates
* Make further updates
* Update test_team_user.py
* Fix issues
* Add migration
* Satisfy mypy
* Remove Signup redirect on logged in
* Use uuid1_macless in Person
* Fix typing
* Update tests
* Update /api/team/signup to /api/organization/signup
* Fix mypy issues and update tests
The remaining failures are actually missing functionality (TDD applied), so filling these in.
* Update 0086_org_live
* Make small improvements
* Implement permissions
* Remove now unneccesary membership check
* Update setup_dev.py
* Make small frontend improvements
* Add drf-nested-routers as requirement
* Remove unused import
* Implemented nested routes
* Remove cruft
* Add relevant org/proj/user name to headings
* Fix imports
* Update migration
* Replace unreliable drf-nested-routers with drf-extensions
* Improve unset team handling
* Make org and team creation proper
* Update migration
* Fix migration
* Update TopContent
* Update command palette for new sidebar structure
* Remove deprecated demo data deletion
* Assume that each org has a project and fix typing
* Require paid plan for multiple orgs and projects
* Make HogFlix demo a separate team
* Update migration
* Slightly improve style
* Adjust page layout bottom padding
* Make user dropdown nicer
* Fix base app tests
* Satisfy mypy
* Fix test_leave_organization
* Improve wording
* Possibly fix import
* Remove misplaced None check
* Enhance org and teams APIs and add tests
* Fix /api/projects for particular Team
* Improve invites and demo data
* Address feedback
* Put everything related to billing on Organization
* Fix minor issues
* Simplify invitation creation
* Update team model
* Make orgs and projects premium only on self-hosted
* Improve testing
* Update migration
* Remove extra License import
* Fix minor issues
* Fix Django tests
* Fix Cypress
* Fix yarn build
* Fix TeamSignupViewset
* Fix posthog-production incompatibility
* Remove extraneous insight endpoint registration
* Adjust tests for posthog-production
* Simplify invitations and fix email validation
* Address all feedback
* Satisfy mypy
* Update migration
* Fix constraint removal in migration
* Update tests
* Fix test creation edge case
* Run posthog-production CI tests against this branch and teams-live
* Ensure that js_posthog_api_key is always passed
* Fix preflight check pre-login
* Update cypress tests
* Update instanceStatus.js
* Bring ee tets up to par
* Bring actions-ux-201012 back
* Cypress retry in cypress.json
* Revert "Run posthog-production CI tests against this branch and teams-live"
This reverts commit d79cb844d8.
Co-authored-by: anna <ms.annaphilips@gmail.com>
Co-authored-by: Anna Philips <aphilips@matmacorp.com>
* fixed negation operators for person and cohorts
* add person negation test
* fix tests
* add leakeage tests
* add space
* add debug statements
* match distinct_id
* change parsing to include operators'
* make properties test into factory
* add clickhouse test implementation and fix another test
* add custom test to clickhouse filter tests
* all tests besides json filtering
* add json test
* fix tests
* fix type errors
* Clickhouse use elements chain
* Fix stuff
* Add action tests and start regex
* Progress
* Progress part deux
* Fix everything
* Add tag name filtering
* Fix funnels
* Fix tag name regex
* Fix ordering
* Fix type issues
* Fix empty nth-child
* Remove commented code
* Split with semicolon and escaped quotes
* Specify all select columns
* working date filter
* frontend for period
* remove period
* add params
* add period
* period working
* add test for period
* merge master
* fix test
* add dropdowns for period date filtering
* pass params and handle properly
* fix labels and readjust tests
* test split for retentino period
* make insight retention test just test for response
* add condition
* feature flag the frontend and bring back ff for the ch endpoint
* fix cohort filtering
* fix cohort querying
* remove init
* remove feature flag
* add back featureflag
* block person modal on retention page if clickhouse enabled
* use distinct_id instead of joining for person_id
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
* Fix person querying
* Fix flake
* Fix person stuff
* Fix test
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* fix mypy error, indent SQL
* add pagination test for postgres
* Run insight API tests in Clickhouse
Co-authored-by: Eric <eeoneric@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
* change inheritance
* "Clickhouse Features V2 (#1565)"
This reverts commit 0b371d43ec.
* fix types
* change to super
* change to super x2
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* Do not shadow Kafka default columns _timestamp and _offset
* drop the columns only on the kafka_ tables
* include data types
* fields cleanups
* datetime -> datetime64
* double import datetime
* Create Omni-Person model for managing people in Clickhouse
* type fixes
* rebase all the things
* cleanups
* id -> uuid for events in clickhouse
* cleanups and type checks
* Further cleanups and uuid conversions
* kafka fix
* break out serializer across kafka clients
* fix a few bugs w/ datetime types
* basic fix for people kafka table
* fix migration errors (copy pasta errors)
* Use KafkaProducer for Omni Person emitting
* setup mock kafka producer
* undo some work for inserting
* Test TestKafkaProducer
* change if order, obvious mistake
* remove unnecessary function arg
* Fix getters for new column
* Test fixes
* mirror columns across element queries
* firm up handling of timestamps
* only return timestamps for handle_timestamp
* Correct heroku config for Kafka
* Use ReplacingMergeTree for elements, remove element_groups and use elements_hash as a virtual "pk"
* remove unused ELEMENT_GROUP_TABLE_SQL
* merge fixes
* use redis cache to avoid writing duplicate elements to clickhouse
* move fakeredis to requirements.txt
* add team_id to cache key
* remove elements_group kafka table references
* add elements_hash to clickhouse element serializer
* fix cache key
* rename few keys
* add test runner to ease pycharm dev
* fix a some mypy errors
* remove typo
Co-authored-by: Eric <eeoneric@gmail.com>
* add test runner to ease pycharm dev
* fix broken import
* drop and recreate the clickhouse test db before running tests
* fix person uuid str json serialization issue
* make kafka optional in tests
* fix inits
* remove need for kafka in person.py
* fix a bunch of mypy errors
* fix function and add process_event to pipeline
* fixed missing params and tests
* change uuid and fix types
* types
* optimize for merge prop test
* make ClickhouseProducer to produce to clickhouse one way or another
* annotate types
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* Query optimizations
* more sql optimizations
* checkpoint
* fix migration
* add UUID field to person
* use django signals to signal that clickhouse needs to be updated
* cleanup person logic
* cleanups
* update migration
* Don't setup the django signals unless we are for sure using ee setup
* expecting back to 30 queries for capturing with new person
* add .venv to .gitignore
* add env back to .gitignore
Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-18.ec2.internal>
* Publish events to Kafka for consumption
* Commit avro idl's for event schemas
* convert client to use github.com/dpkp/kafka-python
* events loaded into clickhouse from Kafka
* remove cruft
* Publish events to Kafka for consumption
* convert client to use github.com/dpkp/kafka-python
* remove cruft
* include kafka migrations
* bugfixes for migrations
* use constants for consistency
* wrap up local migrations
* small fixes
* tune ups
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* change property parsing
* indentation wrong missing calls
* uuid4 instead of call to CH
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
Co-authored-by: James Greenhill <jams@uber.com>
