* Revert "Revert "Adds reserved properties for filtering property definitions (#8291)" (#8340)"
This reverts commit cb6839fe21.
* limit reserved properties to events table, include this in unit tests
* add api level tests for datetime and reserved word filtering of trends
* remove unused constant
* make the cypress test even looser
* delete file that is no longer in master
* extract a query object
* implement reserved attribute querying for date filters
* update datetime cypress test
* separate and clarify tests and ensure date queries only use reserved attributes on event filters
* delete file that isn't in master
* add description to action
* update tests
* no limit to description and use pageheader
* add ee_action
* add description as global relationship:
* fix tests
* revert to using descriptions
* revert to using descriptions
* allow description to be null
* fix tests
* persist mode
* fix e2e
* Address review points, clean up some margins
* Clean up the way editing state is determined
* Restore default description value and improve typing
* Fix typing better
* Update sql.ts
Co-authored-by: Michael Matloka <dev@twixes.com>
* perf(events): limit select clause when getting ingest requirements
Previously we were pulling in all table columns for every event posted
to the `/e/` column. We actually only need the id, such that we can
associate an event with a `Team`, and a setting which says if we should
anonymize ip addresses.
There are still other optimizations to make, and some of the code paths
do several queries to the database, but I have limited the changes here
to just the most straight forward of them.
Refers to https://github.com/PostHog/posthog/issues/8378
* Add a little comment
* Filter by team_id
* Avoid changing error messages
* Update import get_team -> get_ingest_context
* Fix typing
* fix types
* fix mocking
* fix type
* Rename InjestContext -> EventIngestContext
* Update ingestion context fetch methods
* Further rename fetch functions
* remove property_type_format concept
* remove nulls from tests
* deprecate not remove (and see what tests fail)
* plugin server tests care about property_type_format
* fix tests
* wip
* wip
* wip
* can filter events by reserved words - distinct_id, created_at, and timestamp - in the backend
* add reserved properties when showing property definitions
* capitalise name of property
* properties not reserved words
* get cypress working and make search work with reserved properties
* update event/values endpoint to use property_string_expr and so work with reserved properties
* remove created at from reserved words
* test fix
* fix test
* loosen cypress assertion
* exclude specific properties from the taxonomic property filter
* remove FE property definition fangling
* exclude $time and $timestamp property definitions from the backend
* add reserved properties in the API not the FE using a CTE
* use SQL comments in SQL strings
* remove mypy error
* fix enterprise property definitions
* fix tests and improve comment
* exclude materialised column rows with no values when getting event property definitions
* fix cypress test setup
* clean up dead code
* un-remove some not actually dead code
* correct cypress assertion
* try and fix a cypress test
* Don't allow user to set current team that they don't have access to
* Warn when dashboard is available in different project
* update exceptionshog
* fix tests
* fix failing
* fix typing
* tests
* add property type and format and set them for
* add a failing test
* with passing tests on the ee property definition model
* Add a migration to set as a DateTime
* Clarify in failing test that it's only list lookup failing
* add empty property definition fields to assertions
* fix merge error
* clarify tests
* add more supported types
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* proof of concept
* remove old <PropertyFilter> component
* change import paths
* event properties sorted by event names
* get event names from actions if used
* scope event property filters by event names
* fix eslint
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* remove fields we no longer have
* add some instrumentation to figure out what services we can connect to (helps debug errors locally with kafka)
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* merge duplicate code
* use the right prop for event names
* mute unseen events, add info icon
* use the event-property tracker also for non-EE clients
* simplify duplicates
* add test for event properties
* add test for non-EE event properties
* add flag
* revert the label and unmute the text
* change the flag to UNSEEN_EVENT_PROPERTIES
* sort by query_usage_30_day again
* Update frontend/src/types.ts
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* use None if no event_names
* fix type
* improve pagination test
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* feat(correlation): add team wide person property exclusion list
This change just adds the exclusion list to the `Team` model and checks
that it can be updated. Separately we can:
1. add interface for the Project page to display
2. pull and update this list from the funnels correlation page
NOTE: We don't perform any validation on the structure, :fingerscrossed:
this won't be an issue. Alternative would be to use ArrayField, but use
of JSONField is consistent with other fields.
* test(correlation): add test checking exclude properties pulled from team
* feat(correlation): load exclude properties from team settings
Previously we were loading exclude properties from local storage, so we
would not be sharing the exclusion list between users. This change
simply plugs the `excludePropertyNames` into the `teamLogic` for the
purpose of persisting and loading the values.
* use [teamLogic.actionTypes.loadCurrentTeamSuccess]
* get tests passing
* switch frontend to using `correlation_config`
* refactor: use Team.correlation_config for persistence
* add project settings for correlation
* fix lint
* ensure excluded properties are saved to project config
* Add default excluded properties
* format
* make propertyCorrelations not null
* ensure excluded property names config is unique
* rename excludeProperty to excludePropertyFromProject
* update var names
* change to targetProperties
* remove null special casing
* update to filter client side on exclude from project clicked
* update test name to reflect new functionality
* fix tests
* Add `slug` fields to `Organization` and `Team`
* Expose slugs to user
* Add slug autogeneration for new orgs/projects
* Improve slug UX
* Remove slug from settings
* Update org/team instance creation plus add tests
* Only require project slug to be unique for org, not globally
* Fix `get_prep_value`
* Test organization slugification
* Deslugify `Team`
* Clean up changes
* Update test_user.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Random 4 letter suffixes for the win
* Fix import
* Ignore `test_migrations_are_null`
* Fix `RunSQL` query being empty
* Fix `generate_random_short_suffix` testing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Add project-nested version of all project-dependent endpoints
* Fix hook `perform_create`
* Adjust `plugin-configs` to adhere to style of other endpoints
* Refactor `AvailableFeature` from strings to an enum everywhere
* Fix circular dependency and type
* Add "Per-project access" feature flag, premium feature, and organization switch
* Rename `OrganizationMembershipLevel` to `OrganizationAccessLevel`
* Create `ExplicitTeamMembership` model
* Show whether projects are restricted in the project switcher
* Update organizations API code
* Fix migrations
* Move organization tests that require EE to `ee`
* Revert `OrganizationMembershipLevel` rename
* Fix organization tests
* Update migration
* Fix schema and add Members to Project Settings
* Build out test memberships API with security tests
* Update `TeamMembers` and `teamMembersLogic`
* Move "Per-project access" description to tooltip
* Add moar tests
* Fix Project Members list logic
* Add additional membership checks
* Update migrations
* Fix typing
* Adjust explicit team memberships API similarly
* Fix typo
* Unify `ExplicitTeamMemberSerializer`
* Remove old changes to `membersLogic` usage
* Use `effective_membership_level` on `TeamBasicSerializer`
* Clean up organization update tests
* Explicitly disallow enabling per-project access for free
* Fix circular import
* Remove `id` from `UserSerializer`
* Fix typing
* Try to fix import
* Fix fatal typing
* Add more tests
* Update permissioning.ts
* Add clarifying comment to migration
* Fix import
* Revert `TopNavigation` changes
* Restore `TopNavigation` changes
This reverts commit 05fd9e4ed6.
* Try to add a story for `TopNavigation`
* minor clarifications
* Revert `TopNavigation` changes
* Restore `TopNavigation` changes
* Make new access control entirely project-based
* Update migrations
* Add `project_based_permissioning` to `TeamBasicSerializer`
* Update test_team.py
* Fix `isRestricted` in `ProjectRow`
* Disable project creation for non-admins
* Make project icon in top nav itself dynamic as well
* Fix story
* Delete 0169_project_based_permissioning.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Update frontend/src/layout/navigation/TopNavigation.tsx
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Address feedback
* Project-based permissioning full data access restrictions (#6068)
* Fix Access Control restriction tooltip
* Add `TeamMemberAccessPermission` and use it in viewsets
* Add `ErrorProjectUnavailable` scene
* Ignore mypy
* Update MainNavigation.tsx
* Update explicit_team_member.py
* Fix frontend detection of unavailable project
* Fix some tests and edge cases
* Fix basic permissions
* Add more tests
* Simplify `ExplicitTeamMemberViewSet` permissions
* Improve restrictions and add moar tests
* Update frontend
* Fix a couple of things
* Fix import
* Fix some edge cases
* Fix typing errors
* Use hedgehog instead of moth
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Address feedback
* Add proper permissioning to dashboard views
* Update ee/api/test/test_dashboard.py
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Run prettier
* Remove debug code
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Refactor `AvailableFeature` from strings to an enum everywhere
* Fix circular dependency and type
* Add "Per-project access" feature flag, premium feature, and organization switch
* Rename `OrganizationMembershipLevel` to `OrganizationAccessLevel`
* Create `ExplicitTeamMembership` model
* Show whether projects are restricted in the project switcher
* Update organizations API code
* Fix migrations
* Move organization tests that require EE to `ee`
* Revert `OrganizationMembershipLevel` rename
* Fix organization tests
* Update migration
* Fix schema and add Members to Project Settings
* Build out test memberships API with security tests
* Update `TeamMembers` and `teamMembersLogic`
* Move "Per-project access" description to tooltip
* Add moar tests
* Fix Project Members list logic
* Add additional membership checks
* Update migrations
* Fix typing
* Adjust explicit team memberships API similarly
* Fix typo
* Unify `ExplicitTeamMemberSerializer`
* Remove old changes to `membersLogic` usage
* Use `effective_membership_level` on `TeamBasicSerializer`
* Clean up organization update tests
* Explicitly disallow enabling per-project access for free
* Fix circular import
* Remove `id` from `UserSerializer`
* Fix typing
* Try to fix import
* Fix fatal typing
* Add more tests
* Update permissioning.ts
* Add clarifying comment to migration
* Fix import
* Revert `TopNavigation` changes
* Add project member addition button+modal
* minor clarifications
* Revert `TopNavigation` changes
* Make new access control entirely project-based
* Update migrations
* Add `project_based_permissioning` to `TeamBasicSerializer`
* Update test_team.py
* Fix Access Control restriction tooltip
* minor improvements
* fix frontend typing
* Fix frontend typing a bit more
* adjust copy & UI a bit
* Address feedback on field comment
* "Privacy settings" to "Access Control"
* Make `FusedTeamMemberType` comment clearer
* Remove useless `export`
* Delete 0169_project_based_permissioning.py
* Clean some code up a bit
* Project-based permissioning member removal (#6067)
* Fix `teamMembersLogic` loaders
* Allow explicit project members to leave
* Add member removal/leaving button to Members with Project Access
* Restore error message
* Fix error message
* Correct things
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Trigram fuzzy search for property definitions
* support empty search param
* Only use pg_trgm search if pg extension is installed
* suppress exceptions if migration does not succeed
* checks for index
* troubleshooting
* wip: change migration number; strip out conditionals
* wip: copy code from #4110
* remove is_pg_trgm_installed
* try ngram search on CH
* add AND before ngram
* correct some check failures
* remove ordering
* remove ordering from tests
* fix ordering
* rewrite queries for psql, add test bypass for EE features
* remove ordering filter bc of bugs
* remove test code
* check if isinstance of RawQuerySet
* cleanup sql
* add ordering
* typing
* test for ee prop definition
* adjustments to test_ee_property_definition
* add test_ee_event_definition
* restore this prop
* convert to simple term filter
* use term search in EE
* fix EE tests
* fix non-EE tests
* fix test
* fix filter types
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* wrap css classnames and allow for property definition filtering
* event properties section
* properties stats table and object tag functionality
* property descriptions and save all descriptions button
* fix description editing on the table
* fix test failures
* debouncing input is weird
* split up components
* refactor definition drawer kea file
* drawer should only open for taxonomy feature users
* lint errors
* prettier
* property definition filter backend test
* fix test
* populate test with definitions..
* clean up
* clean up property drawer interactions
* fix everything but scroll top issue
* test scroll issue
* table state wip
* update logic after typegen
* prevent undefined error
* only make the clickable rows clickable, not the entire table
* upgrade typegen and fix error with feature flags
* move to reducers and rename action
* update blacklist
* fix event properties description bug
* update typegen once more
* fix editing for events table
* fix editing for properties table
* add script to clean typegen
* update ts blacklist
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* split up files and organize code
* set up definition drawer and logic and add tagging
* add change owner selection
* definition description editing working
* definition drawer graph and events table
* remove graph logic for now
* small fixes
* property definition doesn't have an owner
* minor tweaks
* lots of small fixes
* show tags on table, disable editing for posthog events, fix tags autocomplete
* fix font sizes and alignment
* allow event limiting and hide behind feature flag again
* linter things
* test fix
* lint
* clean up events limit
* limitOffsetPagination in events
* ignore type
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* add description and tag fields to event and property definitions
* set up description and tagging on models
* frontend functionality for description editing
* connect backend and kea logic for description editing
* update event and property definitions model and migration
* delete set null instead of cascade
* migration merge fix
* add owner column
* undo posthog event property taxonomy migrations
* set up definitions on enterprise level
* allow enterprise definitions description editing
* fix licensing conditions and add tests
* proper get and update methods for the multi inheritance table and new column fields for enterprise event model
* check for license to separate routes
* migrate existing definitions to ee definitions tables and render ee vs non-ee definition views based on existing feature conditional
* all the working backend updates
* updated tests
* frontend fixes and linting updates
* feature flag it
