* stores numeric type for property definitions
* writes property definition type for strings
* can write dates for simple string cases as property definition type
* simplify property definition types
* can detect unix timestamps
* rewmove a little more from the test duplication
* some more timestamp thinking
* clearer comment
* rename refactor in tests
* more test correction
* address test failure
* another test fix
* property type as enum
* simplify type detection
* apply typing suggestion
* allow property definition type and format to be set if property is present but they are null
* add more date types for property_type_formats
* add more datetime formats that the backend can accept and filter events with
* delete conflicting migration
* add migration for backend datetime formats
* plugin server can auto detect all the types the backend knows about. And the tests are collapsed from twenty five almost identical tests to two different parameterised blocks
* iso 8601 strings can have fractional seconds
* add link to the RFC 822 regex pattern
* capitalise SQL
* capitalise the SQL the mock looks for too
* remove the under-eager cache change that allowed updating existing property definitions
* convert property type formats to enum
* fix test
* try detecting a property type for existing property definitions but don't keep retrying
* use symbols to make cache state clearer
* fix idiocy
* speculative LRU cache usage with measurement
* tag the statsd guages instead of having team id in the name
* don't pre-emptively optimise the cache
* rename migration
* remove spaces from query strings
* separate property definition code from the team manager
* a little tidying
* correct type definition
* correct test params setup
* Migration to add name_singular and name_plural to groups
* Allow modifying group type names
* Use name_plural and name_singular through the app
* Update tab names
* fixup
* Update snapshots
* add property type and format and set them for
* add a failing test
* with passing tests on the ee property definition model
* Add a migration to set as a DateTime
* Clarify in failing test that it's only list lookup failing
* add empty property definition fields to assertions
* fix merge error
* clarify tests
* add more supported types
* create event property model
* add null
* rename cache vars
* update event properties table on ingestion
* match date formats
* match date formats
* better string handling
* property type can be null too
* pass event timestamp
* update property type later
* perform all updates through a buffer object
* move to EventPropertyCounter
* fix migration
* improve flush last seen at job
* flush job periodically + env
* upsert all event properties in 1 query
* log to statsd
* enable property counter only if experimental mode enabled
* use now() instead of event timestamp
* fix seconds
* add user/pass for default postgres
* add tests
* use big integers
* make query work with 50k props
* processing events saves event properties
* fix script
* test date format detection
* default enabled
* only enable event property counter for specific teams
* eslint fixes
* fix logs double-sync noise in tests
* fix bigint test
* don't do tasks that make no sense
* remove dead code
* simpler test setup
* different contraint name
* refactor team manager
* greatly simplify the system
* fetch cached event properties
* fix team manager and timestamps
* add cached entry
* also don't cache event properties for teams that have it disabled
* remove indexes that are not going to be used
* remove unused imports
* blacked
* remember event properties with a LRU cache
* fix eslint
* clean up the last bits
* move ONE_HOUR to constants
* use sane_repr
* fix typo
* Make settings dynamic for MATERIALIZED_COLUMNS_ENABLED
* migrate MATERIALIZED_COLUMNS_ENABLED to this and test
* allow tests that use materialized columns to hit django model for settings
* nicer installed apps config
* update snapshot from test
* make test_property more deterministic
Co-authored-by: James Greenhill <fuzionech@gmail.com>
Co-authored-by: yakkomajuri <yakko.majuri@gmail.com>
* Rename "dashboard item" to "insight" where applicable
* Use `InsightModel` instead of `InsightType` for former `DashboardItemType`
* Rename some more
* Add migration for enum variant name
* Update migration
* Migration to add version to person_distinct_id
* Update plugin-server type
* Use queueMessages instead of for loops
* Update distinct id versions in postgres
* Add commented out new query
* Add person_distinct_id2 table setup/migration
This will be used for more efficient person_distinct_id queries
* Avoid sharding person_distinct_id2 on cloud
* Write to new distinct ids topic
* Attempt to use version in tests
* Tests attempt 2
* Fixup version - dont send with all messages
* Flush kafka more frequently
* Actually fix tests
* Add another await
* Add partition to person_distinct_id2 table
* SpecialMigration model
* Update latest_migrations.manifest
* update model
* add ph version range to model
* format
* progress -> PositiveSmallIntegerField
* Add group model
```sql
BEGIN;
--
-- Create model Group
--
CREATE TABLE "posthog_group" ("id" serial NOT NULL PRIMARY KEY, "group_key" varchar(400) NOT NULL, "group_type_index" integer NOT NULL, "group_properties" jsonb NOT NULL, "created_at" timestamp with time zone NOT NULL, "properties_last_updated_at" jsonb NOT NULL, "properties_last_operation" jsonb NOT NULL, "version" bigint NOT NULL, "team_id" integer NOT NULL);
--
-- Create constraint unique team_id/group_key/group_type_index combo on model group
--
ALTER TABLE "posthog_group" ADD CONSTRAINT "unique team_id/group_key/group_type_index combo" UNIQUE ("team_id", "group_key", "group_type_index");
ALTER TABLE "posthog_group" ADD CONSTRAINT "posthog_group_team_id_b3aed896_fk_posthog_team_id" FOREIGN KEY ("team_id") REFERENCES "posthog_team" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "posthog_group_team_id_b3aed896" ON "posthog_group" ("team_id");
COMMIT;
```
* Remove a dead import
* Improve typing for groups
* Make groups updating more generic, avoid mutation
This simplifies using the same logic for groups
Note there's a behavioral change: We don't produce a new kafka message
if nothing has been updated anymore.
* Rename a function
* WIP: Handle group property updates
... by storing them in postgres
Uses identical pattern to person property updates, except we handle
first-seen case within updates as well.
* Get rid of boolean option
* WIP continued
* fetchGroup() and upsertGroup()
* Test more edge cases
* Add tests for upsertGroup() in properties-updater
* Rename to PropertyUpdateOperation
* Followup
* Solve typing issues
* Cleanup nits
* Add test showing desired behavior during insert race
* Handle race conditions gracefully
* add window-id to ingestion
* move window_id into snapshot data
* Add tests
* add window_id to snapshot_data only if included in the event
* add window_id to session_recording_events table
* revert ph-js downgrade
* fix data generator
* remove partition change
* update window_id defaults for tests
* missed one window_id default
* rename migration for conflict
* feat(correlation): add team wide person property exclusion list
This change just adds the exclusion list to the `Team` model and checks
that it can be updated. Separately we can:
1. add interface for the Project page to display
2. pull and update this list from the funnels correlation page
NOTE: We don't perform any validation on the structure, :fingerscrossed:
this won't be an issue. Alternative would be to use ArrayField, but use
of JSONField is consistent with other fields.
* test(correlation): add test checking exclude properties pulled from team
* feat(correlation): load exclude properties from team settings
Previously we were loading exclude properties from local storage, so we
would not be sharing the exclusion list between users. This change
simply plugs the `excludePropertyNames` into the `teamLogic` for the
purpose of persisting and loading the values.
* use [teamLogic.actionTypes.loadCurrentTeamSuccess]
* get tests passing
* switch frontend to using `correlation_config`
* refactor: use Team.correlation_config for persistence
* add project settings for correlation
* fix lint
* ensure excluded properties are saved to project config
* Add default excluded properties
* format
* make propertyCorrelations not null
* ensure excluded property names config is unique
* rename excludeProperty to excludePropertyFromProject
* update var names
* change to targetProperties
* remove null special casing
* update to filter client side on exclude from project clicked
* update test name to reflect new functionality
* fix tests
* Add table for group_type_mapping
* Remove materialized columns from events table schema
These are not used and not needed w/ new mat columns work
* WIP: Migration to add group analytics columns
* Remove event table changes temporarily
* Only run `test_migrations_are_null` on new migrations
* Rename `DashboardItem` to `SavedInsight`
* Add explanation to `test_migrations_are_null` script
* Rename `test_dashboard` to `test_saved_insight_model`
* Ditch `DashboardItem{ViewSet,Serializer}` altogether
* Fix refresh via `InsightSerializer`
* "SavedInsight" to just "Insight"
* Update file names
* Update insight.py
* Update dashboardLogic.test.ts
* Update migration
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* initial refactoring
* popup UI
* refactor path cleaning logic
* add nullable
* all ui working
* fix migration
* use regex replacement from team object
* add flag
* add switch
* fix type
* fix type
* UI update
* restore removed arg
* add local path cleaning filters to api
* add test for local path filters
* working new UI
* reduced repeated code
* fix numbering
* minor refactoring
* update copy
* add under advanced features
* address comments, minor cleanup
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Add `slug` fields to `Organization` and `Team`
* Expose slugs to user
* Add slug autogeneration for new orgs/projects
* Improve slug UX
* Remove slug from settings
* Update org/team instance creation plus add tests
* Only require project slug to be unique for org, not globally
* Fix `get_prep_value`
* Test organization slugification
* Deslugify `Team`
* Clean up changes
* Update test_user.py
* Apply suggestions from code review
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Random 4 letter suffixes for the win
* Fix import
* Ignore `test_migrations_are_null`
* Fix `RunSQL` query being empty
* Fix `generate_random_short_suffix` testing
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* add migration for function
* add test
* fix migration name
* approach to address raw sql in migrations
* add more tests, address review
* add equal timestamps test
* update structure
* format
* much better approach, fix tests
* override feature flags in the API
* improve types (thanks mypy!)
* remove unrelated code
* better comment
* fix test query count
* remove v2 in get_active_feature_flags_v2
* store `$feature/{key}` in properties
* Move feature flag overrides to their own model
* little bit of cleanup
* some type check + test cleanup
* respond to comments, add tests, add team permission checks
* add basic analytics
* add team to override model and simplify endpoints
* cleanup
* fix up instrumentation
* restrict a user to only their own overrides
* override query now works for all distinct ids
* typing fix
* fix tests
* sort flags by created date
* fix tests
* Clean up
* remove defualt from override_value field
* update migration
* fix migration dependancy
* run black on feature_flag.py
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Trigram fuzzy search for property definitions
* support empty search param
* Only use pg_trgm search if pg extension is installed
* suppress exceptions if migration does not succeed
* checks for index
* troubleshooting
* wip: change migration number; strip out conditionals
* wip: copy code from #4110
* remove is_pg_trgm_installed
* try ngram search on CH
* add AND before ngram
* correct some check failures
* remove ordering
* remove ordering from tests
* fix ordering
* rewrite queries for psql, add test bypass for EE features
* remove ordering filter bc of bugs
* remove test code
* check if isinstance of RawQuerySet
* cleanup sql
* add ordering
* typing
* test for ee prop definition
* adjustments to test_ee_property_definition
* add test_ee_event_definition
* restore this prop
* convert to simple term filter
* use term search in EE
* fix EE tests
* fix non-EE tests
* fix test
* fix filter types
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add description and tag fields to event and property definitions
* set up description and tagging on models
* frontend functionality for description editing
* connect backend and kea logic for description editing
* update event and property definitions model and migration
* delete set null instead of cascade
* migration merge fix
* add owner column
* undo posthog event property taxonomy migrations
* set up definitions on enterprise level
* allow enterprise definitions description editing
* fix licensing conditions and add tests
* proper get and update methods for the multi inheritance table and new column fields for enterprise event model
* check for license to separate routes
* migrate existing definitions to ee definitions tables and render ee vs non-ee definition views based on existing feature conditional
* all the working backend updates
* updated tests
* frontend fixes and linting updates
* feature flag it
* Save Organization.available_features as a DB column
* `update_available_features()` before an organization is created too
* Run the task half past the hour
* Adjust tests and fix available_features sync on start
* Remove redundant null=False
* Fix `Organization.update_available_features()`
* Run Cloud tests in respect to `posthog-cloud` `4426-fix` branch
* Revert "Run Cloud tests in respect to `posthog-cloud` `4426-fix` branch"
This reverts commit 421e8541b3.
* Extract function for capture
* Add capability to capture internal metrics
- This is behind an environment variable, we'll turn this on by default
on clickhouse-based installation.
- A special Org/Team gets created for this lazily, which is not accessible for
anyone. This makes avoiding issues with stats/billing/properties and easier.
* Capture some useful metrics into posthog
* Set up an unique constraint for for_internal_metrics organization
This avoids some nasty race conditions
* internal_metrics fixes
* Extract component OverviewTab
* New tabbing system for system status page
* Embed ClickhouseTab
* Generate a dashboard for internal metrics on demand
* More metrics
* Add more internal metrics, add tests
* Fix migration files
* Adjust copy
* Update task definitions
* Fix a typing issue
* Rename a component
* Migration fixup
* Try reset patch after test ends
* Clear lru_cache between tests
