* update cypress
* really click something that's actually there
* obey cypress and use done
* run cypress 9 in CI
* no need for before each when only one test
* no need to set window size to the default
* get tests passing file by file
* delay checking for a graph in a test
* be more specific cypress
* use cy command
* select text like a human
* silly cypress
* try and avoid cypress deciding that a visible field is not valid
* select delete button correctly
* find save button differently
* try and avoid not always typing the first character
* better trends selections
* use cy command to navigate
* conitnue trying to get tests to pass in CI
* another try at setting feature flag names in CI
* can CI find undo button without a wait?
* better assertion for cypress
* up to v10
* fix splitting specs with v10 path
* show cypress how to wait for the test to finish
* remove redundant file
* change return to satisfy new cypress
* move import
* fix dayjs
* fix timeouts (we're not strictly speaking running in nodejs)
* export unexported type
* consolidate on a single FormInstance
* no need to rename
* fuse
* forminstance 2
* locationChanged
* BuiltLogic
* remove Type.ts exception
* fix duh
* lay off the bin/check-typescript/strict script
* don't think this is ever used or useful
* no real need to hide the output
* make typescript:check do what the name says
* we're already strict
Co-authored-by: Michael Matloka <dev@twixes.com>
* chore(hobby deployments): various fixes
* default do not check versions for current hobby release
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* feat(object_storage): add unused object storage with health checks
* only prompt debug users if object storage not available at preflight
* safe plugin server health check for unused object storage
* explicit object storage settings
* explicit object storage settings
* explicit object storage settings
* downgrade pip tools
* without spaces?
* like this?
* without updating pip?
* remove object_storage from dev volumes
* named volume on hobby
* lazily init object storage
* simplify conditional check
* reproduced error locally
* reproduced error locally
* object_storage_endpoint not host and port
* log more when checking kafka and clickhouse
* don't filter docker output
* add kafka to hosts before starting stack?
* silly cloud tests (not my brain)
* More secure secret generation
Use a random source designed for secrets/crypto and apply a stronger hash function as MD5 is broken. This shouldn't have too much of an impact in this context, but better safe than sorry.
* Tune `head` params
Co-authored-by: Michael Matloka <dev@twixes.com>
* remove tests that have been off for a year
* remove component tests that are covered by main cypress tests
* remove a bunch of component based test setup and upgrade cypress
* get tests running but not all passing on Cypress 9
* don't upgrade yet
* don't upgrade yet
* feat(sharding): add command to sync tables onto new nodes
clickhouse-operator only syncs some tables onto new nodes. This new
command ensures that when adding new shards, they are automatically
synced up on redeploying
Note that there might be timing concerns here as resharding on altinity
cloud does not redeploy automatically. In practice however what this
means is that new nodes just won't ingest any data until another deploy
* Add test to the new command
* Improve non-replicated test
* Fix docs for hobby and add upgrade instructions
* swallow errors from stop command (should error normally)
* persist env vars in docker-compose for hobby
* remove .hobby from docker-compose
* make file easier to test
* make env vars easier to template
* use docker-compose from git instead of curl
* test envsubst params
* default for sentry because exceptions
* tweaks to default logic
* Add DB user and pass to DB URL for DEBUG mode and to bin tests for dropping test database
* ensmallen the blast radius of the DB connection settings changes
* make PGPORT settable too
* set PG ENV variables in bin tests
* Revert "set PG ENV variables in bin tests"
This reverts commit 4991f28808.
* set debug, test, and primary_db env variables based on context
* remove unnecessary print
* also use pytest.ini for some tests run straight from pytest at the CLI
* use plugin-server in same repo
* remove plugins dir
* update docker files
* clean up docker files:
* visual separation
* run plugin server test stack
* build and push test image
* revert push image
* fix default yarn start
* another fix
* fix some more
* add plugin-server-prod script
* make executable
* update scripts:
* revert prod script
* fixes
* fix tests
* clean
* move build to docker
* build plugin-server in prod dockerfile
* use different entrypoint for e2e test
* fix bugs with ts-node-dev
* restore yarn.lock
* lost a few changes in the rebase
* fix dockerfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
* only load events table from the URL on allow list of URLs
* add return type to function
* pass scene URL into events table
* use urlpattern to check if scene is loaded in events table
* don't need to check if still on the same page as when loaded because urltoaction is now specific
* which means we don't need url pattern matching at all
* add scene url to events scene export's logic props
* use a different action name for each run or retries can never complete as the action cannot be saved
* do not poll results if moved away from sceneUrl
* remove tabs from person/action event tables
* so cypress can run locally
* break the dependency of running one test first so the other can pass
* move check into logic and add an e2e test
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Type math in Entity
* Allow passing group_type_index from FE to BE
* Get a initial query running
* Add group value filter if aggregating by groups
* Add snapshot testing for trends queries
* isort
* Update tests
* Add test for column_optimizer
* Update ee/clickhouse/queries/trends/util.py
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
* Allow passing in token to Personal API Key requests
* fix
* fix accidental commit
* update comment
* dont access self.request.user.team if not needed
* add tests
* Add an assert to make mypy happy
Co-authored-by: Michael Matloka <dev@twixes.com>
From https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb
> By default, preforking Celery workers distribute tasks to their worker processes as soon as they are received, regardless of whether the process is currently busy with other tasks.
> If you have a set of tasks that take varying amounts of time to complete — either deliberately or due to unpredictable network conditions, etc. — this will cause unexpected delays in total execution time for tasks in the queue.
This is 100% the case for us. This should "load balance" the tasks
better across workers.
* Compile requirements-dev.txt with latest pip-tools
* Install pytest
* Avoid picking up factories as tests
* New runner
* Always set TEST env variable running tests
Some of our tests rely on it.
* Remove repetition
* Fix a broken test
* Cut down noise from bin/tests
* Rename test factory
* Fix stickiness filter
* Skip a broken test
This has been broken since numpy removal PR. Sadly tests were not
running for this submodule
* Fix import on ee
* Run ee tests properly
The django_db_setup fixture will be automatically run when running ee/
module tests.
* Make tests run on CI
* Include REDIS_URL, fix cloud
* Set TEST env variable
* Hack cloud tests to work
* Attempt at workflow fix
* Import Person model when running ee tests
This module implicitly adds hooks, so this is needed when running tests
* Respect reuse-db for clickhouse
* Add custom markers to avoid warnings
* pytest: use ch test database always
Accidentally wiped by ch setup a few times without this. Oops
* Remove repetition in tests
* Pytest: Always run migrations
Testing a state cleanup fix
* Use same DB in conftest and main code
* Pytest: autoset TEST setting without env variable
* fix broken test
Co-authored-by: eric <eeoneric@gmail.com>
* Add AUTO_LOG_IN environment variable
This will be used when running tests on CI, setting up demo instances
and with cypress vcr
* Auto log in in review apps, e2e tests
This will speed up tests/reviewing significantly
* Rename AUTO_LOG_IN to AUTO_LOGIN
Co-authored-by: Michael Matloka <dev@twixes.com>
* WIP: get cypress something doing
* Get a test running
* Get css loading, stub posthog
* Move helpers to separate file
* Givens > given2
It plays nicer with beforeEach
* Test for session filters
* Test date navigation
* Test filtering
* Try keep old tests running
* Try get tests running under CI
* Prettify fixtures
* Cleanup, use cypress 6
* Add yarn build
* Fix e2e cypress tests
* given2 => givens
* Rename `frontend-test-runner` to `e2e-test-runner`
* Fix cypress test
* Add webpack-preprocessor as a devDependency
* Improve freezing time
* Make css inclusions automatic for component tests
* Verify no new typescript code breaks "strict" compilation
With this we have a file containing all current errors. As the code
changes, we should not add more errors here.
* remove some lines to show a failure
* Add total count to script
* Update blacklist after rebase
* Remove some fails on purpose.
* Attempt a reorder
* Add missing file
* Update script to write typegen
* Refresh list
* Attempt to use normal tsc
* update blacklist
* Store all errors
* echo for typegen
* Add check command
* typegen 3 times
* typegen 4 times
* typegen 6 times
* typegen 10 times
* remove debug code
* Update kea typegen to latest
* Remove hacks
* Add setting for handing off process_event_ee to plugin server
* Add StatsD settings to KEYS
* bin/plugin-server → start-plugin-server & docker-plugin-server
* Simplify to only add docker-plugin-server
* Bring back original comment
* Turn down verbosity of plugin server install
* Remove redundant if
* Fix comment
* Remove lone newline
* Roll back unsafe script changes
* Simplify dockerized plugins
* Add some depends_on
* Clarify HAND_OFF_INGESTION env var
* Use posthog-plugin-server 1.0.0-alpha.1
* Enhance bin/plugin-server and rm bin/docker-plugin-server
* Move around PLUGIN_SERVER_INGESTION_HANDOFF ifs
* Use posthog-plugin-server@1.0.0-alpha.2
* Support kafka+ssl:// in plugin-server
* Produce to topic events_ingestion_handoff for plugin server
* Use posthog-plugin-server@1.0.0-alpha.3
* Don't import Kafka topics in FOSS
* Use @posthog/plugin-server
* Update yarn.lock
* Add commands for external ClickHouse setup/teardown
* Actually delete test CH teardown command
* ClickhouseTestRunner.setup_test_environment() in setup_test_clickhouse
* Rework test setup script to work with Postgres too
* Restore master plugins dir for merge
* Unset PLUGIN_SERVER_INGESTION_HANDOFF in docker-compose.ch.yml
* Fix unimportant typo
* Build log_event data dict only once
* Make it clear in bin/plugin-server help that it's bin
* Space space
* Add relevant settings to KEYS in bin/plugins-server
* Log all EE events to events_handoff Kafka topic for plugin server
* Clean up settings
* Fix FOSS
* Don't introduce KAFKA_EVENTS_HANDOFF
* Add cosmetic newline
* Add DEBUG WAL print()
* Deprecate dashboard item type and move to display
* Mypy rerors
* fix test
* fix
* Fix test
* Fix another test
* Remove determineInsightType
* Get rid of RetentionTable display types
* Fix update dashboard
* nodejs worker concurrency on heroku
* upgrade plugin server
* upgrade server
* fix another bug with 0.3.2
* plugin server with resilient redis
* plugin server with more resilient workers
* new plugin server
* update
* separate plugin worker for heroku
* plugin server for dev with less concurrency
* add back plugins script
* move beat log to beat
* move starting the beat into the celery worker
* add optional process types for plugins and celery
* premium redis for heroku review apps
* fix broken script
* proc/dyno names are alphanumeric
* singularize
* premium-0 redis for all heroku apps, not just review apps
* premium-0 redis also for review apps
* remove heroku redis modifications
* remove out of scope code
* run beat in bg
* Run E2E tests without celery
I spent ~hour today figuring out why a task was not executing,
restarting frontend test runner each time.
By not running celery in test and forcing all things to evaluate eagerly
we can avoid that pain
* Remove a now always-passing test
* plugin progress
* blah
* add posthog config for plugins
* test gitignore
* new functionality for plugins
* support local plugin paths
* also ignore symlinks
* add positional argument
* fixes
* small fixes
* config polish
* config passed to posthog plugin
* ooooooops
* symlink fix
* cleanse dir before loading
* add cache to plugins
* pickle the goods
* unlink symlink
* pass full config
* unlink even if link points to nothing
* fix fix
* return none if value is empty
* plugin model
* plugins scene
* add config schema to plugins
* install plugins
* save descriptions
* show descriptions
* edit plugin
* save plugin config
* plugin modal
* uninstall plugins
* UX cleanup
* add "required" to plugin config
* open plugin modal after install
* split to subcomponents
* install custom plugins
* rework backend for model plugins
* Plugins on models
* simple reload pubsub
* fix apps not installed
* fix master/main issue
* fix reload command
* use the github api to get the default branch
* init plugins only if not running migrate/makemigrations
* store plugins zip archives in postgres
* tag plugins to specific versions
* save plugins in pluginConfig
* update pluginConfigs instead of adding new rows, remove from redux on uninstall
* remove debug
* run plugins from db by team
* reload when deleting
* remove debug
* smarter handling of dynamic plugins, support local plugins again
* improve typings, add some nicer warnings
* yarn lock file after merge
* squash migrations and add "locked" field to plugins
* error if folder not found in zip
* unregister plugins
* skip plugin init in test mode
* basic plugin test
* avoid mutating the same prop hash
* add pip tools to requirements.txt
* fix mypy, fix manage.py script error
* avoid plugins with mypy
* mypy fix
* abstract redis into plugin and add team_id to reload
* refactor and start work on syncing with posthog.json
* start testing plugin loading from json
* test plugin deletion
* test for syncing plugins from config
* complete and then test local json plugin sync
* test converting back and forward between an local and http path
* remove global plugin config from plugins array in posthog.json
* rename configSchema --> config_schema
* fix migration after merge
* rename from_cli to from_json
* mypy
* import pip after plugin loaded
* show error details
* raise exceptions visible to the frontend
* sync plugins on load
* access control to updating plugins from the web
* access control
* remove posthog.json from git
* test config schema from json
* if you can install via the web, you can also configure
* remove separate view access
* title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository
* add self.team to plugin base class
* add instance_init method
* refactor into files
* sync global plugin config from json
* make global plugins work, add test
* global plugins in interface, make them take precedence over local plugins
* add comments to plugin base class
* reload/reset plugins before each test
* add error field to plugins
* add many plugin zips
* add many plugin zips, fix imports
* store errors on plugin object and test them
* fix types
* add null to error
* can be with any team ID in the test
* save problems running plugins in the plugin_config model
* try to create redis connection pool only once
* throw if no redis
* mypy
* get instance inside heartbeat and not top level
* try caching pubsub
* try pip install with -q
* install pip externally
* remove uuid and typing, now in stdlib
* more verbosity
* add pip back
* catch exceptions
* quiet and no input for pip
* check plugin reload every 10sec on new task
* fix type errors
* fix requirements error message
* use repository.json
* only load and reload plugins on workers
* rename task
* support local js plugins via py-mini-racer
* load js plugins from zip files
* extract jsplugin class and convert to syntax that uses global functions instead of initializing a class
* process events via grpc
* process events with the "posthog-plugins" queue to enable plugins
* remove old native python & mini racer plugin code
* default to false
* change env vars
* fix test
* remove grpc tools
* skip plugins in migrate.py scripts
* fix migration
* change output of settings debug banner to STDERR
* start posthog plugin server with worker
* try to fix python 3.7 test
* add fallback for the optional argument
* annoying CI test debug
* try to finally fix python 3.7 test
* here we go again
* move plugins under instance
* move plugins npm start into its own folder
* more console.log debugginf
* and again
* move plugins to separate script
* more prints
* fix test error
* docker config
* small fixes
* dckerfile fix
* reload plugins via pubsub, upgrade version
* plugins that support team setup code
* sync if made changes from config
* move plugins in menu
* require node 14 in heroku for better plugin support (namely ?. support)
* bump node version in dockerfiles
* update node versions for github actions
* update the concurrency for heroku workers
* update the concurrency for heroku workers (add link)
* Fix migrations after merge
* add ignore_result to process_event tasks
* fix: docker-preview run in parallel bug
* change order of commands
* remove separate plugins server conf script
* clarify intent
* revert castaway change
* add context to plugins/sync.py
* change everything to ValidationError
* delegate destroy to super
* no request to repository url if can't install
* make the if cleaner
* add clarifying line
* add clarifying line
* fix url field type
* rename get_redis_instance to get_client, move to posthog.redis
* remove duplicate validation
* flip if around
* simplify api logic
* simplify plugin_config api, fix global_plugin error
* remove unnecessary field rename
* mypy
* Plugins UI (#2090)
* base UI
* more UI
* load plugin image if available
* toggle enabled plugin
* plugin cards for available
* custom plugins
* change plugin configuration to drawer
* asks for confirmation when enabling or disabling a plugin
* loading state
* separation of concerns, leave new styles for separate PR
* general improvements
* remove button when installation is not available
* preemptively avoid merge conflict with #2114
* move papercups widget & hide bottom bar when drawer is open
* allow clicking the entire plugin card
* address all feedback
* move plugins under "project" menu
* Hide "configure" from globally enabled plugins
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* add plugins opt in toggle to project/plugins
* choose pipeline based on team setting
* add "beta"
* plugin opt-in opt-out pages
* adjust install button
* remove tasks that are never called, remove PLUGINS_ENABLED global key
* fix responsive card display
* fix typo and drawer width
* skeleton fixes
* typo
* use "posthog-plugin-server" npm package
* "posthog-plugin-server" doc
* require the plugin server to be online before enabling plugins
* remove a few needeless "?." cases
* add hint for config_schema
* add hint for errors
* show plugin errors
* stop clicks if clicking on error
* show plugin errors
* loading indicators
* reload plugins when opting in/out
* nicer beta tag
* add frontend type
* fix mypy error
* fix test
* disable plugins if MULTI_TENANCY
* upgrade plugin-server version
* save event with plugin error
* upgrade plugin server
* squashed & optimized migrations
* remove unused import
* updates opt-in copy & hides tech details for cloud version
* fix cypress tests
* compare with None
* change plugins url and add redirect
* remove ellipsis
* use code snippet in plugin errors
* change github regex
* fix loading flickering on installing plugins
* add comment to plugin archive
* fix python style
* remove pip-tools (relic from the python plugin era)
* hard pin plugin server version
* remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway)
* update lockfile
Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
* Clickhouse use elements chain
* Fix stuff
* Add action tests and start regex
* Progress
* Progress part deux
* Fix everything
* Add tag name filtering
* Fix funnels
* Fix tag name regex
* Fix ordering
* Fix type issues
* Fix empty nth-child
* Remove commented code
* Split with semicolon and escaped quotes
* Specify all select columns
* convert sessions table logic to TS
* convert rest of sessions to TS
* sessions table logic refactor, store date in the url
* add back/forward buttons
* load sessions based on the URL, not after mount --> avoids duplicate query if opening an url with a filter
* prevent multiple queries
* throw error if failed instead of returning an empty list
* date from filters
* rename offset to nextOffset
* initial limit/offset block
* indent sql
* support limit + offset
* load LIMIT+1 sessions in postgres, pop last and show load more sign. (was: show sign if exactly LIMIT fetched)
* based offset is always 0
* default limit to 50
* events in clickhouse sessions
* add elements to query results
* add person properties to sessions query response
* show seconds with two digits
* fix pagination, timestamp calculation and ordering on pages 2 and beyond
* mypy
* fix test
* add default time to fix test, fix some any(*) filter issues
* remove reverse
* WIP event list
* Events progress
* Finish off event listing, skip live actions for now
* Fix mypy
* Fix mypy again
* Try fixing mypy
* Fix assertnumqueries
* Fix tests
* Fix tests
* fix test
* Fix tests
* Fix tests
* Fix tests again
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
* change inheritance
* "Clickhouse Features V2 (#1565)"
This reverts commit 0b371d43ec.
* fix types
* change to super
* change to super x2
Co-authored-by: Eric <eeoneric@gmail.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* initial
* migration command
* migrations working
* add modelless views for clickhouse
* initial testing structure
* use test factory
* scaffold for all tests
* add insight and person api
* add basic readme
* add client
* change how migrations are run
* add base tables
* ingesting events
* restore delay
* remove print
* updated testing flow
* changed sessions tests
* update tests
* reorganized sql
* parametrize strings
* element list query
* change to seralizer
* add values endpoint
* retrieve with filter
* pruned code to prepare for staged merge
* working ingestion again
* tests for ee
* undo unneeded tests right now
* fix linting
* more typing errors
* fix tests
* add clickhouse image to workflow
* move to right job
* remove django_clickhouse
* return database url
* run super
* remove keepdb
* reordered calls
* fix type
* fractional seconds
* fix type error
* add checks
* remove retention sql
* fix tests
* add property storage and tests
* merge master
* fix tests
* fix tests
* .
* remove keepdb
* format python files
* update CI env vars
* Override defaults and insecure tests
* Update how ClickHouse database gets evaluated
* remove bootstrapping clickhouse database routine
* Don't initialize the clickhouse connection unless we say it's primary
* .
* fixed id generation
* remove dump
* black settings
* empty client
* add param
* move docker-compose for ch to ee dir
* Add _public_ key to repo for verifying self signed cert on server
* update ee compose file for ee dir
* fix a few issues with tls in migrations
* update migrations to be flexible about storage profile and engine
* black settings
* add elements prop tables
* add elements prop tables
* working filter
* refactored
* better url handling
* add mapping table
* add processing to worker task
* working cohort with actions
* add cohort property filtering
* add cohort property filtering
* reformat and add cohort processing
* prop clauses
* add util
* add more util
* add clickhouse modifier
* Clickhouse Sessions (#1623)
* sessions sql
* skeleton
* add endpoint
* better tests
* sessions list
* merge clickhouse-actions
* added session endpoint
* sessions sql working again
* add clickhouse modifier
* session avg with props working
* add dist
* tests working (no list)
* list working
* add formatting
* more formatting
* fix tests
* dummy commit
* fix types
* remove unnecessary improt
* ignore type when importing from ee in task
* fix test running
* Clickhouse Trends Base (#1609)
* initial working
* date param almost working
* fix date range and labels
* fixed monthly math
* handle compare
* change table
* using new event ingestion
* direct query actions working
* remove interface
* fix date range
* properties initial working
* handle operator
* handle operator
* move timestamp parse
* move more to util
* inital breaking down working
* working cohort breakdown
* some tests running
* fix sessions
* cohort tests
* action and interval test
* reorder cohort filtering
* rename retention test
* fix inits
* change multitenancy tests
* fix types
* fix optional types
* replace ch_client.execute with sync_execute
* replace ch_client.execute with sync_execute, part 2
* Clickhouse Stickiness + Process Event (#1654)
* generate clickhouse uuid script
* set CLICKHOUSE_SECURE=False by default if running in TEST or DEBUG
* convert person_id to UUID, make adding `person_id` optional, add distinct_ids already in the `create_person` function
* Fix test_process_event_ee.py, remove all calls to Person.objects.*
* add back util
* fix broken imports
* improve process_event test clickhouse queries
* Basic stickiness query
* Clickhouse Stickiness tests
* stickiness test [WIP, actions fail]
* generate clickhouse uuid script
* change default test runner if PRIMARY_DB=clickhouse
* fix stickiness test for actions
* fix merge bug
* remove _create_person stub; cohort person_id is UUID now
* fix typing
* Clickhouse trends process math (#1660)
* most of process math works
* all process math
* fix ordering issue
* unusued imports
* update property comparison for process_event_ee
* indentation wrong missing calls
* demo users and events (#1661)
* finish breakdown filtering tests and reformat label function
* add increment to demo_data
* update demo data populating
* Add people endpoint for ch (#1670)
* add people endpoint for ch
* stickiness people
* fix value padding
* add process math to breakdown and
* add limit
* fix tests
* condensed code
* converted test to factory
* add people tests
* add month handling
* add typing fix
* change people test handling
* fix tests
* Clickhouse funnels 2 (#1668)
* add elements to create_event
* WIP closes #1663 Add funnels to clickhouse
* Make funnels work
* Clean up
* Move filtering around
* Add mypy tests and fix
* Performance improvements
* fix person tests again
* add people for funnel endpoint
* fix prop numbering
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Eric <eeoneric@gmail.com>
* merge master
* add retention
* update types
* more typing errors
* fix types
* bug with kafka payload, elements insert, and demo data
* Clickhouse Paths (#1657)
* paths clickhouse test (fails)
* add elements to create_event
* make this fail for clickhouse
* hardcoded query that returns good results for $pageviews, no filters yet
* clean up queries
* bound by time, fix 30min new session boundary
* support screen and custom events
* add properties filter
* paths url
* filter by path start
* better path start test
* even better path start test
* start from the first "path start" in a group
* test for person_id in paths
* partition by person_id for POSTGRES paths
* partition by person_id for Clickhouse paths
* clean up order in paths test
* clean up order in paths test
* join elements
* force element order on element group creation
* remove "order" when creating elements in tests and demo
* get list of elements for paths
* add limit to paths query
* use materialized view
* rename "element_hash" to "elements_hash" (no change in db)
* cull rows that are definitely unused
* simplify query
* New highly optimized paths clickhouse query
* start_point for $autocapture paths
* extract event property values from clickhouse
* prevent crash
* select one element sql
* get elements for event
* remove lodash
* remove host from $pageview path elements if same domain as incoming path
* show metadata based on loaded paths filter, not in flight filter
* fix order (all soures and targets in order, not all sources first, then all targets after) - makes for a better looking graph
* add test that makes the Postgres paths query fail
* fix postgres paths --> no fuzzy matching, breaks "starts with" for urls and gives too many incorrect start points
* create automatic /demo urls that match the real urls (no ending /)
* fix elements queries
* path element joins
* create persons via postgres in paths test
* change serializers back to id
* fix tests with uuid
* fix demo
* more bugs
* fix type
* change now to timezone aware
* [clickhouse] retention filters (#1725)
* implemented target entity and prop filtering
* add insight view override
* fix endpoint and filters
* include tests
* fix tests
* add period filtering
* .
* fix pg param name
* add filtering params to both queries in retention sql
* fix param again
* change to todatetime
* change tz to timezone
* add back timezone in model/event
* [clickhouse] feature flag endpoint requests (#1731)
* add feature flags to endpoints
* add flags to endpoints that check on request
* remove magic strings and fill in missing flags
* fix types
* add missing flag
* change from iso
* fix more timestamps and comparator
* change _people to get_people in actions view
* remove action and cohort populating
Co-authored-by: James Greenhill <jams@uber.com>
Co-authored-by: Marius Andra <marius.andra@gmail.com>
Co-authored-by: Tim Glaser <tim.glaser@hiberly.com>
* - set a custom cookie to tell the toolbar we're logged in
- make other cookies secure
* skip adding new toolbar cookie on API requests
* fix demo page local https
* enable/disable secure cookies as needed
* show django toolbar only for API requests
* update posthog-js and kea versions
* satisfy mypy
* add SECURE_COOKIES=0 to cypress tests, since they run over HTTP
* revert kea version
* add test for toolbar cookie middleware
* try something for cypress
* add debug
* add cypress-terminal-report for better logging
* fix demo api_url
* revert previous change that triggered cypress
It did not run under `fish` shell at all, added a shebang to make it
work.
Also it assumed nodemon is installed globally. Instead we can add
nodemon as a devDependency and use `npx` to run it.
* Closes #877 chunk loading errors
* Closes #877 chunk loading errors
* add chunks to all webpack files, use webpack-html-plugin to make an index.html with the right names
* change to contenthash for better caching
* add dev server on a different port
* store loaded scenes in reducer
* add react-hot-loader
* add react-hot-loader to all code split points
* fix action pages HMR
* ignore cypress screenshots
* generate django login/signup page layout with webpack html plugin
* move to devDependencies
* expose webpack-dev-server ports
* run tests on the production docker image
* start webpack dev server on a custom host if requested
* revert e2e to dev dockerfile
* add test travis config
* add stages
* add travis conf
* cache node and pip
* node 11
* travis ci bump
* node via nvm
* install v12, disable cypress for a moment
* remove 2 commands, test cached build time
* try different yarn cache, add cypress
* add postgres 12
* migrate before cypress
* remove latest postgres for now
* createdb before cypress
* Try different port
* cull packages
* remove hash from main bundle filenames to not break anything
* cypress port 8000
* cypress tests with production docker
* don't use the /code path in e2e test, use docker image
* remove hash from css to work better with editor
* only one export from actions
* remove travis test
Co-authored-by: Marius Andra <marius.andra@gmail.com>
* Split out celery worker into its own container for easier debugging
* Refactor creating people from events and serialize Team to celery worker
* type fixes
* remove todo message
* use the correct marshal (unmarshal)
* limit to just removing try except for person creation
* switch from count to exists
* initial test
* add yaml
* add logs
* without tail
* run again for consistency
* wait longer
* manuall configure cypress
* use latest ubuntu
* try installing all
* yarn install
* yarn install with cypress
* full flow
* run again for consistency
* initial tests for trends
* handle exception
* separate e2e docker-compose
* run on pr
* trends-testing foundation
* run test
* don't pass everything
* handle uncaught exceptions
* fix dashboard tests
* change email domain
* change dataattr to data-attr and boilerplate cleanup
* base url config
* add script for test runner
* change default postgres
* Closes #169 break down by cohort
* Fix test
* Prettier breakdown filters
* Rerender each time it's opened
* Fix tests
* Add 'all users' option in breakdown
* Make people work with cohorts
* Precalculate cohorts
* Return last_calculation
* Show precalculation in the frontend
* Closes #675 fix icon alignment
* Frontend fixes, antd and more
* Make sure unsetting breakdown doesn't break things
* Semantic name for migration
* Separate data migration + cohort key bugfix
* Correctly filter people by event
* Typing
* ignore copying frontend/dist folder - otherwise whatever you build in docker will get overridden by local build artifacts if any exist in frontend/dist
* support configuring redis with POSTHOG_REDIS_HOST and other vars in addition to REDIS_URL
* remove "the next version" in worker requirement modal
* split beat and celery scripts
* remove chart folder
* celery heartbeat every 10sec, reduce distributed beat lock hold time
* remove dockerfile local link
* add localhost redis url for tests
* add warning script to settings and docker-worker in case REDIS_URL is not found
* test celery heartbeat
* save celery heartbeat in redis
* /_stats/ with worker heartbeat
* refactor top content
* show worker stats
* heartbeat delay to 90sec
* update worker error message
* remove line
* add redbeat schdeduler
* add redbeat for local development
