0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-12-01 12:21:02 +01:00
Commit Graph

24 Commits

Author SHA1 Message Date
Marius Andra
b5a02e2ca7
Bundle our frontend via esbuild (#6758)
* esbuild package

* almost get esbuild working

* fix react-virtualized imports

* add splitting

* fix funny import reorder bug

* fix squeakAudio referring to itself

* write index.html file

* fix some bad imports

* update antd paths

* remove raw-loader usage, it didn't work anyway

* refactor and copy public

* build app and toolbar

* get toolbar working, but without styles

* make toolbar and its styles work

* shared dashboards

* clean frontend build before rebuilding

* add watch mode

* reorder tasks

* revert js url

* incremental builds of app with debounced chokidar watching

* common build/watch script

* improve logs

* watch during firrst build

* fix toolbar url

* fix wrongly exported scene

* create sceneProxyLogic to untangle sceneLogic from all bundles

* disconnect sceneLogic and refactor setPageTitle

* live reloading server

* rename utils file

* only wait for /static

* fix encoding

* simplify

* add missing dayjs plugins

* fix pathless logics

* simplify options

* add jsx for webapck

* slight delay to catch changes

* a type is a type

* fix build

* esbuild in start

* funnelLogic path

* include all files with a "." (so .mjs, etc) in /frontend/ to docker

* rename to "utils.mjs", make "build.mjs" executable

* improve erroring

* revert some needless changes

* more reverts

* change some scripts

* remove setuff

* clarify function

* make "--host 0.0.0.0" work

* fix import order issue in webpack

* remove webpack css inlining for toolbar to simplify config

* make toolbar with external styles work in storybook

* move live server injection into django

* fix undefined bug

* simplify setup to work with injection directly in http://localhost:8000 (no proxying needed on :8234)

* add comments

* Fix `fse` usage

I was getting this otherwise:

$ node frontend/build.mjs
file:///Users/twixes/Developer/posthog/frontend/utils.mjs:46
    fse.copySync(srcDir, destDir, { overwrite: true }, function (err) {
        ^

TypeError: fse.copySync is not a function
    at copyPublicFolder (file:///Users/twixes/Developer/posthog/frontend/utils.mjs:46:9)
    at file:///Users/twixes/Developer/posthog/frontend/build.mjs:5:1
    at ModuleJob.run (internal/modules/esm/module_job.js:146:23)
    at async Loader.import (internal/modules/esm/loader.js:165:24)
    at async Object.loadESM (internal/process/esm_loader.js:68:5)

* Mock `process` for VFile used by ReactMarkdown

I was getting this otherwise:

core.js:55 Uncaught ReferenceError: process is not defined
    at new VFile (core.js:55)
    at VFile (core.js:49)
    at Function.parse (index.js:273)
    at ReactMarkdown2 (react-markdown.js:42)
    at renderWithHooks (react-dom.development.js:14803)
    at mountIndeterminateComponent (react-dom.development.js:17482)
    at beginWork (react-dom.development.js:18596)
    at HTMLUnknownElement.callCallback2 (react-dom.development.js:188)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:237)
    at invokeGuardedCallback (react-dom.development.js:292)

* Mock `process.env` for VFile used by ReactMarkdown

I was getting this otherwise:

platform.ts:73 Uncaught TypeError: Cannot read properties of undefined (reading 'ENABLE_VSCODE_BROWSER_CODE_LOADING')
    at platform.ts:73
    at platform.ts:79
    at Function.r._invokeFactory (loader.js:1118)
    at r.complete (loader.js:1128)
    at r._onModuleComplete (loader.js:1754)
    at r._resolve (loader.js:1714)
    at r.defineModule (loader.js:1357)
    at _ (loader.js:1804)
    at numbers.ts:10
    at fake:1

* pass the heavy appScenes to sceneLogic through props via App.tsx

* remove sceneProxyLogic

* remove exported variables

* fix sceneLogic test

Co-authored-by: Michael Matloka <dev@twixes.com>
2021-11-03 09:50:24 +01:00
Marius Andra
05d7817d14
update pycharm plugin server run configurations (#6742) 2021-10-29 13:48:39 +02:00
Karl-Aksel Puulmann
437b2982cd
Benchmarking suite for ClickHouse queries (#6187)
* Add asv code

* Get a benchmark running

* better output folder

* Move benchmark file

* Cleanup of config

* Run a query and benchmark

* Skip benchmarks for isort

* Set up materialized columns before test

* Better skipping logic

* Add first proper benchmark, add some documentation

* Person property filtering

* Add new workflow

* Show stderr in test run

* Continue debugging actions

* Try to save benchmark results to separate repo

* Output to main directory

* Scheduling

* Remove if temporarily

* Int for ms

* Comment on the PR

* Collapsible section

* Update README

* Clarification

* Remove h3

* Remove pyproject.toml, .gitignore changes that are unneeded
2021-10-01 15:20:58 +03:00
Marius Andra
c080d5f114
Deploy storybook.posthog.net (#5833)
* Deploy storybook.posthog.net

* test why this fails

* another experiment

* again...

* turn addon panels off by default

* debug ci script

* add welcome story

* try with other paths

* commit via action

* better welcome story

* remove branch exception

* keep the CNAME file

* remove branch exception

* remove duplicate checkout, fix paths

* remove branch exception
2021-09-07 12:00:44 +00:00
Marius Andra
a9f098a659
Storybooks via Chromatic (#5736)
* add chromatic

* install chromatic only in a github action

* remove chromatic script (published via github action)
2021-08-27 09:43:15 +02:00
Michael Matloka
427e4fac6e
Improve handling of .git* files/dirs (#3141)
* Create .gitattributes

* Remove .github/ in posthog-foss sync

* .gitignore whole directories (recursively ofc)

* Remove .gitignore line from ages past
2021-02-01 09:25:40 +01:00
Marius Andra
19b453c996
Kea TypeGen types in the same folder as logic (#3048)
* update kearc to generate types in the same folder, add types to gitignore

* update type imports
2021-01-22 10:21:48 +01:00
Michael Matloka
96e4ee8512
Nest endpoints of project-based models under /api/project/ – LITE (#2485)
* Nest endpoints under /project/ with StructuredViewSetMixin

* Rewrite URLs

* isort

* Update utils.py

* Fix errors

* Fix almoast all the errors

Last left to do: shared dashboards and permission classes.

* isort

* Adjust for master

* Add compatbility with shared dashboards

* Debug ClickHouse

* Remove some # type: ignores

* Simplify CursorPagination

* Move test base from posthog.api.test to posthog.test

* Improve API structure

* Bring back legacy endpoints

* Fix legacy compatibility

* Fix bugs and typing

* isort

* Fix hooks test

* Try fixing errors

* Fix oversight

* isort

* Fix problems

* isort

* Be more tolerant

* Fix naming and remove redundant code

* Fix imports

* Update deleteWithUndo

* Roll back

* Roll back more

* Update .gitignore

* Rollll back

* Rollllllll

* back

* Betterify

* Address feedback
2020-11-24 23:26:28 +01:00
Marius Andra
97b5d8faac
deprecate from_web and from_json (#2449) 2020-11-20 16:51:06 +01:00
Marius Andra
fce86ef46d
Plugin v8 (#1946)
* plugin progress

* blah

* add posthog config for plugins

* test gitignore

* new functionality for plugins

* support local plugin paths

* also ignore symlinks

* add positional argument

* fixes

* small fixes

* config polish

* config passed to posthog plugin

* ooooooops

* symlink fix

* cleanse dir before loading

* add cache to plugins

* pickle the goods

* unlink symlink

* pass full config

* unlink even if link points to nothing

* fix fix

* return none if value is empty

* plugin model

* plugins scene

* add config schema to plugins

* install plugins

* save descriptions

* show descriptions

* edit plugin

* save plugin config

* plugin modal

* uninstall plugins

* UX cleanup

* add "required" to plugin config

* open plugin modal after install

* split to subcomponents

* install custom plugins

* rework backend for model plugins

* Plugins on models

* simple reload pubsub

* fix apps not installed

* fix master/main issue

* fix reload command

* use the github api to get the default branch

* init plugins only if not running migrate/makemigrations

* store plugins zip archives in postgres

* tag plugins to specific versions

* save plugins in pluginConfig

* update pluginConfigs instead of adding new rows, remove from redux on uninstall

* remove debug

* run plugins from db by team

* reload when deleting

* remove debug

* smarter handling of dynamic plugins, support local plugins again

* improve typings, add some nicer warnings

* yarn lock file after merge

* squash migrations and add "locked" field to plugins

* error if folder not found in zip

* unregister plugins

* skip plugin init in test mode

* basic plugin test

* avoid mutating the same prop hash

* add pip tools to requirements.txt

* fix mypy, fix manage.py script error

* avoid plugins with mypy

* mypy fix

* abstract redis into plugin and add team_id to reload

* refactor and start work on syncing with posthog.json

* start testing plugin loading from json

* test plugin deletion

* test for syncing plugins from config

* complete and then test local json plugin sync

* test converting back and forward between an local and http path

* remove global plugin config from plugins array in posthog.json

* rename configSchema --> config_schema

* fix migration after merge

* rename from_cli to from_json

* mypy

* import pip after plugin loaded

* show error details

* raise exceptions visible to the frontend

* sync plugins on load

* access control to updating plugins from the web

* access control

* remove posthog.json from git

* test config schema from json

* if you can install via the web, you can also configure

* remove separate view access

* title as "Plugins" instead of "Installed Plugins" if we can't install ourselves and don't see the repository

* add self.team to plugin base class

* add instance_init method

* refactor into files

* sync global plugin config from json

* make global plugins work, add test

* global plugins in interface, make them take precedence over local plugins

* add comments to plugin base class

* reload/reset plugins before each test

* add error field to plugins

* add many plugin zips

* add many plugin zips, fix imports

* store errors on plugin object and test them

* fix types

* add null to error

* can be with any team ID in the test

* save problems running plugins in the plugin_config model

* try to create redis connection pool only once

* throw if no redis

* mypy

* get instance inside heartbeat and not top level

* try caching pubsub

* try pip install with -q

* install pip externally

* remove uuid and typing, now in stdlib

* more verbosity

* add pip back

* catch exceptions

* quiet and no input for pip

* check plugin reload every 10sec on new task

* fix type errors

* fix requirements error message

* use repository.json

* only load and reload plugins on workers

* rename task

* support local js plugins via py-mini-racer

* load js plugins from zip files

* extract jsplugin class and convert to syntax that uses global functions instead of initializing a class

* process events via grpc

* process events with the "posthog-plugins" queue to enable plugins

* remove old native python & mini racer plugin code

* default to false

* change env vars

* fix test

* remove grpc tools

* skip plugins in migrate.py scripts

* fix migration

* change output of settings debug banner to STDERR

* start posthog plugin server with worker

* try to fix python 3.7 test

* add fallback for the optional argument

* annoying CI test debug

* try to finally fix python 3.7 test

* here we go again

* move plugins under instance

* move plugins npm start into its own folder

* more console.log debugginf

* and again

* move plugins to separate script

* more prints

* fix test error

* docker config

* small fixes

* dckerfile fix

* reload plugins via pubsub, upgrade version

* plugins that support team setup code

* sync if made changes from config

* move plugins in menu

* require node 14 in heroku for better plugin support (namely ?. support)

* bump node version in dockerfiles

* update node versions for github actions

* update the concurrency for heroku workers

* update the concurrency for heroku workers (add link)

* Fix migrations after merge

* add ignore_result to process_event tasks

* fix: docker-preview run in parallel bug

* change order of commands

* remove separate plugins server conf script

* clarify intent

* revert castaway change

* add context to plugins/sync.py

* change everything to ValidationError

* delegate destroy to super

* no request to repository url if can't install

* make the if cleaner

* add clarifying line

* add clarifying line

* fix url field type

* rename get_redis_instance to get_client, move to posthog.redis

* remove duplicate validation

* flip if around

* simplify api logic

* simplify plugin_config api, fix global_plugin error

* remove unnecessary field rename

* mypy

* Plugins UI (#2090)

* base UI

* more UI

* load plugin image if available

* toggle enabled plugin

* plugin cards for available

* custom plugins

* change plugin configuration to drawer

* asks for confirmation when enabling or disabling a plugin

* loading state

* separation of concerns, leave new styles for separate PR

* general improvements

* remove button when installation is not available

* preemptively avoid merge conflict with #2114

* move papercups widget & hide bottom bar when drawer is open

* allow clicking the entire plugin card

* address all feedback

* move plugins under "project" menu

* Hide "configure" from globally enabled plugins

Co-authored-by: Marius Andra <marius.andra@gmail.com>

* add plugins opt in toggle to project/plugins

* choose pipeline based on team setting

* add "beta"

* plugin opt-in opt-out pages

* adjust install button

* remove tasks that are never called, remove PLUGINS_ENABLED global key

* fix responsive card display

* fix typo and drawer width

* skeleton fixes

* typo

* use "posthog-plugin-server" npm package

* "posthog-plugin-server" doc

* require the plugin server to be online before enabling plugins

* remove a few needeless "?." cases

* add hint for config_schema

* add hint for errors

* show plugin errors

* stop clicks if clicking on error

* show plugin errors

* loading indicators

* reload plugins when opting in/out

* nicer beta tag

* add frontend type

* fix mypy error

* fix test

* disable plugins if MULTI_TENANCY

* upgrade plugin-server version

* save event with plugin error

* upgrade plugin server

* squashed & optimized migrations

* remove unused import

* updates opt-in copy & hides tech details for cloud version

* fix cypress tests

* compare with None

* change plugins url and add redirect

* remove ellipsis

* use code snippet in plugin errors

* change github regex

* fix loading flickering on installing plugins

* add comment to plugin archive

* fix python style

* remove pip-tools (relic from the python plugin era)

* hard pin plugin server version

* remove copying of posthog.json from dev dockerfile (breaks if file doesn't exist, copied later anyway)

* update lockfile

Co-authored-by: James Greenhill <fuziontech@gmail.com>
Co-authored-by: Paolo D'Amico <paolodamico@users.noreply.github.com>
2020-11-02 15:08:30 +01:00
Michael Matloka
2e39923ad8
gitignore __pycache__ directories (#1945) 2020-10-20 17:29:28 +02:00
James Greenhill
97a665d277
Leverage Postgres for Persons and reorder order by's on clickhouse" (#1681)
* Query optimizations

* more sql optimizations

* checkpoint

* fix migration

* add UUID field to person

* use django signals to signal that clickhouse needs to be updated

* cleanup person logic

* cleanups

* update migration

* Don't setup the django signals unless we are for sure using ee setup

* expecting back to 30 queries for capturing with new person

* add .venv to .gitignore

* add env back to .gitignore

Co-authored-by: Ubuntu <ubuntu@ip-172-31-73-18.ec2.internal>
2020-09-22 13:41:08 +01:00
James Greenhill
dedf5582c1
first chunk of clickhouse framework (#1613)
* first chunk of clickhouse framework

* prod, not dev docker-compose.yml

* add clickhouse sql files
2020-09-08 16:12:27 -07:00
James Greenhill
c2e03a3a46
Revert this to bring back the working copy of ee / clickhouse (#1588) 2020-09-04 21:12:07 -07:00
James Greenhill
5506135c3c
Bring back clickhouse changes along with queue size metrics (#1579)
* Revert "Revert "Clickhouse setup (#1463)""

This reverts commit 7f2cab4b93.

* add queue backlog to _stats endpoint

* celery queue length into heartbeat -> statsd

* reformat

* type check fixes

* bump expected number of queries to 8

Co-authored-by: Eric <eeoneric@gmail.com>
2020-09-04 13:44:53 -07:00
James Greenhill
7f2cab4b93 Revert "Clickhouse setup (#1463)"
This reverts commit a0327587cb.
Time to process events shot way up and logs are missing.
2020-09-03 19:27:02 -07:00
Eric Duong
a0327587cb
Clickhouse setup (#1463)
* initial

* migration command

* migrations working

* add modelless views for clickhouse

* initial testing structure

* use test factory

* scaffold for all tests

* add insight and person api

* add basic readme

* add client

* change how migrations are run

* add base tables

* ingesting events

* restore delay

* remove print

* updated testing flow

* changed sessions tests

* update tests

* reorganized sql

* parametrize strings

* element list query

* change to seralizer

* add values endpoint

* retrieve with filter

* pruned code to prepare for staged merge

* working ingestion again

* tests for ee

* undo unneeded tests right now

* fix linting

* more typing errors

* fix tests

* add clickhouse image to workflow

* move to right job

* remove django_clickhouse

* return database url

* run super

* remove keepdb

* reordered calls

* fix type

* fractional seconds

* fix type error

* add checks

* remove retention sql

* fix tests

* add property storage and tests

* merge master

* fix tests

* fix tests

* .

* remove keepdb

* format python files

* update CI env vars

* Override defaults and insecure tests

* Update how ClickHouse database gets evaluated

* remove bootstrapping clickhouse database routine

* Don't initialize the clickhouse connection unless we say it's primary

* .

* fixed id generation

* remove dump

* black settings

* empty client

* add param

* move docker-compose for ch to ee dir

* Add _public_ key to repo for verifying self signed cert on server

* update ee compose file for ee dir

* fix a few issues with tls in migrations

* update migrations to be flexible about storage profile and engine

* black settings

* add elements prop tables

Co-authored-by: James Greenhill <jams@uber.com>
2020-09-03 10:27:45 -07:00
Marius Andra
45f10a8eaf
Kea TypeGen 0.2 (#1427)
* kea-typegen 0.2

* remove old Type.ts files

* more setup for typegen-0.2, update yarn scripts, remove webpackbar

* import types from the right paths

* update ENV in build script
2020-08-14 17:02:57 +02:00
Tim Glaser
bcf427cb55
Webpack HMR, Add hashes to chunk filenames. (#878)
* Closes #877 chunk loading errors

* Closes #877 chunk loading errors

* add chunks to all webpack files, use webpack-html-plugin to make an index.html with the right names

* change to contenthash for better caching

* add dev server on a different port

* store loaded scenes in reducer

* add react-hot-loader

* add react-hot-loader to all code split points

* fix action pages HMR

* ignore cypress screenshots

* generate django login/signup page layout with webpack html plugin

* move to devDependencies

* expose webpack-dev-server ports

* run tests on the production docker image

* start webpack dev server on a custom host if requested

* revert e2e to dev dockerfile

* add test travis config

* add stages

* add travis conf

* cache node and pip

* node 11

* travis ci bump

* node via nvm

* install v12, disable cypress for a moment

* remove 2 commands, test cached build time

* try different yarn cache, add cypress

* add postgres 12

* migrate before cypress

* remove latest postgres for now

* createdb before cypress

* Try different port

* cull packages

* remove hash from main bundle filenames to not break anything

* cypress port 8000

* cypress tests with production docker

* don't use the /code path in e2e test, use docker image

* remove hash from css to work better with editor

* only one export from actions

* remove travis test

Co-authored-by: Marius Andra <marius.andra@gmail.com>
2020-06-02 10:39:53 +01:00
Marius Andra
9134e13177
Kea 2 (beta) upgrade (#663)
* upgrade to kea2 beta

* remove empty createStore

* add yalc to .gitignore

* add kea 2 and babel-plugin-kea
2020-04-27 16:39:21 +01:00
Marius Andra
66780ceef7
Make workers optional for now, add a warning if they are offline (#494)
* add warning script to settings and docker-worker in case REDIS_URL is not found

* test celery heartbeat

* save celery heartbeat in redis

* /_stats/ with worker heartbeat

* refactor top content

* show worker stats

* heartbeat delay to 90sec

* update worker error message

* remove line

* add redbeat schdeduler

* add redbeat for local development
2020-04-08 13:52:29 +01:00
Marius Andra
ba1fca11d0 gitignore .idea folder to ignore jetbrains IDE settings 2020-03-21 00:56:59 +01:00
Marius Andra
df500243f9 webpack proof of concept 2020-03-18 13:38:15 +01:00
Tim Glaser
f7ee996231 Initial commit 2020-01-23 14:46:27 -08:00