mirror of
https://github.com/PostHog/posthog.git
synced 2024-11-25 11:17:50 +01:00
cd82caab01
* feat(ingestion): remove Graphile worker as initial ingest dependency At the moment if the Graphile enqueing of an anonymous event fails e.g. due to the database that it is uing to store scheduling information fails, then we end up pushing the event to the Dead Letter Queue and to not do anything with it further. Here, instead of directly sending the event to the DB, we first push it to Kafka, an `anonymous_events_buffer`, which is then committed to the Graphile database. This means that if the Graphile DB is down, but then comes back up, we will end up with the same results as if it was always up* (*) not entirely true as what is ingested also depends on the timings of other events being ingested * narrow typing for anonymous event consumer * fix types import * chore: add comment re todos for consumer * wip * wip * wip * wip * wip * wip * fix typing * Include error message in warning log * Update plugin-server/jest.setup.fetch-mock.js Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com> * Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com> * include warning icon * fix crash message * Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts * Update plugin-server/src/main/ingestion-queues/anonymous-event-buffer-consumer.ts Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com> * setup event handlers as KafkaQueue * chore: instrument buffer consumer * missing import * avoid passing hub to buffer consumer * fix statsd reference. * pass graphile explicitly * explicitly cast * add todo for buffer healthcheck * set NODE_ENV=production * Update comment re. failed batches * fix: call flush on emitting to buffer. * chore: flush to producer * accept that we may drop some anonymous events * Add metrics for enqueue error/enqueued * fix comment * chore: add CONVERSION_BUFFER_TOPIC_ENABLED_TEAMS to switch on buffer topic Co-authored-by: Guido Iaquinti <4038041+guidoiaquinti@users.noreply.github.com> Co-authored-by: Yakko Majuri <38760734+yakkomajuri@users.noreply.github.com>
185 lines
4.9 KiB
Docker
185 lines
4.9 KiB
Docker
#
|
||
# This Dockerfile is used for self-hosted production builds.
|
||
#
|
||
# Note: for PostHog Cloud remember to update 'Dockerfile.cloud' as appropriate.
|
||
#
|
||
|
||
#
|
||
# Build the frontend artifacts
|
||
#
|
||
FROM node:18.8-alpine3.16 AS frontend
|
||
|
||
WORKDIR /code
|
||
|
||
COPY package.json yarn.lock ./
|
||
RUN yarn config set network-timeout 300000 && \
|
||
yarn install --frozen-lockfile
|
||
|
||
COPY frontend/ frontend/
|
||
COPY ./bin/ ./bin/
|
||
COPY babel.config.js tsconfig.json webpack.config.js ./
|
||
RUN yarn build
|
||
|
||
#
|
||
# Build the plugin-server artifacts. Note that we still need to install the
|
||
# runtime deps in the main image
|
||
#
|
||
FROM node:18.8-alpine3.16 AS plugin-server
|
||
|
||
WORKDIR /code/plugin-server
|
||
|
||
# Install python, make and gcc as they are needed for the yarn install
|
||
RUN apk --update --no-cache add \
|
||
"make~=4.3" \
|
||
"g++~=11.2" \
|
||
"gcc~=11.2" \
|
||
"python3~=3.10"
|
||
|
||
# Compile and install Yarn dependencies.
|
||
#
|
||
# Notes:
|
||
#
|
||
# - we explicitly COPY the files so that we don't need to rebuild
|
||
# the container every time a dependency changes
|
||
COPY ./plugin-server/package.json ./plugin-server/yarn.lock ./plugin-server/tsconfig.json ./
|
||
RUN yarn config set network-timeout 300000 && \
|
||
yarn install
|
||
|
||
# Build the plugin server
|
||
#
|
||
# Note: we run the build as a separate actions to increase
|
||
# the cache hit ratio of the layers above.
|
||
COPY ./plugin-server/src/ ./src/
|
||
RUN yarn build
|
||
|
||
# Build the posthog image, incorporating the Django app along with the frontend,
|
||
# as well as the plugin-server
|
||
FROM python:3.8.14-alpine3.16
|
||
|
||
ENV PYTHONUNBUFFERED 1
|
||
|
||
WORKDIR /code
|
||
|
||
# Install OS dependencies needed to run PostHog
|
||
#
|
||
# Note: please add in this section runtime dependences only.
|
||
# If you temporary need a package to build a Python or npm
|
||
# dependency take a look at the sections below.
|
||
RUN apk --update --no-cache add \
|
||
"libpq~=14" \
|
||
"libxslt~=1.1" \
|
||
"nodejs-current~=18" \
|
||
"chromium~=102" \
|
||
"chromium-chromedriver~=102" \
|
||
"xmlsec~=1.2"
|
||
|
||
# Curl the GeoLite2-City database that will be used for IP geolocation within Django
|
||
#
|
||
# Notes:
|
||
#
|
||
# - We are doing this here because it makes sense to ensure the stack will work
|
||
# even if the database is not available at the time of boot.
|
||
# It's better here to fail at build then it is to fail at boot time.
|
||
|
||
RUN apk --update --no-cache --virtual .geolite-deps add \
|
||
"curl~=7" \
|
||
"brotli~=1.0.9" \
|
||
&& \
|
||
mkdir share \
|
||
&& \
|
||
( curl -L "https://mmdbcdn.posthog.net/" | brotli --decompress --output=./share/GeoLite2-City.mmdb ) \
|
||
&& \
|
||
chmod -R 755 ./share/GeoLite2-City.mmdb \
|
||
&& \
|
||
apk del .geolite-deps
|
||
|
||
|
||
# Compile and install Python dependencies.
|
||
#
|
||
# Notes:
|
||
#
|
||
# - we explicitly COPY the files so that we don't need to rebuild
|
||
# the container every time a dependency changes
|
||
#
|
||
# - we need few additional OS packages for this. Let's install
|
||
# and then uninstall them when the compilation is completed.
|
||
COPY requirements.txt ./
|
||
RUN apk --update --no-cache --virtual .build-deps add \
|
||
"bash~=5.1" \
|
||
"g++~=11.2" \
|
||
"gcc~=11.2" \
|
||
"cargo~=1.60" \
|
||
"git~=2" \
|
||
"make~=4.3" \
|
||
"libffi-dev~=3.4" \
|
||
"libxml2-dev~=2.9" \
|
||
"libxslt-dev~=1.1" \
|
||
"xmlsec-dev~=1.2" \
|
||
"postgresql13-dev~=13" \
|
||
"libmaxminddb~=1.6" \
|
||
&& \
|
||
pip install -r requirements.txt --compile --no-cache-dir \
|
||
&& \
|
||
apk del .build-deps
|
||
|
||
RUN addgroup -S posthog && \
|
||
adduser -S posthog -G posthog
|
||
|
||
RUN chown posthog.posthog /code
|
||
|
||
USER posthog
|
||
|
||
# Add in Django deps and generate Django's static files
|
||
COPY manage.py manage.py
|
||
COPY posthog posthog/
|
||
COPY ee ee/
|
||
COPY --from=frontend /code/frontend/dist /code/frontend/dist
|
||
|
||
RUN SKIP_SERVICE_VERSION_REQUIREMENTS=1 SECRET_KEY='unsafe secret key for collectstatic only' DATABASE_URL='postgres:///' REDIS_URL='redis:///' python manage.py collectstatic --noinput
|
||
|
||
# Add in the plugin-server compiled code, as well as the runtime dependencies
|
||
WORKDIR /code/plugin-server
|
||
COPY ./plugin-server/package.json ./plugin-server/yarn.lock ./
|
||
|
||
# Switch to root and install yarn so we can install runtime deps. Node that we
|
||
# still need yarn to run the plugin-server so we do not remove it.
|
||
USER root
|
||
RUN apk --update --no-cache add "yarn~=1"
|
||
|
||
# NOTE: we need make and g++ for node-gyp
|
||
# NOTE: npm is required for re2
|
||
RUN apk --update --no-cache add "make~=4.3" "g++~=11.2" "npm~=8" --virtual .build-deps \
|
||
&& yarn install --frozen-lockfile --production=true \
|
||
&& yarn cache clean \
|
||
&& apk del .build-deps
|
||
|
||
USER posthog
|
||
|
||
# Add in the compiled plugin-server
|
||
COPY --from=plugin-server /code/plugin-server/dist/ ./dist/
|
||
|
||
WORKDIR /code/
|
||
USER root
|
||
COPY ./plugin-server/package.json ./plugin-server/
|
||
|
||
# We need bash to run the bin scripts
|
||
RUN apk --update --no-cache add "bash~=5.1"
|
||
COPY ./bin ./bin/
|
||
USER posthog
|
||
|
||
ENV CHROME_BIN=/usr/bin/chromium-browser \
|
||
CHROME_PATH=/usr/lib/chromium/ \
|
||
CHROMEDRIVER_BIN=/usr/bin/chromedriver
|
||
|
||
COPY gunicorn.config.py ./
|
||
|
||
ENV NODE_ENV=production
|
||
|
||
# Expose container port and run entry point script
|
||
EXPOSE 8000
|
||
|
||
# Expose the port from which we serve OpenMetrics data
|
||
EXPOSE 8001
|
||
|
||
CMD ["./bin/docker"]
|