0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-12-01 04:12:23 +01:00
posthog/.github/workflows/container-images-cd.yml

230 lines
10 KiB
YAML
Raw Normal View History

#
# Build and push PostHog and PostHog Cloud container images
#
# - posthog_build: build and push the PostHog container image to DockerHub
#
# - posthog_cloud_build: build the PostHog Cloud container image using
# as base image the container image from the previous step. The image is
# then pushed to AWS ECR.
#
name: Container Images CD
on:
push:
branches:
- master
paths-ignore:
- 'rust/**'
chore: move livestream to posthog monorepo (#23044) * 🔥 initial commit * update readme * Update README.md * Update README.md * deploy scripts * very basic consumer setup * add some configs and docker-compose * formatting for testing * add tailscale * flip from dev to prod flag * set default to be not prod * default for group_id * tailscale up * update gitignore * basic geolocation * remove unused localServer * document mmdb * just make configs an example * drop raw print * add a start script (downloads the mmdb) * add readme and update configs.example * ts working * if in start * update start script * fix start * fix start * fix more * add sql endpoints for tokenId and Person lookups * work towards filter * sub channel * fix subChan * hardcode team2 token * add cors * only allow get and head * add atomicbool * add channel to kafka * add logs * verbose logs * make array * drop sub ptrs * more logs * helps to loop * drop some logigng * move sub branch * logging * drop log * hog * Deal with numeric distinct ids later * logs * api_key * send 1/1000 * remove log * remove more logs * change response payload * set timestamp if needed * fill in person_id if team_id is set * require teamid, convert to token * clean up subs on disconnect * log * check for token in another place * clean up subs on disconnect * drop modulo and log * fix no assign * don't reuse db conn for now * drop a log * add back commented out log * Don't block on send to client channel * add geo bool * only geo events * use wrapper ip * don't require team in geo mode * add an endpoint and stats keeper for teams * remove stats keeper * start stats keeper * wire it up * change the shape of the response * omit empty error * omit empty on the stats as well * enable logging on back pressure * add jwt endpoint for testing * support multiple event types * Get Auth Setup * jwt team is float so turn that into int * logs * add auth for stats endpoint * remove tailscale and use autoTLS on public endpoints * default to :443 for auto tls * remove un-needed endpoints and handlers * Use compression because... a lot of data (#9) * add dockerfile and CI/CD (#10) * add dockerfile and CI/CD * Use ubuntu not alpine couldn't build in alpine :'( * Add MMDB download to Dockerfile (#11) * Use clearer name for MMDB * Don't connect to Kafka over SSL in dev * Fix JWT token in example config * Add postgres.url to example config * Add expected scope * Fix const syntax * Put scope validation where claims are known * Fix audience validation * moves * ignore livestream for ci * main -> master * move GA to root * docker lint fix * fix typo * fixes for docker builds * test docker build * livestream build docker * dang * Update .github/workflows/livestream-docker-image.yml Co-authored-by: Neil Kakkar <neilkakkar@gmail.com> * Update .github/workflows/livestream-docker-image.yml Co-authored-by: Neil Kakkar <neilkakkar@gmail.com> * don't build posthog container when PR is pushed for rust or livestream * Update .github/workflows/livestream-docker-image.yml Co-authored-by: Neil Kakkar <neilkakkar@gmail.com> * add a lot of paths-ignore * Update .github/workflows/livestream-docker-image.yml Co-authored-by: Neil Kakkar <neilkakkar@gmail.com> * Dorny filters are handling most of what I was trying to do * remove tailscale to speed up builds * maybe? * push container to github.com/posthog/postog * don't build container on PR * remove more filters because dorny --------- Co-authored-by: Brett Hoerner <brett@bretthoerner.com> Co-authored-by: Zach Waterfield <zlwaterfield@gmail.com> Co-authored-by: Frank Hamand <frankhamand@gmail.com> Co-authored-by: Michael Matloka <michal@matloka.com> Co-authored-by: Neil Kakkar <neilkakkar@gmail.com>
2024-06-18 17:38:53 +02:00
- 'livestream/**'
workflow_dispatch:
jobs:
posthog_build:
name: Build and push PostHog
if: github.repository == 'PostHog/posthog'
runs-on: ubuntu-latest
permissions:
id-token: write # allow issuing OIDC tokens for this workflow run
contents: read # allow at least reading the repo contents, add other permissions if necessary
packages: write # allow push to ghcr.io
steps:
- name: Check out
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Depot CLI
uses: depot/setup-action@v1
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Login to Amazon ECR
id: aws-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build and push container image
id: build
uses: depot/build-push-action@v1
with:
buildx-fallback: false # the fallback is so slow it's better to just fail
push: true
tags: posthog/posthog:${{ github.sha }},posthog/posthog:latest,${{ steps.aws-ecr.outputs.registry }}/posthog-cloud:master
platforms: linux/arm64,linux/amd64
build-args: COMMIT_HASH=${{ github.sha }}
- name: get deployer token
id: deployer
uses: getsentry/action-github-app-token@v3
with:
app_id: ${{ secrets.DEPLOYER_APP_ID }}
private_key: ${{ secrets.DEPLOYER_APP_PRIVATE_KEY }}
- name: get PR labels
id: labels
uses: ./.github/actions/get-pr-labels
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Trigger PostHog Cloud deployment from Charts
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "posthog",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ steps.labels.outputs.labels }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
}
- name: Check for changes in plugins directory
id: check_changes_plugins
run: |
echo "changed=$((git diff --name-only HEAD^ HEAD | grep -q '^plugin-server/' && echo true) || echo false)" >> $GITHUB_OUTPUT
- name: Trigger Ingestion Cloud deployment
if: steps.check_changes_plugins.outputs.changed == 'true'
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "ingestion",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ toJson(steps.labels.outputs.labels) }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
}
- name: Check for changes that affect batch exports temporal worker
id: check_changes_batch_exports_temporal_worker
run: |
echo "changed=$((git diff --name-only HEAD^ HEAD | grep -qE '^posthog/temporal/common|^posthog/temporal/batch_exports|^posthog/batch_exports/|^posthog/management/commands/start_temporal_worker.py$|^requirements.txt$' && echo true) || echo false)" >> $GITHUB_OUTPUT
- name: Trigger Batch Exports Sync Temporal Worker Cloud deployment
if: steps.check_changes_batch_exports_temporal_worker.outputs.changed == 'true'
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "temporal-worker",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ steps.labels.outputs.labels }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
}
- name: Trigger Batch Exports Temporal Worker Cloud deployment
if: steps.check_changes_batch_exports_temporal_worker.outputs.changed == 'true'
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "temporal-worker-batch-exports",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ steps.labels.outputs.labels }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
}
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
- name: Check for changes that affect general purpose temporal worker
id: check_changes_general_purpose_temporal_worker
run: |
echo "changed=$((git diff --name-only HEAD^ HEAD | grep -qE '^posthog/temporal/common|^posthog/temporal/proxy_service|^posthog/management/commands/start_temporal_worker.py$|^requirements.txt$' && echo true) || echo false)" >> $GITHUB_OUTPUT
- name: Trigger General Purpose Temporal Worker Cloud deployment
if: steps.check_changes_general_purpose_temporal_worker.outputs.changed == 'true'
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "temporal-worker-general-purpose",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ steps.labels.outputs.labels }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
}
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
- name: Check for changes that affect data warehouse temporal worker
id: check_changes_data_warehouse_temporal_worker
run: |
echo "changed=$((git diff --name-only HEAD^ HEAD | grep -qE '^posthog/temporal/common|^posthog/temporal/data_imports|^posthog/warehouse/|^posthog/management/commands/start_temporal_worker.py$|^requirements.txt$' && echo true) || echo false)" >> $GITHUB_OUTPUT
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
- name: Trigger Data Warehouse Temporal Worker Cloud deployment
if: steps.check_changes_data_warehouse_temporal_worker.outputs.changed == 'true'
uses: peter-evans/repository-dispatch@v3
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
with:
token: ${{ steps.deployer.outputs.token }}
repository: PostHog/charts
event-type: commit_state_update
client-payload: |
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
{
"values": {
"image": {
"sha": "${{ steps.build.outputs.digest }}"
}
},
"release": "temporal-worker-data-warehouse",
"commit": ${{ toJson(github.event.head_commit) }},
"repository": ${{ toJson(github.repository) }},
"labels": ${{ steps.labels.outputs.labels }},
"timestamp": "${{ github.event.head_commit.timestamp }}"
feat(data-warehouse): DLT + temporal (#18700) * testing * draft * tests * working rough draft * split workflow and activity workers * temp * split task queues * add schedule * working with schedule * add bucket stitching activity * sync data steps for pipeline * add comment * update default endpoints * workflow with all steps * update heartbeat * update env var, update folders so there are common and specific workflow modules * update workflow * update imports * reformat * already sync * format * adjust async methods * make it readable * remove extra * remove unnecessary config * add invoices to default endpoint * change heartbeat name * remove * update dev packages * update migration * package and tests * update test import path * restore task queue env var and remove unnecessary ones, update github action to only trigger deploy for respective worker * fix tests * typing * revert task queue change * update github action and env vars * poll reload * update retry * change activity stiching function back to sync for now * typing * try to import differently * Update query snapshots * Update query snapshots * remap * move * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (2) * Update UI snapshots for `chromium` (1) * sort * update import * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) * typing * add notes on logging * make sure datawarehousetable filtering works right * update timeout and how datawarehouse tables are linked * sort * add delete * Update UI snapshots for `chromium` (1) * Update UI snapshots for `chromium` (1) --------- Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 20:05:24 +01:00
}