0
0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-25 02:49:32 +01:00
Commit Graph

18 Commits

Author SHA1 Message Date
Frank Hamand
7a14421e78
feat: Set keepalive to 60 on gunicorn (#17897)
Set keepalive to 60 on gunicorn

The default is 2 seconds, the default for ALBs is 30 seconds

This can cause a race condition where gunicorn closes the connection
as the ALB sends a request, resulting in a 502.
2023-10-11 08:54:48 +01:00
Frank Hamand
c660212989
Revert "feat: Add kafka rack aware injection on docker-server" (#17710)
Revert "feat: Add kafka rack aware injection on docker-server (#17541)"

This reverts commit 789bc23c91.
2023-10-02 10:21:00 +00:00
Frank Hamand
69eb2e8fd3
Allow us to set gunicorn backlog (#17559) 2023-09-20 22:14:16 +01:00
Frank Hamand
789bc23c91
feat: Add kafka rack aware injection on docker-server (#17541)
Add kafka rack aware injection on docker-server

Currently we only inject this in plugin-server (aka reading from kafka)

Let's add it as well to docker-server (aka capture/writing to kafka)
2023-09-20 10:14:58 +01:00
Marius Andra
85b240aa21
fix(funnels): increase GET request size limit to 16k (#12381) 2022-10-21 14:27:10 +02:00
Harry Waye
3813795d22
chore(gunicorn): increase thread count for workers (#11507)
* chore(gunicorn): increase thread count for workers

We currently run with 2 worker processes each with 4 threads. It seems
occasionally we see spikes in the number of pending requests within a
worker in the 20-25 region. I suspect this is due to 4 slow requests
blocking the thread pool.

I'd suggest that the majority of work is going to be IO bound, thus it's
hopefully going to not peg the CPU. If it does then it should end up
triggering the HPA and hopefully resolve itself :fingerscrossed:

There is however gzip running on events, which could be intensive
(suggest we offload this to a downstream at some point). If lots of
large requests come in this could be an issue. Some profiling here
wouldn't go amiss.

Note these are the same settings between web and event ingestion
workload. At some point we may want to split.

I've added a Dashboard for gunicorn worker stats
[here](https://github.com/PostHog/charts-clickhouse/pull/559) which we
can monitor to see the effects.

Aside: if would be wise to be able to specify these settings from the
chart itself such that we do not have to consider chart/posthog version
combos, and allow tweaking according to the deployment.

* reduce to 8 threads
2022-08-26 10:36:40 +00:00
Harry Waye
8d38003aeb
fix(gunicorn): ensure gunicorn processes signals appropriately (#11417)
Prior to this the containing script was recieving the TERM signal e.g.
from Kubernetes eviction. This was as a result terminating the root
process without waiting on gunicorn.

We solve this by avoiding spawning a new process and rather have
gunicorn replace the current process.
2022-08-22 16:14:43 +00:00
Ellie Huxtable
912e258e12
chore: Do not set statsd prefix (#11391)
Do not set statsd prefix
2022-08-19 11:29:55 +00:00
Ellie Huxtable
7ef39f6402
chore: Enable statsd instrumenting for gunicorn (#11386)
* chore: Allow instrumentation of gunicorn with statsd (#11372)

* chore: Allow instrumentation of gunicorn with statsd

In order to ensure that gunicorn is performing optimally, it helps to
monitor it with statsd.

This change allows us to include the flags needed to send UDP packets to
a statsd instance.

Docs: https://docs.gunicorn.org/en/stable/instrumentation.html

* Update bin/docker-server

Co-authored-by: Harry Waye <harry@posthog.com>

* Update bin/docker-server

Co-authored-by: Harry Waye <harry@posthog.com>

Co-authored-by: Harry Waye <harry@posthog.com>

* Include the STATSD_PORT correctly

Co-authored-by: Harry Waye <harry@posthog.com>
2022-08-19 11:03:57 +01:00
Harry Waye
eda2cd5ce7
revert: chore: Allow instrumentation of gunicorn with statsd (#11385)
Revert "chore: Allow instrumentation of gunicorn with statsd (#11372)"

This reverts commit 84ea166e9b.
2022-08-19 11:39:44 +03:00
Ellie Huxtable
84ea166e9b
chore: Allow instrumentation of gunicorn with statsd (#11372)
* chore: Allow instrumentation of gunicorn with statsd

In order to ensure that gunicorn is performing optimally, it helps to
monitor it with statsd.

This change allows us to include the flags needed to send UDP packets to
a statsd instance.

Docs: https://docs.gunicorn.org/en/stable/instrumentation.html

* Update bin/docker-server

Co-authored-by: Harry Waye <harry@posthog.com>

* Update bin/docker-server

Co-authored-by: Harry Waye <harry@posthog.com>

Co-authored-by: Harry Waye <harry@posthog.com>
2022-08-19 08:40:37 +01:00
Harry Waye
a70b4b28c6
chore(web): add django-prometheus exposed on /_metrics (#11000)
* chore(web): add django-prometheus exposed on /_metrics

This exposes a number of metrics, see
97d5748664/documentation/exports.md
for details. It includes histogram of timings by viewname before and
after middleware.

I'm not particularly interested in these right now, but rather would
like to expose Kafka Producer metrics as per
https://github.com/PostHog/posthog/pull/10997

* Refactor to use gunicorn server hooks

* also add expose to dockerfile

* wip
2022-07-27 20:37:44 +01:00
Tiina Turban
0fb19f87d7
Check for all necessary migrations completed before worker, plugins start (#8504) 2022-02-17 17:56:24 +01:00
Dawid Janik
6da21e2428
Increase request line limit for Gunicorn. (#8184) 2022-01-31 12:14:04 +02:00
Yakko Majuri
909d6bf7c6
do not start worker, plugins, or events if migrations not complete (#7823) 2022-01-06 10:30:47 -03:00
Karl-Aksel Puulmann
67debacff2
Log web access in docker (#2230)
This is needed to close https://github.com/PostHog/deployment/issues/32
and debug https://github.com/PostHog/posthog/issues/2135
2020-11-05 12:07:39 +02:00
Marius Andra
afa3617316 add docker gunicorn workers 2020-04-02 12:06:57 +02:00
Marius Andra
93fab5498f run gunicorn in docker-server script 2020-04-02 00:55:37 +02:00