mirrors/posthog
0
0
Fork 0
mirror of https://github.com/PostHog/posthog.git synced 2024-11-28 18:26:15 +01:00
Code Issues Releases Activity
4b1c9167d0
posthog/ee/kafka_client/topics.py

7 lines
308 B
Python
Raw Normal View History

wire up the length to the proto message (#2089) * wire up the length to the proto message * we are so deep into the proto weeds we are using proto private methods
2020-10-29 01:41:13 +01:00
KAFKA_EVENTS = "clickhouse_events_proto"
Publish events to Kafka for consumption (#1644) * Publish events to Kafka for consumption * Commit avro idl's for event schemas * convert client to use github.com/dpkp/kafka-python * events loaded into clickhouse from Kafka * remove cruft * Publish events to Kafka for consumption * convert client to use github.com/dpkp/kafka-python * remove cruft * include kafka migrations * bugfixes for migrations * use constants for consistency * wrap up local migrations * small fixes * tune ups
2020-09-16 05:04:38 +02:00
KAFKA_PERSON = "clickhouse_person"
KAFKA_PERSON_UNIQUE_ID = "clickhouse_person_unique_id"
Session recording on clickhouse, separate tables and retention cronjob (#2051) * Add scheduled task to wipe session recordings * Create a new table for session recording * Save snapshot events to different table * Use SessionRecordingEvent over Events everywhere We can remove a ton of cruft this way as well * Add missing signature * Extract util from models/event * Attempt to update ingest side of clickhouse session recording events Note that it's using main kafka topic - not sure if a good idea. * Get separate table in ch working for session recording events * WIP: query sessions * Make both session recording queries work * Make linter happy * Rebase migration * Make tests work * Apply a TTL to session recordings and other configuration: - toYYYYMMDD partitioning should be smoother with TTL setup - TTL achieves not needing to archive the data ourselves - index_granularity will enable smaller reads per session_id - ORDER BY clause is to make single session as well as time range query reasonable * Convert retention cronjob to new model * Add tests to process_event changes * Add test for ee_capture change * Fixup migration * Make clickhouse tests drop/create session recording tables * Make TTL not be there in tests Otherwise writes get eaten by it during tests when mocking time * Fix retention task Co-authored-by: Tim Glaser <tim@glsr.nl>
2020-10-28 21:22:16 +01:00
KAFKA_SESSION_RECORDING_EVENTS = "clickhouse_session_recording_events"
Enable PLUGIN_SERVER_INGESTION (#3107) * Enable PLUGIN_SERVER_INGESTION_HANDOFF = get_bool_from_env("PLUGIN_SERVER_INGESTION_HANDOFF * Don't set PLUGIN_SERVER_INGESTION_HANDOFF in worker * Add comments * Remove _HANDOFF from PLUGIN_SERVER_INGESTION * add stats counter for plugin server handoff, so we can verify events out and events in * add whitelisted posthog and kea organizations * disable ingestion this round --> first let's just check the plugin server can talk to kafka & clickhouse before sending real events to it * enable ingestion in docker-compose.ch.yml * eliminate bad merge * async action event matching when using postgres plugin server ingestion (#3182) * fix org * remove _HANDOFF from topic * add plugin_ to plugin server ingestion topic * update plugin server to 0.7.0 Co-authored-by: Marius Andra <marius.andra@gmail.com>
2021-02-04 16:17:24 +01:00
KAFKA_EVENTS_PLUGIN_INGESTION = "events_plugin_ingestion"
Plugin log entries (#3482) * Add Postgres model PluginLogEntry * Add equivalent PluginLogEntry to Kafka+ClickHouse * Add migration * Add PluginLogEntry.Type.LOG & make PluginLogEntry.message a TextField * Update 0130_pluginlogentry.py * Add PluginLogEntry.instance_id * Update migration * Update migration * Add plugin log entries API * Test plugin log entries DB fetching * Add PluginLogs component prototype * Fix API * Improve PluginLogs component * Remove almost unused plugin Feedback button * Update migration * Fixed typing * Fix org permission error test asserts * Fix plugin log entry tests * Fix CH plugin log entry timestamp string * Update CH test_plugin_log_entry.py * Fix plugin log entry tests across PG/CH * Satisfy mypy * Add search and limit to plugin log entry API * Send team_id in plugin config API * Rework plugin logs UI * Add plugin config team ID in tests * Add plugin config team ID in tests actually * Fix code quality * Make logs plugin config-based * Fix CH queries * Fix typing * Improve UX and fix things * Polish plugin logs logic * Update migration * Add Celery task to delete old plugin logs * Fix UX bug with loading more plugin logs * Fix missing import * Remove OrganizationMemberPermissions message change * Make mypy happy * Add PluginLogEntry.is_system * Optimize CH plugin_log_entires PARTITION/ORDER * Increment migration * Adjust plugin logs drawer display * Fix plugin_log_factory_ch * Fix plugin_log_factory_ch fix * Replace PluginLogEntry.is_system with source * Adjust PluginLogEntrySerializer * Update CH fetch_plugin_log_entries * Make kea-typegen happy
2021-05-06 09:54:32 +02:00
KAFKA_PLUGIN_LOG_ENTRIES = "plugin_log_entries"
Copy Permalink