Skip to content
Streaming in Prefect: contributor HQ
Streaming in Prefect HQ
Notes
Design
Project
More
Share
Explore
Contrib meetings

Contrib meeting 7/15/2020

Perspective on how people out there are marrying Python and Kafka together ideas (AC)
Python client libraries but you have to do it yourself
All in on Faust which does everything for you
Fuast emphasizes properly sized/structured messages like protobufs but Prefect is into pickling a n y t h i n g
Ruby environment experience (PC)
handled mostly with the client libraries
if you want to control them in more detail probably need to go to the original Java implementation
Interface to look into the events
Confluent has one that we could base it off of (PC) (
https://www.confluent.io/product/confluent-platform
⁠
)
Where does the cursor/polling/event transferance etc actually happen? (PC)
in the FlowRunner (idea 1)
in something akin to the scheduler that will submit a new flow run (PIN 14 idea)
Is it the agent?
is it in the streaming services itself (like do we have an integration Kafka consumer published that knows how to call create_flow_run and then just leave everything meaningful on the streaming infra side?)
The UI situation
is there a way we can make multiple flows that are associated with one stream look different in the UI?
As above, is there a research project with Confluent or other systems to see how this is designed elsewhere and then we can back into how hard copying it/deriving it would be in our UI?
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.