Artwork

A tartalmat a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® biztosítja. Az összes podcast-tartalmat, beleértve az epizódokat, grafikákat és podcast-leírásokat, közvetlenül a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® vagy a podcast platform partnere tölti fel és biztosítja. Ha úgy gondolja, hogy valaki az Ön engedélye nélkül használja fel a szerzői joggal védett művét, kövesse az itt leírt folyamatot https://hu.player.fm/legal.
Player FM - Podcast alkalmazás
Lépjen offline állapotba az Player FM alkalmazással!

If Streaming Is the Answer, Why Are We Still Doing Batch?

43:58
 
Megosztás
 

Manage episode 346518870 series 2355972
A tartalmat a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® biztosítja. Az összes podcast-tartalmat, beleértve az epizódokat, grafikákat és podcast-leírásokat, közvetlenül a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® vagy a podcast platform partnere tölti fel és biztosítja. Ha úgy gondolja, hogy valaki az Ön engedélye nélkül használja fel a szerzői joggal védett művét, kövesse az itt leírt folyamatot https://hu.player.fm/legal.

Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful upfront investment. Some are even running dbt in micro batches to simulate an effect similar to streaming, without having to make the full transition. Will streaming ever fully take over?
In this episode, Kris talks to a panel of industry experts with decades of experience building and implementing data systems. They discuss the state of streaming adoption today, if streaming will ever fully replace batch, and whether it even could (or should). Is micro batching the natural stepping stone between batch and streaming? Will there ever be a unified understanding on how data should be processed over time? Is the lack of agreement on best practices for data streaming an insurmountable obstacle to widespread adoption? What exactly is holding teams back from fully adopting a streaming model?
Recorded live at Current 2022: The Next Generation of Kafka Summit, the panel includes Adi Polak (Vice President of Developer Experience, Treeverse), Amy Chen (Partner Engineering Manager, dbt Labs), Eric Sammer (CEO, Decodable), and Tyler Akidau (Principal Software Engineer, Snowflake).
EPISODE LINKS

  continue reading

Fejezetek

1. Intro (00:00:00)

2. Is the Lambda Architecture here to stay? (00:02:58)

3. What is preventing streaming adoption today? (00:06:27)

4. Is streaming a semantic model? (00:10:00)

5. Should we push for stream processing? (00:20:53)

6. When should we use streaming vs. batch processing? (00:26:15)

7. What is the future of stream processing? (00:37:10)

8. It's a wrap! (00:41:48)

265 epizódok

Artwork
iconMegosztás
 
Manage episode 346518870 series 2355972
A tartalmat a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® biztosítja. Az összes podcast-tartalmat, beleértve az epizódokat, grafikákat és podcast-leírásokat, közvetlenül a Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® vagy a podcast platform partnere tölti fel és biztosítja. Ha úgy gondolja, hogy valaki az Ön engedélye nélkül használja fel a szerzői joggal védett művét, kövesse az itt leírt folyamatot https://hu.player.fm/legal.

Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful upfront investment. Some are even running dbt in micro batches to simulate an effect similar to streaming, without having to make the full transition. Will streaming ever fully take over?
In this episode, Kris talks to a panel of industry experts with decades of experience building and implementing data systems. They discuss the state of streaming adoption today, if streaming will ever fully replace batch, and whether it even could (or should). Is micro batching the natural stepping stone between batch and streaming? Will there ever be a unified understanding on how data should be processed over time? Is the lack of agreement on best practices for data streaming an insurmountable obstacle to widespread adoption? What exactly is holding teams back from fully adopting a streaming model?
Recorded live at Current 2022: The Next Generation of Kafka Summit, the panel includes Adi Polak (Vice President of Developer Experience, Treeverse), Amy Chen (Partner Engineering Manager, dbt Labs), Eric Sammer (CEO, Decodable), and Tyler Akidau (Principal Software Engineer, Snowflake).
EPISODE LINKS

  continue reading

Fejezetek

1. Intro (00:00:00)

2. Is the Lambda Architecture here to stay? (00:02:58)

3. What is preventing streaming adoption today? (00:06:27)

4. Is streaming a semantic model? (00:10:00)

5. Should we push for stream processing? (00:20:53)

6. When should we use streaming vs. batch processing? (00:26:15)

7. What is the future of stream processing? (00:37:10)

8. It's a wrap! (00:41:48)

265 epizódok

Minden epizód

×
 
Loading …

Üdvözlünk a Player FM-nél!

A Player FM lejátszó az internetet böngészi a kiváló minőségű podcastok után, hogy ön élvezhesse azokat. Ez a legjobb podcast-alkalmazás, Androidon, iPhone-on és a weben is működik. Jelentkezzen be az feliratkozások szinkronizálásához az eszközök között.

 

Gyors referencia kézikönyv