shepherd's Blog

spark streaming batch size 본문

Data Engineering

spark streaming batch size

shepherd.dev 2023. 7. 21. 00:26

spark streaming kafka에서 batch size 조절 옵션

df = (
    stream_reader
    .option("maxOffsetsPerTrigger", "100000")
    .load()
)

https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html

'Data Engineering' 카테고리의 다른 글

Trino - HIVE_TABLE_LOCK_NOT_ACQUIRED  (0) 2023.12.25
'Data Engineering' Related Articles more