Data Engineering
spark streaming batch size
shepherd.dev
2023. 7. 21. 00:26
spark streaming kafka에서 batch size 조절 옵션
df = (
stream_reader
.option("maxOffsetsPerTrigger", "100000")
.load()
)https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html