> For the complete documentation index, see [llms.txt](https://ubiradio.gitbook.io/dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ubiradio.gitbook.io/dev/spark/structured-streaming.md).

# Structured Streaming

Structured Streaming 은 데이터 스트림을 테이블로 관리한다. 매 미니매치 마다 새로운 데이터 스트림 유입되면 테이블에 추가된다.

### Basic Concepts <a href="#basic-concepts" id="basic-concepts"></a>

![출처: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html](https://2257590298-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkqRosNMc2ax_2cv-w6%2Fuploads%2FA7RT7Gtto4tqNUAYIyGc%2Fimage.png?alt=media\&token=68c162d1-7769-4876-ba7f-280b50fa712b)

![출처: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html](https://2257590298-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkqRosNMc2ax_2cv-w6%2Fuploads%2F8hQcYlaVtaNvorG98PY2%2Fimage.png?alt=media\&token=fdcff6a7-271c-4481-9439-24d6e3bd912f)

### **Types of time windows**

![출처: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html](https://2257590298-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkqRosNMc2ax_2cv-w6%2Fuploads%2FVJq2e5RneG6PtP5cBsQR%2Fimage.png?alt=media\&token=8b64d94a-d8ee-4397-9dbf-a189e26a60a8)

### **Handling Late Data and Watermarking**

![출처: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html](https://2257590298-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkqRosNMc2ax_2cv-w6%2Fuploads%2FQxlU4f2SzbnlzbhLKJH2%2Fimage.png?alt=media\&token=7f64b69f-83a9-4698-af3b-b164eae872c6)

```scala
import spark.implicits._

val words = ... // streaming DataFrame of schema { timestamp: Timestamp, word: String }

// Group the data by window and word and compute the count of each group
val windowedCounts = words
    .withWatermark("timestamp", "10 minutes")
    .groupBy(
        window($"timestamp", "10 minutes", "5 minutes"),
        $"word")
    .count()
```

###

### 참고자료

Spark Documentation, [https://docs.microsoft.com/ko-kr/azure/databricks/getting-started/spark/streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html)