¿Qué es un sumidero en Flink?

What is a sink in Flink?

This connector provides a Sink that writes partitioned files to filesystems supported by the Flink FileSystem abstraction. The streaming file sink writes incoming data into buckets. Given that the incoming streams can be unbounded, data in each bucket are organized into part files of finite size.

Sink is the extension of the BaseStreamingSink contract for streaming sinks that can add batches to an output. Used exclusively when MicroBatchExecution stream execution engine (Micro-Batch Stream Processing) is requested to add a streaming batch to a sink (addBatch phase) while running an activated streaming query.

A Flink program consists of multiple tasks (transformations/operators, data sources, and sinks). A task is split into several parallel instances for execution and each parallel instance processes a subset of the task’s input data. The number of parallel instances of a task is called its parallelism.

A checkpoint in Flink is a global, asynchronous snapshot of application state that’s taken on a regular interval and sent to durable storage (usually, a distributed file system). In the event of a failure, Flink restarts an application using the most recently completed checkpoint as a starting point.

Apache Spark is an open-source, distributed processing system used for big data workloads. It utilizes in-memory caching, and optimized query execution for fast analytic queries against data of any size.

Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into sophisticated dataflow topologies.

How to scale your Flink Applications

1. Stop your application and trigger a savepoint.
2. Write your application state — store in the savepoint — in a distributed file system or object store.
3. Load the state from the distributed file system and reassign it to the scaled operators.

There is no out of the box PostgreSQL sink for Flink. This does not mean, however, that you have to start from scratch! The JDBCOutputFormatclass can be used to turn any database with a JDBC database driver into a sink. JDBCOutputFormatis/was part of the Flink Batch API, however it can also be used as a sink for the Data Stream API.

Reusability: efficient batch and stream processing under the same API would allow you to easily switch between both execution modes without rewriting any code. So, a job could be easily reused to process real-time and historical data.

There are already many impressive projects built on top of Flink; their users include Uber, Netflix, Alibaba, and more. Flink emerged from here in Berlin; one of the main driving forces behind its development is a local company called data Artisans .

The JDBCOutputFormat class can be used to turn any database with a JDBC database driver into a sink. JDBCOutputFormat is/was part of the Flink Batch API, however it can also be used as a sink for the Data Stream API. It seems to be the recommended approach, judging from a few discussions I found on the Flink user group.