¿Cómo comprimo texto en Kafka?

How do I compress text in Kafka?

Message Compression Format To compress the data, a ‘compression. type’ is used. This lets users decide the type of compression. The type can be ‘gzip’, ‘snappy’, ‘lz4’, or ‘none'(default).

This feature introduces the end-to-end block compression feature in Kafka. If enabled, data will be compressed by the producer, written in compressed format on the server and decompressed by the consumer. Compression will improve the consumer throughput for some decompression cost.

Other good option could be a usage of compression algorithm by the Kafka Producer both on the record level as well as batch level. Using a larger batch.size makes compression more efficient. The producer config has a property called – compression.type. By default it is set to none and hence producer doesn’t compress the data.

The producer config property linger.ms defaults to 0. You can set this so that the Producer will wait this long before sending if batch size not exceeded. This setting allows the Producer to group together any records that arrive before they can be sent into a batch.

Connector configuration for the default topic creation group topic.creation.default.replication.factor defines the replication factor for topics created by the default group. replication.factor is mandatory for the default group but optional for custom groups. Custom groups will fall back to the default group’s value if not set.

The Kafka client API for Producers are thread safe. A Kafka Producer has a pool of buffer that holds to-be-sent records. The producer has background, I/O threads for turning records into request bytes and transmitting requests to Kafka cluster. The producer must be closed to not leak resources, i.e., connections, thread pools, buffers.