apache kafka

apache kafka

What is Kafka? How to use Apache Kafka? How to start with this real-time stream processing tool? Is Kafka the “magic tool” for real-time processings?

什么是卡夫卡？ 如何使用Apache Kafka？ 如何开始使用此实时流处理工具？ Kafka是实时处理的“魔术工具”吗？

Kafka is an open-source and real-time stream-processing software platform written in Scala and Java. Be careful to get JDK 8 on your environment. Otherwise, download it. This technology works with APIs to stream data between a producer and a consumer. The basic architecture with this tool is the following:

Kafka是一个用Scala和Java编写的开源实时流处理软件平台。 注意在您的环境中安装JDK 8。 否则，请下载它。 该技术与API配合使用，可以在生产者和消费者之间流传输数据。 该工具的基本架构如下：

A Kafka cluster is basically built with some brokers and Zookeeper servers. Each broker in a Kafka cluster is a single server that can receive messages from producers, assigns offsets to them, and commits the messages to storage on disk. It also services consumers, responding to fetch requests for partitions and responding with the messages that have been committed to disk.

Kafka集群基本上是由一些代理和Zookeeper服务器构建的。 Kafka群集中的每个代理都是一台服务器，可以接收来自生产者的消息，为它们分配偏移量，并将消息提交到磁盘上的存储中。 它还为使用者提供服务，响应对分区的提取请求并响应已提交到磁盘的消息。

Secondly, Apache Kafka uses Zookeeper to store metadata about the Kafka cluster, as well as consumer client details.

其次，Apache Kafka使用Zookeeper来存储有关Kafka集群的元数据以及消费者客户端详细信息。

These details about the Kafka cluster are summarized in the diagram below:

下图总结了有关Kafka群集的这些详细信息：

Within a cluster of brokers, one broker will also function as the cluster controller (elected automatically from the live members of the cluster). The controller is responsible for administrative operations, including assigning partitions to brokers and monitoring for broker failures. A partition is owned by a single broker in the cluster, and that broker is called the leader of the partition. A partition may be assigned to multiple brokers, which will result in the partition being replicated to provide a reduncy of messages within the partition.

在代理群集中，一个代理还将充当群集控制器(从群集的实时成员中自动选出)。 控制器负责管理操作，包括将分区分配给代理并监视代理故障。 分区由集群中的单个代理拥有，该代理称为分区的领导者。 可以将一个分区分配给多个代理，这将导致该分区被复制以提供该分区内消息的冗余。

The following figure shows a replication of partitions in a cluster:

下图显示了集群中分区的复制：

To stream data with Kafka, we need to add to this cluster bloc a streams bloc. Kafka Stream processes event in real time (i.e. no micro-batching like Spark Streaming) and allows you handle the late arrival of data seamlessly by using different logical components (Source: Confluent):

要使用Kafka传输数据，我们需要向该集群块添加一个流块。 Kafka Stream实时处理事件(即没有像Spark Streaming这样的微批量处理)，并且允许您使用不同的逻辑组件无缝处理数据的后期到达(来源：Confluent)：

1. Stream: Unbounded, continuously updated dataset, composed of stream partition

1. 流 ：无限制的，不断更新的数据集，由流分区组成

2. Stream partition: a part of the stream of data.

2. 流分区 ：数据流的一部分。

3. Processor topology: a graph of stream and stream processor which describes how the streams are to be processed. For those familiar with Spark, it resembles a logical plan.

3. 处理器拓扑 ：流和流处理器的图形，描述了如何处理流。 对于熟悉Spark的人来说，它类似于一个合理的计划。

4. Stream processor: a step to be performed on a stream (map, filter, etc…). There is two special stream processor in a topology, the source processor and the sink processor (read/write data). Two APIs are available, the Kafka Stream DSL or the low-level Processor API.

4. 流处理器 ：要在流上执行的步骤(地图，过滤器等)。 拓扑中有两个特殊的流处理器，即源处理器和宿处理器(读/写数据)。 有两种API，Kafka Stream DSL或低级处理器API。

5. Stream task: an instance of the processor topology attached to a partition of the input topic and stream processor’s steps on that subset only.

5. 流任务 ：附加到输入主题分区的处理器拓扑实例，仅在该子集上流处理器的步骤。

6. Stream thread: can run one or more stream task, is used to scale out the application, if the number of stream threads is greater than the number of partition, some instances will stay idle. If a streaming task fails, an idle instance will take its place.

6. 流线程 ：可以运行一个或多个流任务，用于扩展应用程序，如果流线程数大于分区数，则某些实例将保持空闲状态。 如果流任务失败，则将使用空闲实例。

7. Record: A key-value pair, streams are composed of records

7. 记录 ：键值对，流由记录组成

8. Tables: used to maintain a state inside a Kafka Stream application

8. 表 ：用于维护Kafka Stream应用程序内部的状态

The following figure describes this Streams architecture:

下图描述了此Streams体系结构：

After understanding the context of this tool, the following steps describe the installation, setting up and using of a Kafka environment:

了解此工具的上下文后，以下步骤描述了Kafka环境的安装，设置和使用：

Get Kafka to download the lastest version from this page

让Kafka从此页面下载最新版本

Extract it with the following commands:

使用以下命令将其解压缩： $ tar -xzf kafka_2.13-2.6.0.tgz$ cd kafka_2.13-2.6.0

3. Start the ZooKeeper service with the command:

3.使用以下命令启动ZooKeeper服务：

$ bin/zookeeper-server-start.sh config/zookeeper.properties

4. Open another terminal session to start the Kafka broker service and to run:

4.打开另一个终端会话以启动Kafka代理服务并运行：

$ bin/kafka-server-start.sh config/server.properties

5. Create a topic to store the events by running:

5.通过运行以下命令创建一个主题来存储事件：

$ bin/kafka-topics.sh --create --topic name --bootstrap-server localhost:9092

It is possible to create replication by adding to the previous command:

可以通过添加到上一个命令来创建复制：

--replication-factor 3 --partitions 1 --topic name

6. You can create a communication between the producer console and the consumer console. Let’s write an event on the producer console:

6.您可以在生产者控制台和使用者控制台之间创建通信。 让我们在生产者控制台上编写一个事件：

$ bin/kafka-console-producer.sh --topic name --bootstrap-server localhost:9092First event

And on another terminal let’s read this event on the consumer console by running:

在另一个终端上，让我们通过运行以下命令在使用者控制台上读取此事件：

$ bin/kafka-console-consumer.sh --topic name --from-beginning --bootstrap-server localhost:9092First event

7. It is also possible to export and/or to import data with Kafka connect. This feature allows us to continuously ingest data from external systems into and out of Kafka.

7.也可以使用Kafka connect导出和/或导入数据。 此功能使我们能够不断地将来自外部系统的数据导入和导出Kafka。

8. Once your data is stored in Kafka as events, you can process the data with the Kafka Streams client library for Java/Scala. It allows you to implement mission-critical real-time applications and microservices, where the input and/or output data is stored in Kafka topics.

8.将数据作为事件存储在Kafka中之后，就可以使用Java / Scala的Kafka Streams客户端库处理数据了。 它允许您实现关键任务实时应用程序和微服务，其中输入和/或输出数据存储在Kafka主题中。

For instance, we can write the popular WordCount algorithm:

例如，我们可以编写流行的WordCount算法：

KStream<String, String> textLines = builder.stream("name");KTable<String, Long> wordCounts = textLines .flatMapValues(line -> Arrays.asList(line.toLowerCase().split(" "))) .groupBy((keyIgnored, word) -> word) .count();wordCounts.toStream().to("output-topic"), Produced.with(Serdes.String(), Serdes.Long()));

9. Let’s stop the consoles, the Kafka broker and the Zookeeper servers with Ctrl-C and we can remove the Kafka environment by running:

9.让我们使用Ctrl-C停止控制台，Kafka代理和Zookeeper服务器，我们可以通过运行以下命令删除Kafka环境：

$ rm -rf /tmp/kafka-logs /tmp/zookeeper

There are many advantages with Kafka, especially if you work with a data lake:

Kafka有许多优点，尤其是在使用数据湖的情况下：

It is enable to handle high-velocity, high-variety and high-volume data (i.e. thousands of messages per second and a variety of consumers written in a variety of languages) and with the very low latency of the range of milliseconds.

它能够处理高速，大量和大容量的数据(即每秒数千条消息，以及使用各种语言编写的各种使用方)，并且具有极低的毫秒级延迟。 It is resistant to node/machine failure within a cluster.

它可以抵抗群集中的节点/机器故障。 The distributed architecture of Kafka makes it scalable using features such as replication and partitioning.

Kafka的分布式体系结构使其可以使用复制和分区等功能进行扩展。 It could also be employed for batch-like use cases and can also do the work of a traditional ETL, due to its capability of persists messages.

它也可以用于类似批处理的用例，并且由于它具有持久消息的功能，因此也可以完成传统ETL的工作。 It can handle real-time data pipeline.

它可以处理实时数据管道。

However, some disadvantages can be associated with Kafka:

但是，Kafka可能有一些缺点：

It lacks a full set of management and monitoring tools.

它缺少全套的管理和监视工具。 Kafka’s performance reduces if the message needs some tweaking and it changes and if the brokers and consumers start compressing the messages as the size increases; and, when the number of queues in a Kafka cluster increases.

如果消息需要一些调整并且发生变化，并且代理和使用者随着大小的增加开始压缩消息，则Kafka的性能会降低。 当Kafka集群中的队列数量增加时。 For certain use cases, it is missing some messaging paradigms (i.e. request and reply, point-to-point queues).

对于某些用例，它缺少某些消息传递范例(即，请求和答复，点对点队列)。

翻译自: https://medium.com/@thibautdg/a-brief-introduction-of-apache-kafka-3f48cced3aee

apache kafka