Kafka 集群环境配置

2016-10-04|来源:

下载解压

官网地址:http://kafka.apache.org/

Kafka 0.9.0.1在下面的地址可以下载:

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz

tar -zxvf kafka_2.10-0.9.0.1.tgz



配置

[hadoop@master kafka_2.10-0.9.0.1]$ vim config/server.properties

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.

broker.id=1

############################# Socket Server Settings #############################

listeners=PLAINTEXT://192.168.56.101:9092

# The port the socket server listens on

#port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces

#host.name=master

# Hostname the broker will advertise to producers and consumers. If not set, it uses the

# value for "host.name" if configured.  Otherwise, it will use the value returned from

# java.net.InetAddress.getCanonicalHostName().

#advertised.host.name=<hostname routable by clients>

#这个是配置PRODUCER/CONSUMER连上来的时候使用的地址

advertised.host.name=master

# The port to publish to ZooKeeper for clients to use. If this is not set,

# it will publish the same port that the broker binds to.

#advertised.port=<port accessible by clients>

# The number of threads handling network requests

num.network.threads=3

# The number of threads doing disk I/O

num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server

socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server

socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)

socket.request.max.bytes=104857600

############################# Log Basics #############################

# A comma seperated list of directories under which to store log files

#kafka存放消息文件的路径

log.dirs=/opt/kafka/data/kafka-logs

# The default number of log partitions per topic. More partitions allow greater

# parallelism for consumption, but this will also result in more files across

# the brokers.

#topic的默认分区数

num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

# This value is recommended to be increased for installations with data dirs located in RAID array.

num.recovery.threads.per.data.dir=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync

# the OS cache lazily. The following configurations control the flush of data to disk.

# There are a few important trade-offs here:

#    1. Durability: Unflushed data may be lost if you are not using replication.

#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.

#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.

# The settings below allow one to configure the flush policy to flush data after a period of time or

# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk

#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush

#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can

# be set to delete segments after a period of time, or after a given size has accumulated.

# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens

# from the end of the log.

# The minimum age of a log file to be eligible for deletion

#kafka接收日志的存储目录(目前我们保存7天数据log.retention.hours=168)

log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining

# segments don't drop below log.retention.bytes.

#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.

log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according

# to the retention policies

log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).

# This is a comma separated host:port pairs, each corresponding to a zk

# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

# You can also append an optional chroot string to the urls to specify the

# root directory for all kafka znodes.

zookeeper.connect=192.168.56.101:2181

# Timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=6000




集群

把配置好的kafla打包到其他两台机器

scp -r kafka_2.10-0.9.0.1 slave1:/opt/kafka/

scp -r kafka_2.10-0.9.0.1 slave2:/opt/kafka/


更改server.properties配置文件中的broker.id和listeners



启动

分别在三台机器上启动zookeeper、kafka

启动kafka命令:

/opt/kafka/kafka_2.10-0.9.0.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.10-0.9.0.1/config/server.properties &


使用zkCli查看节点

[zk: localhost:2181(CONNECTED) 10] ls /brokers

[seqid, topics, ids]

[zk: localhost:2181(CONNECTED) 12] ls /brokers/ids

[3, 2, 1]



相关问答

更多

kafka脱离了zookeeper可以集群吗

不可以,kafka必须要依赖一个zookeeper集群才能运行。kafka系群里面各个broker都是通过zookeeper来同步topic列表以及其它broker列表的,一旦连不上zookeeper,kafka也就无法工作。

kafka集群是否启动成功?

你的问题在于如何让一个程序一直在后台运行,还是只是窗口运行, (1)如果要kafka进程一直常驻并且在后台不关闭,启动如下: cd /home/kafka/kafka_2.10-0.9.0.0/ nohup bin/kafka-server-start.sh config/server.properties & (2)如果只是想窗口运行,启动如下: cd /home/kafka/kafka_2.10-0.9.0.0/ bin/kafka-server-start.sh config/server. ...

storm+kafka运行时问题

你好湮魂,从多个角度全面讲解Storm实时数据处理技术和最佳实践,为快速掌握并灵活应用Storm提供实用指南   从实际问题出发,系统介绍Storm的基本应用、多语言特性、完整业务系统实现和产品交付的最佳实践方法;从产品持续交付角度,分析并实践集成、测试和交付的所有步骤   《大数据技术丛书:Storm实时数据处理》涵盖搭建基于Storm的开发环境和测试实时系统的许多实用方法与实战用例,以及如何应用交付最佳实践来将系统部署至云端。

kafka集群重启,怎样确

1、查询当前线程是否包含kafka 2、查询当前端口占用线程是否归属kafka 3、开启consumer线程查看运行 4、借助第三方监控工具查看集群运行情况

kafka哪个配置文件里有版本号

kafka没有像别的软件一样有,kafka -verison的命令,但是你可以查看kafka/libs 下的库文件,知道kafka的版本号: find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*'[root@kafka1 kafka]# find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*'kafka_2.9.2-0.8.1.1.jar.asc

专题教程

JAVA概述
第一部分:java入门基础
第二部分:java常用类
第三部分:jdbc系列教程
第四部分:java高级特征
Gson教程
快速了解 jdk8 新特征

相关文章

更多

最近更新

更多