Kafka 内存陡然降低问题

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

已注册用户请登录

这是一个创建于 2468 天前的主题，其中的信息可能已经有所发展或是发生改变。

最近一直出现 Kafka 集群中所有节点内存瞬间降低的问题。
然后当时的日志里显示：
[2019-01-09 01:08:05,297] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,296] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,447] ERROR Unreasnable buffer length: 100000 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,547] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,564] WARN Connection broken for id 8444275087900673, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.IOException: Received packet with invalid packet: 0
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
[2019-01-09 01:08:05,565] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,595] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1094)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:74)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:929)
[2019-01-09 01:08:05,597] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,597] INFO Received connection request (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,646] WARN Connection broken for id 3377699988963328, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.IOException: Received packet with invalid packet: 0
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
[2019-01-09 01:08:05,670] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,670] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)

请问 ERROR Unreasonable buffer length: 100000 这个是什么原因造成的，应该如何解决？

Kafka

内存

日志

降低

11 条回复 2019-01-09 21:38:09 +08:00

leonme

2019-01-09 10:17:24 +08:00

问大家一个问题：

leonme

2019-01-09 10:17:50 +08:00

@leonme 忽略

petelin

2019-01-09 11:27:44 +08:00

我搜了一下网上的资料, 有一个贴了相关的源码, 看起来是要发送端发送的数据太大了(你这个例子是 10 万字节), 超过了缓冲区, 然后就 close socket 了. (一条消息不会这么大吧, 感觉也有可能是协议解析有问题, 读到是 10 万字节,其实没这么多)
试试发送小一点的数据?

期待下文.

qwefdrt

2019-01-09 11:41:10 +08:00

@petelin 我也是这样认为的，不知道为什么会有这么大的数据。请问有什么方法限制吗？

petelin

2019-01-09 11:45:50 +08:00 via iPhone

@qwefdrt 客户端业务逻辑限制一下？另外内存陡然下降是因为释放了 socket 吗？报错之后还能不能发消息接收消息

qwefdrt

2019-01-09 12:25:48 +08:00 via iPhone

@petelin 目前还能收发消息，我一遇到这个问题就通过重启来解决哈哈

hackerwin7

2019-01-09 13:17:20 +08:00 via iPhone

kafka 版本？

qwefdrt

2019-01-09 13:44:43 +08:00

@hackerwin7 kafka_2.11-2.0.0

hackerwin7

2019-01-09 14:18:12 +08:00 via iPhone

Kafka 日志？写 zookeeper 写太大了

qwefdrt

2019-01-09 18:04:14 +08:00

@hackerwin7 我贴的是 Kafka 日志。Zookeeper 写太大了具体指的是？

hackerwin7

2019-01-09 21:38:09 +08:00 via iPhone

@qwefdrt 这是 zk 的日志，你是不是起的内嵌的 zk ？然后我没有看到 broker 相关的日志所以才叫你贴 Kafka 的日志，然后这个 error unreasonable buffer length 其实是 zk 抛出来的，zk 在 parse 的时候一旦 datainpustream 的 length 小于 0 或者超过 Max 就会报这个