metadataUpdater

最新推荐文章于 2024-08-23 21:00:57 发布

better_mouse

最新推荐文章于 2024-08-23 21:00:57 发布

阅读量390

点赞数

CC 4.0 BY-SA版权

分类专栏： kafka源码

本文链接：https://blog.youkuaiyun.com/better_mouse/article/details/81948145

kafka源码专栏收录该内容

4 篇文章

订阅专栏

本文探讨了Kafka中Metadata类的功能及元数据更新机制。Metadata类负责管理Kafka集群的元数据信息，并确保这些信息的及时更新。当Producer尝试发送消息时，会检查元数据并触发更新流程。默认情况下，每5分钟会自动更新一次元数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Metadata

Metadata 是封装了kafka元数据的类,是一个管理类

metadata的元数据第一次什么时候更新

每次调用producer.send方法,都会去metadata.cluster中去查找元数据,如果没有,就会标记需要更新,并且通过version等待更新.

public synchronized int requestUpdate() {
       //需要更新元数据
        this.needUpdate = true;
        return this.version;
    }

真正的更新是由sender线程去操作的,sender线程,依靠NetworkClient来完成网络的IO操作

        /**
         * Add a metadata request to the list of sends if we can make one
         */
        private void maybeUpdate(long now, Node node) {
            if (node == null) {
                log.debug("Give up sending metadata request since no node is available");
                // mark the timestamp for no node available to connect
                this.lastNoNodeAvailableMs = now;
                return;
            }
            String nodeConnectionId = node.idString();

            if (canSendRequest(nodeConnectionId)) {
                this.metadataFetchInProgress = true;
                MetadataRequest metadataRequest;
                if (metadata.needMetadataForAllTopics())
                    metadataRequest = MetadataRequest.allTopics();
                else
                    metadataRequest = new MetadataRequest(new ArrayList<>(metadata.topics()));
                ClientRequest clientRequest = request(now, nodeConnectionId, metadataRequest);
                log.debug("Sending metadata request {} to node {}", metadataRequest, node.id());
                doSend(clientRequest, now);
            } else if (connectionStates.canConnect(nodeConnectionId, now)) {
                // we don't have a connection to this node right now, make one
                log.debug("Initialize connection to node {} for sending metadata request", node.id());
                initiateConnect(node, now);
                // If initiateConnect failed immediately, this node will be put into blackout and we
                // should allow immediately retrying in case there is another candidate node. If it
                // is still connecting, the worst case is that we end up setting a longer timeout
                // on the next round and then wait for the response.
            } else { // connected, but can't send more OR connecting
                // In either case, we just need to wait for a network event to let us know the selected
                // connection might be usable again.
                this.lastNoNodeAvailableMs = now;
            }
        }

    }

这里可以看到请求元数据的时候,把topic的信息也发送过去了,如果在服务端配置了创建topic,就会自动的去创建topic了.

MetadataUpdater sender在循环发送数据的过程中,就会调用 mayBeupdate 来判断是否要更新元数据.

                                .define(METADATA_MAX_AGE_CONFIG,
                                        Type.LONG,
                                        5 * 60 * 1000,
                                        atLeast(0),
                                        Importance.LOW,
                                        CommonClientConfigs.METADATA_MAX_AGE_DOC)

默认情况下5分钟要更新一次元数据