Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] brokies ledger data cannot be released #23599

Open
2 of 3 tasks
wangrenyi opened this issue Nov 14, 2024 · 15 comments
Open
2 of 3 tasks

[Bug] brokies ledger data cannot be released #23599

wangrenyi opened this issue Nov 14, 2024 · 15 comments
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@wangrenyi
Copy link

Search before asking

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

openjdk 11.0.18 2023-01-17 LTS
apache-zookeeper-3.6.4
pulsar 2.10.6

Minimal reproduce step

broker conf:
broker.txt

bookkeeper conf:
bookkeeper.txt

zookeeper conf:

根据以上配置：在四台服务器上面部署服务，其中三台服务器上面都分别部署 broker，bookie，zookeeper服务，另外一台服务器上面仅部署了broker，bookie服务。

异常状态：
其中两台服务器存储满了，2T的磁盘仅剩余100GB可用。

What did you expect to see?

磁盘满之前 diskUsageThreshold=0.95 ，满了之后我把所有节点都修改为 diskUsageThreshold=0.9。

四个bookie节点，其中两个节点上面的存储一直无法释放，我重新设置了消息的 set-message-ttl，set-persistence，set-retention，都设置在一个非常小的范围，但是观察两天后资源还是无法释放，中间有进行过多次bookie，broker重启，也没有生效

What did you see instead?

bookkeeper shell listbookies -rw -h 执行结果如下：
2024-11-14T10:33:43,456+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - ReadWrite Bookies :
2024-11-14T10:33:43,679+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.118:3181, IP:172.2.10.118, Port:3181, Hostname:172.2.10.118
2024-11-14T10:33:43,681+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.121:3181, IP:172.2.10.121, Port:3181, Hostname:pulsar4

bookkeeper shell listbookies -ro -h 执行结果如下：
2024-11-14T10:35:51,841+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - Readonly Bookies :
2024-11-14T10:35:51,879+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.119:3181, IP:172.2.10.119, Port:3181, Hostname:172.2.10.119
2024-11-14T10:35:52,053+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:172.2.10.120:3181, IP:172.2.10.120, Port:3181, Hostname:172.2.10.120

bin/bookkeeper shell bookiesanity 在172.2.10.120节点执行如下：
2024-11-14T10:40:54,054+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xae3ccd27, L:/172.2.10.120:38590 - R:172.2.10.120/172.2.10.120:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2024-11-14T10:40:54,078+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9062267, 0) to bookie (0, 172.2.10.120:3181): Attempting to write on ReadOnly bookie
2024-11-14T10:40:54,109+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9062267,loopId=19ebc2d1) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-14T10:40:54,127+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9062267, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-14T10:40:54,127+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9062267 due to NotEnoughBookiesException: Not enough non-faulty bookies available
2024-11-14T10:40:54,129+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9062267 E0
2024-11-14T10:40:54,130+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1811) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:575) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-14T10:40:54,143+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9062267
2024-11-14T10:40:54,143+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9062267,loopId=6d3717ff) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:547) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:538) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
2024-11-14T10:40:54,144+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
2024-11-14T10:40:54,144+0800 [main-EventThread] ERROR org.apache.bookkeeper.meta.CleanupLedgerManager - Failed on operating ledger metadata: -25
2024-11-14T10:40:54,145+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9062267
2024-11-14T10:40:54,145+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.120:3181
2024-11-14T10:40:54,150+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xae3ccd27, L:/172.2.10.120:38590 ! R:172.2.10.120/172.2.10.120:3181]
2024-11-14T10:40:54,151+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.120:3181
2024-11-14T10:40:54,278+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x300000618a40092
2024-11-14T10:40:54,278+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x300000618a40092 closed

zookeeper异常日志如下：
2024-11-14 08:13:53,325 [myid:2] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started.
2024-11-14 08:13:53,325 [myid:2] - INFO [PurgeTask:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false
2024-11-14 08:13:53,326 [myid:2] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed.
2024-11-14 08:19:37,602 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x20000004643007a
2024-11-14 08:19:38,369 [myid:2] - INFO [NIOWorkerThread-2:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.119:60822, session = 0x20000004643007b
2024-11-14 08:19:40,478 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007d
2024-11-14 08:19:41,461 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007e
2024-11-14 08:19:41,569 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008b
2024-11-14 08:26:16,633 [myid:2] - INFO [NIOWorkerThread-11:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56702, session = 0x20000004643007c
2024-11-14 08:26:16,633 [myid:2] - INFO [NIOWorkerThread-10:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56706, session = 0x20000004643007d
2024-11-14 08:26:18,835 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x20000004643007e
2024-11-14 08:26:20,074 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a4007f
2024-11-14 08:26:20,207 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x20000004643007f
2024-11-14 08:34:54,679 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40080
2024-11-14 08:34:55,786 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40081
2024-11-14 08:34:55,863 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430080
2024-11-14 08:41:31,688 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x20000004643007e
2024-11-14 08:41:31,886 [myid:2] - INFO [NIOWorkerThread-7:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.120:56724, session = 0x20000004643007f
2024-11-14 08:41:34,077 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008c
2024-11-14 08:41:35,321 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008d
2024-11-14 08:41:35,430 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008e
2024-11-14 08:50:07,103 [myid:2] - INFO [NIOWorkerThread-14:NIOServerCnxn@330] - Unable to read additional data from client, it probably closed the socket: address = /172.2.10.119:60836, session = 0x200000046430080
2024-11-14 08:50:08,898 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a008f
2024-11-14 08:50:09,848 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430081
2024-11-14 08:50:09,976 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0090
2024-11-14 08:56:49,294 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000046430082
2024-11-14 08:56:50,504 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40082
2024-11-14 08:56:50,620 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0091
2024-11-14 09:05:20,614 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x200000046430081
2024-11-14 09:05:23,088 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40083
2024-11-14 09:05:24,087 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40084
2024-11-14 09:05:24,200 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40085
2024-11-14 09:12:02,063 [myid:2] - INFO [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global closeSession request for session 0x200000046430082
2024-11-14 09:12:04,318 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0092
2024-11-14 09:12:05,479 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x300000618a40086
2024-11-14 09:12:05,589 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x1004929423a0093

bookie服务异常如下：
journalctl -f | grep ERROR
Nov 14 10:44:48 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:44:48,842+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515342848, Used space fraction: 0.94621605 > threshold 0.9.
Nov 14 10:44:58 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:44:58,843+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515363328, Used space fraction: 0.94621605 > threshold 0.9.
Nov 14 10:45:08 DFPulsar-172-2-10-120 pulsar[7418]: 2024-11-14T10:45:08,842+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 107515314176, Used space fraction: 0.9462161 > threshold 0.9.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@wangrenyi wangrenyi added the type/bug The PR fixed a bug or issue reported a bug label Nov 14, 2024
@wangrenyi
Copy link
Author

zookeeper config:
Uploading zoo.txt…

@ChenJhua
Copy link

I have the same problem, have you solved it?

@wangrenyi
Copy link
Author

I have the same problem, have you solved it?

I tried many ways to solve it, but it didn't work. If you have a good way, please let me know, thank you

@dao-jun
Copy link
Member

dao-jun commented Nov 15, 2024

I don't see you set backlogQuota.

@dao-jun
Copy link
Member

dao-jun commented Nov 15, 2024

Maybe messages are in backlog that the ledgers can't be deleted.

@wangrenyi
Copy link
Author

Maybe messages are in backlog that the ledgers can't be deleted.

I do the following，make no difference
bin/pulsar-admin namespaces set-backlog-quota --limit 100G --policy producer_request_hold api-open-test/data-push

image

@wangrenyi
Copy link
Author

bin/bookkeeper shell bookiesanity ,the results are as follows:

2024-11-15T16:55:44,433+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
2024-11-15T16:55:44,620+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,620+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.72:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,623+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,623+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-15T16:55:44,718+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9117303
2024-11-15T16:55:44,719+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9117303
2024-11-15T16:55:44,857+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0xd15f532d, L:/172.2.10.119:38528 - R:172.2.10.119/172.2.10.119:3181]
2024-11-15T16:55:44,868+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xd15f532d, L:/172.2.10.119:38528 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2024-11-15T16:55:44,889+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9117303, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie
2024-11-15T16:55:44,902+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9117303,loopId=7a0bfa43) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9117303, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:334) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1947) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:134) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:123) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1966) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1915) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:378) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2168) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2225) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2204) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:1382) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,918+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9117303 due to NotEnoughBookiesException: Not enough non-faulty bookies available
2024-11-15T16:55:44,921+0800 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9117303 E0
2024-11-15T16:55:44,921+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:431) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1811) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:575) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.14.8.jar:4.14.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-11-15T16:55:44,930+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9117303
2024-11-15T16:55:44,930+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9117303,loopId=6a2ca3d5) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:547) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.14.8.jar:4.14.8]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:538) ~[org.apache.zookeeper-zookeeper-3.9.1.jar:3.9.1]
2024-11-15T16:55:44,932+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
2024-11-15T16:55:44,932+0800 [main-EventThread] ERROR org.apache.bookkeeper.meta.CleanupLedgerManager - Failed on operating ledger metadata: -25
2024-11-15T16:55:44,932+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9117303
2024-11-15T16:55:44,932+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181
2024-11-15T16:55:44,936+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181
2024-11-15T16:55:44,936+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xd15f532d, L:/172.2.10.119:38528 ! R:172.2.10.119/172.2.10.119:3181]
2024-11-15T16:55:45,060+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a01c5 closed
2024-11-15T16:55:45,060+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a01c5

@wangrenyi wangrenyi changed the title [Bug] bookie 存储无法释放 [Bug] brokies ledger data cannot be released Nov 15, 2024
@lhotari
Copy link
Member

lhotari commented Nov 15, 2024

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

@wangrenyi
Copy link
Author

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks，I will try to upgrade version to 3.x LTS

@visxu
Copy link
Contributor

visxu commented Nov 19, 2024

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks，I will try to upgrade version to 3.x LTS

@wangrenyi Does the problem still exist after upgrading to 3.x LTS ?

@wangrenyi
Copy link
Author

pulsar 2.10.6

@wangrenyi this is an unsupported version of Pulsar. Please upgrade to a supported version of Pulsar and check if the problem reproduces.

thanks，I will try to upgrade version to 3.x LTS

@wangrenyi Does the problem still exist after upgrading to 3.x LTS ?

yes，yesterday, I upgraded pulsar from version 2.10.6 to 3.0.7, but the above issues still exist

@wangrenyi
Copy link
Author

upgraded version:
apache-zookeeper-3.6.4
openjdk version "17.0.13" 2024-10-15
apache-pulsar-3.0.7

broker conf:
broker.txt

bookie conf:
bookkeeper.txt

zk conf:
zoo.txt

status of pulsar nodes in the cluster is as follows(16c 16g):
172.2.10.118,172.2.10.119,172.2.10.120
deploy: zk，broker，bookie

172.2.10.121
deploy: broker，bookie

172.2.10.119,172.2.10.120 the disk on the node cannot be released，other nodes are working properly
image
image

image

bin/bookkeeper shell bookiesanity
2024-11-20T09:28:32,759+0800 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
2024-11-20T09:28:32,768+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers.
2024-11-20T09:28:32,771+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.9.2-e454e8c7283100c7caec6dcae2bc82aaecb63023, built on 2024-02-12 20:59 UTC
2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar2
2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.13
2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium
2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jdk-17
2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/data/apache-pulsar-3.0.7/conf:::/data/apache-pulsar-3.0.7/lib/com.auth0-java-jwt-4.3.0.jar:/data/apache-pulsar-3.0.7/lib/com.auth0-jwks-rsa-0.22.0.jar:/data/apache-pulsar-3.0.7/lib/com.beust-jcommander-1.82.jar:/data/apache-pulsar-3.0.7/lib/com.carrotsearch-hppc-0.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.seancfoley-ipaddress-5.5.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.gson-gson-2.8.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-failureaccess-1.0.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-guava-32.1.1-jre.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-util-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.re2j-re2j-1.6.jar:/data/apache-pulsar-3.0.7/lib/commons-cli-commons-cli-1.5.0.jar:/data/apache-pulsar-3.0.7/lib/commons-codec-commons-codec-1.15.jar:/data/apache-pulsar-3.0.7/lib/commons-configuration-commons-configuration-1.10.jar:/data/apache-pulsar-3.0.7/lib/commons-io-commons-io-2.14.0.jar:/data/apache-pulsar-3.0.7/lib/commons-lang-commons-lang-2.6.jar:/data/apache-pulsar-3.0.7/lib/commons-logging-commons-logging-1.1.1.jar:/data/apache-pulsar-3.0.7/lib/com.spotify-completable-futures-0.3.6.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.sun.activation-javax.activation-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-memory-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/dev.failsafe-failsafe-3.3.2.jar:/data/apache-pulsar-3.0.7/lib/io.airlift-aircompressor-0.27.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-all-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-alts-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-api-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-auth-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-context-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-core-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-grpclb-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-rls-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-services-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-stub-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-util-1.60.0.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-xds-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.gsonfire-gson-fire-1.8.5.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-api-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-proto-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-buffer-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-haproxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http2-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-socks-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-proxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-classes-macos-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-windows-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-classes-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-classes-epoll-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-unix-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-api-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-proto-0.2.0.jar:/data/apache-pulsar-3.0.7/lib/io.perfmark-perfmark-api-0.26.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus.jmx-collector-0.16.1.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-annotations-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-core-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-models-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-auth-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-bridge-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-core-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/data/apache-pulsar-3.0.7/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/data/apache-pulsar-3.0.7/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/data/apache-pulsar-3.0.7/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.validation-validation-api-1.1.0.Final.jar:/data/apache-pulsar-3.0.7/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/data/apache-pulsar-3.0.7/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/data/apache-pulsar-3.0.7/lib/jline-jline-2.14.6.jar:/data/apache-pulsar-3.0.7/lib/net.bytebuddy-byte-buddy-1.14.12.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.jcip-jcip-annotations-1.0.jar:/data/apache-pulsar-3.0.7/lib/net.jodah-typetools-0.5.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-protobuf-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-circe-checksum-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-cpu-affinity-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-native-io-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-statelib-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-collections4-4.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-compress-1.26.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-lang3-3.11.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-text-1.10.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-client-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-framework-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-recipes-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6-tests.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-protocol-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-bouncy-castle-bc-3.0.7-pkg.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-jetcd-core-shaded-3.0.7-shaded.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-managed-ledger-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-config-validation-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-instance-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-proto-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-utils-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-worker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-io-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-metadata-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-proxy-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-testclient-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-websocket-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-testmocks-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.thrift-libthrift-0.14.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-jute-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.bitbucket.b_c-jose4j-0.9.4.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcpkix-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcprov-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcutil-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.checkerframework-checker-qual-3.33.0.jar:/data/apache-pulsar-3.0.7/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-continuation-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-http-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-io-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-proxy-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-security-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlets-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-ajax-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-api-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-common-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-client-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-common-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-server-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.inject-jersey-hk2-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-multipart-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/data/apache-pulsar-3.0.7/lib/org.javassist-javassist-3.25.0-GA.jar:/data/apache-pulsar-3.0.7/lib/org.jctools-jctools-core-2.1.2.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains-annotations-13.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jline-jline-3.21.0.jar:/data/apache-pulsar-3.0.7/lib/org.jvnet.mimepull-mimepull-1.9.15.jar:/data/apache-pulsar-3.0.7/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.roaringbitmap-RoaringBitmap-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-slf4j-api-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/data/apache-pulsar-3.0.7/lib/org.yaml-snakeyaml-2.0.jar:/data/apache-pulsar-3.0.7/lib/org.zeroturnaround-zt-zip-1.17.jar:
2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1160.25.1.el7.x86_64
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/data/apache-pulsar-3.0.7
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1420MB
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1536MB
2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=1536MB
2024-11-20T09:28:32,778+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5db99216
2024-11-20T09:28:32,783+0800 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-11-20T09:28:32,923+0800 [main] INFO org.apache.zookeeper.common.X509Util - Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello]
2024-11-20T09:28:32,930+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
2024-11-20T09:28:32,937+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181.
2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-11-20T09:28:33,004+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:53878, server: 172.2.10.118/172.2.10.118:2181
2024-11-20T09:28:33,011+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0501, negotiated timeout = 30000
2024-11-20T09:28:33,013+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2024-11-20T09:28:33,139+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.229:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=10.2.20.229, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-20T09:28:33,219+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-20T09:28:33,221+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-20T09:28:33,222+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]}
2024-11-20T09:28:33,282+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9247158
2024-11-20T09:28:33,284+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9247158
2024-11-20T09:28:33,371+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181]
2024-11-20T09:28:33,382+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
2024-11-20T09:28:33,401+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9247158, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie
2024-11-20T09:28:33,411+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=19a2b0ba) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9247158, change-id:0000000001)][attempt:1] Exception changing ensemble
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9247158 due to NotEnoughBookiesException: Not enough non-faulty bookies available
2024-11-20T09:28:33,420+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9247158 E0
2024-11-20T09:28:33,420+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1802) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.client.LedgerHandle.lambda$doAsyncCloseInternal$5(LedgerHandle.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9247158
2024-11-20T09:28:33,427+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=156d9542) Error writing metadata to store
org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:539) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9247158
2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181
2024-11-20T09:28:33,432+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181
2024-11-20T09:28:33,432+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x1cbd0b25, L:/172.2.10.119:34984 ! R:172.2.10.119/172.2.10.119:3181]
2024-11-20T09:28:33,433+0800 [main] WARN org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly
2024-11-20T09:28:33,551+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a0501 closed
2024-11-20T09:28:33,551+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a0501

@wangrenyi
Copy link
Author

wangrenyi commented Nov 22, 2024 •
edited
Loading

when bookeeper is started, the log is as follows:
there was a problem with this place during init
image
image

-- Logs begin at Wed 2024-11-20 13:16:45 CST. --
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "compactionMaxOutstandingRequests" : "100000",
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "journalPreAllocSizeMB" : "16",
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: "journalWriteData" : "true"
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: }
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,597+0800 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Binding bookie-rpc endpoint to 0.0.0.0/0.0.0.0:3181
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,656+0800 [main] INFO org.apache.bookkeeper.server.EmbeddedServer$Builder - Load lifecycle component : bookie-server
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,690+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers.
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,691+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@6b3f6585
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,691+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
Nov 22 17:08:01 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:01,692+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,734+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,735+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,736+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:39010, server: 172.2.10.118/172.2.10.118:2181
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,740+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0599, negotiated timeout = 30000
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,741+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,813+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7a0ef219
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,813+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,814+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,815+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar2/172.2.10.119:2181.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,816+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,817+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:50334, server: pulsar2/172.2.10.119:2181
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,820+0800 [main-SendThread(172.2.10.119:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar2/172.2.10.119:2181, session id = 0x2000000464305e3, negotiated timeout = 30000
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,821+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.119, using default rack for it : /default-rack.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ Bookie:172.2.10.119:0 @ /default-rack : org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,989+0800 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,994+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]}
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,995+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]}
Nov 22 17:08:03 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:03,995+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,005+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.118, using default rack for it : /default-rack.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,005+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/172.2.10.118:3181
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,006+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for 172.2.10.121, using default rack for it : /default-rack.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,006+0800 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/172.2.10.121:3181
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,011+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,042+0800 [main] INFO org.apache.bookkeeper.server.EmbeddedServer$Builder - Load lifecycle component : autorecovery
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,047+0800 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Starting component bookie-server.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,060+0800 [main] INFO org.eclipse.jetty.util.log - Logging initialized @5680ms to org.eclipse.jetty.util.log.Slf4jLog
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,122+0800 [main] INFO org.eclipse.jetty.server.Server - jetty-9.4.54.v20240208; built: 2024-02-08T19:42:39.027Z; git: cef3fbd6d736a21e7d541a5db490381d95a2047d; jvm 17.0.13+11
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,153+0800 [main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@6680f714{/,null,AVAILABLE}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,165+0800 [main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@7fc420b8{HTTP/1.1, (http/1.1)}{0.0.0.0:8000}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,166+0800 [main] INFO org.eclipse.jetty.server.Server - Started @5787ms
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,166+0800 [main] INFO org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider - Started Prometheus stats endpoint at 0.0.0.0/:8000
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,177+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Replaying journal 1680073544285 from position 516
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,179+0800 [main] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25d.txn
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Replaying journal 1680073544286 from position 0
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25e.txn
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,180+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Finished replaying journal in 5 ms.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,184+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.LedgerDirsManager - No writable ledger dirs below diskUsageThreshold. But Dirs that can accommodate 1073741824 are: [/data/bookkeeper/ledgers/current]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,190+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.Journal - garbage collected journal 1872c2fd258.txn
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,191+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.SyncThread - Flush ledger storage at checkpoint CheckpointList{checkpoints=[LogMark: logFileId - 1680073544286 , logFileOffset - 516]}.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,191+0800 [SyncThread-7-1] INFO org.apache.bookkeeper.bookie.LedgerDirsManager - No writable ledger dirs below diskUsageThreshold. But Dirs that can accommodate 1073741824 are: [/data/bookkeeper/ledgers/current]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,193+0800 [main] INFO org.apache.bookkeeper.bookie.Bookie - Finished reading journal, starting bookie
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,195+0800 [BookieStateManagerService-0] INFO org.apache.bookkeeper.bookie.BookieStateManager - Re-registering the bookie
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,195+0800 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.Journal - Starting journal on /data/bookkeeper/journal1/current
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,204+0800 [ForceWriteThread] INFO org.apache.bookkeeper.bookie.Journal - ForceWrite Thread started
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,211+0800 [BookieJournal-3181] INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal /data/bookkeeper/journal1/current/1872c2fd25f.txn
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,218+0800 [BookieStateManagerService-0] WARN org.apache.bookkeeper.discover.ZKRegistrationManager - No writable bookie registered node /ledgers/available/172.2.10.119:3181 when transitioning to readonly
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/available/172.2.10.119:3181
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:117) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.KeeperException.create(KeeperException.java:53) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2101(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:773) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:767) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.zookeeper.ZooKeeperClient.delete(ZooKeeperClient.java:767) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.discover.ZKRegistrationManager.doRegisterReadOnlyBookie(ZKRegistrationManager.java:305) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.discover.ZKRegistrationManager.registerBookie(ZKRegistrationManager.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:293) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager$2.call(BookieStateManager.java:244) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.bookie.BookieStateManager$2.call(BookieStateManager.java:239) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,225+0800 [main] INFO org.apache.bookkeeper.bookie.UncleanShutdownDetectionImpl - Created dirty file in ledger dir: /data/bookkeeper/ledgers/current
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,480+0800 [main] INFO org.apache.bookkeeper.common.component.ComponentStarter - Started component bookie-server.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,497+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,611+0800 [bookkeeper-io-15-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.121:3181 [id: 0xd9b2f0c6, L:/172.2.10.119:34396 - R:172.2.10.121/172.2.10.121:3181]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,611+0800 [bookkeeper-io-15-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.118:3181 [id: 0x815b954d, L:/172.2.10.119:40516 - R:172.2.10.118/172.2.10.118:3181]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,618+0800 [bookkeeper-io-15-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xd9b2f0c6, L:/172.2.10.119:34396 - R:172.2.10.121/172.2.10.121:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,618+0800 [bookkeeper-io-15-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x815b954d, L:/172.2.10.119:40516 - R:172.2.10.118/172.2.10.118:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,656+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181,Bookie:172.2.10.121:3181].
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181].
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,658+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,668+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906044 for 0 number of times, so deferring the ledger lock release by 9375 msecs
Nov 22 17:08:04 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:04,670+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,680+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181].
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,690+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,691+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,691+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,698+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906707 for 0 number of times, so deferring the ledger lock release by 9375 msecs
Nov 22 17:08:09 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:09,698+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,176+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,178+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906926592, Used space fraction: 0.9480211 > threshold 0.9.
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,708+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181,Bookie:172.2.10.121:3181].
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,746+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,777+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 9001906 for 0 number of times, so deferring the ledger lock release by 9375 msecs
Nov 22 17:08:14 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:14,778+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,796+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181].
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,797+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,801+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8961414 for 0 number of times, so deferring the ledger lock release by 9375 msecs
Nov 22 17:08:19 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:19,801+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,175+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,179+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906902016, Used space fraction: 0.9480212 > threshold 0.9.
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,824+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181].
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,825+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,831+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906044 for 1 number of times, so deferring the ledger lock release by 18750 msecs
Nov 22 17:08:24 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:24,832+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,838+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,847+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181].
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,848+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:796) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:716) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:605) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:205) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:226) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:488) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:1092) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1138) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: at java.lang.Thread.run(Thread.java:840) ~[?:?]
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,856+0800 [ReplicationWorker] ERROR org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker failed to replicate Ledger : 8906707 for 1 number of times, so deferring the ledger lock release by 18750 msecs
Nov 22 17:08:29 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:29,856+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - failed while replicating fragments
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,176+0800 [LedgerDirsMonitorThread] WARN org.apache.bookkeeper.bookie.LedgerDirsMonitor - LedgerDirsMonitor check process: All ledger directories are non writable
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,177+0800 [LedgerDirsMonitorThread] ERROR org.apache.bookkeeper.util.DiskChecker - Space left on device /data/bookkeeper/ledgers/current : 103906889728, Used space fraction: 0.9480212 > threshold 0.9.
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,870+0800 [main-EventThread] INFO org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager - Latch countdown due to ZK event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/ledgers/underreplication/locks zxid: -1
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie node from network location /default-rack, the bookies in the network location are [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], excluded bookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], current ensemble org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$EnsembleForReplacementWithNoConstraints@1cfb248, fallback to choose bookie randomly from the cluster.
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.121:3181, Bookie:172.2.10.118:3181].
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], fallback to choose bookie randomly from the cluster.
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,914+0800 [ReplicationWorker] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [Bookie:172.2.10.118:3181, Bookie:172.2.10.121:3181], allBookies [Bookie:172.2.10.118:3181,Bookie:172.2.10.121:3181].
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: 2024-11-22T17:08:34,915+0800 [ReplicationWorker] WARN org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment
Nov 22 17:08:34 DFPulsar-172-2-10-119 pulsar[32699]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available

@visxu
Copy link
Contributor

visxu commented Nov 24, 2024

upgraded version: apache-zookeeper-3.6.4 openjdk version "17.0.13" 2024-10-15 apache-pulsar-3.0.7

broker conf: broker.txt

bookie conf: bookkeeper.txt

zk conf: zoo.txt

status of pulsar nodes in the cluster is as follows(16c 16g): 172.2.10.118,172.2.10.119,172.2.10.120 deploy: zk，broker，bookie

172.2.10.121 deploy: broker，bookie

172.2.10.119,172.2.10.120 the disk on the node cannot be released，other nodes are working properly image image

image

bin/bookkeeper shell bookiesanity 2024-11-20T09:28:32,759+0800 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2024-11-20T09:28:32,768+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers. 2024-11-20T09:28:32,771+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.9.2-e454e8c7283100c7caec6dcae2bc82aaecb63023, built on 2024-02-12 20:59 UTC 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar2 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.13 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jdk-17 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/data/apache-pulsar-3.0.7/conf:::/data/apache-pulsar-3.0.7/lib/com.auth0-java-jwt-4.3.0.jar:/data/apache-pulsar-3.0.7/lib/com.auth0-jwks-rsa-0.22.0.jar:/data/apache-pulsar-3.0.7/lib/com.beust-jcommander-1.82.jar:/data/apache-pulsar-3.0.7/lib/com.carrotsearch-hppc-0.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.seancfoley-ipaddress-5.5.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.gson-gson-2.8.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-failureaccess-1.0.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-guava-32.1.1-jre.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-util-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.re2j-re2j-1.6.jar:/data/apache-pulsar-3.0.7/lib/commons-cli-commons-cli-1.5.0.jar:/data/apache-pulsar-3.0.7/lib/commons-codec-commons-codec-1.15.jar:/data/apache-pulsar-3.0.7/lib/commons-configuration-commons-configuration-1.10.jar:/data/apache-pulsar-3.0.7/lib/commons-io-commons-io-2.14.0.jar:/data/apache-pulsar-3.0.7/lib/commons-lang-commons-lang-2.6.jar:/data/apache-pulsar-3.0.7/lib/commons-logging-commons-logging-1.1.1.jar:/data/apache-pulsar-3.0.7/lib/com.spotify-completable-futures-0.3.6.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.sun.activation-javax.activation-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-memory-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/dev.failsafe-failsafe-3.3.2.jar:/data/apache-pulsar-3.0.7/lib/io.airlift-aircompressor-0.27.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-all-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-alts-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-api-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-auth-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-context-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-core-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-grpclb-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-rls-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-services-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-stub-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-util-1.60.0.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-xds-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.gsonfire-gson-fire-1.8.5.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-api-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-proto-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-buffer-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-haproxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http2-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-socks-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-proxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-classes-macos-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-windows-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-classes-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-classes-epoll-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-unix-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-api-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-proto-0.2.0.jar:/data/apache-pulsar-3.0.7/lib/io.perfmark-perfmark-api-0.26.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus.jmx-collector-0.16.1.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-annotations-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-core-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-models-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-auth-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-bridge-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-core-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/data/apache-pulsar-3.0.7/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/data/apache-pulsar-3.0.7/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/data/apache-pulsar-3.0.7/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.validation-validation-api-1.1.0.Final.jar:/data/apache-pulsar-3.0.7/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/data/apache-pulsar-3.0.7/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/data/apache-pulsar-3.0.7/lib/jline-jline-2.14.6.jar:/data/apache-pulsar-3.0.7/lib/net.bytebuddy-byte-buddy-1.14.12.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.jcip-jcip-annotations-1.0.jar:/data/apache-pulsar-3.0.7/lib/net.jodah-typetools-0.5.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-protobuf-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-circe-checksum-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-cpu-affinity-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-native-io-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-statelib-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-collections4-4.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-compress-1.26.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-lang3-3.11.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-text-1.10.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-client-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-framework-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-recipes-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6-tests.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-protocol-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-bouncy-castle-bc-3.0.7-pkg.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-jetcd-core-shaded-3.0.7-shaded.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-managed-ledger-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-config-validation-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-instance-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-proto-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-utils-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-worker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-io-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-metadata-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-proxy-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-testclient-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-websocket-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-testmocks-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.thrift-libthrift-0.14.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-jute-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.bitbucket.b_c-jose4j-0.9.4.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcpkix-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcprov-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcutil-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.checkerframework-checker-qual-3.33.0.jar:/data/apache-pulsar-3.0.7/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-continuation-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-http-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-io-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-proxy-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-security-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlets-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-ajax-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-api-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-common-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-client-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-common-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-server-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.inject-jersey-hk2-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-multipart-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/data/apache-pulsar-3.0.7/lib/org.javassist-javassist-3.25.0-GA.jar:/data/apache-pulsar-3.0.7/lib/org.jctools-jctools-core-2.1.2.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains-annotations-13.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jline-jline-3.21.0.jar:/data/apache-pulsar-3.0.7/lib/org.jvnet.mimepull-mimepull-1.9.15.jar:/data/apache-pulsar-3.0.7/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.roaringbitmap-RoaringBitmap-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-slf4j-api-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/data/apache-pulsar-3.0.7/lib/org.yaml-snakeyaml-2.0.jar:/data/apache-pulsar-3.0.7/lib/org.zeroturnaround-zt-zip-1.17.jar: 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1160.25.1.el7.x86_64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/data/apache-pulsar-3.0.7 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1420MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1536MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=1536MB 2024-11-20T09:28:32,778+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5db99216 2024-11-20T09:28:32,783+0800 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2024-11-20T09:28:32,923+0800 [main] INFO org.apache.zookeeper.common.X509Util - Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello] 2024-11-20T09:28:32,930+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes 2024-11-20T09:28:32,937+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181. 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2024-11-20T09:28:33,004+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:53878, server: 172.2.10.118/172.2.10.118:2181 2024-11-20T09:28:33,011+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0501, negotiated timeout = 30000 2024-11-20T09:28:33,013+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2024-11-20T09:28:33,139+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.229:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=10.2.20.229, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,219+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,221+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,222+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,282+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9247158 2024-11-20T09:28:33,284+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9247158 2024-11-20T09:28:33,371+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,382+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2024-11-20T09:28:33,401+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9247158, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie 2024-11-20T09:28:33,411+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=19a2b0ba) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9247158, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9247158 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2024-11-20T09:28:33,420+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9247158 E0 2024-11-20T09:28:33,420+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1802) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$doAsyncCloseInternal$5(LedgerHandle.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9247158 2024-11-20T09:28:33,427+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=156d9542) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:539) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9247158 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x1cbd0b25, L:/172.2.10.119:34984 ! R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,433+0800 [main] WARN org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly 2024-11-20T09:28:33,551+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a0501 closed 2024-11-20T09:28:33,551+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a0501

The disk space reaches threshold, seems no such good way to resolve it. You need expand the disk or delete some datas as you wish.

@wangrenyi
Copy link
Author

upgraded version: apache-zookeeper-3.6.4 openjdk version "17.0.13" 2024-10-15 apache-pulsar-3.0.7
broker conf: broker.txt
bookie conf: bookkeeper.txt
zk conf: zoo.txt
status of pulsar nodes in the cluster is as follows(16c 16g): 172.2.10.118,172.2.10.119,172.2.10.120 deploy: zk，broker，bookie
172.2.10.121 deploy: broker，bookie
172.2.10.119,172.2.10.120 the disk on the node cannot be released，other nodes are working properly image image
image
bin/bookkeeper shell bookiesanity 2024-11-20T09:28:32,759+0800 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 2024-11-20T09:28:32,768+0800 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://172.2.10.118:2181;172.2.10.119:2181;172.2.10.120:2181/ledgers : zkServers = 172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181, ledgersRootPath = /ledgers. 2024-11-20T09:28:32,771+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.9.2-e454e8c7283100c7caec6dcae2bc82aaecb63023, built on 2024-02-12 20:59 UTC 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar2 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.13 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Eclipse Adoptium 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jdk-17 2024-11-20T09:28:32,772+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/data/apache-pulsar-3.0.7/conf:::/data/apache-pulsar-3.0.7/lib/com.auth0-java-jwt-4.3.0.jar:/data/apache-pulsar-3.0.7/lib/com.auth0-jwks-rsa-0.22.0.jar:/data/apache-pulsar-3.0.7/lib/com.beust-jcommander-1.82.jar:/data/apache-pulsar-3.0.7/lib/com.carrotsearch-hppc-0.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/data/apache-pulsar-3.0.7/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/data/apache-pulsar-3.0.7/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.seancfoley-ipaddress-5.5.0.jar:/data/apache-pulsar-3.0.7/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/data/apache-pulsar-3.0.7/lib/com.google.code.gson-gson-2.8.9.jar:/data/apache-pulsar-3.0.7/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-failureaccess-1.0.1.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-guava-32.1.1-jre.jar:/data/apache-pulsar-3.0.7/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/data/apache-pulsar-3.0.7/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.protobuf-protobuf-java-util-3.25.5.jar:/data/apache-pulsar-3.0.7/lib/com.google.re2j-re2j-1.6.jar:/data/apache-pulsar-3.0.7/lib/commons-cli-commons-cli-1.5.0.jar:/data/apache-pulsar-3.0.7/lib/commons-codec-commons-codec-1.15.jar:/data/apache-pulsar-3.0.7/lib/commons-configuration-commons-configuration-1.10.jar:/data/apache-pulsar-3.0.7/lib/commons-io-commons-io-2.14.0.jar:/data/apache-pulsar-3.0.7/lib/commons-lang-commons-lang-2.6.jar:/data/apache-pulsar-3.0.7/lib/commons-logging-commons-logging-1.1.1.jar:/data/apache-pulsar-3.0.7/lib/com.spotify-completable-futures-0.3.6.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/data/apache-pulsar-3.0.7/lib/com.sun.activation-javax.activation-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-memory-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/data/apache-pulsar-3.0.7/lib/dev.failsafe-failsafe-3.3.2.jar:/data/apache-pulsar-3.0.7/lib/io.airlift-aircompressor-0.27.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-all-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-alts-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-api-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-auth-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-context-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-core-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-grpclb-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-rls-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-services-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-stub-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-util-1.60.0.jar:/data/apache-pulsar-3.0.7/lib/io.grpc-grpc-xds-1.55.3.jar:/data/apache-pulsar-3.0.7/lib/io.gsonfire-gson-fire-1.8.5.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-api-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.kubernetes-client-java-proto-18.0.0.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-buffer-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-haproxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http2-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-http-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-codec-socks-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-handler-proxy-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-classes-macos-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-resolver-dns-native-macos-4.1.113.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-osx-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-boringssl-static-2.0.66.Final-windows-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-tcnative-classes-2.0.66.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-classes-epoll-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-aarch_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-epoll-4.1.113.Final-linux-x86_64.jar:/data/apache-pulsar-3.0.7/lib/io.netty-netty-transport-native-unix-common-4.1.113.Final.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-api-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/data/apache-pulsar-3.0.7/lib/io.opencensus-opencensus-proto-0.2.0.jar:/data/apache-pulsar-3.0.7/lib/io.perfmark-perfmark-api-0.26.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus.jmx-collector-0.16.1.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/data/apache-pulsar-3.0.7/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-annotations-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-core-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.swagger-swagger-models-1.6.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-auth-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-bridge-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-core-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/io.vertx-vertx-web-common-4.5.10.jar:/data/apache-pulsar-3.0.7/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/data/apache-pulsar-3.0.7/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/data/apache-pulsar-3.0.7/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/data/apache-pulsar-3.0.7/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/data/apache-pulsar-3.0.7/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.validation-validation-api-1.1.0.Final.jar:/data/apache-pulsar-3.0.7/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/data/apache-pulsar-3.0.7/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/data/apache-pulsar-3.0.7/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/data/apache-pulsar-3.0.7/lib/jline-jline-2.14.6.jar:/data/apache-pulsar-3.0.7/lib/net.bytebuddy-byte-buddy-1.14.12.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/data/apache-pulsar-3.0.7/lib/net.jcip-jcip-annotations-1.0.jar:/data/apache-pulsar-3.0.7/lib/net.jodah-typetools-0.5.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.avro-avro-protobuf-1.11.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-circe-checksum-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-cpu-affinity-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-native-io-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-statelib-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-proto-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-server-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-collections4-4.4.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-compress-1.26.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-lang3-3.11.jar:/data/apache-pulsar-3.0.7/lib/org.apache.commons-commons-text-1.10.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-client-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-framework-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.curator-curator-recipes-5.1.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-common-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-core-4.16.6-tests.jar:/data/apache-pulsar-3.0.7/lib/org.apache.distributedlog-distributedlog-protocol-4.16.6.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/data/apache-pulsar-3.0.7/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-bouncy-castle-bc-3.0.7-pkg.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-jetcd-core-shaded-3.0.7-shaded.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-managed-ledger-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-broker-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-config-validation-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-api-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-instance-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-proto-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-utils-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-functions-worker-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-io-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-metadata-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-core-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-proxy-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-testclient-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-common-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-pulsar-websocket-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.pulsar-testmocks-3.0.7.jar:/data/apache-pulsar-3.0.7/lib/org.apache.thrift-libthrift-0.14.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-jute-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/data/apache-pulsar-3.0.7/lib/org.bitbucket.b_c-jose4j-0.9.4.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcpkix-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcprov-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.bouncycastle-bcutil-jdk18on-1.78.1.jar:/data/apache-pulsar-3.0.7/lib/org.checkerframework-checker-qual-3.33.0.jar:/data/apache-pulsar-3.0.7/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-alpn-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-continuation-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-http-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-io-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-proxy-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-security-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-servlets-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty-jetty-util-ajax-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-api-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-client-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-common-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-server-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.54.v20240208.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-client-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-common-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.core-jersey-server-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.inject-jersey-hk2-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.glassfish.jersey.media-jersey-media-multipart-2.42.jar:/data/apache-pulsar-3.0.7/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/data/apache-pulsar-3.0.7/lib/org.javassist-javassist-3.25.0-GA.jar:/data/apache-pulsar-3.0.7/lib/org.jctools-jctools-core-2.1.2.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains-annotations-13.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/data/apache-pulsar-3.0.7/lib/org.jline-jline-3.21.0.jar:/data/apache-pulsar-3.0.7/lib/org.jvnet.mimepull-mimepull-1.9.15.jar:/data/apache-pulsar-3.0.7/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/data/apache-pulsar-3.0.7/lib/org.roaringbitmap-RoaringBitmap-1.2.0.jar:/data/apache-pulsar-3.0.7/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.slf4j-slf4j-api-1.7.32.jar:/data/apache-pulsar-3.0.7/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/data/apache-pulsar-3.0.7/lib/org.yaml-snakeyaml-2.0.jar:/data/apache-pulsar-3.0.7/lib/org.zeroturnaround-zt-zip-1.17.jar: 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2024-11-20T09:28:32,773+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-1160.25.1.el7.x86_64 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/data/apache-pulsar-3.0.7 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1420MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1536MB 2024-11-20T09:28:32,774+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=1536MB 2024-11-20T09:28:32,778+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=172.2.10.118:2181,172.2.10.119:2181,172.2.10.120:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5db99216 2024-11-20T09:28:32,783+0800 [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2024-11-20T09:28:32,923+0800 [main] INFO org.apache.zookeeper.common.X509Util - Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello] 2024-11-20T09:28:32,930+0800 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes 2024-11-20T09:28:32,937+0800 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 172.2.10.118/172.2.10.118:2181. 2024-11-20T09:28:32,999+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2024-11-20T09:28:33,004+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.2.10.119:53878, server: 172.2.10.118/172.2.10.118:2181 2024-11-20T09:28:33,011+0800 [main-SendThread(172.2.10.118:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.2.10.118/172.2.10.118:2181, session id = 0x1004929423a0501, negotiated timeout = 30000 2024-11-20T09:28:33,013+0800 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 2024-11-20T09:28:33,139+0800 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 10.2.20.229:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=10.2.20.229, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,218+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.118:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.118, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,219+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) 172.2.10.121:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.121, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,221+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.119:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.119, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,222+0800 [main-EventThread] INFO org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (readonly bookie) 172.2.10.120:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=3181, host=172.2.10.120, protocol=bookie-rpc, auth=[], extensions=[]}]} 2024-11-20T09:28:33,282+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [172.2.10.119:3181] for ledger: 9247158 2024-11-20T09:28:33,284+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Create ledger 9247158 2024-11-20T09:28:33,371+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: 172.2.10.119:3181 [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,382+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0x1cbd0b25, L:/172.2.10.119:34984 - R:172.2.10.119/172.2.10.119:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 2024-11-20T09:28:33,401+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (9247158, 0) to bookie (0, 172.2.10.119:3181): Attempting to write on ReadOnly bookie 2024-11-20T09:28:33,411+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=19a2b0ba) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:9247158, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.replaceBookie(LocalBookieEnsemblePlacementPolicy.java:89) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:333) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:69) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$9(LedgerHandle.java:1940) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:133) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1959) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1907) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:353) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2195) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleResponse(PerChannelBookieClient.java:2252) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.handleV2Response(PerChannelBookieClient.java:2231) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.run(PerChannelBookieClient.java:1402) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,417+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 9247158 due to NotEnoughBookiesException: Not enough non-faulty bookies available 2024-11-20T09:28:33,420+0800 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L9247158 E0 2024-11-20T09:28:33,420+0800 [main] WARN org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Error in bookie sanity test org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:249) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:92) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:406) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1802) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.client.LedgerHandle.lambda$doAsyncCloseInternal$5(LedgerHandle.java:565) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[org.apache.bookkeeper-bookkeeper-common-4.16.6.jar:4.16.6] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=9247158 2024-11-20T09:28:33,427+0800 [main-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=9247158,loopId=156d9542) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:614) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:539) ~[org.apache.zookeeper-zookeeper-3.9.2.jar:3.9.2] 2024-11-20T09:28:33,427+0800 [main-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Deleted ledger 9247158 2024-11-20T09:28:33,428+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 172.2.10.119:3181 2024-11-20T09:28:33,432+0800 [bookkeeper-io-3-1] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0x1cbd0b25, L:/172.2.10.119:34984 ! R:172.2.10.119/172.2.10.119:3181] 2024-11-20T09:28:33,433+0800 [main] WARN org.apache.bookkeeper.client.BookKeeper - The mainWorkerPool did not shutdown cleanly 2024-11-20T09:28:33,551+0800 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1004929423a0501 closed 2024-11-20T09:28:33,551+0800 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1004929423a0501

The disk space reaches threshold, seems no such good way to resolve it. You need expand the disk or delete some datas as you wish.

Thanks for your reply，I will try to expand disk from 2T to 4T

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lhotari @visxu @dao-jun @wangrenyi @ChenJhua