hbase master在一次网络问题中无法自动切换主节点，hbase client一直报错，无法连接服务

org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:989)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1966)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
        at java.lang.Thread.run(Thread.java:745)
 on 10-10-102-166,16020,1594127715229, tracking started null, retrying after=20029ms, replay=1ops

度娘后，都说可能是hdfs进入了safemode，随后对hdfs的safemode进行关闭。

hdfs dfsadmin -safemode leave

关闭后发现hbase client还是无法连接，hbase的master一直在自动重启切换。查看日志发现新的问题，master节点启动时，会加载/apps/hbase/data/MasterProcWALs里的日志文件，而日志文件数量太多，导致一直启动失败。

把/apps/hbase/data/MasterProcWALs里的log文件移动备份到其他地方后，hbase master终于可以成功启动。

转载请注明出处: http://www.julyme.com/20200922/111.html