HDFSbalancer怎么用-创新互联
小编给大家分享一下HDFS balancer怎么用，相信大部分人都还不怎么了解，因此分享这篇文章给大家参考一下，希望大家阅读完这篇文章后大有收获，下面让我们一起去了解一下吧！
HDFS balancer
在hadoop的hdfs集群中新添加节点后，在新节点上运行./start-balancer.sh。
运行的日志信息如下，重复判断、迁移，速度相当慢。
为了降低集群负荷、避免干扰其他用户，均衡器被设计为在后台运行。在不同节点之间复制数据的带宽也是受限的，默认值是1MB/s。
2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]]
3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]]
Need to move 1.71 TB to make the cluster balanced.
Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:50010
Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:50010
Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:50010
Will move 30 GB in this iteration
2015-10-13 09:21:29,585 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: namenodes = [hdfs://hmaster91/] 2015-10-13 09:21:29,598 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0] 2015-10-13 09:21:30,350 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-10-13 09:21:32,371 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:50010 2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:50010 2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:50010 2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:50010 2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:50010 2015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:50010 2015-10-13 09:21:32,374 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:50010 2015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]] 2015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]] 2015-10-13 09:21:32,380 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.71 TB to make the cluster balanced. 2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:50010 2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:50010 2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:50010 2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration 2015-10-13 09:31:41,945 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515941_8775312 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010 2015-10-13 09:31:47,378 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610315_8869686 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:50010 2015-10-13 09:31:59,425 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082639980_8899357 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.94:50010 2015-10-13 09:32:02,929 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515919_8775290 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010 2015-10-13 09:32:03,938 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082510026_8769397 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010 2015-10-13 09:32:05,370 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1078753887_5013250 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.92:50010 2015-10-13 09:32:06,871 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610299_8869670 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:50010 2015-10-13 09:32:10,923 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610274_8869645 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.94:50010 2015-10-13 09:32:12,930 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515963_8775334 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:50010 2015-10-13 09:32:13,424 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509872_8769243 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010 2015-10-13 09:32:13,429 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610182_8869553 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.151:50010 2015-10-13 09:32:17,423 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509962_8769333 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:50010 2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:50010 2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:50010 2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:50010 2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:50010 2015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:50010 2015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:50010 2015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:50010 2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.598206824722595], Source[192.168.70.92:50010, utilization=64.2691664228416]] 2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.151:50010, utilization=38.248797525595634], BalancerDatanode[192.168.60.154:50010, utilization=8.900929142083783], BalancerDatanode[192.168.60.153:50010, utilization=37.771715581234325]] 2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.68 TB to make the cluster balanced. 2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.151:50010 2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.154:50010 2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.153:50010 2015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration 2015-10-13 09:42:49,653 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424981_8684352 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:50010 2015-10-13 09:42:51,028 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082269380_8528751 with size=125931484 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:50010 2015-10-13 09:43:12,136 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558990_8818361 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:50010 2015-10-13 09:43:17,073 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082547829_8807200 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:50010 2015-10-13 09:43:17,135 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558979_8818350 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:50010 2015-10-13 09:43:18,541 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082544853_8804224 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:50010 2015-10-13 09:43:20,635 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685915_8945292 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.94:50010 2015-10-13 09:43:24,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685889_8945266 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.92:50010 2015-10-13 09:43:25,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082423441_8682812 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:50010 2015-10-13 09:43:27,540 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675683_8935060 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:50010 2015-10-13 09:43:27,563 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675703_8935080 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:50010 2015-10-13 09:43:28,065 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424970_8684341 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:50010
以上是“HDFS balancer怎么用”这篇文章的所有内容，感谢各位的阅读！相信大家都有了一定的了解，希望分享的内容对大家有所帮助，如果还想学习更多知识，欢迎关注创新互联行业资讯频道！
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
本文名称：HDFSbalancer怎么用-创新互联
URL标题：http://shouzuofang.com/article/dscejh.html