Hadoop常见错误及解决办法

2019年9月11日 13:15:12Linux大全评论991 views阅读模式

1：Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out

Answer：
程序里面需要打开多个文件，进行分析，系统一般默认数量是1024，（用ulimit -a可以看到）对于正常使用是够了，但是对于程序来讲，就太少了。
修改办法：
修改2个文件。
/etc/security/limits.conf
vi /etc/security/limits.conf
加上：
* soft nofile 102400
* hard nofile 409600

$cd /etc/pam.d/
$sudo vi login
添加 session required /lib/security/pam_limits.so

针对第一个问题我纠正下答案：
这是reduce预处理阶段shuffle时获取已完成的map的输出失败次数超过上限造成的，上限默认为5。引起此问题的方式可能会有很多种，比如网络连接不正常，连接超时，带宽较差以及端口阻塞等。。。通常框架内网络情况较好是不会出现此错误的。

2：Too many fetch-failures
Answer:
出现这个问题主要是节点间的连通不够全面。
1) 检查所有机子上的/etc/hosts文件
要求本机ip 对应服务器名
要求要包含所有的服务器ip + 服务器名
2) 检查 .ssh/authorized_keys
要求包含所有服务器（包括其自身）的public key

3：处理速度特别的慢出现map很快但是reduce很慢而且反复出现 reduce=0%
Answer:
结合第二点，然后
修改 conf/Hadoop-env.sh 中的export HADOOP_HEAPSIZE=4000

4：Hdfs出现：能够启动datanode，但无法访问，也无法结束的错误
在重新格式化一个新的分布式文件时，需要将你NameNode上所配置的dfs.name.dir这一namenode用来存放NameNode 持久存储名字空间及事务日志的本地文件系统路径删除，同时将各DataNode上的dfs.data.dir的路径 DataNode 存放块数据的本地文件系统路径的目录也删除。如本此配置就是在NameNode上删除/home/hadoop/NameData，在DataNode上删除/home/hadoop/DataNode1和/home/hadoop/DataNode2。这是因为Hadoop在格式化一个新的分布式文件系统时，每个存储的名字空间都对应了建立时间的那个版本（可以查看/home/hadoop /NameData/current目录下的VERSION文件，上面记录了版本信息），在重新格式化新的分布式系统文件时，最好先删除NameData 目录。必须删除各DataNode的dfs.data.dir。这样才可以使namedode和datanode记录的信息版本对应。
注意：删除是个很危险的动作，不能确认的情况下不能删除！！做好删除的文件等通通备份！！

5：java.io.IOException: Could not obtain block: blk_194219614024901469_1100 file=/user/hive/warehouse/src_20090724_log/src_20090724_log
出现这种情况大多是节点断了，没有连接上。

6：java.lang.OutOfMemoryError: Java heap space
出现这种异常，明显是jvm内存不够得原因，要修改所有的datanode的jvm内存大小。
Java -Xms1024m -Xmx4096m
一般jvm的最大内存使用应该为总内存大小的一半，我们使用的8G内存，所以设置为4096m，这一值可能依旧不是最优的值。

Hadoop添加节点的方法

自己实际添加节点过程：

1. 先在slave上配置好环境，包括ssh，jdk，相关config，lib，bin等的拷贝；

2. 将新的datanode的host加到集群namenode及其他datanode中去；

3. 将新的datanode的ip加到master的conf/slaves中；

4. 重启cluster,在cluster中看到新的datanode节点；

5. 运行bin/start-balancer.sh，这个会很耗时间

备注：

1. 如果不balance，那么cluster会把新的数据都存放在新的node上，这样会降低mr的工作效率；

2. 也可调用bin/start-balancer.sh 命令执行，也可加参数 -threshold 5

threshold 是平衡阈值，默认是10%，值越低各节点越平衡，但消耗时间也更长。

3. balancer也可以在有mr job的cluster上运行，默认dfs.balance.bandwidthPerSec很低，为1M/s。在没有mr job时，可以提高该设置加快负载均衡时间。

其他备注：

1. 必须确保slave的firewall已关闭;

2. 确保新的slave的ip已经添加到master及其他slaves的/etc/hosts中，反之也要将master及其他slave的ip添加到新的slave的/etc/hosts中

mapper及reducer个数

url地址： http://wiki.apache.org/hadoop/HowManyMapsAndReduces

HowManyMapsAndReduces

Partitioning your job into maps and reduces

Picking the appropriate size for the tasks for your job can radically change the performance of Hadoop. Increasing the number of tasks increases the framework overhead, but increases load balancing and lowers the cost of failures. At one extreme is the 1 map/1 reduce case where nothing is distributed. The other extreme is to have 1,000,000 maps/ 1,000,000 reduces where the framework runs out of resources for the overhead.

Number of Maps

The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.

Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the [WWW] InputFormat determines the number of maps.

The number of map tasks can also be increased manually using the JobConf's conf.setNumMapTasks(int num). This can be used to increase the number of map tasks, but will not set the number below that which Hadoop determines via splitting the input data.

Number of Reduces

The right number of reduces seems to be 0.95 or 1.75 * (nodes * mapred.tasktracker.tasks.maximum). At 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. At 1.75 the faster nodes will finish their first round of reduces and launch a second round of reduces doing a much better job of load balancing.

Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces << heapSize). This will be fixed at some point, but until it is it provides a pretty firm upper bound.

The number of reduces also controls the number of output files in the output directory, but usually that is not important because the next map/reduce step will split them into even smaller splits for the maps.

The number of reduce tasks can also be increased in the same way as the map tasks, via JobConf's conf.setNumReduceTasks(int num).

自己的理解：

mapper个数的设置：跟input file 有关系，也跟filesplits有关系，filesplits的上线为dfs.block.size，下线可以通过mapred.min.split.size设置，最后还是由InputFormat决定。

较好的建议：

The right number of reduces seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * mapred.tasktracker.reduce.tasks.maximum).increasing the number of reduces increases the framework overhead, but increases load balancing and lowers the cost of failures.

<property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>

<value>2</value>

<description>The maximum number of reduce tasks that will be run

simultaneously by a task tracker.

</description>

</property>

单个node新加硬盘

1.修改需要新加硬盘的node的dfs.data.dir，用逗号分隔新、旧文件目录

2.重启dfs