Monday, October 18, 2010

Solved: java.io.IOException: Content-Length header is not provided by the namenode

Seems like a non-trivial number of people have been having problems with communication between their primary and secondary name nodes under various releases of Hadoop 0.20. I found a solution today and, since getsatisfaction.com (stupidly, IMHO) makes you register to post comments, figured I'd post it here instead.

This problem first manifested itself when we migrated from Cloudera release CDH3b2 to CDH3b3. During the DFS image transfer process the secondary name node was issuing the following request:

http://<name node>:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-19:280794060:0:1287441076000:1287440760971

To which the name node responded with

WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.io.IOException: Content-Length header is not provided by the namenode when trying to fetch http://0.0.0.0:50090/getimage?getimage=1

Note that the name name was trying to contact 0.0.0.0, presumably based on the "machine=0.0.0.0" parameter posted by the secondary name node. After tracing through a bunch of source code it turns out that in release 0.20.2 the "machine=" is (eventually) derived from the value of dfs.secondary.http.address. Editing hdfs-site.xml to explicitly point dfs.secondary.http.address to the secondary name node rather than accepting the default value of "0.0.0.0:50090" resolved the issue for me.

2 Comments:

Blogger Christian Hargraves said...

I just wanted to verify that this worked for me too. I thought maybe the configuration variable was wrong as I wasn't able to find it in the hdfs-default.html file, but it is correct.

3:13 PM  
Anonymous Leandro said...

Thanks a lot! Worked like a charm to me!

3:03 PM  

Post a Comment

Links to this post:

Create a Link

<< Home

Blog Information Profile for gg00