Solved: java.io.IOException: Content-Length header is not provided by the namenode

Seems like a non-trivial number of people have been having problems with communication between their primary and secondary name nodes under various releases of Hadoop 0.20. I found a solution today and, since getsatisfaction.com (stupidly, IMHO) makes you register to post comments, figured I'd post it here instead.

This problem first manifested itself when we migrated from Cloudera release CDH3b2 to CDH3b3. During the DFS image transfer process the secondary name node was issuing the following request:

http://<name node>:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-19:280794060:0:1287441076000:1287440760971

To which the name node responded with

WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.io.IOException: Content-Length header is not provided by the namenode when trying to fetch http://0.0.0.0:50090/getimage?getimage=1

Note that the name name was trying to contact 0.0.0.0, presumably based on the "machine=0.0.0.0" parameter posted by the secondary name node. After tracing through a bunch of source code it turns out that in release 0.20.2 the "machine=" is (eventually) derived from the value of dfs.secondary.http.address. Editing hdfs-site.xml to explicitly point dfs.secondary.http.address to the secondary name node rather than accepting the default value of "0.0.0.0:50090" resolved the issue for me.