Reading/WRITING UTF-8 enabled file
Sometimes, we could have encountered issues in which Spark returns non-ASCII characters in the wrong format. Especially in CDH-flavored Hadoop versions(say CDH 6.*), Spark/MR jobs will decode non-ASCII characters with the wrong format. Rather than using UTF-8 encoding format, Spark/MR would have used ”ANSI_X.3.4-1968”.
On investigating further, we could see that Yarn logs of Spark/MR would have traced java encoding properties file.encoding and sun.jnu.encoding as UTF-8.
grep "ANSI" /application_*************_*****.log file.encoding = ANSI_X3.4-1968 sun.jnu.encoding = ANSI_X3.4-1968
But if we collect java properties using the command “java -XshowSettings -version”, we can see java has been set UTF-8 as encoding options
awt.toolkit = sun.awt.X11.XToolkit file.encoding = UTF-8 file.encoding.pkg = sun.io .. .. sun.io.unicode.encoding = UnicodeLittle sun.java.launcher = SUN_STANDARD sun.jnu.encoding = UTF-8 Also we can see the same behavior from output of unix command “locale” which returns , $ locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8"
Why, then, Spark/MR application returns characters in the wrong format? Why Spark/MR does not consider the encoding format as UTF-8(expected one) but used “ANSI_X3.4-1968” in their applications?
The reason behind this issue is, in some CDH versions(especially in CDH 6.*), CDH blocks some parameters on the Linux system by default. The encoding of most Linux systems should be UTF-8, but CDH6 restricts these parameters by default, so Java uses the JVM default ANSI_X.3.4-1968.
We can identify this behavior by below simple Spark code,
println("Default charset: " + Charset.defaultCharset().displayName()) println("File encoding: " + System.getProperty("file.encoding")) println("Scala encoding: " + scala.util.Properties.encodingString)
How to resolve this issue?
We can resolve this issue, at the job level, by passing file.encoding and sun.jnu.encoding as UTF-8 to Spark/MR application
spark.driver.extraJavaOptions="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8" spark.executor.extraJavaOptions="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
But in order to resolve this issue, in the cluster level, we can add the following parameters to Yarn to make Yarn use the system default parameters:
CM -> YARN -> Configuration -> Search for “Containers Environment Variables Whitelist” -> Add the parameters HADOOP_HOME,PATH,LANG,TZ. Click on “Save Changes”. Restart all NodeManagers.