River IQ

Image

Databricks Log4j Configuration

  Ashish Kumar      Databricks January 15, 2020

System.out.println("Caught Error") This will write output on console window. This is fine when you are developing or running manually but what if you are scheduling as job or automating your application. In that case these output or log should go to some persistent location. You can persist it anywhere database table, email server or in a log file. So here we are discussing how to write our log into log file and the one of solution is Log4j. Here I won't be explaining much about Log4j. I'm sure you must be knowing or you can ...

Read more
Image

Log4j Configuration with spark-submit

  Ashish Kumar      java January 15, 2020

This is 2nd part of log4j configuration for spark application. For more understanding about log4j you can follow below link.https://www.linkedin.com/pulse/databricks-log4j-configuration-ashish-kumar/Spark-submit/usr/hdp/3.0.1.0-187/spark2/bin/spark-submit --master yarn --queue dev --deploy-mode client --class com.riveriq.log4jExample --driver-java-options "-Dlog4j.configuration=file:/home/riveriq/log4j/conf/log4j.xml" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/home/riveriq/log4j/conf/log4j.xml" --num-exe...

Read more