River IQ

Image

Databricks Log4j Configuration

  Ashish Kumar Databricks January 15, 2020

System.out.println("Caught Error") This will write output on console window. This is fine when you are developing or running manually but what if you are scheduling as job or automating your application. In that case these output or log should go to some persistent location. You can persist it anywhere database table, email server or in a log file. So here we are discussing how to write our log into log file and the one of solution is Log4j. Here I won't be explaining much about Log4j. I'm sure you must be knowing or you can ...

Read more
Image

Log4j Configuration with spark-submit

  Ashish Kumar java January 15, 2020

This is 2nd part of log4j configuration for spark application. For more understanding about log4j you can follow below link.https://www.linkedin.com/pulse/databricks-log4j-configuration-ashish-kumar/Spark-submit/usr/hdp/3.0.1.0-187/spark2/bin/spark-submit --master yarn --queue dev --deploy-mode client --class com.riveriq.log4jExample --driver-java-options "-Dlog4j.configuration=file:/home/riveriq/log4j/conf/log4j.xml" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/home/riveriq/log4j/conf/log4j.xml" --num-exe...

Read more
Image

Azure Databricks Notebook - How to get current workspace name

  Ashish Kumar Databricks January 15, 2020

Sometimes you also have been in some situation where you feel something should be very easy but once you started looking for that, you found it's not Here some sort of things happened with me also and sharing my learning with you all. I was looking to get current workspace name from notebook. As wanted to get environment(dev/test/stg/prod) type from workspace name and using same in notebook configuration. I did some research but couldn't succeed or I would say it won't be possible to get workspace details from notebook and reason be...

Read more