River IQ

Image

Sqoop import to Text, Avro, Parquet, Sequence

  Ashish Kumar sqoop January 27, 2019

In my previous article I explained how we can sqoop data in avro file, what kind of error it can throw and how we can resolve them...Now here I am going to show you how we can sqoop import into multiple file format and build table on top of that. As we know that we can sqoop data into multiple file format but sqoop support direct import for four file format. File Format Argument Description Avro Data Files --as-avrodatafile Imports data to Avro Data Files ...

Read more
Image

Hive Integration with Spark

  Ashish Kumar Spark January 22, 2019

Are you struggling to access hive using spark?Is your hive table is not showing in spark?No worry here I am going to show you the key changes made in HDP 3.0 for hive and how we can access hive using spark. Now in HDP 3.0 both spark and hive ha their own meta store. Hive uses the "hive" catalog, and Spark uses the "spark" catalog. With HDP 3.0 in Ambari you can find below configuration for spark.As we know before we could access hive table in spark using HiveContext/SparkSession but now in HDP 3.0 we can access hive using Hive ...

Read more
Image

Sqoop Import in Avro Files

  Ashish Kumar sqoop January 22, 2019

Here today I will show you how we can sqoop data into avro file format.Yeah, we know it very simple put --as-avrodatafile with your sqoop import command as per all apache documentation but in real life does all documented command works as simple as written???Defiantly not…And it happened same as others… so no worry here I’m goanna show you all probable issue you can face and how you need to debug and resolution for the same and if you have some different issue please comment. We will try to solve it together.But before talking to th...

Read more