River IQ

Image

Sqoop import to Text, Avro, Parquet, Sequence

  Ashish Kumar sqoop January 27, 2019

In my previous article I explained how we can sqoop data in avro file, what kind of error it can throw and how we can resolve them...Now here I am going to show you how we can sqoop import into multiple file format and build table on top of that. As we know that we can sqoop data into multiple file format but sqoop support direct import for four file format. File Format Argument Description Avro Data Files --as-avrodatafile Imports data to Avro Data Files ...

Read more
Image

Hive Integration with Spark

  Ashish Kumar Spark January 22, 2019

Are you struggling to access hive using spark?Is your hive table is not showing in spark?No worry here I am going to show you the key changes made in HDP 3.0 for hive and how we can access hive using spark. Now in HDP 3.0 both spark and hive ha their own meta store. Hive uses the "hive" catalog, and Spark uses the "spark" catalog. With HDP 3.0 in Ambari you can find below configuration for spark.As we know before we could access hive table in spark using HiveContext/SparkSession but now in HDP 3.0 we can access hive using Hive ...

Read more
Image

Sqoop Import in Avro Files

  Ashish Kumar sqoop January 22, 2019

Here today I will show you how we can sqoop data into avro file format.Yeah, we know it very simple put --as-avrodatafile with your sqoop import command as per all apache documentation but in real life does all documented command works as simple as written???Defiantly not…And it happened same as others… so no worry here I’m goanna show you all probable issue you can face and how you need to debug and resolution for the same and if you have some different issue please comment. We will try to solve it together.But before talking to th...

Read more
Image

RECOMMENDATION ENGINE CONTENT-BASED FILTERING & COLLABORATIVE FILTERING

RECOMMENDATION ENGINE content-based filtering & COLLABORATIVE FILTERING  Recommendation engines are probably among the best types of machine learning model known to the general public. Even if people do not know exactly what a recommendation engine is, they have most likely experienced one through the use of popular websites such as Amazon, Netflix, YouTube, Twitter, LinkedIn, and Facebook. Recommendations are a core part of all these businesses, and in some cases, they drive significant percentages of their revenue.The idea be...

Read more
Image

CRUD Operation Using Xrm.WebApi

  Dipali Vaish Microsoft CRM September 30, 2018

In Dynamics 365 One of the important enhancement is Xrm.WebApi. Dynamics 365 v9.0 which will help to make developers life simple. We can perform below operations in Xrm.WebApi.Note: Supports Dynamics 365 (online), version 9.xCreateCreate a record in an entity.Syntax:-Xrm.WebApi.createRecord(entityLogicalName, data).then(successCallback, errorCallback); Example:-// define the data to create new account var _accountData =     {         "name": "Sam test", ...

Read more
Image

Apache Hive 3.0

  Ashish Kumar Hive September 26, 2018

Apache Hive 3.0 Workload managementManaging resources was always a concern to gain hive performance. With LLAP it improved the performance. With hive 3.0 using workload management, you can create resource pools and allocate resources to match your needs and prevent contention for those resources.  Workload management improves parallel query execution and cluster sharing for queries running on Hive LLAP, and improves performance of non-LLAP queries. Transaction processing improvementsMature versions of ACID (Atomicity, Consistency, I...

Read more
Image

Spark Performance Tuning

  Ashish Kumar Spark September 26, 2018

Apache Spark overview Analytics is increasingly an integral part of day-to-day operations at today's leadingbusinesses, and transformation is also occurring through huge growth in mobile and digitalchannels. Previously acceptable response times and delays for analytic insight are no longerviable, with more push toward real-time and in-transaction analytics. In addition, data science skills are increasingly in demand. As a result, enterprise organizations are attempting to leverage analytics in new ways and transition existing analy...

Read more
Image

Facebook Page Analytics

  Ashish Kumar other September 23, 2018

Facebook Insights has helped us determine the tastes and interests of our customer demographics.     OverviewThis section provides a snapshot of the last seven days of your Pages performance. It focuses on 3 core areas:·         Page Likes: Total and new likes for your Page·         Post Reach: Total number of unique people who were shown your Page and posts·  &...

Read more
Image

Page Insights and Post Insights Matrices mapping

  Ashish Kumar other September 22, 2018

/{object-id}/insightsFacebook Insights is a product available to all Pages and Apps on Facebook using the Insights dashboard. This object represents a single Insights metric that is tied to another particular Graph API object (Page, Post, etc.). This object is returned by the following edges:/{page-id}/insights/{metric}/{post-id}/insights/{metric} Key Metrics.Column Name(Page Level) Metrics Name Lifetime Total Likes page_fans Daily New Likes page_fan_adds_unique Daily Unlikes ...

Read more
Image

ADF Spark Activity

  Ashish Kumar Azure September 21, 2018

Introduction Spark Activity is one of the data transformation activities supported by Azure Data Factory. This activity runs the specified Spark program on your Apache Spark cluster in Azure HDInsight. Prerequisite for ADF Spark Activity1.     Create a Azure Storage Account and select account type as Blob storage.2.     Create an Apache Spark cluster in Azure HDInsight and Associate the Azure storage account (Blob storage). While creating HDInsight Spar...

Read more