River IQ

Image

spark udf with withColumn

  Ashish Kumar Spark February 14, 2020

import org.apache.spark.sql.functions._ val events = Seq ( (1,1,2,3,4), (2,1,2,3,4), (3,1,2,3,4), (4,1,2,3,4), (5,1,2,3,4)).toDF("ID","amt1","amt2","amt3","amt4") var prev_amt5=0 var i=1 def getamt5value(ID:Int,amt1:Int,amt2:Int,amt3:Int,amt4:Int) : Int = {     if(i==1){ i=i+1 prev_amt5=0   }else{ i=i+1   }   if (ID == 0)   { if(amt1==0) {   val cur_amt5= 1   prev_amt5=cur_amt5   cur_amt5 }else{   val cur_amt5=1*(amt2+amt3)   prev_amt5=cur_amt5   cur_amt5 }   }el...

Read more
Image

transfer files widow to Linux

  Ashish Kumar other February 14, 2020

riveriq_copytoedge.bat@echo off set timestamp=%DATE:/=-%_%TIME::=-%C:\"Program Files (x86)"\WinSCP\WinSCP /script="E: iveriqrtifactscopytoedgeconfig iveriq_copytoedge.txt" /log="E: iveriqloggingcopytoedgeconfig%timestamp%.log" riveriq_copytoedge.txt# Being intended for interactive session, we are not enabling batch mode# Connectopen sftp://testuser:password123@testserver.riveriq.com/# Synchronize paths provided via environment variablessynchronize remote E: iveriq estsourcedatacleansedDataconfig /data/home/riveriq/testsource/data...

Read more
Image

Read JCEKS Containing Secret Keys using java

  Ashish Kumar java February 14, 2020

package com.riveriq.db2con.driver;import java.sql.Connection;import java.sql.DriverManager;import java.sql.SQLException;import java.util.Properties;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import com.riveriq.exception.CustomException;import com.riveriq.util.ReadJceks;public class DB2Connection { private Connection conn; private static DB2Connection db2connection; private static Logger LOGGER = LoggerFactory.getLogger(DB2Connection.class); private DB2Connection() { } public Connection getConnection(Properties prop) throws SQLExcept...

Read more
Image

How to remove new lines within double quotes

  Ashish Kumar other February 14, 2020

!/usr/bin/perluse warnings;use strict;use Path::Tiny; use Text::CSV;use Time::Piece;use File::Path qw( make_path );use diagnostics;use Try::Tiny;#use File::NCopy;use File::Copy::Recursive qw(fcopy rcopy dircopy fmove rmove dirmove);use Time::HiRes qw( time );my $start = time();my $date = localtime->strftime('%Y%m%d');my $feed_date = $date;if(exists($ARGV[3])){  $feed_date = $ARGV[3];}# build source directory path ==>my $source_feed_dir = $ARGV[0];my $source_feed_dir_path = path($source_feed_dir);# process i.e. current date ...

Read more