Apache spark:- Maximum temperature in a file using scala

HI, in this blog I will explain how to find the maximum temperature in a text file

 

 

We need to configure eclipse with spark first which is already described in earlier blog

 

 

Sample input:-

91    67    Agra  Uttar Pradesh     33    19

90    67    Allahabad   Uttar Pradesh     32    19

87    60    Amritsar    Punjab      30    15

89    66    Bhopal      Madhya Pradesh    32    19

87    62    Chandigarh  Chandigarh  30    17

82    60    Dehradun    Uttarakhand 28    15

89    64    Indore      Madhya Pradesh    32    18

89    65    Lucknow     Uttar Pradesh     40    18

 

Objective: – find maximum temperature of particular state

 

 

Here comes our program:-

 

 

import org.apache.spark._;

 

object testfilter extends App {

val conf=new SparkConf().setMaster(“local[2]”).setAppName(“testfilter”)

val sc = new SparkContext(conf)

System.setProperty(“hadoop.home.dir”, “c://winutil//”)

val input=sc.textFile(“file:///D://sparkprog//temp//stats.txt”)

val line=input.map(x=>x.split(“\t”))

val city=line.map(x=>(x(3)+”\t” + x(4)))

val rdd3=city.map(x=>x.split(“\t”))

val maintemp=rdd3.map(x=>((x(0),x(1))))

val grp= maintemp.groupByKey()

val main = grp.map{case (x,iter) => (x,iter.toList.max)}

 

 

for ( i<- main)

{

print(i)

print(“\n”)

}

 

}

 

Run the above program and we will get output like:-

(Jammu and Kashmir,20)(Madhya Pradesh,32)

 

(Bihar,31) and so on ..

 

 

sample text file:-  ( please make data tab separated )

 

91 67 Agra Uttar Pradesh 33 19
90 67 Allahabad Uttar Pradesh 32 19
87 60 Amritsar Punjab 30 15
89 66 Bhopal Madhya Pradesh 32 19
87 62 Chandigarh Chandigarh 30 17
82 60 Dehradun Uttarakhand 28 15
89 64 Indore Madhya Pradesh 32 18
89 65 Lucknow Uttar Pradesh 40 18
86 63 Ludhiana Punjab 30 17
65 47 Mukteswar Uttarakhand 19 8
88 66 New Delhi Delhi 31 19
89 67 Patna Bihar 31 20
62 50 Shimla Himachal Pradesh 17 10
67 45 Srinagar Jammu and Kashmir 20 7
90 67 Varanasi Uttar Pradesh 32 19

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s