Apache Spark :- Configure Spark on Windows with Eclipse

Hi Guys , in this post i am going to write about integration of spark with eclipse using Maven in Windows

 

Open eclipse and install scala IDE using eclipse marketplace

Help ->eclipse marketplace

Now let’s create a scala program with spark by following below steps:

 

  • Create new project (scala project)
  • Change this project to maven project :- rightclick -> configure :- convert to Maven project

 

  • Go to dependences , add following values
  • Save all
  • At bottom watch ;; workspace will be building
  • Once done add a scala file as below
  • Write code for word count simple in a file

import org.apache.spark._

 

object samplesparktest {

def main(args: Array[String]) {

System.setProperty(“hadoop.home.dir”, “c://winutil//”)

val conf = new SparkConf().setAppName(“Simple Application”).setMaster(“local[2]”)

val sc = new SparkContext(conf)

val    input=sc.textFile(“file:///C://Users//HA848869//Desktop//sparkdata//textfile.txt”)

val counts =input.flatMap(line => line.split(” “))

.map(word => (word, 1))

.reduceByKey(_ + _)

 

counts.collect()

for (i – counts)

{

print(i)

}

}

}

 

 

 

Run the above program as Run As Scala application and watch the console:-

 

Also you can see the spark web ui also:-

Web UI :- http://192.168.61.1:4040/

Advertisements

2 thoughts on “Apache Spark :- Configure Spark on Windows with Eclipse

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s