Spark 2.0.1 –

Import org.apache.spark.sql.SparkSession

   val spark = SparkSession.builder()
     .master("local")
     .appName("Spark sql")
     .config("spark.some.config.option", "some-value").
     .getOrCreate()

 

Instead of ‘sparkContext’ can use ‘SparkSession’ to load data,

 

val diamonds = sqlContext.read.format(“com.databricks.spark.csv”)
.option(“header”, “true”)
.option(“inferSchema”, “true”)
.load(“/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv”)

val diamonds1 = spark.read.format(“com.databricks.spark.csv”)
.option(“header”, “true”)
.option(“inferSchema”, “true”)
.load(“/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv”)