Spark

March 11, 2020

Documentation

sbt configuration

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.0"

Environments

Examples

Demos

These files are literate Scala, and can be run using Codedown:

$ curl https://earldouglas.com/posts/scala/spark.md |
  codedown sbt > build.sbt
$ curl https://earldouglas.com/posts/scala/spark/local.md |
  codedown scala > Spark.scala

Group by Key

$ curl https://earldouglas.com/posts/scala/spark/group-by-key.md |
  codedown scala > Main.scala
$ sbt run
Name(List(John, Jane),Doe)
Name(List(Alice),Bob)

Pi

$ curl https://earldouglas.com/posts/scala/spark/pi.md |
  codedown scala > Main.scala
$ sbt run
Pi: 3.1415926635898956