Swarm: Transparent Scalability Through Portable Continuations

James Earl Douglas

September 12, 2011

The Problem

MapReduce (Hadoop)

void map(KEYIN key, VALUEIN value, Mapper.Context context)

void reduce(KEYIN key, Iterable values, Reducer.Context context)

The domain must fit the MapReduce paradigm
To easily scale later, MapReduce must be an early design choice
The void return types imply reliance on side effects

The Solution

The Holy Grail

Initially build our application cheaply and quickly
Later scale without changing the software

Follow the Data

Let Swarm worry about where in the cluster data is located
Use Swarm's Ref class to dereference the data
Transparently relocate execution to the node with the data

Swarm

val local = new InetLocation(java.net.InetAddress.getLocalHost, 9998)
val remote = new InetLocation(java.net.InetAddress.getLocalHost, remotePort)

val a = Ref(local, "bumble bee")
val b = Ref(local, "honey bee")
val c = Ref(remote, "stingless bee")

println(a())
println(b())
println(c())

Delimited Continuations

object AsyncConsoleIO extends App {

  type cont = cpsParam[Unit, Unit]

  reset {
    var i = 5
    while (i > 0) {
      shiftIt()
      i -= 1
    }
  }

  def shiftIt() = shift { k: (Unit => Unit) =>
    val name = prompt()
    k()
    greet(name)
  }

  def prompt() = { println("name> "); readLine }

  def greet(s: String) = { Thread.sleep(1000); println("hello, " + s) }
}

Delimited Continuations

def workflow(): NodeSeq @imp = {
  val name = getName()
  val age = getAge()

  <html>
    <body>
      Hello {name}, you are {age} years old!
    </body>
  </html>
}

def getName(input: Option[String] = None): String @imp =
  input match {
    case None => getName(prompt(form("Name")))
    case Some(name) => name
}

def getAge(input: Option[String] = None): String @imp =
  input match {
  case None => getAge(prompt(form("Age")))
  case Some(age) if age.matches("\\d+") => age
  case _ => getAge(prompt(
    form("Age", Some("Your age must be a number."))))
}

Swarm: Transparent Scalability Through Portable Continuations

The Problem

The Problem

MapReduce (Hadoop)

The Solution

The Holy Grail

Follow the Data

Swarm

Delimited Continuations

Delimited Continuations

Delimited Continuations

Delimited Continuations

Status of Swarm

Status of Swarm

Lots of cool stuff to do

Reference