Logging with Scala and ELK

September 03, 2017

Scala service

build.sbt:

scalaVersion := "2.12.3"

libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.11"
libraryDependencies += "net.logstash.logback" % "logstash-logback-encoder" % "4.9"

libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.5.0"

libraryDependencies += "org.http4s" %% "http4s-blaze-server" % "0.15.8a"
libraryDependencies += "org.http4s" %% "http4s-dsl" % "0.15.8a"

src/main/resources/logback.xml:

<configuration>

  <appender name="logstash" class="net.logstash.logback.appender.LogstashSocketAppender">
    <host>localhost</host>
    <port>4560</host>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <customFields>{ "service": "hello-world" }</customFields>
    </encoder>
  </appender>

  <root level="debug">
    <appender-ref ref="logstash" />
  </root>

</configuration>

src/main/scala/Main.scala:

import com.typesafe.scalalogging.LazyLogging
import org.http4s.HttpService
import org.http4s.dsl._
import org.http4s.server.Server
import org.http4s.server.ServerApp
import org.http4s.server.blaze.BlazeBuilder
import scalaz.concurrent.Task

object Main extends ServerApp with LazyLogging {

  val helloService: HttpService =
    HttpService {
      case GET -> Root =>
        logger.info("Responding to GET /")
        Ok("Hello, world!")
    }

  override def server(args: List[String]): Task[Server] =
    BlazeBuilder
      .bindHttp(8080, "localhost")
      .mountService(helloService, "/")
      .start

}

Logstash configuration

02-udp-input.conf:

input {
 udp {
   port => 4560
   codec => json
 }
}

Docker ELK server

$ docker run --rm \
    -v /path/to/02-udp-input.conf:/etc/logstash/conf.d/02-udp-input.conf \
    -p 5601:5601 -p 4560:4560/udp -it --name elk sebp/elk

This may fail with an error message about vm.max_map_count:

elasticsearch:5.0.0 max virtual memory areas vm.max_map_count [65530]
likely too low, increase to at least [262144]

Fix it with sysctl, then run the above docker command again:

$ sudo sysctl -w vm.max_map_count=262144

Usage

Start the Scala service:

$ sbt run

Submit some requests:

$ curl http://localhost:8080/
Hello, world!
$ curl http://localhost:8080/
Hello, world!

Point a browser to Kibana at localhost:5601, configure a basic index, and run a search for service:hello-world:

Kibana screenshot
Kibana screenshot