Ascension from Dependency Hell

Scala has a rapidly-changing ecosystem of interconnected libraries. This tends to lead to dependency hell, in which a project's transitive dependency graph contains version mismatches. Not only is this painful to manage, but current approaches leave projects vulnerable to defects that can be hard to catch in time.

In this post, we explore shading as a simple and robust approach to deconflicting transitive dependencies in Scala projects.

Example

Our example project has two dependencies, each of which depend on different versions of a common library:

         .--------.
         | logger |
         | * v1 <----.
      .--> * v2   |  |
      |  '--------'  |
      |              |
.------------.   .-------.
| multiplier |   | adder |
'------------'   '-------'
      ^              ^
      |              |
      '-----.  .-----'
            |  |
          .------.
          | main |
          '------'

• Main depends on Multiplier and Adder
• Multiplier depends on Logger v2
• Adder depends on Logger v1

Pulling both Logger v1 and Logger v2 into our project is likely to create namespace collisions, since the code in both versions likely reuses names of packages, classes, methods, etc.

If we try to opt for one version of Logger and evict the other, we are liable to break the dependency that expects the evicted one. Using shading, our project can utilize both versions of Logger without conflicts.

Logger v1

Version 1 of Logger defines a log function for printing to standard output.

build.sbt:

version := "1.0.0"

logger.scala:

package logger

def log(line: String): Unit =
  println(s"[log] ${line}")

Logger v2

Version 2 of Logger renames the log function to info.

build.sbt:

version := "2.0.0"

logger.scala:

package logger

def info(line: String): Unit =
  println(s"[info] ${line}")

Adder

Adder uses Logger v1 to log its operation.

build.sbt:

libraryDependencies += "com.example" %% "logger" % "1.0.0"

adder.scala:

package adder

def add(x: Int, y: Int): Int =
  val z: Int = x + y
  logger.stdout(s"add(${x}, ${y}) = ${z}")
  z

Multiplier

Multiplier uses Logger v2 to log its operation.

build.sbt:

libraryDependencies += "com.example" %% "logger" % "2.0.0"

multiplier.scala:

package multiplier

def multiply(x: Int, y: Int): Int =
  val z: Int = x * y
  logger.info(s"multiply(${x}, ${y}) = ${z}")
  z

Main Project

Our main project uses both Adder, which uses Logger v1, and Multiplier, which uses Logger v2.

main.scala:

@main def main: Unit =
  val x: Int = adder.add(2, 3)
  val y: Int = multiplier.multiply(x, 7)
  println(y)

Without Shading

Normally, we consume both libraries as regular dependencies:

build.sbt:

libraryDependencies += "com.example" %% "adder" % "1.0.0"
libraryDependencies += "com.example" %% "multiplier" % "1.0.0"

The dependency graph looks like this:

• Main
  • Adder
    • Logger v1
  • Multiplier
    • Logger v2

Main has four dependencies in total, including two versions of Logger.

This compiles just fine, but it fails to run because one version of Logger clobbers the namespace of the other:

Exception in thread "sbt-bg-threads-1" java.lang.NoSuchMethodError:
    'void logger.Logger$package$.stdout(java.lang.String)'
        at adder.Adder$package$.add(Adder.scala:5)
        at Main$package$.main(Main.scala:2)
        at main.main(Main.scala:1)

Shading with sbt-shading

With sbt-shading, we can rename the logger package in the Logger v1 library to loggerv1, and update references in the Adder library to match:

build.sbt:

lazy val shadedAdder = project
  .enablePlugins(ShadingPlugin)
  .settings(
    libraryDependencies += "com.example" %% "adder" % "1.0.0",
    shadedDependencies += "com.example" %% "adder" % "<ignored>",
    shadingRules += ShadingRule.moveUnder("logger", "loggerv1"),
    shadingVerbose := true,
    validNamespaces += "adder",
    validNamespaces += "loggerv1",
  )

We can rename the logger package in the Logger v2 library to loggerv2, and update references in the Multiplier library to match:

build.sbt:

lazy val shadedMultiplier = project
  .enablePlugins(ShadingPlugin)
  .settings(
    libraryDependencies += "com.example" %% "multiplier" % "1.0.0",
    shadedDependencies += "com.example" %% "multiplier" % "<ignored>",
    shadingRules += ShadingRule.moveUnder("logger", "loggerv2"),
    shadingVerbose := true,
    validNamespaces += "multiplier",
    validNamespaces += "loggerv2",
  )

In our main project, we can bring the shaded Adder and Multiplier dependencies in as unmanaged jars:

build.sbt:

lazy val main = project
  .in(file("."))
  .settings(
    Compile / unmanagedJars ++=
      Seq(
        (shadedAdder / shadedPackageBin).value,
        (shadedMultiplier / shadedPackageBin).value,
      ),
  )
  .aggregate(shadedAdder, shadedMultiplier)

Shading with sbt-assembly

We can do something similar with sbt-assembly and some extra boilerplate.

As before, we can rename the logger package in the Logger v1 library to loggerv1, and update references in the Adder library to match. We also want to explicitly exclude the Scala libraries, which will be provided automatically.

build.sbt:

lazy val shadedAdder =
  project
    .settings(
      libraryDependencies += ("com.example" %% "adder" % "1.0.0"),

      assembly / assemblyShadeRules +=
        ShadeRule.rename("logger.**" -> "loggerv1.@1")
          .inLibrary(
            // Note that using %% here is unsupported:
            // <https://github.com/sbt/sbt-assembly/issues/235>
            "com.example" % "logger_3" % "1.0.0",
            "com.example" % "adder_3" % "1.0.0",
          ),

      // exclude scala3-library
      assembly / assemblyExcludedJars :=
        (assembly / fullClasspath)
          .value
          .filter {
            _.data.getName ==
              s"scala3-library_3-${scalaVersion.value}.jar"
          },

      // exclude scala-library
      assemblyPackageScala / assembleArtifact := false,
    )

As before, we can rename the logger package in the Logger v2 library to loggerv2, and update references in the Multiplier library to match. Again we explicitly exclude the Scala libraries.

build.sbt:

lazy val shadedMultiplier =
  project
    .settings(
      libraryDependencies += "com.example" %% "multiplier" % "1.0.0",

      assembly / assemblyShadeRules +=
        ShadeRule.rename("logger.**" -> "loggerv2.@1")
          .inLibrary(
            // Note that using %% here is unsupported:
            // <https://github.com/sbt/sbt-assembly/issues/235>
            "com.example" % "logger_3" % "2.0.0",
            "com.example" % "multiplier_3" % "1.0.0",
          ),

      // exclude scala3-library
      assembly / assemblyExcludedJars :=
        (assembly / fullClasspath)
          .value
          .filter {
            _.data.getName ==
              s"scala3-library_3-${scalaVersion.value}.jar"
          },

      // exclude scala-library
      assemblyPackageScala / assembleArtifact := false,
    )

As before, we can bring the shaded Adder and Multiplier dependencies into our main project as unmanaged jars:

build.sbt:

lazy val main =
  project
    .in(file("."))
    .settings(
      Compile / unmanagedJars += (shadedAdder / assembly).value,
      Compile / unmanagedJars += (shadedMultiplier / assembly).value,
    )
    .aggregate(shadedAdder, shadedMultiplier)

Future Work

It might be possible to extend sbt-shading so that it automatically shades every dependency in the graph, prepending each dependency's version number to each namespace in the dependency. This would keep us out of dependency hell by ensuring that every namespace is unique, and no two ever collide even if they come from different versions of the same library.

Other Solutions

OSGi uses separate class loaders to isolate subgraphs of dependencies that might otherwise have namespace conflicts. Rust does something like this too.

Unison just hashes everything, so that all references are content-addressed rather than namespace-addressed.