How do I update a multi project sbt build that depends on macro libraries to scala 3 in stages?

Question

I am trying to update a large multi module repository to Scala 3 in stages.

According to the Scala 3 migration guide, there is good forward and backward runtime compatiblity, so I should be able to update modules arbitrarily

This assumes that three things are true,

1. you provide the -Ytasty-reader scalac option to Scala 2 projects that depend on scala 3,
2. any macro libraries have implemented Macro Mixing
3. you depend on the Scala 3 versions of any Macro libraries (with .cross(CrossVersion.for2_13Use3))

If macro mixing has not been implemented, this won't work as the Scala 2 compiler won't have the Macro definitions that it needs at compile time.

I tried offering to update the upstream library, but this proposal was rejected by the maintainer on account of complexity as it's a temporary situation. In general, macro mixing is not commonly implemented so this also affects other upstream libraries (scala-logging, play-json, circe, etc).

Since it is not always possible or feasible to ensure all upstream macro libraries support Macro mixing, how can I efficiently update my large interdependent multi module build to Scala 3 in stages?

Answer 1

From what I see there is several thing to unpack here:

• you want to migrate project from 2 to 3
• you want to start using Scala 3 features as soon as possible
• you want to do it gradually, I guess you want to avoid gigantic unreviewable PRs
• you are using macro-based libraries

For simplicity I'll assume that you have a project like this:

module A <- module B <- module C

where arrow means "depends on".

The first assumption that you would have to operate on is that "there is only 1 version of particular library in the runtime". So if you need e.g. Circe, you will only use Circe compiled for 2.13 or Circe compiled against 3 - you are able to call (non-macro) Scala 2.13 methods from Scala 3 and vice-versa, so it is not an issue, but something you have to keep in mind.

The other assumption is that macro-mixing is not a thing. Let me explain why, from library maintainer's POV:

• for macro mixing, in the class that uses macro methods from Scala 2 and Scala 3 I have to have access to both:
  • Scala 2.13 methods operating on scala.reflect.macros.blackbox.Context
  • Scala 3 methods operating on scala.quotesQuotes
• while parts of the API working with: path-dependent types, apply, unapply, mixins, abstract types can be cross-compiles (so you can technically write Scala 2.13 macro that will be compiled with Scala 3) Scala 2's quasiquotes and Scala 3 quotes only works on their dedicated compiler version
• tremendous amount of Scala 2 macros are written with quasiquotes, because they make things a lot easier than manually putting trees together with various combinators, so it's hardly ever a thing
• that already forces you to have a project design:

  scala213macroImpl (2.13) <- scalaMacroMixing (3)
  

• but these macros works with some types, usually your own types, and macro has to see these types to be able to work with them, so it's more like

  runtimeTypes (2.13 OR 3) <- scala213macroImpl (2.13) <- scalaMacroMixing (3)
  

• but your 2.13 users would expect library_2.13 name in Maven, not library_3 which would force them to use this for213use3, so it might even become

  runtimeTypes (2.13 OR 3) <- scala213macroImpl (2.13) <- scalaMacroMixing (3) <- emptyProjectJustFor213 (2.13)
  

  Some projects would be fine with it, some would not, it's a choice between friction in the library and the friction in downstream build definitions
• we already have 3-4 projects instead of 1 so the maintenance burden grew up considerable (you 3-4 times the directories to look for implementations), but it would still work under 1 condition: you don't have a use case like e.g. "I need to do some automatic derivation, and I achieve that by putting macro in my types companion object" - because your type is in runtimeTypes, so its companion is also in runtimeTypes but suddenly it requires something that can only be defined in scalaMacroMixing, 2 modules later!
• then there is a matter of 2.12 because, yes, it is not dead and some of us are still supporting it

So when the library maintainer weights the costs of having "macro mixing" (huge) against the benefits (it only benefits people who would migrate a piece of project at a time, like you, but I'm afraid that you are a minority) then it's easy to understand why it didn't caught up.

So how can you work under such constraints?

Gradual cross-compilation

You start to cross-compile projects from bottom up:

module A (cross-compiled 2.13 & 3) <- module B (2.13) <- module C (2.13)

then

module A (cross-compiled 2.13 & 3) <- module B (cross-compiled 2.13 & 3) <- module C (2.13)

then

module A (cross-compiled 2.13 & 3) <- module B (cross-compiled 2.13 & 3) <- module C (cross-compiled 2.13 & 3)

and then you drop 2.13

module A (3) <- module B (3) <- module C (3)

How do you cross-compile each module? There are 2 ways:

• lowest common denominator - you simply never use anything that is only available to only 1 version of Scala (various flags can make that easier)
• you use version-specific implementation, like library's maintainers are usually forced to do:

  src/
    main/
      scala/ <- here goes shared code
      scala-2/ <- here comes Scala 2 only code
      scala-3/ <- here comes Scala 3 only code
  

  Your version-specific implementation might differ, even bytecode can differ as long as the shared code that calls it looks the same. This is where you could use Scala 3 specific features but unfortunately only in such a way that you could implement its Scala 2 counterpart

I'd say if people want to migrate to Scala 3 and they do it gradually... they simply do not use any Scala 3 specific features until they are done. Then they start using Scala 3 in subsequent PRs. Such a migration usually is easier r review because it only updates build.sbt and introduces the least amount of changes that would make the code compile. Ideally, no changes at all.

Using macros only in 1 module

Since your issue are macros there is another approach to try out.

While each macro can only be expanded in its dedicated Scala version the code emitted by Scala 2.13 can be read by both 2.13 and 3, and same is true about code emmittd by 3.

So let's say you want to use e.g. Jsoniter with its Scala 3 macros and Cats Kittens in Scala 2.13 (both of these are released for both 2.13 and 3, but I needed some examples).

You can then arrange your code like this:

module A (2.13)         <- module B (3)             <- module C (2.13 OR 3)
|                               |                            |
| -depends on Cats Kittens      |                            |
| -uses macros to generate Show |                            |
                                |                            |
                      -depends on Jsoniter                   |
                      -uses macros to generate Codec         |
                      -does NOT call Kittens macros          |
                      -CAN use Kittens and Cats runtime      |
                                                             |
                                                    -uses Cats Kittens runtime
                                                    -uses Jsoniter runtime
                                                    -doesn't derive anything

The issue here is that:

• if you had some definition in module B that required 2.13 macro, you'd have to move that definition to module A and expand that macro there
• if you wanted to put some 2.13 AND 3 macro expansion inside the same companion object - you cannot. You have to expand macros somewhere else and them import the results of the expansion
• you might discover that you are creating new modules just for the sake of using a different Scala there (e.g. 2.13), just to expand some macro and then use that expansion result in another module (e.g. with 3)

This approach IMHO only makes sense if at least 1 of your macro dependencies is not published for Scala 3 at all.

If all your macro dependencies are available on Scala 3, I would

• change Scala version in all modules at once
• possibly adapt scalacOptions
• use for3use213 only for those non-macro libraries that weren't cross-published yet
• and avoid any unnecessary changes to the code - each Scala 3 feature that I would be eager to use, I would still leave for another PR
  • I expect that if you do not use Scala 3 features in the same PR in which you move to Scala 3 (which makes sense for a lot of reasons) you should end up with a diff made mostly of build.sbt changes, and possibly several small changes (adding type ascription, suppressing some warning, etc) that should be pretty trivial to review, making it an easy change

I'm afraid that any other approach (like migrating only 1 module at once, but immediately using new features) would actually increase the amount of work rather than make it easier.