Cats 3 Roadmap

[larsrh]

Continued from #2296.

• @joroKr21 suggests that Cats 3 is Scala-3-only.
• @LukaJCB and @kailuowang advocated that we only bring break binary compatibility changes to new Scala major versions.
• @kailuowang drafted a plan in 2019.

[joroKr21]

I think there is tension when relying on major Scala releases to sneak in breaking changes. When there is a new Scala release users and downstream libraries want to have a Cats release ASAP. This leaves us no time to experiment with new features of the language and making Cats better. Take for example Scala 3.0 - we're not using any of the features geared towards better FP (type classes, extension methods, opaque types). But we will release Cats for Scala 3.0 immediately after it's final. So after that we can't use the new features either because we would break binary compatibility. So when would be a good time? Note that eventually the current set of features we use (implicits, implicit conversions, AnyVal) will be deprecated and removed. So if we are aiming for a future Scala 3.1 release, then we should start sooner rather than later because when it comes it will be too late.

[larsrh]

My take on this: we should strive to be cross-compatible for as long as possible between Scala 2 and 3. I'd personally think that we should wait after the Scala 3 final release for community uptake. Only then can we consider refactoring our code to be Scala-3-only. I'd rather avoid any kind of perceived pressure, even when Scala 3.1 comes out.

[kailuowang]

If Cats are allowed to evolve with breaking changes, the community has to pay some cost - think of the Scala collection breaking change in 2.13. Some argues that paying this cost on top of the cost of migrating to Scala 3.0 is too much burden for the community, and I agree that it's probably true, but my evalution is that the cost of trying to do it independent of Scala major release is inhibitive. That's because the only way to support such migration is to cross release and cross-compile for every libs in the community. All the tooling and infrastructures already support cross scala versions, it's hard to imagine to expand it to support a matrix of scala + cats versions.

[djspiewak]

My take: it's okay to break binary things under the surface between Scala 2 and Scala 3, simply because cross-compilation is already happening (see @kailuowang's point), but it's not okay to break source things simply because the churn is too intense. Cats 2 has a particular API surface area, and that API surface area should not change between Scala 2 and Scala 3 (and at present, it doesn't).

When I think of Cats 3, though, I think of a more far-future vision. A couple years from now, more than likely. When that happens, I'd like to redo the typeclass encoding (which is very definitely showing its age and doesn't take advantage of the goodies we have here in Scala 3), as well as more explicitly leverage newer features in the type system introduced by Scala 3. This will have the effect of locking out Scala 2 users, which is precisely why I think it has to be more of a far-future thing. It also has the benefit of significantly evolving the library and the ecosystem, which while painful, is often necessary.

[johnynek]

So, I think the main scala 3 goodie I would like are putting the operators directly on the type class. I can't think of any major problems otherwise. What problems are you seeing with the type class encoding, @djspiewak ?

that said, the main win for users would be to avoid having to do the syntax import.

I think it would also be interesting to consider if some match types or inline methods could be used to improve safety or performance, but I don't have any examples on the tip of my tongue.

Finally, opaque type could be very nice for the monad transformer types to remove having to allocate the wrapper types. Similar with nested. Also, Validated could conceivably become an opaque type, which would allow zero cost conversion between Validated and Either.

All that said: I strongly agree that breaking source compatibility is a tax on users that slows adoption. It would be really nice to have a test suite for source compatibility, something similar to mima for source.

[djspiewak]

So, I think the main scala 3 goodie I would like are putting the operators directly on the type class. I can't think of any major problems otherwise. What problems are you seeing with the type class encoding, @djspiewak ?

It is effectively impossible to have multipath resolution. The infamous : Traverse: Applicative problem is almost the only instance of this in the current hierarchy, but Cats Effect has a ton of problems related to this. The CE hierarchy is distorted appreciably by the fact that it needs to be linear on top of Monad, and even with that distortion it still has effectively two encodings baked into it.

Scato isn't really the answer either because the linearization is entirely static, but I think we can do better.

that said, the main win for users would be to avoid having to do the syntax import.

Which is, justifiably, the number one complaint about Cats' usability right now.

Finally, opaque type could be very nice for the monad transformer types to remove having to allocate the wrapper types. Similar with nested. Also, Validated could conceivably become an opaque type, which would allow zero cost conversion between Validated and Either.

Potentially. Nesting very quickly becomes a problem, and a lot of transformers are more complex than simple datatype wrappers (like Kleisli).

All that said: I strongly agree that breaking source compatibility is a tax on users that slows adoption. It would be really nice to have a test suite for source compatibility, something similar to mima for source.

Basically you want a distributed Cats community build. I have some ideas on this but haven't had time to push them forward.

[johnynek]

re: testing:

if we solved this issue:

scala-steward-org/scala-steward#1877

then we could possibly use scala-steward to pull all the projects that depend on cats, bump to a snapshot version, and see if they compile (or maybe even have their tests pass). That would be pretty heavy weight for every PR potentially, but maybe before release we could do it.

[joroKr21]

I think it's pretty clear that Cats is prime for a rewrite with Scala 3 features. Source compatibility should definitely be a goal but binary compatibility is just not possible. Still not clear to me is if we want to tie this to some future breaking Scala release or we can do this on our own timeline.

Scato isn't really the answer either because the linearization is entirely static, but I think we can do better.

Curious, what do you have in mind? Note that in Scala 3 we can use regular scope (e.g. a local given) to disambiguate.

Potentially. Nesting very quickly becomes a problem, and a lot of transformers are more complex than simple datatype wrappers (like Kleisli).

Kleisli is just a wrapper around a function. I don't see a reason why nesting opaque types would be any more complex than nesting case classes.

[mpilquist]

I think it's pretty clear that Cats is prime for a rewrite with Scala 3 features.

I'm not yet convinced of this and worry that if we assume it, we'll end up rewriting things that don't need to be rewritten. I agree that binary compatibility is not a target but source compatibility should be. Timeline wise, I think we're looking at 1-2 years in the future at the earliest.

[joroKr21]

All that said: I strongly agree that breaking source compatibility is a tax on users that slows adoption. It would be really nice to have a test suite for source compatibility, something similar to mima for source.

Actually source compatibility for implementing type class instances might be not so straightforward precisely because we would like to use extension methods.

[johnynek]

Here's an old idea, but maybe useful.

We could have a cats.v3. package. We would try to be source compatible (will be hard to test this), but embrace all of scala3.

The goal would be to change import from cats to cats.v3 and still have things compile.

I know people usually react negatively to versioning in package names but frankly scala 3 is going to take many years to adopt I expect (imagine how long spark will block us). By using this technique we can in parallel continue to build cats for 2.12, 2.13, 3.0 and only the 3.0 build contains v3.