REPL with -Ybest-effort throws NullPointerException

[jirijakes]

When calling sbt console on a 3.5.0 project with Best Effort Compilation enabled, the console opens but any evaluation of Scala expression ends up with NullPointerException.

From stack trace, it seems to be related to dotty rather than SBT. SBT commands like :help work.

Compiler version

Scala: 3.5.0
Java OpenJDK 64-Bit Server VM, 22.0.2
SBT: 1.10.1

Minimized code

build.sbt:

scalaVersion := "3.5.0"

scalacOptions ++= List("-Ybest-effort", "-Ywith-best-effort-tasty")

$> sbt console
scala> 1

Output

[error] java.lang.NullPointerException: Cannot invoke "java.nio.file.Path.toAbsolutePath()" because the return value of "dotty.tools.io.AbstractFile.jpath()" is null
[error]         at dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:416)
[error]         at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
[error]         at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error]         at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error]         at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
[error]         at dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
[error]         at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:384)
[error]         at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:396)
[error]         at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
[error]         at dotty.tools.dotc.Run.compileUnits(Run.scala:396)
[error]         at dotty.tools.dotc.Run.compileUnits(Run.scala:288)
[error]         at dotty.tools.repl.ReplCompiler.compile(ReplCompiler.scala:88)
[error]         at dotty.tools.repl.ReplDriver.compile(ReplDriver.scala:321)
[error]         at dotty.tools.repl.ReplDriver.interpret(ReplDriver.scala:283)
[error]         at dotty.tools.repl.ReplDriver.loop$1(ReplDriver.scala:196)
[error]         at dotty.tools.repl.ReplDriver.runUntilQuit$$anonfun$1(ReplDriver.scala:199)
[error]         at dotty.tools.repl.ReplDriver.withRedirectedOutput(ReplDriver.scala:238)
[error]         at dotty.tools.repl.ReplDriver.runBody$$anonfun$1(ReplDriver.scala:212)
[error]         at dotty.tools.runner.ScalaClassLoader$.asContext(ScalaClassLoader.scala:80)
[error]         at dotty.tools.repl.ReplDriver.runBody(ReplDriver.scala:212)
[error]         at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:199)
[error]         at xsbt.ConsoleInterface.run(ConsoleInterface.java:52)
[error]         at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[error]         at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[error]         at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
[error]         at sbt.internal.inc.AnalyzingCompiler.console(AnalyzingCompiler.scala:233)
[error]         at sbt.Console.console0$1(Console.scala:65)
[error]         at sbt.Console.$anonfun$apply$5(Console.scala:75)
[error]         at sbt.Run$.executeSuccess(Run.scala:187)
[error]         at sbt.Console.$anonfun$apply$4(Console.scala:75)
[error]         at sbt.internal.util.Terminal.withRawInput(Terminal.scala:146)
[error]         at sbt.internal.util.Terminal.withRawInput$(Terminal.scala:144)
[error]         at sbt.internal.util.Terminal$ProxyTerminal$.withRawInput(Terminal.scala:424)
[error]         at sbt.Console.$anonfun$apply$3(Console.scala:75)
[error]         at sbt.internal.util.Terminal$TerminalImpl.withRawOutput(Terminal.scala:1028)
[error]         at sbt.internal.util.Terminal$ProxyTerminal$.withRawOutput(Terminal.scala:463)
[error]         at sbt.Console.apply(Console.scala:72)
[error]         at sbt.Console.apply(Console.scala:50)
[error]         at sbt.Console.apply(Console.scala:42)
[error]         at sbt.Defaults$.$anonfun$consoleTask$1(Defaults.scala:2287)
[error]         at sbt.Defaults$.$anonfun$consoleTask$1$adapted(Defaults.scala:2273)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]         at sbt.Execute.work(Execute.scala:292)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]         at java.base/java.lang.Thread.run(Thread.java:1570)
[error] (Compile / console) java.lang.NullPointerException: Cannot invoke "java.nio.file.Path.toAbsolutePath()" because the return value of "dotty.tools.io.AbstractFile.jpath()" is null

[noti0na1]

We should probably remove the unsafeNulls and update the types in dotty.tools.io.* as well.

[Gedochao]

Replicable with Scala CLI /scala.

scala-cli -Ybest-effort -Ywith-best-effort-tasty
# Welcome to Scala 3.5.0 (17, Java OpenJDK 64-Bit Server VM).
# Type in expressions for evaluation. Or try :help.
scala> 1
# Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.nio.file.Path.toAbsolutePath()" because the return value of "dotty.tools.io.AbstractFile.jpath()" is null
# 	at dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:416)
# 	at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
# 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
# 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
# 	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
# 	at dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
# 	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:384)
# 	at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:396)
# 	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
# 	at dotty.tools.dotc.Run.compileUnits(Run.scala:396)
# 	at dotty.tools.dotc.Run.compileUnits(Run.scala:288)
# 	at dotty.tools.repl.ReplCompiler.compile(ReplCompiler.scala:88)
# 	at dotty.tools.repl.ReplDriver.compile(ReplDriver.scala:321)
# 	at dotty.tools.repl.ReplDriver.interpret(ReplDriver.scala:283)
# 	at dotty.tools.repl.ReplDriver.loop$1(ReplDriver.scala:196)
# 	at dotty.tools.repl.ReplDriver.runUntilQuit$$anonfun$1(ReplDriver.scala:199)
# 	at dotty.tools.repl.ReplDriver.withRedirectedOutput(ReplDriver.scala:238)
# 	at dotty.tools.repl.ReplDriver.runBody$$anonfun$1(ReplDriver.scala:212)
# 	at dotty.tools.runner.ScalaClassLoader$.asContext(ScalaClassLoader.scala:80)
# 	at dotty.tools.repl.ReplDriver.runBody(ReplDriver.scala:212)
# 	at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:199)
# 	at dotty.tools.repl.ReplDriver.tryRunning(ReplDriver.scala:136)
# 	at dotty.tools.repl.Main$.main(Main.scala:7)
# 	at dotty.tools.repl.Main.main(Main.scala)

[jchyb]

@jirijakes Hi, was there any specific intent in using the best effort compilation in repl? It was designed as an IDE feature only (which is why it has the -Y private option signifier) - the idea is Metals IDE will enable it themselves whenever it needs to. Did you add those options manually in the sbt project?

We should still probably make the repl error out and quit gracefully when those options are used.

[jirijakes]

Hi, @jchyb.

No, no specific intent. I added it to build.sbt manually, yes, after reading https://scala-lang.org/api/3.5.0/docs/docs/internals/best-effort-compilation.html believing that it might make working with Metals faster or smoother. I did not understand that this is supposed to be done by IDE. Then after I added it to build.sbt, I started to experience this issue when entering console.

So if the issue is me using this option in an unexpected way, then perhaps mentioning this in the aformentioned documentation would be helpful.

[jchyb]

Yes, for sure! Thank you for specifying where you found them, we'll update the docs to make things clearer. If you still want to try best effort compilation you can add -Dmetals.enable-best-effort=true to Metals: Server Properties and it will be available while using the bloop build server (no changes to the build definition necessary). With that said, there are still a few bugs we are ironing out on the build server side, so It's best to wait a little bit for that (which is why the option is so hidden for now).
Apologies for the confusion!

[jirijakes]

Now I understand. Thanks a lot!

[jirijakes]

Apologies for adding to this issue but is it possible that the same thing causes that worksheets do not evaluate? I use Emacs LSP.

With:

(setq lsp-metals-server-args '( "-J-Dmetals.enable-best-effort=false"))

worksheets are evaluated as expected. No best-effort configuration in build.sbt

However when best effort is enabled, the same worksheets do not evaluate anymore.

I tried to repeat multiple times, cleaned the project every time and always got the same result. The only message I could find was:

error: src/main/scala/w.worksheet.sc: skipping file, the compiler produced no classfiles and reported no errors to explain what went wrong during compilation. Please report an issue to https://github.com/scalameta/mdoc/issues.

Is there something else I could try to provide more information? Or is it issue rather for mdoc or metals?

[tgodzik]

Doesn't seem like I can reproduce, but worksheet should not have betasty options enabled, so everything should work ok.

If you are able to provide a reproduction we would be glad to help out.