New evaluation and costing #639

aslesarenko · 2020-01-31T12:59:56Z

This new implementation of ErgoTree v2 is based on #637, #697
The code of this PR can be merged into v3.x/v4.0 branches, BUT NOT ACTIVATED.
The actual activation should be done in v5.0 release as it is described in https://github.com/ScorexFoundation/sigmastate-interpreter/blob/develop/docs/aot-jit-switch.md.
That said, this PR MUST NOT introduce consensus breaking changes in the existing (AOT based) implementation of the interpreter.

Motivation

ErgoTree is simple declarative intermediate representation for Ergo contracts.
It is designed to be compact in serialized form and directly executable, i.e. no additional transformation is necessary before it can be efficiently executed.

This PR implements big-step recursive ErgoTreeEvaluator that works directly with ErgoTree.
Because of this the evaluator is very simple and follows denotational semantics of ErgoTree.
Or, the other way around, this implementation of ErgoTreeEvaluator is purely functional with immutable data structures and can be used as definition of ErgoTree's semantics.

Implementation

ErgoTreeEvaluator takes ErgoTree directly as it is deserialized as part of a transaction.
No additional transformations is performed.
ErgoTree is interpreted directly and all intermediate data are stored in runtime types.
Runtime types are such types as Coll, SigmaProp, AvlTree, BigInt, etc.
It also use immutable Map to keep current DataEnv of computed ValDefs, as a result only the addition operation is used from the map, and the deletion is essentially done by the garbage collection.

Performance

Since this interpreter directly works with SigmaDsl types (Coll, BigInt, SigmaProp etc), it turns out to be fast.
Since it also does JIT style costing instead of AOT style, it is 5-6x faster then existing implementation.

Milestone

New evaluation of ErgoTree required changes in how isProven handled.
In particular isProven is not supported in ErgoTree that is passed to prove/verify methods.
This required changes in tests and also in the coverage report.
Unfortunately those changes may break the consensus, because isProven can be serialized and present in ErgoTree.
This prevents ErgoTreeEvaluator to be used in v3.x

…ByteArray, ByteArrayToLong, ByteArrayToBigInt, CalcBlake2b256, CalcSha256, SubstConstants

…NoRef, ExtractCreationInfo

…IsDefined

…nd, SigmaOr, OR, AND, AtLeast, BinOr, BinAnd, If

coveralls · 2020-01-31T13:34:42Z

Pull Request Test Coverage Report for Build 5222

209 of 781 (26.76%) changed or added relevant lines in 24 files are covered.
2864 unchanged lines in 194 files lost coverage.
Overall coverage decreased (-0.8%) to 33.918%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
sigma-api/src/main/scala/special/sigma/SigmaDsl.scala	0	1	0.0%
sigma-impl/src/main/scala/special/sigma/SigmaDslOverArrays.scala	0	1	0.0%
sigma-library/src/main/scala/special/sigma/SigmaDslUnit.scala	0	1	0.0%
sigmastate/src/main/scala/org/ergoplatform/ErgoBox.scala	0	1	0.0%
sigmastate/src/main/scala/org/ergoplatform/ErgoBoxCandidate.scala	0	1	0.0%
sigmastate/src/main/scala/sigmastate/basics/DLogProtocol.scala	0	1	0.0%
sigmastate/src/main/scala/sigmastate/interpreter/ProverResult.scala	0	1	0.0%
sigmastate/src/main/scala/sigmastate/serialization/ValueSerializer.scala	0	1	0.0%
sigmastate/src/main/scala/sigmastate/eval/BigIntegerOps.scala	1	4	25.0%
sigmastate/src/main/scala/sigmastate/eval/Evaluation.scala	1	5	20.0%

Files with Coverage Reduction	New Missed Lines	%
common/src/main/scala/scalan/ExactIntegral.scala	1	20.0%
common/src/main/scala/scalan/ExactNumeric.scala	1	53.85%
common/src/main/scala/scalan/OverloadHack.scala	1	28.57%
core/src/main/scala/scalan/Modules.scala	1	33.33%
core/src/main/scala/scalan/staged/ProgramGraphs.scala	1	11.11%
library-api/src/main/scala/special/collection/package.scala	1	20.0%
library-api/src/main/scala/special/SpecialPredef.scala	1	14.29%
library/src/main/scala/wrappers/special/impl/WSpecialPredefsImpl.scala	1	24.32%
sigma-api/src/main/scala/special/sigma/SigmaDsl.scala	1	16.22%
sigma-impl/src/main/scala/special/sigma/Extensions.scala	1	33.33%

Totals
Change from base Build 5219:	-0.8%
Covered Lines:	5836
Relevant Lines:	17206

💛 - Coveralls

…Group

aslesarenko · 2020-06-17T09:53:27Z

sigmastate/src/main/scala/sigmastate/lang/Terms.scala

+        val argsV = args.map(a => a.evalTo[Any](E, env))
+        f(argsV)
+      }
+      // TODO JITC


TO REVIEWER: I added a separate issue #662 to implement all TODO JITC in a separate PR

aslesarenko · 2020-06-17T12:42:18Z

sigmastate/src/main/scala/sigmastate/utxo/JitCostTable.scala

+import sigmastate.lang.SigmaParser
+import sigmastate.lang.Terms.OperationId
+
+case class JitCostTable(operCosts: Map[OperationId, Int]) extends (OperationId => Int) {


To reviewer:
This is version is mostly the same as the old CostTable, but the plan is to make completely new Cost Model which better reflect JIT costs.
The main invariant which will be guaranteed is that always JIT cost < AOT cost for all non spam scripts.
This shouldn't be hard to do since existing op costs are very conservative and new costs may be reduced proportional to new JIT interpreter speedup. (e.g. 2-3x)

aslesarenko · 2020-06-17T12:46:57Z

sigmastate/src/test/scala/sigmastate/JitCostingSpecification.scala

+import sigmastate.utxo.CostTable._
+import special.sigma.{SigmaTestingData, AvlTree}
+
+class JitCostingSpecification extends SigmaTestingData {


To reviewer:
This spec will be further updated (in separate PR as part of implementing #665)
The old CostingSpecification remains unchanged since the AOT version should remain unchanged.

kushti · 2020-06-26T10:55:07Z

sigmastate/src/main/scala/org/ergoplatform/ErgoLikeInterpreter.scala

@@ -3,9 +3,12 @@ package org.ergoplatform
 import sigmastate.SCollection.SByteArray
 import sigmastate.Values._
 import sigmastate.eval.IRContext
-import sigmastate.interpreter.Interpreter
+import sigmastate.interpreter.{Interpreter, ErgoTreeEvaluator}
+import sigmastate.interpreter.Interpreter.{ScriptEnv, ReductionResult}


unused imports added

kushti · 2020-06-26T10:55:36Z

sigmastate/src/main/scala/org/ergoplatform/ErgoLikeInterpreter.scala

 import sigmastate.utxo._

+import scala.util.Try


unused import added

kushti · 2020-06-26T10:59:24Z

sigmastate/src/main/scala/sigmastate/eval/BigIntegerOps.scala

  trait BigIntIsIntegral extends Integral[BigInt] {
    def quot(x: BigInt, y: BigInt): BigInt = x.divide(y)
-    def rem(x: BigInt, y: BigInt): BigInt = x.remainder(y)
+    def rem(x: BigInt, y: BigInt): BigInt = x.mod(y)


Looks like consensus-breaking change (however, mod is more proper operation, so should be added to 4.0).

added comment

kushti · 2020-06-26T11:19:25Z

sigmastate/src/main/scala/sigmastate/interpreter/ErgoTreeEvaluator.scala

+    exp.evalTo[Any](this, env)
+  }
+
+  def addCostOf(opName: String, opType: SFunc) = {


please do ScalaDoc for all the public methods

kushti · 2020-06-26T11:21:13Z

sigmastate/src/main/scala/sigmastate/interpreter/Interpreter.scala

@@ -97,10 +118,21 @@ trait Interpreter extends ScorexLogging {
    (res, currContext.value)
  }

+  // TODO after HF: merge with old version (`applyDeserializeContext`)
+  def applyDeserializeContextJITC(context: CTX, exp: Value[SType]): (SigmaPropValue, CTX) = {


kushti · 2020-06-26T11:21:20Z

sigmastate/src/main/scala/sigmastate/interpreter/Interpreter.scala

@@ -83,6 +97,13 @@ trait Interpreter extends ScorexLogging {
    case x => throw new Error(s"Context-dependent pre-processing should produce tree of type Boolean or SigmaProp but was $x")
  }

+  // TODO after HF: merge with old version (`toValidScriptType`)
+  def toValidScriptTypeJITC(exp: SValue): SigmaPropValue = exp match {


this is helper, made it private.

kushti · 2020-06-26T11:25:28Z

sigmastate/src/main/scala/sigmastate/utxo/JitCostTable.scala

+import sigmastate.lang.SigmaParser
+import sigmastate.lang.Terms.OperationId
+
+case class JitCostTable(operCosts: Map[OperationId, Int]) extends (OperationId => Int) {


class removed

kushti · 2020-06-26T11:28:23Z

sigmastate/src/main/scala/sigmastate/Values.scala

+
+    import sigmastate.utxo.JitCostTable
+
+    def costOf(opName: String, opType: SFunc): Int = {


Please do ScalaDoc for all the public methods

methods removed

# Conflicts: # build.sbt # sigma-api/src/main/scala/special/sigma/SigmaDsl.scala # sigmastate/src/main/scala/org/ergoplatform/ErgoLikeContext.scala # sigmastate/src/main/scala/sigmastate/Values.scala # sigmastate/src/main/scala/sigmastate/eval/BigIntegerOps.scala # sigmastate/src/main/scala/sigmastate/eval/CostingDataContext.scala # sigmastate/src/main/scala/sigmastate/eval/Evaluation.scala # sigmastate/src/main/scala/sigmastate/interpreter/Interpreter.scala # sigmastate/src/main/scala/sigmastate/interpreter/InterpreterContext.scala # sigmastate/src/main/scala/sigmastate/interpreter/ProverInterpreter.scala # sigmastate/src/main/scala/sigmastate/lang/Terms.scala # sigmastate/src/main/scala/sigmastate/serialization/ConstantPlaceholderSerializer.scala # sigmastate/src/main/scala/sigmastate/trees.scala # sigmastate/src/main/scala/sigmastate/types.scala # sigmastate/src/main/scala/sigmastate/utxo/transformers.scala # sigmastate/src/test/scala/sigmastate/helpers/SigmaTestingCommons.scala # sigmastate/src/test/scala/sigmastate/utxo/examples/OracleExamplesSpecification.scala # sigmastate/src/test/scala/special/sigma/SigmaDslTest.scala

aslesarenko · 2020-11-19T11:27:02Z

The work continues in #699, as the approach to costing will be slightly different.
All comments are addressed there.

aslesarenko added 22 commits January 21, 2020 10:53

scalaDoc + perf improvement

a155978

towards simple ErgoTree interpreter

50e9a4e

some first Value.eval methods

ac06d33

first tests pass

f7a5c05

eval for Upcast, Downcast

a85cb6c

eval for EQ, SelectField

58e6875

eval for ArithOp

fac1af6

eval for BlockValue

3ccceee

eval for Global

df2f1c3

eval for DecodePoint, Exponentiate

4043d51

eval for MethodCall, MapCollection

0b71cd1

eval for ByIndex, ExtractAmount, ExtractId, ExtractRegisterAs, LongTo…

83a7888

…ByteArray, ByteArrayToLong, ByteArrayToBigInt, CalcBlake2b256, CalcSha256, SubstConstants

eval for OUTPUTS, Negate, ExtractScriptBytes, XorOf, LogicalNot

f649960

eval for Height, Inputs, Self, Filter, ExtractBytes, ExtractBytesWith…

bb9bf09

…NoRef, ExtractCreationInfo

passing context to func[A,B]

e397daa

eval for Xor, GT, BinXor, Tuple

c4a4bfc

eval for Exists, ForAll, SizeOf

b620262

handle extra descriptors in MethodCall

a618992

eval for Append, Slice, Fold, SigmaPropBytes, OptionGetOrElse, Option…

d126f5d

…IsDefined

eval for BoolToSigmaProp, CreateProveDlog, CreateProveDHTuple, SigmaA…

506a33d

…nd, SigmaOr, OR, AND, AtLeast, BinOr, BinAnd, If

adding javaMethod declarations

65e2e40

code cleanup

7904135

aslesarenko added 7 commits January 31, 2020 17:41

avoid creating isProven at the root

cf0e323

eval for ConcreteCollection, LT, LE, GE

351ac36

eval for Context, NEQ

ce5915a

eval for MinerPubkey, LastBlockUtxoRootHash, GroupGenerator, Multiply…

9e3dec2

…Group

fix substConstants

a86c239

fix tests (removed isProven)

7b1253a

make Value abstract class, make eval protected and use evalTo

023b866

aslesarenko added 2 commits May 21, 2020 17:12

All Transformer nodes documented

ceec472

trySoftForkable documented

ec71a6f

aslesarenko self-assigned this Jun 9, 2020

aslesarenko added 7 commits June 10, 2020 14:20

remove implicit Elem descriptor from substConstants method

1fd5f3f

TODO HF description for contextVar helper method

f69a524

TODO JITC in ErgoLikeContext.scala

95e3fc2

removing type Idn + ScalaDocs

9bbc122

more TODO JITC + ScalaDocs

e41191a

rollback HF changes in reduceToCrypto

c6f74df

separate JitCostTable

a481b31

aslesarenko commented Jun 17, 2020

View reviewed changes

aslesarenko added 2 commits June 17, 2020 13:19

added TODO JITC for transformers.scala

111ee9c

separate JitCostingSpecification.scala

af3e733

aslesarenko commented Jun 17, 2020

View reviewed changes

Merge branch 'regression-tests' into new-eval

c0547f3

kushti requested changes Jun 26, 2020

View reviewed changes

aslesarenko modified the milestones: v3.x, v5.0, v4.0 Sep 10, 2020

aslesarenko added the 40h Approximately 40 hours or work (1 week) label Sep 16, 2020

aslesarenko removed the S-ready-for-merge Status: This PR is ready for merge label Nov 5, 2020

aslesarenko added 2 commits November 5, 2020 14:20

new eval: fixes after merge to make it compile

3c8c760

aslesarenko changed the base branch from develop to i664-todo-coverage November 5, 2020 12:01

aslesarenko closed this Nov 19, 2020

aslesarenko mentioned this pull request Nov 19, 2020

New direct ErgoTree evaluation and JIT costing #699

Closed

18 tasks

aslesarenko deleted the new-eval branch March 12, 2022 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New evaluation and costing #639

New evaluation and costing #639

aslesarenko commented Jan 31, 2020 •

edited

Loading

coveralls commented Jan 31, 2020 •

edited

Loading

aslesarenko Jun 17, 2020 •

edited

Loading

aslesarenko Jun 17, 2020

aslesarenko Jun 17, 2020

kushti Jun 26, 2020

aslesarenko Nov 18, 2020

kushti Jun 26, 2020

aslesarenko Nov 18, 2020

kushti Jun 26, 2020

aslesarenko Nov 18, 2020

kushti Jun 26, 2020

aslesarenko Nov 19, 2020

kushti Jun 26, 2020

aslesarenko Nov 19, 2020

kushti Jun 26, 2020

aslesarenko Nov 19, 2020

kushti Jun 26, 2020

aslesarenko Nov 19, 2020

kushti Jun 26, 2020

aslesarenko Nov 19, 2020

aslesarenko commented Nov 19, 2020


		import sigmastate.utxo.JitCostTable

		def costOf(opName: String, opType: SFunc): Int = {

New evaluation and costing #639

New evaluation and costing #639

Conversation

aslesarenko commented Jan 31, 2020 • edited Loading

Motivation

Implementation

Performance

Milestone

coveralls commented Jan 31, 2020 • edited Loading

Pull Request Test Coverage Report for Build 5222

💛 - Coveralls

aslesarenko Jun 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aslesarenko commented Nov 19, 2020

aslesarenko commented Jan 31, 2020 •

edited

Loading

coveralls commented Jan 31, 2020 •

edited

Loading

aslesarenko Jun 17, 2020 •

edited

Loading