add TransformBatch for back-to-back Transforms

[sequencer]

Contributor Checklist

• Did you add Scaladoc to every public function/method?
• Did you update the FIRRTL spec to include every new feature/behavior?
• Did you add at least one test demonstrating the PR?
• Did you delete any extraneous printlns/debugging code?
• Did you specify the type of improvement?
• Did you state the API impact?
• Did you specify the code generation impact?
• Did you request a desired merge strategy?
• Did you add text to be included in the Release Notes for this change?

Type of Improvement

• new feature/API

API Impact

• deprecate SeqTransfrom
• add TransformBatch

Backend Code Generation Impact

No impact

Desired Merge Strategy

• Squash: The PR will be squashed and merged (choose this if you have no preference.

Release Notes

It support add multi transforms wrapped in a single Transform like SeqTransform does, but:
- Automatically generate prerequisites(The sum of all transforms)
- Automatically adding necessary dependents and prerequisite to allTransforms.

With this implementation, back-to-back transform can be easily implemented like this:

class SomeTransform extends TransformBatch {
  def transforms = Seq(
    new TransformA,
    new TransformC,
    new TransformD
  )
}

Reviewer Checklist (only modified by reviewer)

• Did you add the appropriate labels?
• Did you mark the proper milestone (1.2.x, 1.3.0, 1.4.0) ?
• Did you review?
• Did you check whether all relevant Contributor checkboxes have been checked?
• Did you mark as Please Merge?

[seldridge]

I can see a need for a DependencyManager where the prerequisites are derived from the targets directly as opposed to being set by currentState. This makes a lot of sense to me and this PR pushes towards that.

I don't think that the default definitions of dependencies for DependencyManager are correct. Perhaps correcting the way those are derived from the targets would do exactly what this PR is trying to do.

I do think there are a couple of rough edges that need to be discussed:

1. TransformBatch redefines the meaning of dependents, but only for itself. This may introduce weird behavior. It may be better to have an expandDependents(a: Seq[Transform]): Seq[Transform] that can be used to expand some targets into targets and dependents. Example usage would be something like: TransformManager(targets = expandDependents(targets)).
2. dependents are only added one-level deep for TransformBatch. Why does it stop there as opposed to recursively expanding dependents?
3. Prerequisites are transitive. Meaning: it shouldn't be necessary to recursively expand prerequisites when setting targets. I'm not following the logic for invalidates in the PR, so the recursive expansion may be something that's needed here.
4. Getting the behavior right between prereqs, dependents (mandatory dependents here), and invalidates while also making sure that the resulting solution is something close to optimal is really, really hard to get right... This motivates tons of tests, but I'd like to sort out if the existing TransformManager is sufficient to describe what this PR does before asking you to invest a bunch of time writing tests.

[sequencer]

Although I'm already dizzy now.(stay up for tester2), I can remember the reason why I didn't directly use DependencyManager is that it needs currentState. And I don't think that's a good idea. So in order to not breaking current API, I choose to add a new trait.

[sequencer]

TransformBatch redefines the meaning of dependents, but only for itself. This may introduce weird behavior.

Yes, I have removed the usage of dependents in targets generation, I think all optionalPrerequisiteOf should be added manually now.

It may be better to have an expandDependents(a: Seq[Transform]): Seq[Transform] that can be used to expand some targets into targets and dependents. Example usage would be something like: TransformManager(targets = expandDependents(targets)).

Do you think expandDependents should be added by this PR? I think it could be a helper under DependencyAPI.

dependents are only added one-level deep for TransformBatch. Why does it stop there as opposed to recursively expanding dependents?

In my current implementation, optionalPrerequisiteOf should be added by user, not automatically generated.

Prerequisites are transitive. Meaning: it shouldn't be necessary to recursively expand prerequisites when setting targets. I'm not following the logic for invalidates in the PR, so the recursive expansion may be something that's needed here.

After I read detail of dependencyAPI I think that is my mistake. I'm working on a new version of this PR.

[seldridge]

I think this notion of a transform as solving a DependencyManager sub-problem makes sense. The DependencyManager is recursively solving sub-problems to handle re-lowering. It's natural that users would want to use this pattern.

Fortunately, I think what you're trying to do can be more tersely expressed with something like:

import firrtl.stage.transforms.Compiler

class SubCompiler(
  targets: Seq[TransformDependency],
  knownObjects: Set[Transform] = Set.empty)
    extends Compiler(
  targets=targets,
  currentState=targets.flatMap(_.getObject.prerequisites),
  knownObjects=knownObjects)

This is basically saying, "Figure out how to run these transforms. Their prerequisites are already satisfied."

Mandatory dependents, while not something I'm in favor of, can be expressed by recursively expanding the dependents of your targets:

class DependentsCompiler(
  targets: Seq[TransformDependency],
  knownObjects: Set[Transform] = Set.empty)
    extends Compiler(
  targets = targets ++ expandDependents(targets),
  currentState = targets.flatMap(_.getObject.prerequisites),
  knownObjects = knownObjects)

Granted, if you have Foo with dependents Bar and Baz, I'd prefer to treat the subproblem as targets=Seq(Foo, Bar, Baz) and not targets=Seq(Foo). Nonetheless, a user is entirely free to reinterpret dependents this way. A transform doing this would likely not work outside of it's batch, but that can be worked around by making the transform package private.