feat: break out smoke test goldens into directories based on flutter version

[btrautmann]

Description

The goal of this PR is to allow us to maintain different versions of golden files depending on the Flutter version being used to run the tests. This is a necessity to allow for things like #123 which adds a matrix workflow to run Alchemist's smoke tests against a variety of Flutter versions. A couple smoke tests begin failing on Flutter 3.22.x with changes that seem to be outside of Alchemist's control. Using the code in this PR, we can generate new goldens just for that Flutter version and the workflow can continue to run without notifying of new failures (unless something else changes).

Note: Right now, we have one golden test that does not use Alchemist that sits outside of test/smoke_tests which is not supported by this code; we need to figure out a solution for this. @jeroen-meijer I'm curious if you recall why we didn't use Alchemist for that test (I'm asking this before I have looked deeply at it, so it may be that I realize the answer before you get back to me... If that's the case I'll edit below 😄 ).
Edit: I realized after the fact that the diff here was due to a change in Scaffold background color. I set one explicitly and now all versions are passing (with re-generated goldens for 3.16.x) for all versions 🎉

Note: These changes are isolated to flutter_test_config and therefore should not impact consumers.

Type of Change

• ✨ New feature (non-breaking change which adds functionality)
• 🛠️ Bug fix (non-breaking change which fixes an issue)
• ❌ Breaking change (fix or feature that would cause existing functionality to change)
• 🧹 Code refactor
• ✅ Build configuration change
• 📝 Documentation
• 🗑️ Chore

[jeroen-meijer]

Great work! Two quick Qs:

1. So if I understand correctly, does this mean that Alchemist will now separate golden files based on the Flutter version by default? If so, that's a breaking change we should communicate in the docs, and I'm not sure this use-case applies to all consumers of the lib.
2. I saw you passed an environment variable into the Flutter test call in the CI config. Is this required for all users from now? And, can't we find out the Flutter version in runtime and do it that way to omit having to pass it in manually?

[btrautmann]

1. So if I understand correctly, does this mean that Alchemist will now separate golden files based on the Flutter version by default? If so, that's a breaking change we should communicate in the docs, and I'm not sure this use-case applies to all consumers of the lib.

So, I'm intentionally not adding this as a feature of Alchemist, but rather a "feature" of our test suite. This means that it won't change anything about how consumers use Alchemist (though they could totally do this themselves! I imagine the use case where this would be powerful is in a UI SDK development...You may want to ensure your SDK looks the same when built against a variety of Flutter versions). For normal "app" development, this doesn't seem as useful because (most likely) you are targeting/consuming a single Flutter version. The primary hook that is utilized here is Alchemist's FilePathResolver which made it pretty easy to do this type of file path manipulation.

2. I saw you passed an environment variable into the Flutter test call in the CI config. Is this required for all users from now? And, can't we find out the Flutter version in runtime and do it that way to omit having to pass it in manually?

As per the above, consumers don't need to pass this, it's just for us. And yeah, I was thinking the same RE: grabbing this at runtime. AFAIK the only way to do this would be to do it via Process.run. Something like this does the job:

flutter --version | head -n 1 | awk '{print $2}'

I'll give that a try in the next few commits.

Right now, I'm trying to see if I can "fix" the diff I'm seeing on flutter 3.22.x by specifying an explicit set of colors for the ElevatedButton in interactions_smoke_test. Our smoke tests are sensitive to the Material color token changes which isn't ideal! That's actually what made me want to do this, so that if there just so happens to be a fundamental non-backwards-compatible change introduced by Flutter, we can simply generate goldens for that version and keep on chugging.

Let me know if you have any thoughts on ANY of the above @jeroen-meijer! Really value your input.

[btrautmann]

We'll analyze against the lowest version we support. This means that contributors should be using that version (fvm is helpful here). We can/should probably add something that details that in our CONTRIBUTING.md.

[btrautmann]

Test against all supported versions of flutter, up to latest stable version. This will take some manual work to keep this up to date; I'll noodle on how to make this more dynamic as a follow-up.

[btrautmann]

Gonna look at allowing us to drop this env variable and instead grab it via Process.run. This will ensure that the version Alchemist uses for golden comparison is guaranteed to match the actual running version.

[jeroen-meijer]

Process.run sounds good to me, but be aware that:

1. If people run fvm, simply running a flutter command might give incorrect output.
2. Instead of doing some string voodoo, you could do

import 'dart:convert';

final output = Process.run('flutter --version --machine');
final flutterData = json.decode(output);
final flutterVersion = flutterData['flutterVersion'];

[btrautmann]

Ohh, that's way better haha. I just pushed a commit that uses that; if everything succeeds, I think this is good to merge now 👀

Also, 🎉 good news. I was able to explicitly set button colors in the interactions test which resolved the breaking changes that we were inheriting from flutter 3.22.x, meaning we don't need a separate directory for that version, everything is passing on 3.16.x and beyond 🎉

[jeroen-meijer]

Awesome, thanks for the explanation @btrautmann. Nice work! Seems like it's working wonderfully 😄
Lovely to see all those versions crunching numbers!

[jeroen-meijer]

🎉

[btrautmann]

@jeroen-meijer as mentioned in #126 (comment), this is now ready I think 😄

I may make some updates to the contributing guide in a different PR!

[jeroen-meijer]

@jeroen-meijer as mentioned in #126 (comment), this is now ready I think 😄

I may make some updates to the contributing guide in a different PR!

Send it~