Convert proto output from internal string encoding to Unicode

[fmeum]

Work towards #374

[fmeum]

@bazel-io fork 8.1.0

[aiuto]

Fabian: Can you help me understand the subtle change of this PR.

It looks to me like:

• protobuf builder setXXX() methods expect a normal Java String
• bazel internal strings are actually UTF-8 representation under the guise of a String
• Thus, previous behavior was for setXXX() to whomp down the bytes an convert each octet to UTF-8. This is essentially double encoding
• The fix is to decode the Bazel secret representation to Unicode as UTF-16 (or whatever Java uses), so that the protobuf setter sees the kind thing it expects and makes it UTF-8 on the wire.

So essentially equivalent to "copy the bytes without interpreting them", which is what Bazel does elsewhere like in ctx.actions.write()

If that's what you intend, then LGTM.

[fmeum]

That's exactly correct!

The StringEncoding class has a long comment about the different types of string encodings relevant with Bazel (turns out there are three). I added these central conversion methods to avoid a comment on each use site.

[tjgq]

I think this conversion is incorrect, since all callsites already perform the conversion to Unicode (also for the other three below)?

If you agree, I will remove it, and also add some integration tests in bazel_query_test.sh when I import.

[fmeum]

I think you are right, but I would personally drop the internalToUnicode calls from the callsites rather than this function as addStringListValue etc. are the actual sinks for valid Unicode. I don't have a strong opinion though, feel free to do what you think is easier to understand

Thanks for taking care of the tests!

[tjgq]

While I generally agree with the principle of putting the conversion as close to the sink as possible, in this case it would cause writeAttributeValueToBuilder to end up with mixed application/non-application of internalToUnicode, as not every proto field is set through the AttributeValueBuilderAdapter interface. That seems less readable to me, because it's harder to convince oneself that all fields have been covered. (It might be possible to do better with additional indirection, but I'd rather not do that now.)