Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nullable bug fixes #34891

Closed
wants to merge 18 commits into from
Closed

Nullable bug fixes #34891

wants to merge 18 commits into from

Conversation

jaredpar
Copy link
Member

@jaredpar jaredpar commented Apr 9, 2019

This comprises the following changes:

  1. Renames SetUnknownNullability* to SetObliviousNullability*
  2. Fixes a stack overflow in SetObliviousNullabilityForReferenceTypes (StackOverflowException in SetUnknownNullabilityForReferenceTypes. #30023).
  3. Completes the duplication of Follow up on comments in ImplicitConversions_07 unit-test #29898 to Update methods from initial binding based on inferred nullability #29605 by updating a comment

@jaredpar jaredpar requested a review from a team as a code owner April 9, 2019 23:12
@jaredpar
Copy link
Member Author

jaredpar commented Apr 9, 2019

@dotnet/roslyn-compiler for review

elementType = array.ElementTypeWithAnnotations;
array = elementType.TypeKind == TypeKind.Array
? (ArrayTypeSymbol)elementType.Type
: null;
Copy link
Member

@gafter gafter Apr 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe this is faster than array = elementType.Type as ArrayTypeSymbol;
In fact I bet it is slower. #Resolved

// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head
// recursive solution here. Need to take an iterative approach to building up the annotations here.
var builder = ArrayBuilder<ArrayTypeSymbol>.GetInstance();

Copy link
Member

@gafter gafter Apr 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good, but a stack would be clearer, if you were motivated to add a pooled stack. #Resolved

Copy link
Member Author

@jaredpar jaredpar Apr 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. See so many ArrayBuilder I sometimes forget that we have stacks 😉 #Resolved

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using ArrayBuilderExtensions.Push() and Pop() instead.


In reply to: 273743425 [](ancestors = 273743425)

Copy link
Member

@333fred 333fred Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Would prefer a stack here, it would make the second part of this much easier to reason about. #Resolved

builder.Free();
return array;
}
return WithElementType(ElementTypeWithAnnotations.SetObliviousNullabilityForReferenceTypes());
Copy link
Member

@gafter gafter Apr 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return [](start = 12, length = 6)

return [](start = 12, length = 6)

nit: newline above. #Resolved

gafter
gafter previously approved these changes Apr 9, 2019
Copy link
Member

@gafter gafter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gafter
Copy link
Member

gafter commented Apr 9, 2019

I wanted to call my PR "Nullable bug fixes". Now I have to think up a new name.

builder.Free();
return array;
}
return WithElementType(ElementTypeWithAnnotations.SetObliviousNullabilityForReferenceTypes());
Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WithElementType [](start = 19, length = 15)

WithElementType [](start = 19, length = 15)

Is this different than the base case in the iterative approach above? In short, do we need to differentiate the case where the top-level element type is not an array? #Resolved

@@ -36,8 +36,7 @@ public void Bug18280()
var arr = x.Type;

arr.GetHashCode();
// https://github.com/dotnet/roslyn/issues/30023: StackOverflowException in SetUnknownNullabilityForReferenceTypes.
//arr.SetUnknownNullabilityForReferenceTypes();
arr.SetObliviousNullabilityForReferenceTypes();
Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arr [](start = 12, length = 3)

arr [](start = 12, length = 3)

Consider checking nullability on each element type, before and after call. #Resolved

@333fred
Copy link
Member

333fred commented Apr 10, 2019

I wanted to call my PR "Nullable bug fixes". Now I have to think up a new name.

@gafter "Nullabler nullable bug fixes"

@jaredpar
Copy link
Member Author

@gafter, @cston thanks, updated based on your feedback.

}
// Fixup the element type on the most nested array
elementType = elementType.SetObliviousNullabilityForReferenceTypes();
array = builder.Pop().WithElementType(elementType);
Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this Pop() case necessary of would it be handled by the while loop below? #Resolved

Copy link
Member Author

@jaredpar jaredpar Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it's necessary. The first WithElementType is a special case because it needs to be the result of TypeWithAnnotations.SetObliviousNullabilityFoReferenceTypes. This specific type is opaque here, or not array, we don't special case it in any way. The rest of the WithElementType calls are operating on ArrayTypeSymbol instances and they have to manually re-create the TypeWithAnnotations
#Resolved

Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the fixup be written as:

elementType = elementType.SetObliviousNullabilityForReferenceTypes();
while (builder.Count > 0)
{
    array = builder.Pop().WithElementType(elementType);
    elementType = TypeWithAnnotations.Create(array, 
        NullableAnnotation.Oblivious, array.ElementTypeWithAnnotations.CustomModifiers);
}

In reply to: 274030114 [](ancestors = 274030114)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure that messes up the custom modifiers. Let's say that arrayA has element type arrayB. In this sample the first Pop returns arrayB and sets elementType to have the custom modifiers of the original elementType (outside the loop). What it really needs though is to take the custom modifiers off of the element on arrayA.


In reply to: 274056977 [](ancestors = 274056977,274030114)

do
{
var next = arrayType.ElementType as ArrayTypeSymbol;
if (next is object)
Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this check needed? #Resolved

Copy link
Member Author

@jaredpar jaredpar Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The final element type is int and this method is iterating over arrays. #Resolved

Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I missed that. In that case, consider having the Action check x.NullableAnnotation rather than x.Element.NullableAnnotation. #Resolved

type.GetHashCode();
foreachArray(x => Assert.Equal(NullableAnnotation.NotAnnotated, x.ElementTypeWithAnnotations.NullableAnnotation));
type = type.SetObliviousNullabilityForReferenceTypes();
foreachArray(x => Assert.Equal(NullableAnnotation.Oblivious, x.ElementTypeWithAnnotations.NullableAnnotation));
Copy link
Member

@cston cston Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor point: It might be simpler to pass in the expected NullableAnnotation to the local function rather than an Action. #Resolved

cston
cston previously approved these changes Apr 10, 2019
333fred
333fred previously approved these changes Apr 10, 2019
{
return SetNullabilityForReferenceTypes(s_setUnknownNullability);
}
internal abstract TypeSymbol SetObliviousNullabilityForReferenceTypes();

private readonly static Func<TypeWithAnnotations, TypeWithAnnotations> s_setUnknownNullability =
Copy link
Contributor

@AlekseyTs AlekseyTs Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s_setUnknownNullability [](start = 79, length = 23)

s_setUnknownNullability [](start = 79, length = 23)

Is this delegate used anywhere? #Closed

Copy link
Member Author

@jaredpar jaredpar Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be deleted. #Resolved

{
return WithElementType(transform(ElementTypeWithAnnotations));
// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head
// recursive solution here. Need to take an iterative approach to building up the annotations here.
Copy link
Contributor

@AlekseyTs AlekseyTs Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to take an iterative approach to building up the annotations here. [](start = 40, length = 71)

Why are we concerned with a recursion only within arrays, what about a recursion for pointer types or generic types? Also, it looks like we are fine with relying on recursion for AddNullableTransforms, ApplyNullableTransforms and MergeNullability. Why SetObliviousNullabilityForReferenceTypes is more important? Are we addressing a real problem? #Closed

Copy link
Member Author

@jaredpar jaredpar Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why SetObliviousNullabilityForReferenceTypes is more important?

The bug filed was for this case. If we want to expand out to the other cases that's fine to.

what about a recursion for pointer types or generic types?

It's the level of nesting that is the concern. The compiler supports nesting levels of ~500 for generics. That's a reasonable depth for recursive solutions. The tests for this array case indicate we support arrays at a depth of 10,000+ which is not suitable for recursion (blows the stack every time). There are no tests I'm aware of for pointers that suggest we support that level of nesting.

#Resolved

Copy link
Contributor

@AlekseyTs AlekseyTs Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bug filed was for this case. If we want to expand out to the other cases that's fine to.

I think that in this case we should be concerned about any API that takes recursive approach over arrays. If we don't have a targeted test, that doesn't mean we don't have a problem. I personally not convinced that we have to handle 10000+ deep arrays.


In reply to: 274079994 [](ancestors = 274079994)

Copy link
Member Author

@jaredpar jaredpar Apr 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the original customer bug for the test case. They do indeed have arrays of incredible depth in their code base (see Class1pf3fbcd0.nxt.cs in the attachment for an example). It's to validate their serialization framework (from the look of it).

http://vstfdevdiv:8080/DevDiv2/DevDiv/_workitems/edit/531552 #Resolved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the original customer bug for the test case. They do indeed have arrays of incredible depth in their code base (see Class1pf3fbcd0.nxt.cs in the attachment for an example). It's to validate their serialization framework (from the look of it).

There are always people that push things to the extreme. If we think we absolutely have to support them, then I think we should be concerned with AddNullableTransforms, ApplyNullableTransforms and MergeNullability as well.


In reply to: 274092328 [](ancestors = 274092328)

AlekseyTs
AlekseyTs previously approved these changes Apr 10, 2019
Copy link
Contributor

@AlekseyTs AlekseyTs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (iteration 7)

@jaredpar jaredpar dismissed stale reviews from AlekseyTs, 333fred, cston, and gafter April 11, 2019 18:22

I changed a bunch

@jaredpar
Copy link
Member Author

The PR is now updated to make ApplyNullableTransforms, AddNullableTransforms, Equals and MergeNullability all iterative.

Note: happy to help explain the style of transform I added to ArrayTypeSymbol here. This type of transform, bears a resemblance to CSP, is necessary in order to make this process iterative. Don't believe there is a good way to simplify this without lots of duplication. If this makes us re-evaluate if we want to support this scenario then that might be a good discussion to have.

@cston, @AlekseyTs, @gafter, @333fred I dismissed your reviews as this was a substantial change to the PR and warrants a re-review.

if ((object)other == null || !other.HasSameShapeAs(this) ||
!other.ElementTypeWithAnnotations.Equals(ElementTypeWithAnnotations, comparison))
// We don't want to blow the stack if we have a type like T[][][][][][][][]....[][],
// so we do not recurse until we have a non-array. Rather, hash all the ranks together
Copy link
Member

@gafter gafter Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hash [](start = 71, length = 4)

Please adapt this comment to Equals. Perhaps something like

            // We don't want to blow the stack if we have a type like T[][][][][][][][]....[][],
            // so we do not recurse until we have a non-array. Rather, check all the ranks
            // and then check the final "T" type.
``` #Resolved

/// Represents a sequence of <see cref="NullableAnnotation"/> to apply to a type. The sequence of
/// flags is specific to how the individual <see cref="TypeSymbol"/> walk their nested types.
/// </summary>
internal sealed class NullableTransformStream
Copy link
Member

@gafter gafter Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NullableTransformStream [](start = 26, length = 23)

Perhaps move to its own source file? #Resolved

{
private static readonly ObjectPool<NullableTransformStream> Pool = new ObjectPool<NullableTransformStream>(() => new NullableTransformStream(), Environment.ProcessorCount);

private ImmutableArray<byte> _transforms;
Copy link
Member

@gafter gafter Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private [](start = 8, length = 7)

readonly #WontFix

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be mutated as it's a pooled object.


In reply to: 274613981 [](ancestors = 274613981)

}

public static NullableTransformStream Create(byte defaultTransform)
{
Copy link
Member

@gafter gafter Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/// <summary>Returns an infinite stream of the given transform value.</summary> #Resolved

}
while (true);

Copy link
Member

@gafter gafter Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simpler, I think, to use a for loop:

for (var array = this; array != null; array = array.ElementType as ArrayTypeSymbol)
{
    builder.Push((array, stream.GetNextTransform()));
}

#Resolved

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or like this:

            var array = this;
            do
            {
                builder.Push((array, stream.GetNextTransform()));
                array = array.ElementType as ArrayTypeSymbol;
            }
            while (!(array is null));

In reply to: 275028963 [](ancestors = 275028963)

lastState.array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream),
lastState.transform);
var finalArray = lastState.array.WithElementType(lastElementType);

Copy link
Member

@gafter gafter Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this would be simpler if you started with the final element type (0 case) rather than the final array (1 case).

            var finalType = builder.Peek().array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream);
            while (builder.Count > 0)
            {
                var state = builder.Pop();
                var elementType = apply(state.array.ElementTypeWithAnnotations, finalType, state.transform);
                finalType = state.array.WithElementType(elementType);
            }

            builder.Free();
            return finalType;

after which apply can be inlined.


// First build up the state for each of the ArrayTypeSymbol in the chain of arrays.
var array = this;
builder.Push(array);
Copy link
Member

@gafter gafter Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builder [](start = 12, length = 7)

Similar suggestions here. #Resolved


// First build up the pair of arrays that need to be processed.
var state = (thisArray: this, otherArray: (ArrayTypeSymbol)other);
builder.Push(state);
Copy link
Member

@gafter gafter Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly here. #Resolved

@gafter
Copy link
Member

gafter commented Apr 12, 2019

Personally I much prefer the latter iteration (custom iterative algorithm), particularly if simplified as suggested.

Copy link
Member

@gafter gafter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

var finalArray = lastState.array.WithElementType(lastElementType);

// Unwind the stack and rebuild the array chain.
while (builder.Count > 0)
Copy link
Contributor

@AlekseyTs AlekseyTs Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while (builder.Count > 0) [](start = 11, length = 26)

Consider the following refactoring:

            TypeSymbol result = builder.Peek().array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream);

            do
            {
                var state = builder.Pop();

                var twa = TypeWithAnnotations.Create(result, state.array.ElementTypeWithAnnotations.NullableAnnotation, state.array.ElementTypeWithAnnotations.CustomModifiers);
                if (state.transform.HasValue && twa.ApplyNullableTransformShallow(state.transform.GetValueOrDefault()) is TypeWithAnnotations other)
                {
                    twa = other;
                }
                else
                {
                    stream.SetHasInsufficientData();
                }

                result = state.array.WithElementType(twa);
            }
            while (builder.Count != 0);

            return result;
``` #Closed

@AlekseyTs
Copy link
Contributor

AlekseyTs commented Apr 12, 2019

Done with review pass (iteration 16) #Closed

finalArray = state.thisArray.WithElementType(elementType);
}

return finalArray;
}
Copy link
Member Author

@jaredpar jaredpar Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice I have a missing builder.Free call here. Everyone missed it on the review. I only noticed it as I was applying the refactorings suggested.

This is what worries me about the separate implementations. Subtle issues like this. #ByDesign

@jaredpar
Copy link
Member Author

Updated based on latest suggestions.


In reply to: 482710139 [](ancestors = 482710139)

state.array.ElementTypeWithAnnotations.NullableAnnotation,
state.array.ElementTypeWithAnnotations.CustomModifiers);
if (state.transform.HasValue &&
elementType.ApplyNullableTransformShallow(state.transform.Value) is TypeWithAnnotations other)
Copy link
Contributor

@AlekseyTs AlekseyTs Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Value [](start = 78, length = 5)

Minor: GetVAlueOrDefault()? To save on the null check that we already done #Closed

var finalArray = builder.Peek().array.ElementType.ApplyNullableTransforms(stream);
do
{
var state = builder.Pop();
Copy link
Member

@cston cston Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var state [](start = 16, length = 9)

Please consider deconstructing:

var (array, transform) = builder.Pop();
``` #Closed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hadn' tthought o f that.


In reply to: 275071261 [](ancestors = 275071261)

Copy link
Contributor

@AlekseyTs AlekseyTs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (iteration 17)

TypeWithAnnotations newElementType;
// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head
// recursive solution here. Need to take an iterative approach.
var builder = ArrayBuilder<(ArrayTypeSymbol array, byte? transform)>.GetInstance();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ArrayTypeSymbol array, byte? transform) [](start = 39, length = 40)

I thought the latest guidance was to use leading uppercase for tuple element names.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest from CoreFX. Our code base hasn't adjusted yet. I'm using the existing style. Will fix it all up in one go once everything is stetled.


// First build up the pair of arrays that need to be processed.
for (
(ArrayTypeSymbol array, ArrayTypeSymbol other) it = (this, (ArrayTypeSymbol)other);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ArrayTypeSymbol array, ArrayTypeSymbol other) [](start = 16, length = 46)

Leading uppercase for tuple element names.

var finalArray = state.thisArray.ElementType.MergeNullability(state.otherArray.ElementType, variance);
do
{
state = builder.Pop();
Copy link
Member

@cston cston Apr 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

state [](start = 16, length = 5)

Consider deconstructing. #Closed

var stream = NullableTransformStream.GetInstance(flagsBuilder.ToImmutableAndFree());
var result = resultType.ApplyNullableTransforms(stream);
Debug.Assert(stream.IsComplete);
resultType = stream.IsComplete ? result : resultType;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r [](start = 12, length = 1)

stream.Free();

return result;
}

// No NullableAttribute applied to the target symbol, or flags do not line-up, return unchanged metadataType.
stream.Free();
return metadataType;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider reordering to free in a single code path.

if (!stream.IsComplete) result = metadataType;
stream.Free();
return result;

/// Returns whether or not there is data remaining in the series. When a single flag is used this
/// will always return false.
/// </summary>
public bool HasUnusedTransforms => !IsDefault && _positionOrDefault >= 0 && _positionOrDefault < _transforms.Length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public [](start = 8, length = 6)

private


public void Free()
{
_transforms = default;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_transforms = default; [](start = 12, length = 22)

Consider resetting _positionOrDefault as well.

@jcouv jcouv added this to the 16.2 milestone Apr 23, 2019
@RikkiGibson
Copy link
Contributor

Is this PR still active?

@jcouv jcouv added the PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it. label Jun 25, 2019
@jinujoseph jinujoseph modified the milestones: 16.2, Backlog Aug 5, 2019
@jaredpar jaredpar closed this Nov 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Compilers PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants