-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nullable bug fixes #34891
Nullable bug fixes #34891
Conversation
@dotnet/roslyn-compiler for review |
elementType = array.ElementTypeWithAnnotations; | ||
array = elementType.TypeKind == TypeKind.Array | ||
? (ArrayTypeSymbol)elementType.Type | ||
: null; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't believe this is faster than array = elementType.Type as ArrayTypeSymbol;
In fact I bet it is slower. #Resolved
// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head | ||
// recursive solution here. Need to take an iterative approach to building up the annotations here. | ||
var builder = ArrayBuilder<ArrayTypeSymbol>.GetInstance(); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is good, but a stack would be clearer, if you were motivated to add a pooled stack. #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. See so many ArrayBuilder
I sometimes forget that we have stacks 😉 #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider using ArrayBuilderExtensions.Push()
and Pop()
instead.
In reply to: 273743425 [](ancestors = 273743425)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. Would prefer a stack here, it would make the second part of this much easier to reason about. #Resolved
builder.Free(); | ||
return array; | ||
} | ||
return WithElementType(ElementTypeWithAnnotations.SetObliviousNullabilityForReferenceTypes()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return [](start = 12, length = 6)
return [](start = 12, length = 6)
nit: newline above. #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to call my PR "Nullable bug fixes". Now I have to think up a new name. |
builder.Free(); | ||
return array; | ||
} | ||
return WithElementType(ElementTypeWithAnnotations.SetObliviousNullabilityForReferenceTypes()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WithElementType [](start = 19, length = 15)
WithElementType [](start = 19, length = 15)
Is this different than the base case in the iterative approach above? In short, do we need to differentiate the case where the top-level element type is not an array? #Resolved
@@ -36,8 +36,7 @@ public void Bug18280() | |||
var arr = x.Type; | |||
|
|||
arr.GetHashCode(); | |||
// https://github.com/dotnet/roslyn/issues/30023: StackOverflowException in SetUnknownNullabilityForReferenceTypes. | |||
//arr.SetUnknownNullabilityForReferenceTypes(); | |||
arr.SetObliviousNullabilityForReferenceTypes(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
arr [](start = 12, length = 3)
arr [](start = 12, length = 3)
Consider checking nullability on each element type, before and after call. #Resolved
@gafter "Nullabler nullable bug fixes" |
} | ||
// Fixup the element type on the most nested array | ||
elementType = elementType.SetObliviousNullabilityForReferenceTypes(); | ||
array = builder.Pop().WithElementType(elementType); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this Pop()
case necessary of would it be handled by the while
loop below? #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it's necessary. The first WithElementType
is a special case because it needs to be the result of TypeWithAnnotations.SetObliviousNullabilityFoReferenceTypes
. This specific type is opaque here, or not array, we don't special case it in any way. The rest of the WithElementType
calls are operating on ArrayTypeSymbol
instances and they have to manually re-create the TypeWithAnnotations
#Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can the fixup be written as:
elementType = elementType.SetObliviousNullabilityForReferenceTypes();
while (builder.Count > 0)
{
array = builder.Pop().WithElementType(elementType);
elementType = TypeWithAnnotations.Create(array,
NullableAnnotation.Oblivious, array.ElementTypeWithAnnotations.CustomModifiers);
}
In reply to: 274030114 [](ancestors = 274030114)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pretty sure that messes up the custom modifiers. Let's say that arrayA has element type arrayB. In this sample the first Pop
returns arrayB and sets elementType
to have the custom modifiers of the original elementType
(outside the loop). What it really needs though is to take the custom modifiers off of the element on arrayA.
In reply to: 274056977 [](ancestors = 274056977,274030114)
do | ||
{ | ||
var next = arrayType.ElementType as ArrayTypeSymbol; | ||
if (next is object) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this check needed? #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. The final element type is int
and this method is iterating over arrays. #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I missed that. In that case, consider having the Action
check x.NullableAnnotation
rather than x.Element.NullableAnnotation
. #Resolved
type.GetHashCode(); | ||
foreachArray(x => Assert.Equal(NullableAnnotation.NotAnnotated, x.ElementTypeWithAnnotations.NullableAnnotation)); | ||
type = type.SetObliviousNullabilityForReferenceTypes(); | ||
foreachArray(x => Assert.Equal(NullableAnnotation.Oblivious, x.ElementTypeWithAnnotations.NullableAnnotation)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor point: It might be simpler to pass in the expected NullableAnnotation
to the local function rather than an Action
. #Resolved
{ | ||
return SetNullabilityForReferenceTypes(s_setUnknownNullability); | ||
} | ||
internal abstract TypeSymbol SetObliviousNullabilityForReferenceTypes(); | ||
|
||
private readonly static Func<TypeWithAnnotations, TypeWithAnnotations> s_setUnknownNullability = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s_setUnknownNullability [](start = 79, length = 23)
s_setUnknownNullability [](start = 79, length = 23)
Is this delegate used anywhere? #Closed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can be deleted. #Resolved
{ | ||
return WithElementType(transform(ElementTypeWithAnnotations)); | ||
// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head | ||
// recursive solution here. Need to take an iterative approach to building up the annotations here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to take an iterative approach to building up the annotations here. [](start = 40, length = 71)
Why are we concerned with a recursion only within arrays, what about a recursion for pointer types or generic types? Also, it looks like we are fine with relying on recursion for AddNullableTransforms
, ApplyNullableTransforms
and MergeNullability
. Why SetObliviousNullabilityForReferenceTypes
is more important? Are we addressing a real problem? #Closed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why SetObliviousNullabilityForReferenceTypes is more important?
The bug filed was for this case. If we want to expand out to the other cases that's fine to.
what about a recursion for pointer types or generic types?
It's the level of nesting that is the concern. The compiler supports nesting levels of ~500 for generics. That's a reasonable depth for recursive solutions. The tests for this array case indicate we support arrays at a depth of 10,000+ which is not suitable for recursion (blows the stack every time). There are no tests I'm aware of for pointers that suggest we support that level of nesting.
#Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The bug filed was for this case. If we want to expand out to the other cases that's fine to.
I think that in this case we should be concerned about any API that takes recursive approach over arrays. If we don't have a targeted test, that doesn't mean we don't have a problem. I personally not convinced that we have to handle 10000+ deep arrays.
In reply to: 274079994 [](ancestors = 274079994)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the original customer bug for the test case. They do indeed have arrays of incredible depth in their code base (see Class1pf3fbcd0.nxt.cs in the attachment for an example). It's to validate their serialization framework (from the look of it).
http://vstfdevdiv:8080/DevDiv2/DevDiv/_workitems/edit/531552 #Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is the original customer bug for the test case. They do indeed have arrays of incredible depth in their code base (see Class1pf3fbcd0.nxt.cs in the attachment for an example). It's to validate their serialization framework (from the look of it).
There are always people that push things to the extreme. If we think we absolutely have to support them, then I think we should be concerned with AddNullableTransforms
, ApplyNullableTransforms
and MergeNullability
as well.
In reply to: 274092328 [](ancestors = 274092328)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (iteration 7)
The PR is now updated to make Note: happy to help explain the style of transform I added to @cston, @AlekseyTs, @gafter, @333fred I dismissed your reviews as this was a substantial change to the PR and warrants a re-review. |
if ((object)other == null || !other.HasSameShapeAs(this) || | ||
!other.ElementTypeWithAnnotations.Equals(ElementTypeWithAnnotations, comparison)) | ||
// We don't want to blow the stack if we have a type like T[][][][][][][][]....[][], | ||
// so we do not recurse until we have a non-array. Rather, hash all the ranks together |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hash [](start = 71, length = 4)
Please adapt this comment to Equals. Perhaps something like
// We don't want to blow the stack if we have a type like T[][][][][][][][]....[][],
// so we do not recurse until we have a non-array. Rather, check all the ranks
// and then check the final "T" type.
``` #Resolved
/// Represents a sequence of <see cref="NullableAnnotation"/> to apply to a type. The sequence of | ||
/// flags is specific to how the individual <see cref="TypeSymbol"/> walk their nested types. | ||
/// </summary> | ||
internal sealed class NullableTransformStream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NullableTransformStream [](start = 26, length = 23)
Perhaps move to its own source file? #Resolved
{ | ||
private static readonly ObjectPool<NullableTransformStream> Pool = new ObjectPool<NullableTransformStream>(() => new NullableTransformStream(), Environment.ProcessorCount); | ||
|
||
private ImmutableArray<byte> _transforms; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
private [](start = 8, length = 7)
readonly #WontFix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
} | ||
|
||
public static NullableTransformStream Create(byte defaultTransform) | ||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// <summary>Returns an infinite stream of the given transform value.</summary>
#Resolved
} | ||
while (true); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Simpler, I think, to use a for loop:
for (var array = this; array != null; array = array.ElementType as ArrayTypeSymbol)
{
builder.Push((array, stream.GetNextTransform()));
}
#Resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or like this:
var array = this;
do
{
builder.Push((array, stream.GetNextTransform()));
array = array.ElementType as ArrayTypeSymbol;
}
while (!(array is null));
In reply to: 275028963 [](ancestors = 275028963)
lastState.array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream), | ||
lastState.transform); | ||
var finalArray = lastState.array.WithElementType(lastElementType); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suspect this would be simpler if you started with the final element type (0 case) rather than the final array (1 case).
var finalType = builder.Peek().array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream);
while (builder.Count > 0)
{
var state = builder.Pop();
var elementType = apply(state.array.ElementTypeWithAnnotations, finalType, state.transform);
finalType = state.array.WithElementType(elementType);
}
builder.Free();
return finalType;
after which apply
can be inlined.
|
||
// First build up the state for each of the ArrayTypeSymbol in the chain of arrays. | ||
var array = this; | ||
builder.Push(array); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
builder [](start = 12, length = 7)
Similar suggestions here. #Resolved
|
||
// First build up the pair of arrays that need to be processed. | ||
var state = (thisArray: this, otherArray: (ArrayTypeSymbol)other); | ||
builder.Push(state); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similarly here. #Resolved
Personally I much prefer the latter iteration (custom iterative algorithm), particularly if simplified as suggested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var finalArray = lastState.array.WithElementType(lastElementType); | ||
|
||
// Unwind the stack and rebuild the array chain. | ||
while (builder.Count > 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
while (builder.Count > 0) [](start = 11, length = 26)
Consider the following refactoring:
TypeSymbol result = builder.Peek().array.ElementTypeWithAnnotations.Type.ApplyNullableTransforms(stream);
do
{
var state = builder.Pop();
var twa = TypeWithAnnotations.Create(result, state.array.ElementTypeWithAnnotations.NullableAnnotation, state.array.ElementTypeWithAnnotations.CustomModifiers);
if (state.transform.HasValue && twa.ApplyNullableTransformShallow(state.transform.GetValueOrDefault()) is TypeWithAnnotations other)
{
twa = other;
}
else
{
stream.SetHasInsufficientData();
}
result = state.array.WithElementType(twa);
}
while (builder.Count != 0);
return result;
``` #Closed
Done with review pass (iteration 16) #Closed |
finalArray = state.thisArray.WithElementType(elementType); | ||
} | ||
|
||
return finalArray; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Notice I have a missing builder.Free
call here. Everyone missed it on the review. I only noticed it as I was applying the refactorings suggested.
This is what worries me about the separate implementations. Subtle issues like this. #ByDesign
Updated based on latest suggestions. In reply to: 482710139 [](ancestors = 482710139) |
state.array.ElementTypeWithAnnotations.NullableAnnotation, | ||
state.array.ElementTypeWithAnnotations.CustomModifiers); | ||
if (state.transform.HasValue && | ||
elementType.ApplyNullableTransformShallow(state.transform.Value) is TypeWithAnnotations other) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Value [](start = 78, length = 5)
Minor: GetVAlueOrDefault()
? To save on the null check that we already done #Closed
var finalArray = builder.Peek().array.ElementType.ApplyNullableTransforms(stream); | ||
do | ||
{ | ||
var state = builder.Pop(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var state [](start = 16, length = 9)
Please consider deconstructing:
var (array, transform) = builder.Pop();
``` #Closed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (iteration 17)
TypeWithAnnotations newElementType; | ||
// The language supports deeply nested arrays, up to 10,000+ instances. This means we can't implemented a head | ||
// recursive solution here. Need to take an iterative approach. | ||
var builder = ArrayBuilder<(ArrayTypeSymbol array, byte? transform)>.GetInstance(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(ArrayTypeSymbol array, byte? transform) [](start = 39, length = 40)
I thought the latest guidance was to use leading uppercase for tuple element names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Latest from CoreFX. Our code base hasn't adjusted yet. I'm using the existing style. Will fix it all up in one go once everything is stetled.
|
||
// First build up the pair of arrays that need to be processed. | ||
for ( | ||
(ArrayTypeSymbol array, ArrayTypeSymbol other) it = (this, (ArrayTypeSymbol)other); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(ArrayTypeSymbol array, ArrayTypeSymbol other) [](start = 16, length = 46)
Leading uppercase for tuple element names.
var finalArray = state.thisArray.ElementType.MergeNullability(state.otherArray.ElementType, variance); | ||
do | ||
{ | ||
state = builder.Pop(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
state [](start = 16, length = 5)
Consider deconstructing. #Closed
var stream = NullableTransformStream.GetInstance(flagsBuilder.ToImmutableAndFree()); | ||
var result = resultType.ApplyNullableTransforms(stream); | ||
Debug.Assert(stream.IsComplete); | ||
resultType = stream.IsComplete ? result : resultType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
r [](start = 12, length = 1)
stream.Free();
return result; | ||
} | ||
|
||
// No NullableAttribute applied to the target symbol, or flags do not line-up, return unchanged metadataType. | ||
stream.Free(); | ||
return metadataType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider reordering to free in a single code path.
if (!stream.IsComplete) result = metadataType;
stream.Free();
return result;
/// Returns whether or not there is data remaining in the series. When a single flag is used this | ||
/// will always return false. | ||
/// </summary> | ||
public bool HasUnusedTransforms => !IsDefault && _positionOrDefault >= 0 && _positionOrDefault < _transforms.Length; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
public [](start = 8, length = 6)
private
|
||
public void Free() | ||
{ | ||
_transforms = default; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_transforms = default; [](start = 12, length = 22)
Consider resetting _positionOrDefault
as well.
Is this PR still active? |
This comprises the following changes:
SetUnknownNullability*
toSetObliviousNullability*
SetObliviousNullabilityForReferenceTypes
(StackOverflowException in SetUnknownNullabilityForReferenceTypes. #30023).