Default SizeRange too large

[nipunn1313]

Hi proptest community. I'm finding that the default size range of 0..100 is too large, and is ending up broadly taking up most of the time in our testsuite (generating collections).

Would proptest be amenable to making this default range something that's adjustable? I would probably set it to 10 or 20. Perhaps by environment variable. It would be nice to avoid littering the codebase with strategy overrides for basic Vec.

https://docs.rs/proptest/latest/proptest/collection/struct.SizeRange.html

I am happy to submit a PR.
I'm also open to other solution ideas for this problem.

Thanks! Big fan

[rex-remind101]

I have no objections to configuring this via the environment. @matthew-russo @cameron1024 thoughts?

[matthew-russo]

Sorry for the delay in getting to this -- do you have an example of whats problematic?

generating Vecs of specific lengths can just be done with

proptest::collections::vec(any::<T>(), 0..10)

or with imports it cuts down to

vec(any::<T>(), 0..10)

which is pretty terse. the range could be extracted out in to a const so its not hardcoded everywhere as well.

i get the feeling i'm not understanding part of this though

[nipunn1313]

Yep that's exactly the workaround to override default settings.

I'll give an example from our codebase.

#[derive(Debug, Arbitrary)]
struct TestQuery {
    #[proptest(strategy = "prop::collection::vec(any::<TestValue>(), 0..4)")]
    search: Vec<TestValue>,
    filter: Option<TestValue>,
}

The default settings for us routinely lead to tests that spend majority of time in test-case-generation of large Vecs. I've seen such changes reduce individual test time from 20s to <1s w/ minimal impact to test quality (vec of 10 is plenty).

We tend to use the default settings most of the time and override them when they are non-performant. It is particularly likely to become non-performant in a nested collection scenario (eg if the TestValue struct also contains a vec in the example above).

When I grep our codebase (100k loc) for this pattern, I found 32 instances of this sort of pattern, mostly from us whack-a-moling tests that are too slow with default settings. Many more don't use this pattern. (I count ~150 #[derive(Arbitrary)] in the codebase).

@matthew-russo is certainly right - we could do a pass over the codebase and train everyone (or build a sophisticated linter) to stick a #[proptest(strategy = "prop::collection::vec(any::<T>(), 0..4)")] or similar in every spot, but there's gotta be a better way. Perhaps you might have a better idea? I am curious.

Thanks for the work on maintaining the project! We're fans of randomized testing.