Grouping and other changes #273
Conversation
kbattocchi
force-pushed
the
kebatt/robust
branch
3 times, most recently
from
1e9b728 to
b57fe03
Compare
kbattocchi
force-pushed
the
kebatt/robust
branch
from
6d2a999 to
cb53b5c
Compare
kbattocchi
force-pushed
the
kebatt/robust
branch
2 times, most recently
from
1a07974 to
9d29e1b
Compare
| self.z_transformer = None | ||
|
|
||
| if self._n_splits == 1: # special case, no cross validation | ||
| folds = None | ||
| else: | ||
| splitter = check_cv(self._n_splits, [0], classifier=stratify) | ||
| # if check_cv produced a new KFold or StratifiedKFold object, we need to set shuffle and random_state | ||
| # TODO: ideally, we'd also infer whether we need a GroupKFold (if groups are passed) |
There was a problem hiding this comment.
Grouping is more important than stratification for valid inference. So I would prioritize grouping over stratification here, i.e. if groups are enabled then use groupkfold. If not then use stratified if strata is not None else kfold.
Also we should most prob be raising a warning that "cross fitting performed without treatment stratification because grouping was enabled."
Ultimately I feel we should just add our own stratified group kfold that stratifies within each group, so that we really deliver the full version of our API.
There was a problem hiding this comment.
With small sample sizes, failure to stratify can cause first stage model prediction to fail if no examples from one strata make it into a training fold. I agree, though, that we ought to have a mechanism that supports both simultaneously; there is work in progress to add such a feature to sklearn natively.
kbattocchi
force-pushed
the
kebatt/robust
branch
2 times, most recently
from
3e890fd to
a81342d
Compare
kbattocchi
force-pushed
the
kebatt/robust
branch
from
a81342d to
b95b96a
Compare
Enabling grouping, robust linear final models.