LightGBM: average_precision earlyStopping criterion does not seem to work

[edsonaoki]

Hi,

I am trying to train models using MMLSpark LightGBMClassifier, setting the "earlyStopping" criterion to "average_precision". However, it never stops until it reaches the maximum number of iterations, and in fact, based on the logs, it does not seem to be even evaluating the criterion.

For example, when I set earlyStopping to 'auc', I see something like this in the stderr log:

21/01/20 22:11:58 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:11:58 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:11:58 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:11:58 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:11:58 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: Valid auc=0.8800437895547021 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: Valid auc=0.8800437895547021 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: Valid auc=0.8800437895547021 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: Valid auc=0.8800437895547021 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: Valid auc=0.8800437895547021 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:00 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: Valid auc=0.8763379546795153 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: Valid auc=0.8763379546795153 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: Valid auc=0.8763379546795153 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: Valid auc=0.8763379546795153 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: Valid auc=0.8763379546795153 21/01/20 22:12:03 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: Valid auc=0.8427327779203404 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: Valid auc=0.8427327779203404 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: Valid auc=0.8427327779203404 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: Valid auc=0.8427327779203404 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: Valid auc=0.8427327779203404 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:06 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:08 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 22:12:08 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 22:12:08 INFO lightgbm.LightGBMClassifier: Valid auc=0.8436776167278073 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: Valid auc=0.8436776167278073 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: Valid auc=0.8436776167278073 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: Valid auc=0.8436776167278073 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: Valid auc=0.8436776167278073 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:09 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: Valid auc=0.8445991882695754 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: Valid auc=0.8445991882695754 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: Valid auc=0.8445991882695754 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: Valid auc=0.8445991882695754 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: Valid auc=0.8445991882695754 21/01/20 22:12:11 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: Valid auc=0.8461803153101356 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: Valid auc=0.8461803153101356 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: Valid auc=0.8461803153101356 21/01/20 22:12:14 INFO lightgbm.LightGBMClassifier: Valid auc=0.8461803153101356

But when I set earlyStopping to 'average_precision', I get this:

21/01/20 18:32:35 INFO lightgbm.LightGBMClassifier: LightGBM task generating sparse dataset with 210563 rows and 1079 columns 21/01/20 18:32:35 INFO lightgbm.LightGBMClassifier: LightGBM task generating sparse dataset with 210563 rows and 1079 columns 21/01/20 18:32:35 INFO lightgbm.LightGBMClassifier: LightGBM task generating sparse dataset with 210563 rows and 1079 columns 21/01/20 18:32:35 INFO lightgbm.LightGBMClassifier: LightGBM task generating sparse dataset with 210563 rows and 1079 columns 21/01/20 18:32:35 INFO lightgbm.LightGBMClassifier: LightGBM task generating sparse dataset with 210563 rows and 1079 columns 21/01/20 18:32:39 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:39 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:40 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:40 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:40 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 0 with result: 0 and is finished: false 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:42 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 1 with result: 0 and is finished: false 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:44 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 2 with result: 0 and is finished: false 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:46 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 3 with result: 0 and is finished: false 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:48 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 4 with result: 0 and is finished: false 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:51 INFO lightgbm.LightGBMClassifier: LightGBM task calling LGBM_BoosterUpdateOneIter 21/01/20 18:32:54 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 18:32:54 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 18:32:54 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 18:32:54 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false 21/01/20 18:32:54 INFO lightgbm.LightGBMClassifier: LightGBM running iteration: 5 with result: 0 and is finished: false

As you can see, the metric is does not seem to be updated, and predictably, earlyStopping is never triggered.
If it helps, here is an excertp of the stdout:

[LightGBM] [Warning] metric is set=average_precision, metric=average_precision will be ignored. Current value: metric=average_precision [LightGBM] [Warning] metric is set=average_precision, metric=average_precision will be ignored. Current value: metric=average_precision [LightGBM] [Warning] metric is set=average_precision, metric=average_precision will be ignored. Current value: metric=average_precision [LightGBM] [Warning] metric is set=average_precision, metric=average_precision will be ignored. Current value: metric=average_precision [LightGBM] [Warning] metric is set=average_precision, metric=average_precision will be ignored. Current value: metric=average_precision [LightGBM] [Info] Number of positive: 4816, number of negative: 626989 [LightGBM] [Info] Number of positive: 4816, number of negative: 626989 [LightGBM] [Info] Number of positive: 4816, number of negative: 626989 [LightGBM] [Info] Number of positive: 4816, number of negative: 626989 [LightGBM] [Info] Number of positive: 4816, number of negative: 626989 [LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.392420 seconds. You can set force_row_wise=trueto remove the overhead. And if memory is not enough, you can setforce_col_wise=true. [LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.433868 seconds. You can set force_col_wise=trueto remove the overhead. [LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.672994 seconds. You can setforce_col_wise=trueto remove the overhead. [LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.459724 seconds. You can setforce_row_wise=trueto remove the overhead. And if memory is not enough, you can setforce_col_wise=true. [LightGBM] [Info] Total Bins 58687 [LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.593519 seconds. You can set force_row_wise=trueto remove the overhead. And if memory is not enough, you can setforce_col_wise=true. [LightGBM] [Info] Number of data points in the train set: 126387, number of used features: 778 [LightGBM] [Info] Total Bins 58687 [LightGBM] [Info] Total Bins 58687 [LightGBM] [Info] Number of data points in the train set: 126316, number of used features: 778 [LightGBM] [Info] Number of data points in the train set: 126378, number of used features: 778 [LightGBM] [Info] Total Bins 58687 [LightGBM] [Info] Total Bins 58687 [LightGBM] [Info] Number of data points in the train set: 126371, number of used features: 778 [LightGBM] [Info] Number of data points in the train set: 126353, number of used features: 778 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.007627 -> initscore=-4.868356 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.007624 -> initscore=-4.868836 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.007636 -> initscore=-4.867184 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.007574 -> initscore=-4.875429 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.007652 -> initscore=-4.865153 [LightGBM] [Info] Start training from score -4.868991 [LightGBM] [Info] Start training from score -4.868991 [LightGBM] [Info] Start training from score -4.868991 [LightGBM] [Info] Start training from score -4.868991 [LightGBM] [Info] Start training from score -4.868991 [LightGBM] [Info] Finished linking network in 880.123565 seconds [LightGBM] [Info] Finished linking network in 1351.083543 seconds [LightGBM] [Info] Finished linking network in 1363.004313 seconds [LightGBM] [Info] Finished linking network in 1352.197649 seconds [LightGBM] [Info] Finished linking network in 1360.990499 seconds

As you can see there is a warning message, but a nearly identical warning message appears when I use 'auc' instead of 'average_precision'

• MMLSpark Version: 1.0.0-rc3
• Spark Version 2.4.0
• Spark Platform Cloudera

AB#1155244

[edsonaoki]

Hi, can somebody look into this? 'average_precision' is a very important criterion for extremely unbalanced datasets, and with this feature missing, the usefulness of LightGBM is greatly reduced for real-world problems.

[imatiach-msft]

@edsonaoki I think average_precision was only implemented in September 2020 based on this commit:
microsoft/LightGBM@2870490
the older version of lightgbm on mmlspark you are using can't recognize it. I think this newer version I upgraded may work now:
#1015
but I would have to check

[imatiach-msft]

also, according to an email conversation following this issue
#1006
it seems that early stopping in mmlspark is not returning the model trained at the best iteration, but at best iteration + earlyStoppingRound later, which is not correct right now.
Copying a snippet from email conversation with user request:

“Moreover, I can calculate the best iteration value by accessing the attribute numIterations or numTotalModel in
https://mmlspark.blob.core.windows.net/docs/1.0.0-rc3/scala/index.html#com.microsoft.ml.spark.lightgbm.LightGBMBooster
”

I will need to double check this in detail later, but just thinking about this, if early stopping is triggered it will be “earlyStoppingRound” iterations later than the best iterations based on the code:

iters - bestIter(index) >= trainParams.earlyStoppingRound
https://github.com/Azure/mmlspark/blob/master/src/main/scala/com/microsoft/ml/spark/lightgbm/TrainUtils.scala#L296

But we actually want to get the model at the best iteration, not the last model trained at (best iteration) + (earlyStoppingRound later). In that case, if the user hits the early stopping conditions, we should set the default number of iterations to be the best iteration on predict in the lightgbm model, instead of the last model trained. So this isn’t necessarily correct: “I can calculate the best iteration value by accessing the attribute numIterations or numTotalModel” as that wasn’t exactly the best iteration, it was the best iteration + early stopping rounds later if early stopping was triggered.

[imatiach-msft]

note you can try the latest version, it's not an official release, but every PR has maven coordinates:

MMLSpark Build and Release Information
Maven Coordinates
com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-49-659b7743-SNAPSHOT

Maven Resolver
https://mmlspark.azureedge.net/maven

[imatiach-msft]

@edsonaoki I noticed the new average precision metric also needs to be added to the comparison function with AUC, NDCG and MAP, which I did in this PR:
#1034

[imatiach-msft]

@edsonaoki average precision metric for early stopping should be fixed now with PR:
#1034

the latest build on master should have the fix, including this build:
Maven Coordinates
com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-78-ff42a2a4-SNAPSHOT
Maven Resolver
https://mmlspark.azureedge.net/maven

please feel free to reopen this issue or open a new issue if you are still seeing problems in latest build, thank you!

[edsonaoki]

@imatiach-msft thank you very much for looking into this! I will try to use the new build asap and let you know if there are any issues.