feat: add featuresShapCol to LightGBMClassifierModel

[ocworld]

In this pr, featuresShapCol is added to LightGBMClassifierModel.

[ocworld]

@AhnLab-OSS

[ekerazha]

So we have SHAP for LightGBM Ranker and Classifier? Is it now missing only for Regressor?

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

Looks great except for a few minor changes!

[ocworld]

@ekerazha I assume that except regression SHAP is supported.
Please note that it has different usages for getting SHAP between LightGBM Ranker and Classifier in this pr.

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[codecov]

Codecov Report

Merging #863 into master will increase coverage by 0.03%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #863      +/-   ##
==========================================
+ Coverage   85.12%   85.16%   +0.03%     
==========================================
  Files         187      187              
  Lines        8668     8681      +13     
  Branches      520      507      -13     
==========================================
+ Hits         7379     7393      +14     
+ Misses       1289     1288       -1     

Impacted Files	Coverage Δ
...crosoft/ml/spark/lightgbm/LightGBMClassifier.scala	90.09% <87.50%> (+1.46%)	⬆️
.../microsoft/ml/spark/lightgbm/LightGBMBooster.scala	87.87% <100.00%> (+1.51%)	⬆️
.../execution/streaming/continuous/HTTPSourceV2.scala	93.04% <0.00%> (-0.37%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82e7a8e...170e187. Read the comment docs.

[imatiach-msft]

hmm seems the test is failing in the native code:

[info] + Test Verify LightGBM Classifier features shap took 0.583s 
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4ea7779ef2, pid=7254, tid=0x00007f4ea4263700
#
# JRE version: OpenJDK Runtime Environment () (8.0_252-b14) (build 1.8.0_252-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.252-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x7fef2]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/vsts/work/1/s/hs_err_pid7254.log
[thread 139974389262080 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://www.azulsystems.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
timeout: the monitored command dumped core
/home/vsts/work/_temp/azureclitaskscript1589903456723.sh: line 6:  7253 Aborted

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

running another build again, will try to run this branch locally also

[imatiach-msft]

@ocworld great news, I was able to run your code locally and I discovered the issue - SHAP returns the expected value in addition to the feature contributions, which led to memory corruption. Although I was not able to get the SHAP test to fail, I did see another test fail due to memory corruption from the SHAP test, and once I fixed the issue I validated that the memory corruption error is no longer occurring. I pushed the fix directly to your branch, hope you don't mind.

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

missed updating another test in ranker which was checking the number of features, fixed

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

found another minor issue where the new column name was missing in the objects to save which caused a serialization failure, fixed, hopefully build should be green now

[imatiach-msft]

once PR is merged I'll send a corresponding PR for
1.) pyspark API
2.) lightgbm regressor
to make it consistent across codebase

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

@ocworld I noticed this PR won't work for multiclass case, since multiclass shap values are of shape (# features + 1 expected_value) * # classes, but we currently aren't making the array the size of # classes. I can fix this too when I send out the regression PR.

[ocworld]

@imatiach-msft Thanks for your notice