Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) #310

skakker · 2017-12-05T10:41:12Z

checks whether memory is getting spilled and recommendations are given accordingly.

shkhrgpt · 2017-12-07T23:26:34Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+    if(evaluator.severity != Severity.NONE){
+      resultDetails :+ new HeuristicResultDetails("Note", "Your memory is being spilled. Kindly look into it.")
+      if(evaluator.sparkExecutorCores >=4 && evaluator.sparkExecutorMemory >= MemoryFormatUtils.stringToBytes("10GB")) {
+        resultDetails :+ new HeuristicResultDetails("Recommendation", "You can try decreasing the number of cores to reduce the number of concurrently running tasks.")


I am wondering if increasing memory could be a recommendation. That should also reduce the memory spill?

shkhrgpt · 2017-12-07T23:28:27Z

app/com/linkedin/drelephant/spark/fetchers/statusapiv1/statusapiv1.scala

@@ -87,6 +87,7 @@ trait ExecutorSummary{
  def totalShuffleRead: Long
  def totalShuffleWrite: Long
  def maxMemory: Long
+  def totalMemoryBytesSpilled: Long


Does the regular Spark history server REST API provide totalMemoryBytesSpilled? Or you are using a custom Spark history server?

Using the custom Spark History Server

shkhrgpt · 2017-12-07T23:39:16Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+    val fractionOfExecutorsHavingBytesSpilled: Double = executorSummaries.count(_.totalMemoryBytesSpilled > 0).toDouble / executorSummaries.size.toDouble
+    val severity: Severity = {
+      if (fractionOfExecutorsHavingBytesSpilled != 0) {
+        if (fractionOfExecutorsHavingBytesSpilled < 0.2 && maxMemorySpilled < 0.05 * ratioMemoryCores) {


Make the threshold configurable?

shkhrgpt · 2017-12-07T23:39:22Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+        if (fractionOfExecutorsHavingBytesSpilled < 0.2 && maxMemorySpilled < 0.05 * ratioMemoryCores) {
+          Severity.LOW
+        }
+        else if (fractionOfExecutorsHavingBytesSpilled < 0.2 && meanMemorySpilled < 0.05 * ratioMemoryCores) {


Make the threshold configurable?

shkhrgpt · 2017-12-07T23:39:29Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+          Severity.MODERATE
+        }
+
+        else if (fractionOfExecutorsHavingBytesSpilled >= 0.2 && meanMemorySpilled < 0.05 * ratioMemoryCores) {


Make the threshold configurable?

shkhrgpt · 2017-12-07T23:39:36Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+        else if (fractionOfExecutorsHavingBytesSpilled >= 0.2 && meanMemorySpilled < 0.05 * ratioMemoryCores) {
+          Severity.SEVERE
+        }
+        else if (fractionOfExecutorsHavingBytesSpilled >= 0.2 && meanMemorySpilled >= 0.05 * ratioMemoryCores) {


Make the threshold configurable?

edwinalu · 2017-12-14T22:35:35Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+    )
+
+    if(evaluator.severity != Severity.NONE){
+      resultDetails :+ new HeuristicResultDetails("Note", "Your memory is being spilled. Kindly look into it.")


Please change "memory" to "execution memory". We can recommend increasing executor memory, if it isn't too high (perhaps <10G) already.

edwinalu · 2017-12-14T22:47:30Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+
+object ExecutorStorageSpillHeuristic {
+  val SPARK_EXECUTOR_MEMORY = "spark.executor.memory"
+  val SPARK_EXECUTOR_CORES = "spark.executor.cores"


Let's use the number of executors, to get a sense of the amount of memory spilled per executor, rather than dividing by the number of cores per executor.

Remove these variables if not in use.

akshayrai · 2018-01-08T15:20:29Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+
+    if(evaluator.severity != Severity.NONE){
+      resultDetails :+ new HeuristicResultDetails("Note", "Your execution memory is being spilled. Kindly look into it.")
+      if(evaluator.sparkExecutorCores >=4 && evaluator.sparkExecutorMemory >= MemoryFormatUtils.stringToBytes("10GB")) {


4 and 10GB should be made configurable in HeuristicConf

akshayrai · 2018-01-08T15:22:08Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+
+object ExecutorStorageSpillHeuristic {
+  val SPARK_EXECUTOR_MEMORY = "spark.executor.memory"
+  val SPARK_EXECUTOR_CORES = "spark.executor.cores"


Remove these variables if not in use.

akshayrai · 2018-01-08T15:22:22Z

app/com/linkedin/drelephant/spark/heuristics/ExecutorStorageSpillHeuristic.scala

+    }
+
+    lazy val sparkExecutorMemory: Long = (appConfigurationProperties.get(SPARK_EXECUTOR_MEMORY).map(MemoryFormatUtils.stringToBytes)).getOrElse(0)
+    lazy val sparkExecutorCores: Int = (appConfigurationProperties.get(SPARK_EXECUTOR_CORES).map(_.toInt)).getOrElse(0)


Remove if not required.

akshayrai · 2018-01-08T15:22:43Z

app/views/help/spark/helpExecutorStorageSpillHeuristic.scala.html

+* License for the specific language governing permissions and limitations under
+* the License.
+*@
+<p>Spark performs best when data is kept in memory. Spilled execution memory is tracked by memoryBytesSpilled, which is available at the task level, and each stage will have this information for its tasks. memoryBytesSpilled is cumulative for a task -- it is incremented whenever more execution memory is spilled. If execution memory is being spilled, then the warnings are as follows:</p>


Update the help page reflecting the non usage of sparkExecutorCores

…talMemoryBytesSpilled metric) (#310)

…talMemoryBytesSpilled metric) (linkedin#310)

…talMemoryBytesSpilled metric) (#310)

…talMemoryBytesSpilled metric) (linkedin#310)

…talMemoryBytesSpilled metric) (#310)

shkhrgpt reviewed Dec 7, 2017

View reviewed changes

akshayrai force-pushed the master branch from 7c2fd7f to 8b46933 Compare December 12, 2017 05:09

skakker force-pushed the SpillHeuristic branch from 204d5a7 to 681e577 Compare December 14, 2017 13:24

edwinalu reviewed Dec 14, 2017

View reviewed changes

swasti added 3 commits January 8, 2018 10:16

adding spill heuristic

dabc925

changed logic to fetch spilled bytes information from executors

919fd62

review comments

20bd753

skakker force-pushed the SpillHeuristic branch from a769026 to 20bd753 Compare January 8, 2018 06:59

swasti added 2 commits January 8, 2018 12:34

deleted unwanted files

512719f

fixed the error divide by 0

58586e7

akshayrai suggested changes Jan 8, 2018

View reviewed changes

swasti added 3 commits January 9, 2018 13:15

acknowledging review comments

b69d0a0

changing instances to cores

91eb40d

acknowledging review comments and some other required changes

4f3d280

akshayrai approved these changes Jan 10, 2018

View reviewed changes

akshayrai changed the title ~~adding spill heuristic~~ Executor Spill Heuristic Jan 10, 2018

akshayrai changed the title ~~Executor Spill Heuristic~~ Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled) Jan 10, 2018

akshayrai changed the title ~~Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled)~~ Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) Jan 10, 2018

skakker changed the base branch from master to customSHSWork January 10, 2018 05:59

akshayrai merged commit 14ec8f2 into linkedin:customSHSWork Jan 10, 2018

akshayrai pushed a commit that referenced this pull request Feb 21, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

efe5a91

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request Feb 27, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

f948b82

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request Mar 6, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

e8942fd

…talMemoryBytesSpilled metric) (#310)

arpang pushed a commit to arpang/dr-elephant that referenced this pull request Mar 14, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

f5d64ed

…talMemoryBytesSpilled metric) (linkedin#310)

akshayrai pushed a commit that referenced this pull request Mar 19, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

093afcb

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request Mar 19, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

a18bbce

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request Mar 30, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

07129a4

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request Apr 6, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

6b4a3cf

…talMemoryBytesSpilled metric) (#310)

akshayrai pushed a commit that referenced this pull request May 21, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

55a3f2b

…talMemoryBytesSpilled metric) (#310)

pralabhkumar pushed a commit to pralabhkumar/dr-elephant that referenced this pull request Aug 31, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

3ed8ef6

…talMemoryBytesSpilled metric) (linkedin#310)

varunsaxena pushed a commit that referenced this pull request Oct 16, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires to…

668fe5b

…talMemoryBytesSpilled metric) (#310)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) #310

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) #310

skakker commented Dec 5, 2017

shkhrgpt Dec 7, 2017

shkhrgpt Dec 7, 2017

skakker Dec 12, 2017

shkhrgpt Dec 7, 2017

skakker Dec 18, 2017

shkhrgpt Dec 7, 2017

skakker Dec 18, 2017

shkhrgpt Dec 7, 2017

skakker Dec 18, 2017

shkhrgpt Dec 7, 2017

skakker Dec 18, 2017

edwinalu Dec 14, 2017

skakker Dec 18, 2017

edwinalu Dec 14, 2017

skakker Dec 18, 2017

akshayrai Jan 8, 2018

akshayrai Jan 8, 2018

akshayrai Jan 8, 2018

akshayrai Jan 8, 2018

akshayrai Jan 8, 2018

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) #310

Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) #310

Conversation

skakker commented Dec 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment