Skip to content

Commit 7b59ef5

Browse files
Nicole00wey-gu
andauthored
add spark version validation (#69)
* add spark version validation * Add spark version info in Compatibility Matrix --------- Co-authored-by: Wey Gu <weyl.gu@gmail.com>
1 parent 21564be commit 7b59ef5

File tree

2 files changed

+27
-12
lines changed

2 files changed

+27
-12
lines changed

README.md

+15-11
Original file line numberDiff line numberDiff line change
@@ -111,19 +111,23 @@ If you want to write the algorithm execution result into NebulaGraph(`sink: nebu
111111
| bfs | dfs | string |
112112
| jaccard | jaccard | string |
113113
| node2vec | node2vec | string |
114-
114+
115+
116+
<!--- We cannot change name of this chapter, or the version checker code will be broken --->
117+
115118
## Version Compatibility Matrix
116119
117-
| Nebula Algorithm Version | Nebula Version |
118-
|:------------------------:|:--------------:|
119-
| 2.0.0 | 2.0.0, 2.0.1 |
120-
| 2.1.0 | 2.0.0, 2.0.1 |
121-
| 2.5.0 | 2.5.0, 2.5.1 |
122-
| 2.6.0 | 2.6.0, 2.6.1 |
123-
| 2.6.1 | 2.6.0, 2.6.1 |
124-
| 2.6.2 | 2.6.0, 2.6.1 |
125-
| 3.0.0 | 3.0.x, 3.1.x, 3.2.x, 3.3.x |
126-
| 3.0-SNAPSHOT | nightly |
120+
| NebulaGraph Algorithm Version | NebulaGraph Version | Spark Version |
121+
| :---------------------------: | :------------------------: | ------------- |
122+
| 2.0.0 | 2.0.0, 2.0.1 | 2.4 |
123+
| 2.1.0 | 2.0.0, 2.0.1 | 2.4 |
124+
| 2.5.0 | 2.5.0, 2.5.1 | 2.4 |
125+
| 2.6.0 | 2.6.0, 2.6.1 | 2.4 |
126+
| 2.6.1 | 2.6.0, 2.6.1 | 2.4 |
127+
| 2.6.2 | 2.6.0, 2.6.1 | 2.4 |
128+
| 3.0.0, 3.1.x | 3.0.x, 3.1.x, 3.2.x, 3.3.x | 2.4 |
129+
| 3.0-SNAPSHOT | nightly | 2.4 |
130+
127131
128132
## Contribute
129133

nebula-algorithm/src/main/scala/com/vesoft/nebula/algorithm/config/SparkConfig.scala

+12-1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,17 @@ object SparkConfig {
2525
session.config(key, sparkConfigs(key))
2626
}
2727
partitionNum = sparkConfigs.getOrElse("spark.app.partitionNum", "0")
28-
SparkConfig(session.getOrCreate(), partitionNum)
28+
val spark = session.getOrCreate()
29+
validate(spark.version, "2.4.*")
30+
SparkConfig(spark, partitionNum)
31+
}
32+
33+
def validate(sparkVersion: String, supportedVersions: String*): Unit = {
34+
if (sparkVersion != "UNKNOWN" && !supportedVersions.exists(sparkVersion.matches)) {
35+
throw new RuntimeException(
36+
s"""Your current spark version ${sparkVersion} is not supported by the current NebulaGraph Algorithm.
37+
| please visit https://github.com/vesoft-inc/nebula-algorithm#version-compatibility-matrix to know which version you need.
38+
| """.stripMargin)
39+
}
2940
}
3041
}

0 commit comments

Comments
 (0)