Skip to content

Default master value to first check the system variables. #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Kotlin for Apache® Spark™ [![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api-parent.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:org.jetbrains.kotlinx.spark%20AND%20v:1.0.2) [![official JetBrains project](http://jb.gg/badges/incubator.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)
# Kotlin for Apache® Spark™ [![Maven Central](https://img.shields.io/maven-central/v/org.jetbrains.kotlinx.spark/kotlin-spark-api-parent.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:org.jetbrains.kotlinx.spark%20AND%20v:1.0.2) [![official JetBrains project](http://jb.gg/badges/official.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)


Your next API to work with [Apache Spark](https://spark.apache.org/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,27 @@
*/
package org.jetbrains.kotlinx.spark.api

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession.Builder
import org.apache.spark.sql.UDFRegistration
import org.jetbrains.kotlinx.spark.api.SparkLogLevel.ERROR

/**
* Wrapper for spark creation which allows to set different spark params
* Wrapper for spark creation which allows setting different spark params.
*
* @param props spark options, value types are runtime-checked for type-correctness
* @param master [SparkSession.Builder.master]
* @param appName [SparkSession.Builder.appName]
* @param master Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to
* run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster. By default, it
* tries to get the system value "spark.master", otherwise it uses "local[*]"
* @param appName Sets a name for the application, which will be shown in the Spark web UI.
* If no application name is set, a randomly generated name will be used.
* @param logLevel Control our logLevel. This overrides any user-defined log settings.
* @param func function which will be executed in context of [KSparkSession] (it means that `this` inside block will point to [KSparkSession])
*/
@JvmOverloads
inline fun withSpark(
props: Map<String, Any> = emptyMap(),
master: String = "local[*]",
master: String = SparkConf().get("spark.master", "local[*]"),
appName: String = "Kotlin Spark Sample",
logLevel: SparkLogLevel = ERROR,
func: KSparkSession.() -> Unit,
Expand All @@ -58,10 +63,17 @@ inline fun withSpark(

}

/**
* Wrapper for spark creation which allows setting different spark params.
*
* @param builder A [SparkSession.Builder] object, configured how you want.
* @param logLevel Control our logLevel. This overrides any user-defined log settings.
* @param func function which will be executed in context of [KSparkSession] (it means that `this` inside block will point to [KSparkSession])
*/
@JvmOverloads
inline fun withSpark(builder: Builder, logLevel: SparkLogLevel = ERROR, func: KSparkSession.() -> Unit) {
builder
.orCreate
.getOrCreate()
.apply {
KSparkSession(this).apply {
sparkContext.setLogLevel(logLevel)
Expand Down