What is the proper syntax of using broadcast variables in Spark using Scala?

728 views Asked by At

I want to use a broadcast variable in Spark with Scala. But I can't find enough help on how to use them. Say, I have an object of class A, which I normally would declare as follows in Scala.

val a = new A()

What would be the syntax of declaring it as a broadcast variable. And how would I call its methods?

2

There are 2 answers

0
Marth On BEST ANSWER

If sc is a SparkContext, then val broadcasted = sc.broadcast(a) will broadcast a.
You can then access it with broadcasted.value.

1
Beniamino Del Pizzo On

Marth is right. You need also to destroy a broadcast variable by using sc.destroy(blocking) where blocking is a flag. I want to highlight that is recommended to avoid to broadcast small variables.