Spark NLP (5.1.4) no such method error while using bert zero shot classification. Spark version

Question

I have downloaded spark nlp just right now following the instructions from spark nlp website on databricks. Most functions work for me, however zero shot classification throws an exception. Can someone explain?

from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline, PipelineModel

document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer().setInputCols("document").setOutputCol("token")

zero_shot_classifier = BertForZeroShotClassification \
    .pretrained() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("class") \
    .setCandidateLabels(["urgent", "mobile", "travel", "movie", "music", "sport", "weather", "technology"])

pipeline = Pipeline(stages=[
    document_assembler,
    tokenizer,
    zero_shot_classifier
])



text = [["I have a problem with my iphone that needs to be resolved asap!!"],
        ["Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app."],
        ["I have a phone and I love it!"],
        ["I really want to visit Germany and I am planning to go there next year."],
        ["Let's watch some movies tonight! I am in the mood for a horror movie."],
        ["Have you watched the match yesterday? It was a great game!"],
        ["We need to harry up and get to the airport. We are going to miss our flight!"]]

inputDataset = spark.createDataFrame(text, ["text"])
model = pipeline.fit(inputDataset)
predictionDF = model.transform(inputDataset)

Here is my code and I got a result

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/catalyst/encoders/ExpressionEncoder;
---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
File <command-2627974051564241>, line 12
     10 inputData

set = spark.createDataFrame(text, ["text"])
     11 model = pipeline.fit(inputDataset)
---> 12 predictionDF = model.transform(inputDataset)