kedro pyspark starter cannot load spark.SparkDataset

Question

The kedro starter project immediately fails to recognize the spark dataset

kedro new --starter=spaceflights-pyspark-viz
cd projectpath
pip install -r requirements.txt
kedro ipython
Class 'spark.SparkDataset' not found, is this a typo?

Tested with kedro 0.19.1 and .2

any idea?

Answer 1

if you explicitly install pip kedro-datasets[spark.SparkDataset] does it work? This should be part of the requirements.txt