Plotting Monthly data using groupby in dask dataset

963 views Asked by At

I have a large CSV file that it is opened with Dask.

import numpy as np
import pandas as pd
import hvplot.pandas
import hvplot.dask
import intake

data = '../file.csv'
ddf = intake.open_csv(data).to_dask()
ddf.head()

Datetime    latitude    longitude   Temp_2m(C)  
1   1980-01-02 03:00:00     30.605  50.217  5.31
2   1980-01-02 04:00:00     30.605  50.217  5.36
3   1980-01-02 05:00:00     30.605  50.217  7.04
4   1980-01-02 06:00:00     30.605  50.217  10.24

I want to plot Temp_2m(C) monthly with hvplot. Plot with hourly data of Datetime is done correctly, but when I want to group Datetime as follow, It return an error.

# Convert 'Datetime' column to 'datetime64'
ddf["Datetime"] = ddf["Datetime"].astype("M8[us]")

# set index column
ddf = ddf.set_index('Datetime')

g = pd.Grouper(freq='M', key='Datetime')
month_ddf = dff.groupby(g).mean()

# plot
month_ddf.hvplot('Temp_2m(C)')

ERROR: ValueError: all keys need to be the same shape what is my mistake?

for reply @frankr6591:

month_ddf.describe()
Dask DataFrame Structure:
    latitude    longitude   Temp_2m(C)
npartitions=1                                           
    float64     float64     float64
    ...     ...     ... 
Dask Name: describe-numeric, 89 tasks
1

There are 1 answers

3
frankr6591 On

I used to_datetime() and got correct plot with .plot()... ran into problems installing hvplot.

import numpy as np
import pandas as pd
# FIXME : the following does not work
#import hvplot.pandas
%matplotlib inline

d = dict(datetime = ['1980-01-02 02:00:00',
                        '1980-01-02 03:00:00',
                        '1980-01-02 04:00:00',
                        '1980-01-02 05:00:00',
                        '1980-07-02 06:00:00'],
            latitude = [30.605 for n in range(5)],            
            longitude = [50.217 for n in range(5)],
            Temp_2m = [np.random.random()*10 for n in range(5)])
df = pd.DataFrame(d)

df['datetime'] = pd.to_datetime(df['datetime'])
df['mon'] = df['datetime'].dt.to_period('M')
print(df)

ddf = df.groupby('mon').mean()
print(ddf)

# This works on my py3.7
ddf.plot('Temp_2m')

# This fails because hvplot could not be imported. 
ddf.hvplot('Temp_2m')

             datetime  latitude  longitude   Temp_2m      mon
0 1980-01-02 02:00:00    30.605     50.217  2.512897  1980-01
1 1980-01-02 03:00:00    30.605     50.217  0.247358  1980-01
2 1980-01-02 04:00:00    30.605     50.217  7.678030  1980-01
3 1980-01-02 05:00:00    30.605     50.217  0.637331  1980-01
4 1980-07-02 06:00:00    30.605     50.217  2.156502  1980-07


         latitude  longitude   Temp_2m
mon                                   
1980-01    30.605     50.217  5.080373
1980-07    30.605     50.217  1.324140

enter image description here