1. Environment
- Linux (#44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux)
- Anaconda (23.1.0)
- running a proxy (clash)
- conda environment created by python_pangeo_cmip6
2. Logs/tracks
$ python get_cmip6.py inputIPSL_hist.json (pg_cimpr)
using metadta from https://storage.googleapis.com/cmip6/cmip6-zarr-consolidated-stores.csv
Trying to download:
table_id == 'Amon' & variable_id == 'hfls' & experiment_id == 'historical' & source_id == 'IPSL-CM6A-LR' & member_id == 'r1i1p1f1'
Traceback (most recent call last):
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
await waiter
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/hgfs/F/Data/Original/CMIP6/LFMIP-pdLC/python_pangeo_cmip6/get_cmip6.py", line 93, in <module>
ds = xr.open_zarr(mapper, consolidated=True)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/xarray/backends/zarr.py", line 675, in open_zarr
ds = open_dataset(
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/xarray/backends/api.py", line 572, in open_dataset
store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/xarray/backends/zarr.py", line 294, in open_group
zarr_group = zarr.open_consolidated(store, **open_kwargs)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/zarr/convenience.py", line 1178, in open_consolidated
meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/zarr/storage.py", line 2680, in __init__
meta = json_loads(store[metadata_key])
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/fsspec/mapping.py", line 132, in __getitem__
result = self.fs.cat(k)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/fsspec/asyn.py", line 241, in cat
raise ex
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/gcsfs/core.py", line 826, in _cat_file
headers, out = await self._call("GET", u2)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/gcsfs/core.py", line 487, in _call
async with self.session.request(
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__
self._resp = await self._coro
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/home/cool70/anaconda3/envs/pg_cimpr/lib/python3.9/site-packages/aiohttp/connector.py", line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.googleapis.com:443 ssl:default [Connection reset by peer]
The problem is related to the proxy I'm running, it failed to reach google api. Anyone in China has the solution?
I can use other python packages to download CMIP6 data like acccmip6, or through official CMIP6 website. But this repository can regrid the nc files and choose the wanted time period while downloading data, so there would be less data pre-processing.