GES DISC - MERRA2

Reading MERRA2 Data Using Kerchunk Reference File

Many of NASA’s current and legacy data collections are archive in netCDF4 format. By itself, netCDF4 are not cloud optimized and reading these files can take as long from a personal/local work environment as it takes to read the data from a working environment deployed in the cloud. Using Kerchunk, we can treat these files as cloud optimized assets by creating metadata json file describing existing netCDF4 files, their chunks, and where to access them. The json reference files can be read in using Zarr and Xarray for efficient reads and fast processing.

Requirements

1. AWS instance running in us-west-2

NASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.

2. Earthdata Login

An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.

3. netrc File

You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata.

Import required packages

import requests
import xarray as xr
import ujson
import s3fs
import fsspec
from tqdm import tqdm
from glob import glob
import os
import pathlib
import hvplot.xarray

from kerchunk.hdf import SingleHdf5ToZarr
from kerchunk.combine import MultiZarrToZarr

# The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why
import warnings
warnings.simplefilter("ignore")

Create Dask client to process the output json file in parallel

Generating the Kerchunk reference file can take some time depending on the internal structure of the data. Dask allows us to execute the reference file generation process in parallel, thus speeding up the overall process.

import dask
from dask.distributed import Client
client = Client(n_workers=4)
client

Client

Client-3e6c0be3-d18c-11ec-809e-527eee20f3f0

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

64356e89

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 7.48 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-9811e6a9-be9d-4a54-9cc3-f17a8d5ea5bb

Comm: tcp://127.0.0.1:33947	Workers: 4
Dashboard: http://127.0.0.1:8787/status	Total threads: 4
Started: Just now	Total memory: 7.48 GiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:38069	Total threads: 1
Dashboard: http://127.0.0.1:44433/status	Memory: 1.87 GiB
Nanny: tcp://127.0.0.1:43685
Local directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-6b5pkr_f

Worker: 1

Comm: tcp://127.0.0.1:32859	Total threads: 1
Dashboard: http://127.0.0.1:36075/status	Memory: 1.87 GiB
Nanny: tcp://127.0.0.1:35291
Local directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-piqb4cet

Worker: 2

Comm: tcp://127.0.0.1:36601	Total threads: 1
Dashboard: http://127.0.0.1:45185/status	Memory: 1.87 GiB
Nanny: tcp://127.0.0.1:41163
Local directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-d_91r19y

Worker: 3

Comm: tcp://127.0.0.1:34309	Total threads: 1
Dashboard: http://127.0.0.1:38301/status	Memory: 1.87 GiB
Nanny: tcp://127.0.0.1:44119
Local directory: /home/jovyan/earthdata-cloud-cookbook/examples/GESDISC/dask-worker-space/worker-9wa5m1vc