01. Data discovery with CMR

In this tutorial you will learn: - what CMR is;
- how to use the requests package to search data collections and granules;
- how to parse the results of these searches.

We will focus on datasets in the cloud. Currently, DAACs with data in the cloud are ‘ASF’, ‘GES_DISC’, ‘GHRC_DAAC’, ‘LPCLOUD’, ‘ORNL_CLOUD’, ‘POCLOUD’

What is CMR

CMR is the Common Metadata Repository. It catalogs all data for NASA’s Earth Observing System Data and Information System (EOSDIS). It is the backend of Earthdata Search, the GUI search interface you are probably familiar with. More information about CMR can be found here.

Unfortunately, the GUI for Earthdata Search is not accessible from a cloud instance - at least not without some work. Earthdata Search is also not immediately reproducible. What I mean by that is if you create a search using the GUI you would have to note the search criteria (date range, search area, collection name, etc), take a screenshot, copy the search url, or save the list of data granules returned by the search, in order to recreate the search. This information would have to be re-entered each time you or someone else wanted to do the search. You could make typos or other mistakes. A cleaner, reproducible solution is to search CMR programmatically using the CMR API.

What is the CMR API

API stands for Application Programming Interface. It allows applications (software, services, etc) to send information to each other. A helpful analogy is a waiter in a restaurant. The waiter takes your drink or food order that you select from the menu, often translated into short-hand, to the bar or kitchen, and then returns (hopefully) with what you ordered when it is ready.

The CMR API accepts search terms such as collection name, keywords, datetime range, and location, queries the CMR database and returns the results.

How to search CMR from Python

The first step is to import python packages. We will use:
- requests This package does most of the work for us accessing the CMR API using HTTP methods. - pprint to pretty print the results of the search.

A more in depth tutorial on requests is here

import requests
from pprint import pprint

Then we need to authenticate with EarthData Login. Since we’ve already set this up in the previous lesson, here you need to enter your username before executing the cell.

To conduct a search using the CMR API, requests needs the url for the root CMR search endpoint. We’ll build this url as a python variable.

CMR_OPS = 'https://cmr.earthdata.nasa.gov/search'

CMR allows search by collections, which are datasets, and granules, which are files that contain data. Many of the same search parameters can be used for colections and granules but the type of results returned differ. Search parameters can be found in the API Documentation.

Whether we search collections or granules is distinguished by adding "collections" or "granules" to the url for the root CMR endpoint.

We are going to search collections first, so we add collections to the url. I’m using a python format string here.

url = f'{CMR_OPS}/{"collections"}'

In this first example, I want to retrieve a list of collections that are hosted in the cloud. Each collection has a cloud_hosted parameter that is either True if that collection is in the cloud and False if it is not. The migration of NASA data to the cloud is a work in progress. Not all collections tagged as cloud_hosted have granules. To search for only cloud_hosted datasets with granules, I also set has_granules to True.

I also want to get the content in json (pronounced “jason”) format, so I pass a dictionary to the header keyword argument to say that I want results returned as json.

The .get() method is used to send this information to the CMR API. get() calls the HTTP method GET.

response = requests.get(url,
                        params={
                            'cloud_hosted': 'True',
                            'has_granules': 'True',
                        },
                        headers={
                            'Accept': 'application/json',
                        }
                       )

requests returns a Response object.

Often, we want to check that our request was successful. In a notebook or someother interactive environment, we can just type the name of the variable we have saved our requests Response to, in this case the response variable.

response
<Response [200]>

A cleaner and more understandable method is to check the status_code attribute. Both methods return a HTTP status code. You’ve probably seen a 404 error when you have tried to access a website that doesn’t exist.

response.status_code
200

Try changing CMR_OPS to https://cmr.earthdata.nasa.gov/searches and run requests.get again. Don’t forget to rerun the cell that assigns the url variable

The response from requests.get returns the results of the search and metadata about those results in the headers.

More information about the response object can be found by typing help(response).

headers contains useful information in a case-insensitive dictionary. This information is printed below. TODO: maybe some context for where the 2 elements k, v, come from?

for k, v in response.headers.items():
    print(f'{k}: {v}')
Content-Type: application/json;charset=utf-8
Content-Length: 3820
Connection: keep-alive
Date: Tue, 02 Nov 2021 22:07:37 GMT
X-Frame-Options: SAMEORIGIN
Access-Control-Allow-Origin: *
X-XSS-Protection: 1; mode=block
CMR-Request-Id: 510cf611-d65b-4b46-981b-f4719405676a
Strict-Transport-Security: max-age=31536000
CMR-Search-After: [0.0,14000.0,"SENTINEL-1A_RAW","1",1214470561,1320]
CMR-Hits: 917
Access-Control-Expose-Headers: CMR-Hits, CMR-Request-Id, X-Request-Id, CMR-Scroll-Id, CMR-Search-After, CMR-Timed-Out, CMR-Shapefile-Original-Point-Count, CMR-Shapefile-Simplified-Point-Count
X-Content-Type-Options: nosniff
CMR-Took: 435
X-Request-Id: 510cf611-d65b-4b46-981b-f4719405676a
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Server: ServerTokens ProductOnly
X-Cache: Miss from cloudfront
Via: 1.1 3f7e5e686bf8f19b9c786efbe99c7589.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: DEN52-C1
X-Amz-Cf-Id: BXEUlb5ygB9nj9ygUbSZvIc3BE4k0d1Mn1qDd66IkkuX1rC_Z-mN6Q==

We can see that the content returned is in json format in the UTF-8 character set. We can also see from CMR-Hits that 919 collections were found.

Each item in the dictionary can be accessed in the normal way you access a python dictionary but because it is case-insensitive, both

response.headers['CMR-Hits']
'917'

and

response.headers['cmr-hits']
'917'

work.

This is a large number of data sets. I’m going to restrict the search to cloud-hosted datasets from ASF (Alaska SAR Facility) because I’m interested in SAR images of sea ice. To do this, I set the provider parameter to ASF.

You can modify the code below to explore all of the cloud-hosted datasets or cloud-hosted datasets from other providers. A partial list of providers is given below.

DAAC Short Name Cloud Provider On-Premises Provider
NSIDC National Snow and Ice Data Center NSIDC_CPRD NSIDC_ECS
GHRC DAAC Global Hydrometeorology Resource Center GHRC_DAAC GHRC_DAAC
PO DAAC Physical Oceanography Distributed Active Archive Center POCLOUD PODAAC
ASF Alaska Satellite Facility ASF ASF
ORNL DAAC Oak Ridge National Laboratory ORNL_CLOUD ORNL_DAAC
LP DAAC Land Processes Distributed Active Archive Center LPCLOUD LPDAAC_ECS
GES DISC NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC) GES_DISC GES_DISC
OB DAAC NASA’s Ocean Biology Distributed Active Archive Center OB_DAAC
SEDAC NASA’s Socioeconomic Data and Applications Center SEDAC

When search by provider, use Cloud Provider to search for cloud-hosted datasets and On-Premises Provider to search for datasets archived at the DAACs.

provider = 'ASF'
response = requests.get(url,
                        params={
                            'cloud_hosted': 'True',
                            'has_granules': 'True',
                            'provider': provider,
                        },
                        headers={
                            'Accept': 'application/json'
                        }
                       )
response.headers['cmr-hits']
'45'

Search results are contained in the content part of the Response object. However, response.content returns information in bytes.

response.content
b'{"feed":{"updated":"2021-11-02T22:41:34.322Z","id":"https://cmr.earthdata.nasa.gov:443/search/collections.json?cloud_hosted=True&has_granules=True&provider=ASF","title":"ECHO dataset metadata","entry":[{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:39.000Z","dataset_id":"SENTINEL-1A_SLC","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_SLC","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1A_SLC","coordinate_system":"CARTESIAN","summary":"Sentinel-1A slant-range product","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214470488-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2016-04-25T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:17:22.000Z","dataset_id":"SENTINEL-1B_SLC","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1B_SLC","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1B_SLC","coordinate_system":"CARTESIAN","summary":"Sentinel-1B slant-range product","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1327985661-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1B"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:15:56.000Z","dataset_id":"SENTINEL-1A_DUAL_POL_GRD_HIGH_RES","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_DP_GRD_HIGH","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1A_DUAL_POL_GRD_HIGH_RES","coordinate_system":"CARTESIAN","summary":"Sentinel-1A Dual-pol ground projected high and full resolution images","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214470533-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2016-04-25T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:47.000Z","dataset_id":"SENTINEL-1B_DUAL_POL_GRD_HIGH_RES","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1B_DP_GRD_HIGH","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1B_DUAL_POL_GRD_HIGH_RES","coordinate_system":"CARTESIAN","summary":"Sentinel-1B Dual-pol ground projected high and full resolution images","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1327985645-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1B"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2016-04-25T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:49.000Z","dataset_id":"SENTINEL-1B_DUAL_POL_GRD_MEDIUM_RES","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1B_DP_GRD_MEDIUM","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1B_DUAL_POL_GRD_MEDIUM_RES","coordinate_system":"CARTESIAN","summary":"Sentinel-1B Dual-pol ground projected medium resolution images","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1327985660-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1B"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:15:58.000Z","dataset_id":"SENTINEL-1A_DUAL_POL_GRD_MEDIUM_RES","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_DP_GRD_MEDIUM","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1A_DUAL_POL_GRD_MEDIUM_RES","coordinate_system":"CARTESIAN","summary":"Sentinel-1A Dual-pol ground projected medium resolution images","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214471521-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:20.000Z","dataset_id":"SENTINEL-1A_RAW","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_RAW","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1A_RAW","coordinate_system":"CARTESIAN","summary":"Sentinel-1A level zero product","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214470561-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:15.000Z","dataset_id":"SENTINEL-1A_METADATA_SLC","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_META_SLC","organizations":["ASF"],"title":"SENTINEL-1A_METADATA_SLC","coordinate_system":"CARTESIAN","summary":"Metadata for Sentinel-1A slant-range product","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214470496-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2016-04-25T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:17:02.000Z","dataset_id":"SENTINEL-1B_METADATA_SLC","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1B_META_SLC","organizations":["ASF"],"title":"SENTINEL-1B_METADATA_SLC","coordinate_system":"CARTESIAN","summary":"Metadata for Sentinel-1B slant-range product","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1327985617-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1B"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]},{"boxes":["-90 -180 90 180"],"time_start":"2014-04-03T00:00:00.000Z","version_id":"1","updated":"2021-07-15T19:16:26.000Z","dataset_id":"SENTINEL-1A_SINGLE_POL_GRD_HIGH_RES","has_spatial_subsetting":false,"has_transforms":false,"has_variables":false,"data_center":"ASF","short_name":"SENTINEL-1A_SP_GRD_HIGH","organizations":["ASF","ESA/CS1CGS"],"title":"SENTINEL-1A_SINGLE_POL_GRD_HIGH_RES","coordinate_system":"CARTESIAN","summary":"Sentinel-1A Single-pol ground projected high and full resolution images","service_features":{"opendap":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"esi":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false},"harmony":{"has_formats":false,"has_variables":false,"has_transforms":false,"has_spatial_subsetting":false,"has_temporal_subsetting":false}},"orbit_parameters":{},"id":"C1214470682-ASF","has_formats":false,"original_format":"ECHO10","archive_center":"ASF","has_temporal_subsetting":false,"browse_flag":false,"platforms":["Sentinel-1A"],"online_access_flag":true,"links":[{"rel":"http://esipfed.org/ns/fedsearch/1.1/data#","hreflang":"en-US","href":"https://vertex.daac.asf.alaska.edu/"}]}]}}'

It is more convenient to work with json formatted data. I’m using pretty print pprint to print the data in an easy to read way.

Step through response.json(), then to response.json()['feed']['entry'][0]. A reminder that python starts indexing at 0, not 1!

pprint(response.json()['feed']['entry'][0])
{'archive_center': 'ASF',
 'boxes': ['-90 -180 90 180'],
 'browse_flag': False,
 'coordinate_system': 'CARTESIAN',
 'data_center': 'ASF',
 'dataset_id': 'SENTINEL-1A_SLC',
 'has_formats': False,
 'has_spatial_subsetting': False,
 'has_temporal_subsetting': False,
 'has_transforms': False,
 'has_variables': False,
 'id': 'C1214470488-ASF',
 'links': [{'href': 'https://vertex.daac.asf.alaska.edu/',
            'hreflang': 'en-US',
            'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#'}],
 'online_access_flag': True,
 'orbit_parameters': {},
 'organizations': ['ASF', 'ESA/CS1CGS'],
 'original_format': 'ECHO10',
 'platforms': ['Sentinel-1A'],
 'service_features': {'esi': {'has_formats': False,
                              'has_spatial_subsetting': False,
                              'has_temporal_subsetting': False,
                              'has_transforms': False,
                              'has_variables': False},
                      'harmony': {'has_formats': False,
                                  'has_spatial_subsetting': False,
                                  'has_temporal_subsetting': False,
                                  'has_transforms': False,
                                  'has_variables': False},
                      'opendap': {'has_formats': False,
                                  'has_spatial_subsetting': False,
                                  'has_temporal_subsetting': False,
                                  'has_transforms': False,
                                  'has_variables': False}},
 'short_name': 'SENTINEL-1A_SLC',
 'summary': 'Sentinel-1A slant-range product',
 'time_start': '2014-04-03T00:00:00.000Z',
 'title': 'SENTINEL-1A_SLC',
 'updated': '2021-07-15T19:16:39.000Z',
 'version_id': '1'}

The first response is not the result I am looking for TODO: because xyz…but it does show a few variables that we can use to further refine the search. So I want to print the name of the dataset (dataset_id) and the concept id (id). We can build this variable and print statement like we did above with the url variable. TODO: is it worth saying something about what “feed” and “entry” are?

collections = response.json()['feed']['entry']
for collection in collections:
    print(f'{collection["archive_center"]} {collection["dataset_id"]} {collection["id"]}')
ASF SENTINEL-1A_SLC C1214470488-ASF
ASF SENTINEL-1B_SLC C1327985661-ASF
ASF SENTINEL-1A_DUAL_POL_GRD_HIGH_RES C1214470533-ASF
ASF SENTINEL-1B_DUAL_POL_GRD_HIGH_RES C1327985645-ASF
ASF SENTINEL-1B_DUAL_POL_GRD_MEDIUM_RES C1327985660-ASF
ASF SENTINEL-1A_DUAL_POL_GRD_MEDIUM_RES C1214471521-ASF
ASF SENTINEL-1A_RAW C1214470561-ASF
ASF SENTINEL-1A_METADATA_SLC C1214470496-ASF
ASF SENTINEL-1B_METADATA_SLC C1327985617-ASF
ASF SENTINEL-1A_SINGLE_POL_GRD_HIGH_RES C1214470682-ASF

But there is a problem. We know from CMR-Hits that there are 49 datasets but only 10 are printed. This is because CMR restricts the number of results returned by a query. The default is 10 but it can be set to a maximum of 2000. Knowing that there were 49 ‘hits’, I’ll set page_size to 49. Then, we can re-run our for loop for the collections.

response = requests.get(url,
                        params={
                            'cloud_hosted': 'True',
                            'provider': provider,
                            'page_size': 49,
                        },
                        headers={
                            'Accept': 'application/json'
                        }
                       )
collections = response.json()['feed']['entry']
for collection in collections:
    print(f'{collection["archive_center"]} {collection["dataset_id"]} {collection["id"]}')
ASF SENTINEL-1A_SLC C1214470488-ASF
ASF SENTINEL-1B_SLC C1327985661-ASF
ASF SENTINEL-1A_DUAL_POL_GRD_HIGH_RES C1214470533-ASF
ASF SENTINEL-1B_DUAL_POL_GRD_HIGH_RES C1327985645-ASF
ASF SENTINEL-1B_DUAL_POL_GRD_MEDIUM_RES C1327985660-ASF
ASF SENTINEL-1A_DUAL_POL_GRD_MEDIUM_RES C1214471521-ASF
ASF SENTINEL-1A_RAW C1214470561-ASF
ASF SENTINEL-1A_METADATA_SLC C1214470496-ASF
ASF SENTINEL-1B_METADATA_SLC C1327985617-ASF
ASF SENTINEL-1A_SINGLE_POL_GRD_HIGH_RES C1214470682-ASF
ASF SENTINEL-1A_OCN C1214472977-ASF
ASF SENTINEL-1B_RAW C1327985647-ASF
ASF SENTINEL-1A_DUAL_POL_METADATA_GRD_HIGH_RES C1214470576-ASF
ASF SENTINEL-1B_OCN C1327985579-ASF
ASF SENTINEL-1A_METADATA_RAW C1214470532-ASF
ASF SENTINEL-1B_DUAL_POL_METADATA_GRD_HIGH_RES C1327985741-ASF
Alaska Satellite Facility Sentinel-1 Interferograms (BETA) C1595422627-ASF
ASF ALOS_AVNIR_OBS_ORI C1808440897-ASF
ASF SENTINEL-1A_SINGLE_POL_GRD_MEDIUM_RES C1214472994-ASF
ASF SENTINEL-1B_METADATA_RAW C1327985650-ASF
ASF SENTINEL-1B_SINGLE_POL_GRD_HIGH_RES C1327985571-ASF
ASF SENTINEL-1B_SINGLE_POL_GRD_MEDIUM_RES C1327985740-ASF
ASF SENTINEL-1A_METADATA_OCN C1266376001-ASF
ASF SENTINEL-1B_METADATA_OCN C1327985646-ASF
ASF SENTINEL-1A_DUAL_POL_METADATA_GRD_MEDIUM_RES C1214472336-ASF
ASF SENTINEL-1A_SINGLE_POL_METADATA_GRD_HIGH_RES C1214470732-ASF
ASF SENTINEL-1A_SINGLE_POL_METADATA_GRD_MEDIUM_RES C1214473170-ASF
ASF SENTINEL-1B_DUAL_POL_METADATA_GRD_MEDIUM_RES C1327985578-ASF
ASF SENTINEL-1B_SINGLE_POL_METADATA_GRD_HIGH_RES C1327985619-ASF
ASF SENTINEL-1B_SINGLE_POL_METADATA_GRD_MEDIUM_RES C1327985739-ASF
ASF STS-68_BROWSE_GRD C1661710593-ASF
ASF STS-68_BROWSE_SLC C1661710596-ASF
ASF STS-59_BROWSE_GRD C1661710578-ASF
ASF STS-59_BROWSE_SLC C1661710581-ASF
ASF SENTINEL-1A_DUAL_POL_GRD_FULL_RES C1214471197-ASF
ASF SENTINEL-1A_DUAL_POL_METADATA_GRD_FULL_RES C1214471960-ASF
ASF SENTINEL-1A_SINGLE_POL_GRD_FULL_RES C1214472978-ASF
ASF SENTINEL-1A_SINGLE_POL_METADATA_GRD_FULL_RES C1214473165-ASF
ASF SENTINEL-1B_DUAL_POL_GRD_FULL_RES C1327985697-ASF
ASF SENTINEL-1B_DUAL_POL_METADATA_GRD_FULL_RES C1327985651-ASF
ASF SENTINEL-1B_SINGLE_POL_GRD_FULL_RES C1327985644-ASF
ASF SENTINEL-1B_SINGLE_POL_METADATA_GRD_FULL_RES C1327985674-ASF
Alaska Satellite Facility Sentinel-1 Unwrapped Interferogram and Coherence Map (BETA) C1379535600-ASF
Alaska Satellite Facility Sentinel-1 Interferograms - Amplitude (BETA) C1596065640-ASF
Alaska Satellite Facility Sentinel-1 Interferograms - Coherence (BETA) C1596065639-ASF
Alaska Satellite Facility Sentinel-1 Interferograms - Connected Components (BETA) C1596065641-ASF
Alaska Satellite Facility Sentinel-1 Interferograms - Unwrapped Phase (BETA) C1595765183-ASF
ASF STS-59_GRD C1661710583-ASF
ASF STS-59_METADATA_GRD C1661710586-ASF