Accessing Datasets under an Access Control List (ACL)¶

NASA Earthdata API Client 🌍¶

Note: Before we can use earthaccess we need an account with NASA EDL

In [1]:

Copied!

from earthaccess import Auth, DataCollections, DataGranules, Store

auth = Auth()
from earthaccess import Auth, DataCollections, DataGranules, Store

auth = Auth()

Auth()¶

earthaccess's Auth class provides 3 different strategies to authenticate ourselves with NASA EDL.

netrc: Do we have a .netrc file with our EDL credentials? if so, we can use it with earthaccess. If we don't have it and want to create one we can, earthaccess allows users to type their credentials and persist them into a .netrc file.
environment: If we have our EDL credentials as environment variables
- EARTHDATA_USERNAME
- EARTHDATA_PASSWORD
interactive: We will be asked for our EDL credentials with optional persistence to .netrc

To persist our credentials to a .netrc file we have to do the following:

auth.login(strategy="interactive", persist=True)

In this notebook we'll use the environment method followed by the netrc strategy. You can of course use the interactive strategy if you don't have a .netrc file.

In [2]:

Copied!





auth.login(strategy="environment")
# are we authenticated?
if not auth.authenticated:
    auth.login(strategy="netrc")
auth.login(strategy="environment")
# are we authenticated?
if not auth.authenticated:
    auth.login(strategy="netrc")

Authentication with Earthdata Login failed with:
{"error":"invalid_credentials","error_description":"Invalid user credentials"}
NoneType: None

---------------------------------------------------------------------------
LoginAttemptFailure                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 auth.login(strategy="environment")
      2 # are we authenticated?
      3 if not auth.authenticated:

File ~/checkouts/readthedocs.org/user_builds/earthaccess/checkouts/1360/earthaccess/auth.py:152, in Auth.login(self, strategy, persist, system)
    150     self._netrc()
    151 elif strategy == "environment":
--> 152     self._environment()
    154 return self

File ~/checkouts/readthedocs.org/user_builds/earthaccess/checkouts/1360/earthaccess/auth.py:305, in Auth._environment(self)
    302     raise LoginStrategyUnavailable(msg)
    304 logger.debug("Using environment variables for EDL")
--> 305 return self._get_credentials(username, password, token)

File ~/checkouts/readthedocs.org/user_builds/earthaccess/checkouts/1360/earthaccess/auth.py:324, in Auth._get_credentials(self, username, password, user_token)
    322     msg = f"Authentication with Earthdata Login failed with:\n{token_resp.text}"
    323     logger.exception(msg)
--> 324     raise LoginAttemptFailure(msg)
    326 logger.info("You're now authenticated with NASA Earthdata Login")
    328 token = token_resp.json()

LoginAttemptFailure: Authentication with Earthdata Login failed with:
{"error":"invalid_credentials","error_description":"Invalid user credentials"}

Querying for restricted datasets¶

The DataCollection client can query CMR for any collection (dataset) using all of CMR's Query parameters and has built-in functions to extract useful information from the response.

auth.refresh_tokens()

If we belong to an early adopter group within NASA we can pass the Auth object to the other classes when we instantiate them.

# An anonymous query to CMR
Query = DataCollections().keyword('elevation')
# An authenticated query to CMR
Query = DataCollections(auth).keyword('elevation')

and it's the same with DataGranules

# An anonymous query to CMR
Query = DataGranules().keyword('elevation')
# An authenticated query to CMR
Query = DataGranules(auth).keyword('elevation')

Note: Some collections under an access control list are flagged by CMR and won't count when asking about results with hits().

In [3]:

Copied!





# The first step is to create a DataCollections query
Query = DataCollections()

# Use chain methods to customize our query
Query.short_name("ATL06").version("006")

print(f"Collections found: {Query.hits()}")

# filtering what UMM fields to print, to see the full record we omit the fields filters
# meta is always included as
collections = Query.fields(["ShortName", "Version"]).get(5)
# Inspect some results printing just the ShortName and Abstract
collections
# The first step is to create a DataCollections query
Query = DataCollections()

# Use chain methods to customize our query
Query.short_name("ATL06").version("006")

print(f"Collections found: {Query.hits()}")

# filtering what UMM fields to print, to see the full record we omit the fields filters
# meta is always included as
collections = Query.fields(["ShortName", "Version"]).get(5)
# Inspect some results printing just the ShortName and Abstract
collections

Collections found: 0

Out[3]:

[]

In [4]:

Copied!

if not auth.refresh_tokens():
    print("Something went wrong, we may need to regenerate our tokens manually")
if not auth.refresh_tokens():
    print("Something went wrong, we may need to regenerate our tokens manually")

Something went wrong, we may need to regenerate our tokens manually

/tmp/ipykernel_1614/2137380767.py:1: DeprecationWarning: No replacement, as tokens are now refreshed automatically.
  if not auth.refresh_tokens():

In [5]:

Copied!





Query = DataCollections(auth)

# Use chain methods to customize our query
Query.short_name("ATL06").version("006")

# This will say 1, even though we get 2 back.
print(f"Collections found: {Query.hits()}")

collections = Query.fields(["ShortName", "Version"]).get()
# Inspect some results printing just the ShortName and Abstract
collections
Query = DataCollections(auth)

# Use chain methods to customize our query
Query.short_name("ATL06").version("006")

# This will say 1, even though we get 2 back.
print(f"Collections found: {Query.hits()}")

collections = Query.fields(["ShortName", "Version"]).get()
# Inspect some results printing just the ShortName and Abstract
collections

Collections found: 0

Out[5]:

[]

Oh no! What!? only 1 collection found even though we got 2 results back?!

Interpreting the results¶

The hits() method above will tell you the number of query hits, but only for publicly available data sets. In this case because cloud hosted ICESat-2 data are not yet publicly available, CMR will return “1” hits, if you filtered DataCollections by provider = NSIDC_CPRD you'll get 0 hits. For now we need an alternative method of seeing how many cloud data sets are available at NSIDC. This is only temporary until cloud-hosted ICESat-2 become publicly available. We can create a collections object (we’re going to want one of these soon anyhow) and print the len() of the collections object to see the true number of hits.

Note: Since we cannot rely on hits() we need to be aware that get() may get us too many metadata records depending on the dataset and how broad our query is.

In [6]:

Copied!





Query = (
    DataGranules(auth)
    .concept_id("C2153572614-NSIDC_CPRD")
    .bounding_box(-134.7, 58.9, -133.9, 59.2)
    .temporal("2020-03-01", "2020-03-30")
)

# Unfortunately the hits() methods will behave the same for granule queries
print(f"Granules found with hits(): {Query.hits()}")

cloud_granules = Query.get()

print(f"Actual number found: {len(cloud_granules)}")
Query = (
    DataGranules(auth)
    .concept_id("C2153572614-NSIDC_CPRD")
    .bounding_box(-134.7, 58.9, -133.9, 59.2)
    .temporal("2020-03-01", "2020-03-30")
)

# Unfortunately the hits() methods will behave the same for granule queries
print(f"Granules found with hits(): {Query.hits()}")

cloud_granules = Query.get()

print(f"Actual number found: {len(cloud_granules)}")

Granules found with hits(): 0

Actual number found: 0

In [7]:

Copied!

store = Store(auth)
files = store.get(cloud_granules, "./data/C2153572614-NSIDC_CPRD/")
store = Store(auth)
files = store.get(cloud_granules, "./data/C2153572614-NSIDC_CPRD/")

The current session is not authenticated with NASA

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 store = Store(auth)
      2 files = store.get(cloud_granules, "./data/C2153572614-NSIDC_CPRD/")

File ~/checkouts/readthedocs.org/user_builds/earthaccess/checkouts/1360/earthaccess/store.py:264, in Store.__init__(self, auth, pre_authorize)
    262     logger.warning("The current session is not authenticated with NASA")
    263     self.auth = None
--> 264 self.in_region = self._running_in_us_west_2()

File ~/checkouts/readthedocs.org/user_builds/earthaccess/checkouts/1360/earthaccess/store.py:285, in Store._running_in_us_west_2(self)
    284 def _running_in_us_west_2(self) -> bool:
--> 285     session = self.auth.get_session()
    286     try:
    287         # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
    288         token_ = session.put(
    289             "http://169.254.169.254/latest/api/token",
    290             headers={"X-aws-ec2-metadata-token-ttl-seconds": "21600"},
    291             timeout=1,
    292         )

AttributeError: 'NoneType' object has no attribute 'get_session'

In [ ]: