GitLab

sobloo APIs in action: search & download items

APIs & Authentication

Several sobloo APIs require to be authenticated to work. Right now this is the case only for the download API. All the other ones are open and accessible without authentication.

The current version of Python is: v3.7.X

The APIs that can be used that:

API	BASE URL	Requires authentication
REST search	https://sobloo.eu/api/v1/services/search	No
Opensearch	https://sobloo.eu/api/v1/services/opensearch	No
Free Download	https://free.sobloo.eu/api/v1/services/download	Yes
Download	https://sobloo.eu/api/v1/services/download	Yes
Quicklook	https://sobloo.eu/api/v1/services/quicklook	No
Thumbnail	https://sobloo.eu/api/v1/services/thumbnail	No
WMTS	https://sobloo.eu/api/v1/services/wmts	No
WFS	https://sobloo.eu/api/v1/services/wfs	No
CSW	https://sobloo.eu/api/v1/services/csw	No
WCS	https://sobloo.eu/api/v1/services/wcs	No
WMS	https://sobloo.eu/api/v1/services/wms	No

The code below allows accessing sobloo API in authenticated or non-authenticated mode. Basically, to call a protected API, you just have to submit the secret of your API key in the "authorization" header of the http call. To call non authenticated API, you do not have to inject anything in the headers.

As a pratical example, let's check the code below:

def callAPI(cmd, auth):
	    if (auth == 1):
		# we inject here the SECRET APIKEY from our sobloo account page
	        headers={"Authorization":"Apikey " + APIKEY_SECRET}
	        r = requests.get(cmd, headers=headers, verify=False)
	    else:
		# this API does not need authentication	 
		r = requests.get(cmd, verify=False)	 

	    if( r.status_code != requests.codes.ok ):	        
		print("Error: %s  for the request cmd <%s>." % (r.status_code,cmd))
	        return -1	    
	    else:	       
		# return the response
	        return r

The function callAPI takes 2 arguments: the command to send (cmd) and if we want authentication or not (auth). The "cmd" is the URL we want to access to and "auth" can be "1" for authenticated APIs or "0" if no authentication. In the case of authenticated APIs, the secret must be injected in the "Authorization" header. To generate an API key you can have a look to Authentication tutorial.

Let's call the WFS API

With the use of the callAPI function defined above, we just need to submit the URL of the WFS API endpoint as shown in the following code:

def wfsQuery():
	    cmd = "https://sobloo.eu/api/v1/services/wfs?service=WFS&version=3.0.0&request=GetCapabilities"	
	    ret = callAPI(cmd,0)
	    if (ret != -1):
		print(ret.content)

In the example above we query the "GetCapabilities" of the WFS in an unauthenticated way.

You can use the same example to call the CSW API and the opensearch one as they all do not require authentication.

REST Search API in action: filter items

Generally, when you want to use Copernicus Sentinel data, you focus on a particular satellite (or family), an area and a specific time window. To get all the items of a certain satellite family, you can use filters that you inject in your query such as:

def listSentinelProducts(family):
	    #family must be something like Sentinel-1A or Sentinel-2A etc.
	    cmd = "https://sobloo.eu/api/v1/services/search?f=acquisition.missionName:eq:" + family
	    ret = callAPI(cmd,0)
	    if (ret != -1):	
		print(ret.content)

A filter is injected directly in the URL in standard querying form: "?f=". In the above example, we use the filter on the metadata "acquisition.missionName" followed by the unary operator "eq" (equal) and the string we are looking for. The complete sentance will something like: ?f=acquisition.missionName:eq:"Sentinel-1A" to get all Sentinel-1A items available in the catalogue.

In order to retrieve only products that are available in sobloo local storage one can use the specific filter f=state.services.download:eq:internal. The above query with this filter will look like:

cmd = "https://sobloo.eu/api/v1/services/search?f=state.services.download:eq:internal&f=acquisition.missionName:eq:" + family

Business key and internal id

In the catalogue, the items can be retrieved by their "internal id" (iid). However you more likely want to access a product by its name that we call the business key. Hopefully, sobloo offers a simple way to get one or the other as shown in the example below:

def getInternalId(prodName):
	    # query to rest search API to get the internal ID of a product through its name
	    cmd = "https://sobloo.eu/api/v1/services/search?f=identification.externalId:eq:" + prodName + "&include=previews,identification&pretty=true"
	    # this API does not need authentication
	    r = requests.get(cmd, verify=False)
	    if( r.status_code != requests.codes.ok ):
	        print("Error: %s  for the request cmd <%s>." % (r.status_code,cmd))
	        internalId = -1
	    else:
	        # the returned json object contains the internal Id (only one product returned)
	        #print(r.json())
	        j = r.json()
            if j["totalnb"] == 0:
	        print("No matching products found")
	        internalId = -1
	    else:
	        internalId = j['hits'][0]['data']['uid']
	        return internalId

The function above simply returns the internal id of the item associated to the product name submitted (business key). To do so, we use the filter "f=identification.externalId:eq:[business key]" that filter the items that corresponds to "business key" (product name) submitted. If at least an element is found, the code above returns the first internal id that matches the product name.

Get the quicklook of a product

Knowing the name of a product (ex: the Sentinel 2A product "S2A_MSIL1C_20180405T235621_N0206_R116_T01WDV_20180406T014524"), You can download its quicklook simply by using the quicklook API as follows:

def getQuickLook(prodName):
	    iid = getInternalId(prodName)
	    if (iid == -1):
	        print("No corresponding data.")
	    else:
	        cmd = "https://sobloo.eu/api/v1/services/quicklook/" + iid
	        ret = callAPI(cmd,0)
	    if (ret != -1):    
	        # Write the data in the current directory
	        filename = "quicklook_" + prodName + ".png"
	        with open(filename, "w+b") as f:
	            f.write(ret.content)
	        return 0

Using the same way, you can also download the thumbnail by replacing 'quicklook' by 'thumbnail' in the command URL.

Access one product metadata

You can display product metadata (JSON format) as follows:

def oneProductMeta(prodName):
	    # query to get one product from its name
	    cmd = "https://sobloo.eu/api/v1/services/search?f=identification.externalId:eq:" + prodName + "&include=previews,identification&pretty=true"
	    ret = callAPI(cmd,0)
	    if(ret != -1 ):
	        j = ret.json()
	        print("Nb product(s): %s" % (j["totalnb"]))
	        # Check that there is only one product  
	        if( j["totalnb"] != 1):
	            print("Error, there are more than one product to display")
	        else:
	            print(ret.content)

Display a JSON file which contains metadata (coordinates, geometry data, spatial coverage data, … and the internalId).

How can I download a product

The data access download API wants the "internalId of a product. Now that we have the function getInternalId (see above), downloading an item is a straighforward operation as shown in the code below:

def downloadProd(prodName):
	    # Get product internal Id
	    iid = getInternalId(prodName)
	    if (iid == -1):
	        print("Product %s not found" % (prodName))
	        return -1
	    # once we get the internal ID of the product we can start downloading it through the API
	    # note the APIKEY injection in the headers to authenticate
	    cmd = "https://sobloo.eu/api/v1/services/download/" + iid
	    # we inject here the APIKEY got from our account page
	    headers={"Authorization":"Apikey " + APIKEY_SECRET}
	    r = requests.get(cmd, headers=headers, verify=False)
	    if( r.status_code != requests.codes.ok ):
	        print("Error: %s  for the request cmd <%s>." % (r.status_code,cmd))
	        return -1
	    else:
	        # Write the data in the current directory
	        filename = prodName + ".zip"
	        with open(filename, "w+b") as f:
	            f.write(r.content)
	        return 0

Note that the download API requires authentication and that the apikey secret is injected in the http headers.

Let’s take a practical example: I would like to find products in the area of Spain

The geonames information are available within sobloo and can be accessed by playing with opensearch API. You can easily figure out what parameter to use in the "search" queries by pointing your browser to: https://sobloo.eu/api/v1/services/opensearch. In the sample code below, we will use the features "enrichment.geonames" to retrieve results following their geoname tag:

def listProductGeoname(typeGeo,thegeoname):
	    # Careful the search query does not allow going beyond the 10000th element
	    # So a query such as &&from=9999&&size=2 ends with an HTTP 500 error, same for &&from=10000&&size=1
	    if (typeGeo == "country"):
	        filter = "enrichment.geonames.name"
	    elif (typeGeo == "states"):
	        filter = "enrichment.geonames.states.name"
	    elif (typeGeo == "county"):
	        filter = "enrichment.geonames.states.counties.name"
	    elif (typeGeo == "city"):
	        filter = "enrichment.geonames.states.counties.cities.name"
	    elif (typeGeo == "village"):
	        filter = "enrichment.geonames.states.counties.villages.name"
	    else:
	        filter = "all"
	        
	    offset = 0
	    nbReturn=1000
	    # Make a first call to retrieve how many elements will be returned
	    cmd = "https://sobloo.eu/api/v1/services/search?f=" + filter + ":like:" + thegeoname +"&&size=" + str(nbReturn)
	    ret = callAPI(cmd,0)
	    if (ret != -1):
	        try:
	            nbHits=ret.json()["totalnb"]
	            print("nbElements returned: %s" % nbHits)
	            if (nbHits == 0):
	                print("No hits !")
	                return
	            # Iterate the query by steps of nbReturn elements to display them all
	            if (nbHits > nbReturn):
	                nbCalls=int((nbHits/nbReturn))
	                for i in range(0,nbCalls):
	                    cmd = "https://sobloo.eu/api/v1/services/search?f=" + filter + ":like:" + thegeoname + "&&from=" + str(i*nbReturn) + "&&size=" + str(nbReturn)
	                    ret2 = callAPI(cmd,0)
	                    printProductDetail(ret2.json())
	            else:
	                printProductDetail(ret.json())  
	        except:
	            print("Unexpected exception %s" % (sys.exc_info()[0]))    
	    else:
	        print("No elements returned") 

	def is_empty(any_structure):
	    if any_structure:
	        return False
	    else:
	        return True 

	def printProductDetail(jsonObject):
	    # Display the results
	    if is_empty(jsonObject):
	        print("The object to be printed is empty.")
	        return -1
	    
	    for elements in jsonObject["hits"]:
	        uid="empty"
	        try:
	            uid = elements["data"]["uid"]
	            ident=elements["data"]["identification"]
	            acq = elements["data"]["acquisition"]
	            prodName= elements["data"]["identification"]["externalId"]
	            prodType=elements["data"]["identification"]["type"]
	            prodLevel=elements["data"]["production"]["levelCode"]
	            geoname= elements["data"]["enrichment"]["geonames"]
	            uid = elements["data"]["uid"]
	            orbit=elements["data"]["orbit"]
	            sizeMB = elements["data"]["archive"]["size"]
	            location=elements["data"]["spatialCoverage"]["geometry"]["centerPoint"]
	            ident = elements["data"]["identification"]
	            print("uid: %s name: %s acq: %s" % (uid,prodName,acq))
	        except:
	            print("Unexpected exception %s while rendering object with uid: %s ." % (sys.exc_info()[0],uid)) 

	###### MAIN #######
	# Get all products in Spain (country) area
	listProductGeoname("country","Spain")

In this example, you first see how the "geoname" query is built:

cmd = "https://sobloo.eu/api/v1/services/search?f=" + filter + ":like:" + thegeoname +"&&size=" + str(nbReturn)

The parameters "size" allows to fetch a fixed number of items (here nbReturn). The overall number of elements returned by the query is retrieved by: nbHits=ret.json()["totalnb"]. Then we just have to move forward step by step in the elements as follow to list all returned elements:

cmd = "https://sobloo.eu/api/v1/services/search?f=" + filter + ":like:" + thegeoname + "&&from=" + str(i*nbReturn) + "&&size=" + str(nbReturn)

Here we specify to get results "from" (with an offset representing from what position we want to start in the records) and again "size" to say how many elements we want. Finally the function "printProductDetail" displays the elements returned.