The European Space Agency
Contact us

APEx STAC catalogue integration with openEO

The project catalogue provided by APEx can be used in combination with an openEO-based platform.

Do note that openEO is an open standard, and not all platforms may support all use cases. Also the produced STAC metadata may have differences that are relevant to your project. Therefore we always recommend to check the envisioned use cases with your provider of choice.

Supported use cases

  1. Scripted upload of openEO generated results into APEx catalog (supported by any openEO instance).
  2. Direct publishing of results to object storage (requires support for openEO ‘workspace’ API).
  3. Direct publishing of results to STAC catalogue (requires support for openEO ‘workspace’ API, and specifically merging into existing catalogue.)

Use case 1: Scripted upload

This use case is most broadly supported because it simply takes STAC metadata of openEO job results, downloads it, and then ingests it into a catalogue.

The most important drawbacks of this method:

  • Data is first downloaded from openEO backend and then uploaded again, which can take time and bandwidth, and may even be interrupted when facing network instability.
  • Extra project specific code is needed to perform the task.
import openeo
from osgeo_utils.gdal2tiles import GlobalMercator

con = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc()
Authenticated using refresh token.

tilegrid = GlobalMercator()
bounds = tilegrid.TileLatLonBounds(233, 214, 9)
resolution = tilegrid.Resolution(12)
print(bounds)
print(resolution)


cube = con.load_collection(
    "SENTINEL2_L2A",
    bands=["B04", "B03", "B02"],
    temporal_extent="2019-08-19",
    spatial_extent={
        "west": bounds[1],
        "south": abs(bounds[2]),#abs is weird, does GlobalMercator return wrong latitude values?
        "east": bounds[3],
        "north": abs(bounds[0]),
    }
)
cube.result_node().update_arguments(featureflags={"tilesize": 256})#force block size in output tiff
result = cube.resample_spatial(resolution=resolution,projection="EPSG:3857")
job = result.execute_batch("gran_canaria.tiff",title="Gran Canaria",filename_prefix="gran_canaria")
(-28.30438068296276, -16.171874999999993, -27.68352808378777, -15.468750000000012)
38.21851414258813
Authenticated using refresh token.
0:00:00 Job 'j-241106f371e448088b97aeb8d0c9c19f': send 'start'
0:00:14 Job 'j-241106f371e448088b97aeb8d0c9c19f': created (progress 0%)
0:00:21 Job 'j-241106f371e448088b97aeb8d0c9c19f': created (progress 0%)
0:00:27 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:00:36 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:00:46 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:00:58 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:01:14 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:01:33 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:01:58 Job 'j-241106f371e448088b97aeb8d0c9c19f': running (progress N/A)
0:02:28 Job 'j-241106f371e448088b97aeb8d0c9c19f': finished (progress 100%)
job = con.job("j-241106f371e448088b97aeb8d0c9c19f")
job
import pystac

stac_metadata_dict = job.get_results().get_metadata()
stac_metadata_dict["id"] = "gran_canaria_rgb"
#remove collection assets, we will rely on item links
del stac_metadata_dict["assets"]
collection = pystac.Collection.from_dict(stac_metadata_dict)
collection

Convert online openEO collection to local collection

This step cleans up links to avoid that they point to the openEO API, which in some cases requires authentication.

There’s an open issue to integrate this in the API.

#remove collection and canoncial links
collection.remove_links(rel="collection")
collection.remove_links(rel="canonical")

items = list(collection.get_stac_objects(rel=pystac.RelType.ITEM))
for i in items:
    i.remove_links(rel="collection")
    i.remove_links(rel="canonical")
    i.



collection.set_self_href("/tmp/collection.json")
collection.normalize_hrefs('/tmp/', skip_unresolved=True)

collection.license = "CC-BY-4.0"

def asset_transform(name,a):
    a.href = "/tmp/" + name
    return a

#this step can transform asset hrefs as well, 
#c2=catalog.map_assets(asset_transform)



collection.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
collection

Upload to STAC API

The collection metadata has now been cleaned up and can be added to the STAC API. Here we create the full collection, but it’s also possible to only add the generated item to a new collection.

At this point, it is also recommended to improve the metadata quality by adding additional metadata properties.

Important note

At this point, the actual Geotiff is still located on the openEO backend, and can be accessed by any tool via the signed url. This url will however expire, so for more permanent catalogues, the data file needs to be moved to a better location. To do this, simply download the tiff file and upload it to an online location of your preference.

import requests
from owslib.ogcapi.records import Records
from owslib.util import Authentication

class BearerAuth(requests.auth.AuthBase):
    def __init__(self, token):
        self.token = token
    def __call__(self, r):
        r.headers["authorization"] = "Bearer " + self.token
        return r
    
auth = BearerAuth("""PROVIDE_TOKEN_HERE""")

r = Records("https://catalogue.project-a.apex.esa.int",auth=Authentication(auth_delegate=auth))
coll_dict = collection.to_dict()

default_auth = {
    "_auth": {
        "read": ["anonymous"],
        "write": ["stac-openeo-admin", "stac-openeo-editor"]
    }
}

coll_dict.update(default_auth)

response = requests.post("https://catalogue.project-a.apex.esa.int/collections", auth=auth,json=coll_dict)
response
<Response [201]>
for item in collection.get_all_items():
    item_dict = item.to_dict()
    print(item)
    item_dict["collection"] = collection.id
    r.collection_item_create(collection.id, item_dict)
{'type': 'Feature', 'stac_version': '1.0.0', 'id': 'gran_canaria_2019-08-19Z.tif', 'properties': {'datetime': '2019-08-19T00:00:00Z', 'proj:bbox': [-1800244.89, 3209132.196, -1721973.373, 3287403.712], 'proj:epsg': 3857, 'proj:shape': [2048, 2048]}, 'geometry': {'coordinates': [[[-16.17187499999999, 27.68352808378774], [-16.17187499999999, 28.304380682962755], [-15.468749999999991, 28.304380682962755], [-15.468749999999991, 27.68352808378774], [-16.17187499999999, 27.68352808378774]]], 'type': 'Polygon'}, 'links': [{'rel': 'root', 'href': '../collection.json', 'type': 'application/json', 'title': 'Gran Canaria'}, {'rel': 'self', 'href': '/tmp/gran_canaria_2019-08-19Z.tif/gran_canaria_2019-08-19Z.tif.json', 'type': 'application/json'}, {'rel': 'parent', 'href': '../collection.json', 'type': 'application/json', 'title': 'Gran Canaria'}], 'assets': {'gran_canaria_2019-08-19Z.tif': {'href': 'https://openeo.dataspace.copernicus.eu/openeo/1.2/jobs/j-241106f371e448088b97aeb8d0c9c19f/results/assets/MzJjYzdkZGItZjdlMS00YjFjLTk3OTYtZjlmZTM5Y2I4ZmVi/3f3276dfba5bb7235326dea697f556c1/gran_canaria_2019-08-19Z.tif?expires=1731512915', 'type': 'image/tiff; application=geotiff', 'title': 'gran_canaria_2019-08-19Z.tif', 'eo:bands': [{'center_wavelength': 0.6646, 'name': 'B04'}, {'center_wavelength': 0.5598, 'name': 'B03'}, {'center_wavelength': 0.4924, 'name': 'B02'}], 'proj:bbox': [-1800244.89, 3209132.196, -1721973.373, 3287403.712], 'proj:epsg': 3857, 'proj:shape': [2048, 2048], 'raster:bands': [{'name': 'B04', 'statistics': {'maximum': 18494.0, 'mean': 765.91613554958, 'minimum': -301.0, 'stddev': 534.21168059182, 'valid_percent': 100.0}}, {'name': 'B03', 'statistics': {'maximum': 20682.0, 'mean': 699.08895897867, 'minimum': -560.0, 'stddev': 449.31294931756, 'valid_percent': 100.0}}, {'name': 'B02', 'statistics': {'maximum': 21562.0, 'mean': 678.19460010522, 'minimum': -980.0, 'stddev': 412.96003458047, 'valid_percent': 100.0}}], 'roles': ['data']}}, 'bbox': [-16.17187499999999, 27.68352808378774, -15.468749999999991, 28.304380682962755], 'stac_extensions': ['https://stac-extensions.github.io/eo/v1.1.0/schema.json', 'https://stac-extensions.github.io/file/v2.1.0/schema.json', 'https://stac-extensions.github.io/projection/v1.1.0/schema.json'], 'collection': 'j-241106f371e448088b97aeb8d0c9c19f', 'epsg': 3857}

requests.delete("https://catalogue.project-a.apex.esa.int/collections/" + collection.id, auth=auth)
<Response [204]>

Use case 2/3: direct upload by openEO

In this case, we will let openEO export the collection to an external storage.

Because this case depends on the availability of the export_workspace process, we check for its existence first.


con.describe_process("export_workspace")
exported_result = result.export_workspace("my_workspace_id", merge="my_stac_collection")

job = exported_result.execute_batch("gran_canaria.tiff",title="Gran Canaria",filename_prefix="gran_canaria")