The European Space Agency
Contact us

Using the APEx STAC Catalogue with openEO

This notebook demonstrates how the APEx Product Catalogue can be integrated with an openEO-based platform.

Important
Please note that openEO is an open standard, and not all platforms may support every use case. Additionally, the generated STAC metadata may vary based on your project’s requirements. Therefore, we strongly recommend verifying your specific use cases with your chosen provider.

Supported Use Cases

  1. Scripted upload of results generated by openEO(supported by all openEO instances).
  2. Direct publishing of results to object storage (requires support for the openEO ‘workspace’ API).
  3. Direct publishing of results to a STAC catalogue (requires support for the openEO ‘workspace’ API, including merging into an existing catalogue).

In this notebook, we will focus on demonstrating use case 1 in detail, with a brief overview of how to implement use cases 2 and 3.

!pip install openeo owslib

import openeo
import pystac
import requests
import json

from owslib.ogcapi.records import Records
from owslib.util import Authentication
Requirement already satisfied: openeo in /opt/conda/lib/python3.11/site-packages (0.39.1)
Requirement already satisfied: owslib in /opt/conda/lib/python3.11/site-packages (0.32.1)
Requirement already satisfied: requests>=2.26.0 in /opt/conda/lib/python3.11/site-packages (from openeo) (2.32.3)
Requirement already satisfied: shapely>=1.6.4 in /opt/conda/lib/python3.11/site-packages (from openeo) (2.0.7)
Requirement already satisfied: numpy>=1.17.0 in /opt/conda/lib/python3.11/site-packages (from openeo) (2.2.3)
Requirement already satisfied: xarray!=2025.01.2,>=0.12.3 in /opt/conda/lib/python3.11/site-packages (from openeo) (2025.1.1)
Requirement already satisfied: pandas>0.20.0 in /opt/conda/lib/python3.11/site-packages (from openeo) (2.2.3)
Requirement already satisfied: pystac<1.12,>=1.5.0 in /opt/conda/lib/python3.11/site-packages (from openeo) (1.11.0)
Requirement already satisfied: deprecated>=1.2.12 in /opt/conda/lib/python3.11/site-packages (from openeo) (1.2.18)
Requirement already satisfied: lxml in /opt/conda/lib/python3.11/site-packages (from owslib) (5.3.1)
Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.11/site-packages (from owslib) (2.9.0)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.11/site-packages (from owslib) (6.0.2)
Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.11/site-packages (from deprecated>=1.2.12->openeo) (1.17.2)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.11/site-packages (from pandas>0.20.0->openeo) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /opt/conda/lib/python3.11/site-packages (from pandas>0.20.0->openeo) (2025.1)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.11/site-packages (from python-dateutil->owslib) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests>=2.26.0->openeo) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests>=2.26.0->openeo) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests>=2.26.0->openeo) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests>=2.26.0->openeo) (2024.8.30)
Requirement already satisfied: packaging>=23.2 in /opt/conda/lib/python3.11/site-packages (from xarray!=2025.01.2,>=0.12.3->openeo) (24.1)

Building the openEO Processing Graph

Before diving into the different use cases, we will first build a simple openEO workflow. In this example, we will calculate the NDVI for a single Sentinel-2 observation over Gran Canaria. This section assumes the reader has basic knowledge of openEO and will help us focus on the integration with the APEx Product Catalogue.

con = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc()
Authenticated using refresh token.
spatial_extent = {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "coordinates": [
          [
            [
              -15.918605217697035,
              28.205553799401358
            ],
            [
              -15.918605217697035,
              27.72422378541127
            ],
            [
              -15.305188375387814,
              27.72422378541127
            ],
            [
              -15.305188375387814,
              28.205553799401358
            ],
            [
              -15.918605217697035,
              28.205553799401358
            ]
          ]
        ],
        "type": "Polygon"
      }
    }
cube = con.load_collection(
    "SENTINEL2_L2A",
    bands=["B04", "B08"],
    temporal_extent="2019-08-19",
    spatial_extent=spatial_extent
)
cube.result_node().update_arguments(featureflags={"tilesize": 256}) #force block size in output tiff

ndvi = cube.ndvi()

result = ndvi.resample_spatial(resolution=38.21851414258813, projection="EPSG:3857")
result

Use Case 1: Scripted Upload

This use case is the most widely supported, as it involves retrieving the STAC metadata of openEO job results, downloading it, and then ingesting it into an APEx Product Catalogue.

Key drawbacks of this method include:

  • The need to first download data from the openEO backend and then re-upload it, which can be time-consuming and bandwidth-intensive. This process may also be interrupted by network instability. Such cases could be avoided when in use cases 2 and 3.
  • Additional project-specific code is required to automate this task.

We start this example by downloading the result through an openEO batch job.

job = result.execute_batch("gran_canaria.tiff",title="APEx Demo - Gran Canaria - NDVI")
0:00:00 Job 'j-250313114527493e83f90fc346ad8fee': send 'start'
0:00:13 Job 'j-250313114527493e83f90fc346ad8fee': created (progress 0%)
0:00:18 Job 'j-250313114527493e83f90fc346ad8fee': created (progress 0%)
0:00:25 Job 'j-250313114527493e83f90fc346ad8fee': created (progress 0%)
0:00:33 Job 'j-250313114527493e83f90fc346ad8fee': created (progress 0%)
0:00:43 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:00:55 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:01:11 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:01:30 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:01:54 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:02:24 Job 'j-250313114527493e83f90fc346ad8fee': running (progress N/A)
0:03:01 Job 'j-250313114527493e83f90fc346ad8fee': finished (progress 100%)
job = con.job("j-250313110405469ea3095a5b2b06ce85")
job

In this next step we are going to download the job metadata and create a STAC collection using the pystac library.

stac_metadata_dict = job.get_results().get_metadata()
stac_metadata_dict["id"] = "gran_canaria_ndvi"


# #remove collection assets, we will rely on item links
# del stac_metadata_dict["assets"]
collection = pystac.Collection.from_dict(stac_metadata_dict)
collection.license = "CC-BY-4.0"
collection

Converting an online openEO Collection to a local Collection

An important consideration until now is that we are currently using openEO API URLs in the STAC metadata. It’s important to note that some of these URLs may require authentication. Additionally, openEO only stores batch job results temporarily. Therefore, if long-term access to the results is needed, the files should be stored in a separate, accessible storage solution, such as an S3 bucket. Use cases 2 and 3 provide a solution for this.

The code below provides an example of how to update the STAC metadata to point to a URL outside of openEO. For this example, we use a placeholder value (/tmp).

Currently, there is an open issue aimed at improving this behavior in openEO.

# Remove collection and canoncial links from both the collection and their items
collection.remove_links(rel="collection")
collection.remove_links(rel="canonical")

items = list(collection.get_stac_objects(rel=pystac.RelType.ITEM))
for i in items:
    i.remove_links(rel="collection")
    i.remove_links(rel="canonical")


# Normalize all the hrefs to use the new path /tmp
collection.set_self_href("./collection.json")
collection.normalize_hrefs('/tmp/', skip_unresolved=True)

collection.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)
collection

Upload to STAC API

The collection metadata has now been cleaned up and can be added to the STAC API. Here we create the full collection, but it’s also possible to only add the generated item to a new collection.

At this point, it is also recommended to improve the metadata quality by adding additional metadata properties.

Important note

At this point, the actual Geotiff is still located on the openEO backend, and can be accessed by any tool via the signed url. This url will however expire, so for more permanent catalogues, the data file needs to be moved to a better location. To do this, simply download the tiff file and upload it to an online location of your preference.

CATALOGUE = "https://catalogue.demo.apex.esa.int"

Authentication for APEx Product Catalogue

To interact with an APEx Product Catalogue, you need a valid authentication token. This token ensures that you can write metadata to an APEx-initialized STAC catalogue, provided you have been granted sufficient access rights.

To obtain an authentication token, follow the dedicated guide on generating your OIDC token. This step is essential for securely accessing and modifying catalogue records.

# Helper class to support the authentication with owslib and requests libraries
class BearerAuth(requests.auth.AuthBase):
    def __init__(self, token):
        self.token = token
    def __call__(self, r):
        r.headers["authorization"] = "Bearer " + self.token
        return r
    

token = input("Please provide your OIDC token here")
auth = BearerAuth(token)
r = Records(CATALOGUE,auth=Authentication(auth_delegate=auth))
print(json.dumps(r.collections(), indent=2))
{
  "collections": [],
  "links": [
    {
      "rel": "root",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/"
    },
    {
      "rel": "parent",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/"
    },
    {
      "rel": "self",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/collections"
    }
  ]
}
coll_dict = collection.to_dict()

default_auth = {
    "_auth": {
        "read": ["anonymous"],
        "write": ["stac-admin-prod"]
    }
}

coll_dict.update(default_auth)
response = requests.post(f"{CATALOGUE}/collections", auth=auth,json=coll_dict)
response
<Response [201]>
print(json.dumps(r.collections()['collections'][0], indent=2))
{
  "id": "gran_canaria_ndvi",
  "description": "Results for batch job j-250313110405469ea3095a5b2b06ce85",
  "stac_version": "1.0.0",
  "links": [
    {
      "rel": "items",
      "type": "application/geo+json",
      "href": "https://catalogue.demo.apex.esa.int/collections/gran_canaria_ndvi/items"
    },
    {
      "rel": "parent",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/"
    },
    {
      "rel": "root",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/"
    },
    {
      "rel": "self",
      "type": "application/json",
      "href": "https://catalogue.demo.apex.esa.int/collections"
    },
    {
      "href": "https://catalogue.demo.apex.esa.int/eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RDR_20230507T090239.SAFE",
      "rel": "derived_from",
      "type": "application/json",
      "title": "Derived from /eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RDR_20230507T090239.SAFE"
    },
    {
      "href": "https://catalogue.demo.apex.esa.int/eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RCR_20230507T090239.SAFE",
      "rel": "derived_from",
      "type": "application/json",
      "title": "Derived from /eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RCR_20230507T090239.SAFE"
    },
    {
      "href": "https://catalogue.demo.apex.esa.int/eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RDS_20230507T090239.SAFE",
      "rel": "derived_from",
      "type": "application/json",
      "title": "Derived from /eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RDS_20230507T090239.SAFE"
    },
    {
      "href": "https://catalogue.demo.apex.esa.int/eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RCS_20230507T090239.SAFE",
      "rel": "derived_from",
      "type": "application/json",
      "title": "Derived from /eodata/Sentinel-2/MSI/L2A_N0500/2019/08/19/S2A_MSIL2A_20190819T115221_N0500_R123_T28RCS_20230507T090239.SAFE"
    },
    {
      "href": "http://ceos.org/ard/files/PFS/SR/v5.0/CARD4L_Product_Family_Specification_Surface_Reflectance-v5.0.pdf",
      "rel": "card4l-document",
      "type": "application/pdf"
    }
  ],
  "stac_extensions": [
    "https://stac-extensions.github.io/eo/v1.1.0/schema.json",
    "https://stac-extensions.github.io/file/v2.1.0/schema.json",
    "https://stac-extensions.github.io/processing/v1.1.0/schema.json",
    "https://stac-extensions.github.io/projection/v1.1.0/schema.json"
  ],
  "title": "APEx Demo - Gran Canaria - NDVI",
  "type": "Collection",
  "assets": {
    "openEO_2019-08-19Z.tif": {
      "href": "https://openeo.dataspace.copernicus.eu/openeo/1.2/jobs/j-250313110405469ea3095a5b2b06ce85/results/assets/NjM5MTg1MWYtOTA0Mi00MTA4LThiMmEtM2RkMmU4YTlkZDBi/c3b120ff13b6c9bae13fe9ea75c105b1/openEO_2019-08-19Z.tif?expires=1742471533",
      "type": "image/tiff; application=geotiff",
      "title": "openEO_2019-08-19Z.tif",
      "roles": [
        "data"
      ],
      "proj:bbox": [
        -1772060.353,
        3214241.561,
        -1703761.577,
        3274924.983
      ],
      "proj:epsg": 3857,
      "proj:shape": [
        5323,
        5991
      ],
      "raster:bands": [
        {
          "name": "1",
          "statistics": {
            "maximum": 73.0,
            "mean": 0.10199033917873,
            "minimum": -161.0,
            "stddev": 0.16400283077253,
            "valid_percent": 99.999
          }
        }
      ]
    }
  },
  "license": "CC-BY-4.0",
  "extent": {
    "spatial": {
      "bbox": [
        [
          -15.918605217697035,
          27.72422378541127,
          -15.305188375387814,
          28.205553799401358
        ]
      ]
    },
    "temporal": {
      "interval": [
        [
          "2019-08-19T00:00:00Z",
          "2019-08-20T00:00:00Z"
        ]
      ]
    }
  },
  "providers": [
    {
      "name": "VITO",
      "description": "This data was processed on an openEO backend maintained by VITO.",
      "roles": [
        "processor"
      ]
    }
  ],
  "openeo:status": "finished",
  "keywords": [],
  "summaries": {}
}
requests.delete(f"{CATALOGUE}/collections/" + collection.id, auth=auth)
<Response [204]>

Use Case 2 and 3: Using the export_workspace Process

In these use cases, openEO exports the collection directly to external storage. This is enabled through the Workspaces API, which provides greater control over your data—allowing you to define where it is saved and from where it is loaded.

There are multiple options for creating a workspace. You can either provision one through a given backend or register your own storage location, such as an S3 bucket, to directly store openEO results.

Since these use cases rely on the availability of the export_workspace process, we first check whether it is supported.

con.describe_process("export_workspace")
exported_result = result.export_workspace("my_workspace_id", merge="my_stac_collection")
job = exported_result.execute_batch("gran_canaria.tiff",title="APEx Demo - Gran Canaria - NDVI",filename_prefix="gran_canaria")