Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions default-sample.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Authors: Tom Kralidis <[email protected]>
# Angelos Tzotsos <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
# Copyright (c) 2024 Angelos Tzotsos
#
# Permission is hereby granted, free of charge, to any person
Expand Down Expand Up @@ -55,7 +55,20 @@ profiles:
- apiso

federatedcatalogues:
- http://catalog.data.gov/csw
- id: arctic-sdi-csw
type: CSW
title: Arctic SDI
url: https://catalogue.arctic-sdi.org/csw
- id: pycsw-cite-demo
type: OARec
title: pycsw OGC CITE demo and Reference Implementation
uresultsrl: https://demo.pycsw.org/cite
- id: fedcat03
type: STAC-API
title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
url: https://stac.dataspace.copernicus.eu/v1
collections:
- daymet-annual-pr

manager:
transactions: false
Expand Down
7 changes: 5 additions & 2 deletions docker/compose/pycsw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Authors: Tom Kralidis <[email protected]>
# Ricardo Garcia Silva <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
# Copyright (c) 2017 Ricardo Garcia Silva
#
# Permission is hereby granted, free of charge, to any person
Expand Down Expand Up @@ -52,7 +52,10 @@ profiles:
- apiso

federatedcatalogues:
- http://catalog.data.gov/csw
- id: fedcat01
type: CSW
title: Arctic SDI
url: https://catalogue.arctic-sdi.org/csw

manager:
transactions: false
Expand Down
5 changes: 4 additions & 1 deletion docker/helm/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,10 @@ pycsw:
profiles:
- apiso
# federatedcatalogues:
# - http://catalog.data.gov/csw
# - id: fedcat01
# type: CSW
# title: Arctic SDI
# url: https://catalogue.arctic-sdi.org/csw
manager:
transactions: "false"
allowed_ips:
Expand Down
7 changes: 5 additions & 2 deletions docker/kubernetes/pycsw-configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ data:
# Ricardo Garcia Silva <[email protected]>
# Angelos Tzotsos <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
# Copyright (c) 2017 Ricardo Garcia Silva
# Copyright (c) 2024 Angelos Tzotsos
#
Expand Down Expand Up @@ -57,7 +57,10 @@ data:
- apiso

federatedcatalogues:
- http://catalog.data.gov/csw
- id: fedcat01
type: CSW
title: Arctic SDI
url: https://catalogue.arctic-sdi.org/csw

manager:
transactions: false
Expand Down
7 changes: 5 additions & 2 deletions docker/pycsw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Authors: Tom Kralidis <[email protected]>
# Ricardo Garcia Silva <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
# Copyright (c) 2017 Ricardo Garcia Silva
#
# Permission is hereby granted, free of charge, to any person
Expand Down Expand Up @@ -52,7 +52,10 @@ profiles:
- apiso

federatedcatalogues:
- http://catalog.data.gov/csw
- id: fedcat01
type: CSW
title: Arctic SDI
url: https://catalogue.arctic-sdi.org/csw

manager:
transactions: false
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pycsw's runtime configuration is defined by ``default.yml``. pycsw ships with a
- **level**: the logging level (see https://docs.python.org/library/logging.html#logging-levels)
- **logfile**: the full file path to the logfile
- **ogc_schemas_base**: base URL of OGC XML schemas tree file structure (default is http://schemas.opengis.net)
- **federatedcatalogues**: comma delimited list of CSW endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
- **federatedcatalogues**: arrray of distributed catalogue endpoints to be used for distributed searching, if requested by the client (see :ref:`distributedsearching`)
- **pretty_print**: whether to pretty print the output (``true`` or ``false``). Default is ``false``
- **gzip_compresslevel**: gzip compression level, lowest is ``1``, highest is ``9``. Default is off. **NOTE**: if gzip compression is already enabled via your web server, do not enable this directive (or else the server will try to compress the response twice, resulting in degraded performance)
- **domainquerytype**: for GetDomain operations, how to output domain values. Accepted values are ``list`` and ``range`` (min/max). Default is ``list``
Expand Down
60 changes: 53 additions & 7 deletions docs/distributedsearching.rst
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,18 @@ in the CSW-all configuration:
.. code-block:: yaml

federatedcatalogues:
- http://localhost/pycsw/csw.py?config=CSW-1.yml
- http://localhost/pycsw/csw.py?config=CSW-2.yml
- http://localhost/pycsw/csw.py?config=CSW-3.yml
- id: fedcat01
type: CSW
title: Federated catalogue 1
url: http://localhost/pycsw/csw.py?config=CSW-1.yml
- id: fedcat02
type: CSW
title: Federated catalogue 2
url: http://localhost/pycsw/csw.py?config=CSW-2.yml
- id: fedcat03
type: CSW
title: Federated catalogue 3
url: http://localhost/pycsw/csw.py?config=CSW-3.yml

At which point a CSW client request to CSW-all with ``distributedsearch=TRUE``, while specifying an optional ``hopCount``. Query network topology:

Expand Down Expand Up @@ -88,7 +97,7 @@ A very important facet of distributed search is as per Annex B of OGC:CSW 2.0.2.
OGC API - Records
-----------------

Experimental support for distibuted searching is available in pycsw's OGC API - Records support to allow for searching remote services. The implementation uses the same approach as described above, operating in OGC API - Records mode.
Experimental support for distibuted searching is available in pycsw's OGC API - Records support to allow for searching remote services. The implementation uses the same approach as described above, operating in OGC API - Records mode as per `OGC API - Records - Part 4: Federated Search`_ (draft).

.. note::

Expand All @@ -97,9 +106,46 @@ Experimental support for distibuted searching is available in pycsw's OGC API -
.. code-block:: yaml

federatedcatalogues:
- https://example.org/collections/collection1
- https://example.org/collections/collection2
- id: fedcat01
type: OARec
title: Federated catalogue 1
url: https://example.org/collections/collection1
- id: fedcat02
type: OARec
title: Federated catalogue 2
url: https://example.org/collections/collection2

With the above configured, a distributed search can be invoked as follows:

http://localhost/collections/metadata:main/items?distributed=true
http://localhost/collections/metadata:main/items?distributedSearch=true

STAC API
--------

Experimental support for distibuted searching is available in pycsw's STAC API support to allow for searching remote services. The implementation uses the same approach as described above.

.. note::

The ``federatedcatalogues`` directives must point to a STAC API endpoint.

.. code-block:: yaml

federatedcatalogues:
- id: fedcat03
type: STAC-API
title: Copernicus Data Space Ecosystem (CDSE) asset-level STAC catalogue
url: https://stac.dataspace.copernicus.eu/v1
collections:
- daymet-annual-pr


.. note::

To constrain STAC API distributed search to specific collections, define one to many in the `collections` (array) directive.


With the above configured, a distributed search can be invoked as follows:

http://localhost/stac/search?distributedSearch=true

.. _`OGC API - Records - Part 4: Federated Search`: https://github.com/opengeospatial/ogcapi-records/blob/master/extensions/federated-search/document.adoc
8 changes: 7 additions & 1 deletion docs/introduction.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ Features
========

- implements `OGC API - Records - Part 1: Core`_
- implements `OGC API - Records - Part 2: Facets`_
- implements `OGC API - Records - Part 3: Create, Replace, Update, Delete, Harvest`_
- implements `OGC API - Records - Part 4: Federated Search`_
- implements `OGC API - Features - Part 3: Filtering`_
- implements `STAC API`_
- implements `Common Query Language (CQL2)`_
Expand Down Expand Up @@ -235,7 +238,10 @@ Paging

- resumptionToken

.. _`OGC API - Records - Part 1: Core`: https://ogcapi.ogc.org/records
.. _`OGC API - Records - Part 1: Core`: https://docs.ogc.org/is/20-004r1/20-004r1.html
.. _`OGC API - Records - Part 2: Facets`: https://docs.ogc.org/DRAFTS/25-013.html
.. _`OGC API - Records - Part 3: Create, Replace, Update, Delete, Harvest`: https://docs.ogc.org/DRAFTS/25-015.html
.. _`OGC API - Records - Part 4: Federated Search`: https://github.com/opengeospatial/ogcapi-records/blob/master/extensions/federated-search/document.adoc
.. _`OGC API - Features - Part 3: Filtering`: http://docs.ogc.org/DRAFTS/19-079.html
.. _`Common Query Language (CQL2)`: https://docs.ogc.org/DRAFTS/21-065.html
.. _`OGC CSW`: https://www.ogc.org/standards/cat
Expand Down
6 changes: 4 additions & 2 deletions pycsw/core/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Authors: Tom Kralidis <[email protected]>
# Angelos Tzotsos <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
# Copyright (c) 2015 Angelos Tzotsos
#
# Permission is hereby granted, free of charge, to any person
Expand Down Expand Up @@ -632,7 +632,9 @@ def cli_migrate_config(ctx, config, verbosity):
elif name == 'profiles':
dict_[name] = value.split(',')
elif name == 'federatedcatalogues':
dict_[name] = value.split(',')
dict_[name] = []
for count, fc in enumerate(value.split(',')):
dict_[name].append({'id': f'fedcat{count}', 'url': fc})
else:
dict_['server'][name] = get_typed_value(value)

Expand Down
66 changes: 60 additions & 6 deletions pycsw/ogc/api/oapi.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# Authors: Tom Kralidis <[email protected]>
#
# Copyright (c) 2024 Tom Kralidis
# Copyright (c) 2026 Tom Kralidis
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
Expand Down Expand Up @@ -199,10 +199,10 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
'style': 'form',
'explode': False
}
oapi['components']['parameters']['distributed'] = {
'name': 'distributed',
oapi['components']['parameters']['distributedSearch'] = {
'name': 'distributedSearch',
'in': 'query',
'description': 'Whether to invoke distributed mode',
'description': 'Whether to invoke distributed search',
'schema': {
'type': 'boolean',
'default': False
Expand Down Expand Up @@ -390,6 +390,60 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
oapi['paths']['/collections/{collectionId}/queryables'] = path2
oapi['components']['parameters']['collectionId']['default'] = 'metadata:main' # noqa

path = {
'get': {
'tags': ['Federated catalogs'],
'summary': 'Federated catalogs page',
'description': 'Federated catalogs page',
'operationId': 'getFederatedCatalogs',
'parameters': [
{'$ref': '#/components/parameters/collectionId'},
{'$ref': '#/components/parameters/f'}
],
'responses': {
'200': {
'$ref': '#/components/responses/FederatedCatalogs'
},
'500': {
'$ref': '#/components/responses/ServerError'
}
}
}
}

oapi['paths']['/collections/{collectionId}/federatedCatalogs'] = path

path = {
'get': {
'tags': ['Federated catalogs'],
'summary': 'Federated catalogs page',
'description': 'Federated catalogs page',
'operationId': 'getFederatedCatalog',
'parameters': [
{'$ref': '#/components/parameters/collectionId'},
{'name': 'catalogId',
'in': 'path',
'description': 'catalog ID',
'required': True,
'schema': {
'type': 'string'
}
},
{'$ref': '#/components/parameters/f'}
],
'responses': {
'200': {
'$ref': '#/components/responses/FederatedCatalog'
},
'500': {
'$ref': '#/components/responses/ServerError'
}
}
}
}

oapi['paths']['/collections/{collectionId}/federatedCatalogs/{catalogId}'] = path

path = {
'get': {
'tags': ['metadata'],
Expand All @@ -411,7 +465,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
{'$ref': '#/components/parameters/f'},
{'$ref': '#/components/parameters/offset'},
{'$ref': '#/components/parameters/facets'},
{'$ref': '#/components/parameters/distributed'},
{'$ref': '#/components/parameters/distributedSearch'},
{'$ref': '#/components/parameters/vendorSpecificParameters'}
],
'responses': {
Expand Down Expand Up @@ -488,7 +542,7 @@ def gen_oapi(config, oapi_filepath, mode='ogcapi-records'):
'parameters': [
{'$ref': '#/components/parameters/collectionId'},
{'$ref': '#/components/parameters/recordId'},
{'$ref': '#/components/parameters/distributed'},
{'$ref': '#/components/parameters/distributedSearch'},
f
],
'responses': {
Expand Down
Loading
Loading