WqSat_format

Python utilities for transforming Sentinel-2 and Sentinel-3 Level-1 products into analysis-ready stacks, composites, and exports tailored to inland water quality workflows.

---

Overview

The package wraps reusable readers, preprocessing steps, and composite builders behind a single orchestration layer (FormatManager) that:

• loads configuration from YAML files or Python dictionaries,
• validates platform/band/ROI inputs,
• reads SAFE (S2) or SEN3 (S3) tiles while cropping to areas of interest,
• optionally applies atmospheric correction (DOS) and 10 m pansharpening, and
• produces temporal/spatial composites before exporting GeoTIFF or NetCDF products.

Logging is enabled by default and records both console output and timestamped log files under logs/.

---

Key Features

• Read Sentinel-2 and Sentinel-3 scenes with precise spatial subsetting by geographic bounds or pixel windows.
• Apply Dark Object Subtraction (DOS) to Sentinel-2 imagery and optional pansharpening to harmonise bands at 10 m.
• Build temporal composites (min, median, max) and spatial mosaics, or chain both for grouped Sentinel-2 tiles.
• Export multi-resolution or single-resolution outputs as GeoTIFF or NetCDF via dedicated writers.
• Modular architecture (io, preprocess, opt) for reuse in notebooks or bespoke workflows.

---

Requirements

• Python 3.10 or newer (matches the package metadata in setup.cfg).
• Dependencies listed in requirements.txt:
  • pyyaml
  • rasterio
  • numpy
  • matplotlib
  • rioxarray
  • xarray
  • scipy
  • PyWavelets
  • opencv-python

Install dependencies with:

pip install -r requirements.txt

---

Installation

git clone https://github.com/garciadd/WqSat_format.git
cd WqSat_format
python -m venv .venv && source .venv/bin/activate  # optional but recommended
pip install -r requirements.txt
pip install -e .

The editable install exposes wqsat_format for local development and integration with the companion notebooks in notebooks/.

---

Configuration

FormatManager expects the processing parameters in a dictionary (loaded from YAML or supplied inline). Key fields:

Key	Type	Required	Description
platform	str	✔	SENTINEL-2 or SENTINEL-3. Determines reader/writer pairing.
tile_path	str or list	✔	SAFE/SEN3 path (single scene) or list of scenes. For double composites use a list of lists grouped by tile.
output_path	str	✔	Output filename (GeoTIFF/NetCDF). Resolution-specific suffixes are added automatically for multi-resolution data.
bands	list[str]	Optional	Subset of band identifiers. Defaults to all bands provided by the product.
roi_lat_lon	dict	Optional	Geographic bounding box with keys W, S, E, N. Applies to both Sentinel-2 and Sentinel-3 readers.
roi_window	dict	Optional	Pixel window (xmin, ymin, xmax, ymax) referenced to the native grid. Mutually exclusive with roi_lat_lon.
atcor	str	Optional	Set to "DOS" to enable Dark Object Subtraction for Sentinel-2 data.
resample	bool	Optional	When true, Sentinel-2 bands are pansharpened/interpolated to 10 m using the TRWH/IHS workflow.
temporal_composite	str or bool	Optional	min, median, max, or false. Supported for Sentinel-2 tiles only.
spatial_composite	str or bool	Optional	min, median, max, or false. Supported for Sentinel-2 tiles only.

Double composites (temporal_composite and spatial_composite both enabled) expect tile_path to contain groups of tiles that share a grid (e.g., [[tile_a, tile_b], [tile_c, tile_d]]).

Example YAML (WqSat_format_settings.yaml)

platform: SENTINEL-2
tile_path:
  - /data/S2A_MSIL1C_20230705T110621_N0509_R094_T30TXM_20230705T130047.SAFE
  - /data/S2B_MSIL1C_20230708T110619_N0509_R094_T30TXM_20230708T130530.SAFE
output_path: ./outputs/s2_t30txm_composite.tif
bands: [B02, B03, B04, B08, B11, B12]
roi_lat_lon:
  W: -4.8
  S: 43.2
  E: -3.9
  N: 43.7
atcor: DOS
resample: true
temporal_composite: median
spatial_composite: false

---

Usage

Run the processing pipeline from YAML

from wqsat_format.manager import FormatManager

manager = FormatManager(config="WqSat_format_settings.yaml")
manager.run()

Provide configuration inline (Sentinel-3 example)

from wqsat_format.manager import FormatManager

config = {
    "platform": "SENTINEL-3",
    "tile_path": "/data/S3A_OL_1_EFR____20230415T101245_20230415T101545_20230416T150411_0180_097_354_1800_MAR_O_NR_002.SEN3",
    "output_path": "./outputs/s3_stack.tif",
    "bands": ["Oa03", "Oa06", "Oa08"],
    "roi_window": {"xmin": 500, "ymin": 300, "xmax": 1500, "ymax": 1200}
}

manager = FormatManager(config=config)
manager.run()

Command-line helper

wqsat_format/manager.py exposes a main() function for CLI execution:

python -m wqsat_format.manager --config WqSat_format_settings.yaml

The CLI mirrors the programmatic interface. Errors raise non-zero exit codes with additional detail in the latest file under logs/.

---

Logging

Calling FormatManager triggers setup_logging() which:

• writes INFO-level progress to stdout,
• records DEBUG-level diagnostics to timestamped files in logs/, and
• clears previous handlers to avoid duplicated messages in interactive sessions.

Log files contain tile names, per-band operations, and export paths—keep them for traceability when processing large batches.

---

Development

• Format/lint tooling is not standardised yet; stick to black/ruff defaults if contributing.
• Run the test suite (once available) with pytest.
• Exploratory workflows and prototypes live in the Jupyter notebooks under notebooks/.

---

Known Limitations

• Composites (temporal_composite, spatial_composite) are implemented for Sentinel-2 data; Sentinel-3 scenes are processed as standalone stacks.
• Double composites require tile_path to be grouped by grid so temporal aggregation happens before mosaicking.
• DOS correction and pansharpening assume Level-1C Sentinel-2 inputs with accessible sun-angle metadata.
• Atmospheric correction for Sentinel-3 relies on external angle products and does not yet integrate with the pipeline.
• Large scenes can consume significant memory during mosaicking; consider tiling or increasing system swap for production runs.

---

Authors

Daniel García-Díaz & Fernando Aguilar Gómez
Spanish National Research Council (CSIC) – Institute of Physics of Cantabria (IFCA)

---

License

This project is licensed under the Apache License 2.0. See LICENSE for the full text.