Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,26 @@
## Summary

Lecture facilitée de la liste d'une pyramide.
Lecture d'informations sur une donnée raster unique depuis un fichier ou une liste de paramètres.

## Changelog

### [Added]

* Pyramid
* Fonctions de gestion de la liste : chargement et lecture (via un generator)
* Taille du header d'une dalle stockée dans la variable `ROK4_IMAGE_HEADER_SIZE`
* La proriété `tile_extension` : retourne l'extension d'une tuile de la pyramide en fonction du format
* Des exemples d'utilisation des fonctions principales
### [Fixed]
* Raster
* Chargement des informations sur un fichier raster (chemin du fichier, chemin du fichier de masque si applicable, nombre de canaux, boundingbox de l'emprise géographique)
* depuis le fichier raster
* depuis une liste de paramètres provenant d'une utilisation précédente
* Tests unitaires
* Documentation interne des fonctions et classes

### [Changed]

* README.md
* Modification du bloc code de compilation pour utiliser explicitement python3, et installer certaines dépendances.
* Utils
* Fonction de calcul de la boundix box d'une donnée
* Fonction de détermination du format de variable des couleurs dans une donéne raster

* Storage
* Lecture de la taille d'un objet S3 : pas besoin d'enlever des quotes dans le header `Content-Length`
<!--
### [Added]

Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,31 +37,34 @@ Plus d'exemple dans la documentation développeur.
## Compiler la librairie

```sh
apt install python3-venv
apt install python3-venv python3-rados python3-gdal
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade build bump2version
bump2version --allow-dirty --current-version 0.0.0 --new-version x.y.z patch pyproject.toml src/rok4/__init__.py

# Run unit tests
pip install -e .[test]
python3 -m pip install -e .[test]
# To use system installed modules rados and osgeo
echo "/usr/lib/python3/dist-packages/" >.venv/lib/python3.10/site-packages/system.pth
python -c 'import sys; print (sys.path)'
python3 -c 'import sys; print (sys.path)'
# Run tests
coverage run -m pytest
# Get tests report and generate site
coverage report -m
coverage html -d dist/tests/

# Build documentation
pip install -e .[doc]
python3 -m pip install -e .[doc]
pdoc3 --html --output-dir dist/ rok4

# Build artefacts
python3 -m build
```

Remarque :
Lors de l'installation du paquet apt `python3-gdal`, une dépendance, peut demander des interactions de configuration. Pour installer dans un environnement non-interactif, définir la variable shell `DEBIAN_FRONTEND=noninteractive` permet d'adopter une configuration par défaut.

## Publier la librairie sur Pypi

Configurer le fichier `$HOME/.pypirc` avec les accès à votre compte PyPI.
Expand Down
12 changes: 6 additions & 6 deletions src/rok4/Pyramid.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ class PyramidType(Enum):
class SlabType(Enum):
""" Slab's type """
DATA = "DATA" # Slab of data, raster or vector
MASK = "MASK" # Slab of mask, only for raster pyramide, image with one band : 0 is nodata, other values are data
MASK = "MASK" # Slab of mask, only for raster pyramid, image with one band : 0 is nodata, other values are data


ROK4_IMAGE_HEADER_SIZE = 2048
Expand Down Expand Up @@ -794,11 +794,11 @@ def get_level(self, level_id: str) -> 'Level':


def get_levels(self, bottom_id: str = None, top_id: str = None) -> List[Level]:
"""Get sorted levels from bottom to top provided
"""Get sorted levels in the provided range from bottom to top

Args:
bottom_id (str): optionnal specific bottom level id. Defaults to None.
top_id (str): optionnal specific top level id. Defaults to None.
bottom_id (str, optionnal): specific bottom level id. Defaults to None.
top_id (str, optionnal): specific top level id. Defaults to None.

Raises:
Exception: Provided levels are not consistent (bottom > top or not in the pyramid)
Expand Down Expand Up @@ -969,7 +969,7 @@ def get_tile_data_binary(self, level: str, column: int, row: int) -> str:
"""Get a pyramid's tile as binary string

To get a tile, 3 steps :
* calculate slab path from tile indice
* calculate slab path from tile index
* read slab index to get offsets and sizes of slab's tiles
* read the tile into the slab

Expand Down Expand Up @@ -1243,7 +1243,7 @@ def get_tile_data_vector(self, level: str, column: int, row: int) -> Dict:
def get_tile_indices(self, x: float, y: float, level: str = None, **kwargs) -> Tuple[str, int, int, int, int]:
"""Get pyramid's tile and pixel indices from point's coordinates

Used coordinates system have to be the pyramide one. If EPSG:4326, x is latitude and y longitude.
Used coordinates system have to be the pyramid one. If EPSG:4326, x is latitude and y longitude.

Args:
x (float): point's x
Expand Down
139 changes: 139 additions & 0 deletions src/rok4/Raster.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
"""Provide functions to read information on raster data from file path or object path

The module contains the following class :

- 'Raster' - Structure describing raster data.

"""

import re
from enum import Enum

from osgeo import ogr, gdal
from typing import Tuple

from rok4.Storage import exists, get_osgeo_path
from rok4.Utils import ColorFormat, compute_bbox,compute_format

# Enable GDAL/OGR exceptions
ogr.UseExceptions()
gdal.UseExceptions()

class Raster:
"""A structure describing raster data

Attributes :
path (str): path to the file/object (ex: file:///path/to/image.tif or s3://bucket/path/to/image.tif)
bbox (Tuple[float, float, float, float]): bounding rectange in the data projection
bands (int): number of color bands (or channels)
format (ColorFormat): numeric variable format for color values. Bit depth, as bits per channel, can be derived from it.
mask (str): path to the associated mask file or object, if any, or None (same path as the image, but with a ".msk" extension and TIFF format. ex: file:///path/to/image.msk or s3://bucket/path/to/image.msk)
dimensions (Tuple[int, int]): image width and height expressed in pixels
"""

def __init__(self) -> None:
self.bands = None
self.bbox = (None, None, None, None)
self.dimensions = (None, None)
self.format = None
self.mask = None
self.path = None

@classmethod
def from_file(cls, path: str) -> 'Raster':
"""Creates a Raster object from an image

Args:
path (str): path to the image file/object

Examples:

Loading informations from a file stored raster TIFF image

from rok4.Raster import Raster

try:
raster = Raster.from_file("file:///data/images/SC1000_TIFF_LAMB93_FXX/SC1000_0040_6150_L93.tif")

except Exception as e:
print(f"Cannot load information from image : {e}")

Raises:
RuntimeError: raised by OGR/GDAL if anything goes wrong
NotImplementedError: Storage type not handled

Returns:
Raster: a Raster instance
"""

if not exists(path):
raise Exception(f"No file or object found at path '{path}'.")

self = cls()

work_image_path = get_osgeo_path(path)

image_datasource = gdal.Open(work_image_path)
self.path = path

path_pattern = re.compile('(/[^/]+?)[.][a-zA-Z0-9_-]+$')
mask_path = path_pattern.sub('\\1.msk', path)

if exists(mask_path):
work_mask_path = get_osgeo_path(mask_path)
mask_driver = gdal.IdentifyDriver(work_mask_path).ShortName
if 'GTiff' != mask_driver:
raise Exception(f"Mask file '{mask_path}' is not a TIFF image. (GDAL driver : '{mask_driver}'")

self.mask = mask_path
else:
self.mask = None

self.bbox = compute_bbox(image_datasource)
self.bands = image_datasource.RasterCount
self.format = compute_format(image_datasource, path)
self.dimensions = (image_datasource.RasterXSize, image_datasource.RasterYSize)

return self

@classmethod
def from_parameters(cls, path: str, bands: int, bbox: Tuple[float, float, float, float], dimensions: Tuple[int, int], format: ColorFormat, mask: str = None) -> 'Raster':
"""Creates a Raster object from parameters

Args:
path (str): path to the file/object (ex: file:///path/to/image.tif or s3://bucket/path/to/image.tif)
bands (int): number of color bands (or channels)
bbox (Tuple[float, float, float, float]): bounding rectange in the data projection
dimensions (Tuple[int, int]): image width and height expressed in pixels
format (ColorFormat): numeric variable format for color values. Bit depth, as bits per channel, can be derived from it.
mask (str, optionnal): path to the associated mask file or object, if any, or None (same path as the image, but with a ".msk" extension and TIFF format. ex: file:///path/to/image.msk or s3://bucket/path/to/image.msk)

Examples:

Loading informations from parameters, related to a TIFF main image coupled to a TIFF mask image

from rok4.Raster import Raster

try:
raster = Raster.from_parameters(path="file:///data/images/SC1000_TIFF_LAMB93_FXX/SC1000_0040_6150_L93.tif", mask="file:///data/images/SC1000_TIFF_LAMB93_FXX/SC1000_0040_6150_L93.msk", bands=3, format=ColorFormat.UINT8, dimensions=(2000, 2000), bbox=(40000.000, 5950000.000, 240000.000, 6150000.000))

except Exception as e:
print(f"Cannot load information from parameters : {e}")

Raises:
KeyError: a mandatory argument is missing

Returns:
Raster: a Raster instance
"""

self = cls()

self.path = path
self.bands = bands
self.bbox = bbox
self.dimensions = dimensions
self.format = format
self.mask = mask

return self
114 changes: 112 additions & 2 deletions src/rok4/Utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,25 @@
"""

import os
import re

from typing import Dict, List, Tuple, Union
from osgeo import ogr, osr
from osgeo import ogr, osr, gdal
from enum import Enum

ogr.UseExceptions()
osr.UseExceptions()
gdal.UseExceptions()


class ColorFormat(Enum):
"""A color format enumeration.
Except from "BIT", the member's name matches a common variable format name. The member's value is the allocated bit size associated to this format.
"""
BIT = 1
UINT8 = 8
FLOAT32 = 32


__SR_BOOK = dict()
def srs_to_spatialreference(srs: str) -> 'osgeo.osr.SpatialReference':
Expand Down Expand Up @@ -154,4 +168,100 @@ def reproject_point(point: Tuple[float, float], sr_src: 'osgeo.osr.SpatialRefere
ct = osr.CreateCoordinateTransformation(sr_src, sr_dst)
x_dst, y_dst, z_dst = ct.TransformPoint(point[0], point[1])

return (x_dst, y_dst)
return (x_dst, y_dst)


def compute_bbox(source_dataset: gdal.Dataset) -> tuple:
"""Image boundingbox computing method

Args:
source_dataset (gdal.Dataset): Dataset object created from the raster image

Limitations:
Image's axis must be parallel to SRS' axis

Raises:
AttributeError: source_dataset is not a gdal.Dataset instance.
Exception: The dataset does not contain transform data.
"""

bbox = None
transform_vector = source_dataset.GetGeoTransform()

if transform_vector is None:
raise Exception(f"No transform vector found in the dataset created from the following file : {source_dataset.GetFileList()[0]}")

width = source_dataset.RasterXSize
height = source_dataset.RasterYSize

x_range = (
transform_vector[0],
transform_vector[0] + width * transform_vector[1] + height * transform_vector[2]
)

y_range = (
transform_vector[3],
transform_vector[3] + width * transform_vector[4] + height * transform_vector[5]
)

spatial_ref = source_dataset.GetSpatialRef()
if spatial_ref is not None and spatial_ref.GetDataAxisToSRSAxisMapping() == [2, 1]:
# Coordonnées terrain de type (latitude, longitude) => on permute les coordonnées terrain par rapport à l'image
bbox = (
min(y_range),
min(x_range),
max(y_range),
max(x_range)
)
elif spatial_ref is None or spatial_ref.GetDataAxisToSRSAxisMapping() == [1, 2]:
# Coordonnées terrain de type (longitude, latitude) ou pas de SRS => les coordonnées terrain sont dans le même ordre que celle de l'image
bbox = (
min(x_range),
min(y_range),
max(x_range),
max(y_range)
)

return bbox


def compute_format(dataset: gdal.Dataset, path: str = None) -> 'ColorFormat':
"""Image color format computing method

Args:
dataset (gdal.Dataset): Dataset object created from the raster image
path (str, optionnal): path to the original file/object

Raises:
AttributeError: source_dataset is not a gdal.Dataset instance.
Exception: Image has no color band or its color format is unsupported.
"""

format = None

if path is None:
path = dataset.GetFileList()[0]

if dataset.RasterCount < 1:
raise Exception(f"Image {path} contains no color band.")

band_1_datatype = dataset.GetRasterBand(1).DataType
data_type_name = gdal.GetDataTypeName(band_1_datatype)
data_type_size = gdal.GetDataTypeSize(band_1_datatype)
color_interpretation = dataset.GetRasterBand(1).GetRasterColorInterpretation()
color_name = None
if color_interpretation is not None:
color_name = gdal.GetColorInterpretationName(color_interpretation)
compression_regex_match = re.search(r'COMPRESSION\s*=\s*PACKBITS', gdal.Info(dataset))


if data_type_name == "Byte" and data_type_size == 8 and color_name == "Palette" and compression_regex_match:
format = ColorFormat.BIT
elif data_type_name == "Byte" and data_type_size == 8:
format = ColorFormat.UINT8
elif data_type_name == "Float32" and data_type_size == 32:
format = ColorFormat.FLOAT32
else:
raise Exception(f"Unsupported color format for image {path} : '{data_type_name}' ({data_type_size} bits)")

return format
Loading