NaN Value Handling for Signals and Images

[PierreRaybaut]

Summary

Add a dedicated NaN detection and replacement capability to Sigima and DataLab, allowing users to identify and replace NaN values in both 1D signals and 2D images using configurable strategies (zero, mean, interpolation, etc.), with a preview option in DataLab before applying the replacement.

Motivation

NaN values frequently appear in scientific datasets due to sensor failures, division by zero, or incomplete acquisitions. Currently, Sigima and DataLab handle NaN values incidentally (e.g., using np.nanmean in specific functions or setting infinity to NaN), but there is no dedicated tool for users to detect, visualize, and replace NaN values in a controlled manner.

This leads to:

• Silent propagation of NaN through processing pipelines
• Unexpected results in downstream computations
• No user visibility into which data points are affected

Implementation Plan

Phase 1 — Sigima: Computation Functions

1.1 Parameter Class: ReplaceNaNParam

File: sigima/proc/base.py (shared by signal and image)

Define a guidata.dataset.DataSet parameter class with the following items:

Parameter	Type	Description
method	ChoiceItem	Replacement strategy: "zero", "mean", "median", "interpolation" (signal only), "constant"
constant	FloatItem	Custom constant value (enabled only when method == "constant")

Notes:

• The "interpolation" method uses linear interpolation from neighboring valid values; it is only available for signals (1D), not images (2D)
• For images, an additional "inpaint" method could be added (using OpenCV's cv2.inpaint) — this can be deferred to a follow-up iteration
• Provide a create() static method for script-friendly instantiation
• Provide a generate_title() method for title formatting

1.2 Signal Function: replace_nan

File: sigima/proc/signal/processing.py

@computation_function()
def replace_nan(src: SignalObj, p: ReplaceNaNParam) -> SignalObj:
    """Replace NaN values in signal Y data.

    Args:
        src: Input signal (may contain NaN values in Y array)
        p: Replacement parameters (method, constant)

    Returns:
        Signal with NaN values replaced according to the chosen method
    """

Implementation details:

• Operate on src.y (and optionally src.dy uncertainty array)
• Use np.isnan() to build the NaN mask
• Apply replacement strategy:
  • "zero" → y[mask] = 0.0
  • "mean" → y[mask] = np.nanmean(y)
  • "median" → y[mask] = np.nanmedian(y)
  • "interpolation" → np.interp(x[mask], x[~mask], y[~mask])
  • "constant" → y[mask] = p.constant
• Follow existing patterns: dst_1_to_1(), restore_data_outside_roi(), FormatResultTitle.apply()
• If no NaN values are found, return an unmodified copy (with a warning in metadata or title)

1.3 Image Function: replace_nan

File: sigima/proc/image/exposure.py (alongside existing normalize, clip)

@computation_function()
def replace_nan(src: ImageObj, p: ReplaceNaNParam) -> ImageObj:
    """Replace NaN values in image data.

    Args:
        src: Input image (may contain NaN values)
        p: Replacement parameters (method, constant)

    Returns:
        Image with NaN values replaced according to the chosen method
    """

Implementation details:

• Operate on src.data (2D NumPy array)
• Same strategies as signal except "interpolation" is not available (raise a clear error or skip silently)
• For "mean" and "median", use np.nanmean(data) / np.nanmedian(data) to compute the fill value from valid pixels
• Follow existing image patterns: dst_1_to_1(), restore_data_outside_roi()

1.4 Low-Level Tools (Optional)

Files: sigima/tools/signal/ and sigima/tools/image/

If the NaN replacement logic is useful outside the object model (pure NumPy), extract core algorithms into tool functions:

# sigima/tools/signal/nanhandling.py
def replace_nan_in_array(y: np.ndarray, x: np.ndarray | None = None,
                         method: str = "zero", constant: float = 0.0) -> np.ndarray:
    """Replace NaN values in a 1D array."""

# sigima/tools/image/nanhandling.py
def replace_nan_in_2d(data: np.ndarray, method: str = "zero",
                      constant: float = 0.0) -> np.ndarray:
    """Replace NaN values in a 2D array."""

1.5 Exports

File	Action
sigima/proc/signal/__init__.py	Import replace_nan + ReplaceNaNParam, add to __all__
sigima/proc/image/__init__.py	Import replace_nan, add to __all__
sigima/params.py	Re-export ReplaceNaNParam, add to __all__

1.6 Sigima Tests

File: sigima/tests/signal/processing_unit_test.py (or new dedicated file)

Test cases:

• Signal with no NaN → output equals input
• Signal with all NaN → appropriate handling (fill with constant, or raise)
• Signal with scattered NaN → verify each method produces correct values
• Signal with NaN at edges → verify interpolation handles boundary conditions
• ROI interaction: NaN replacement only within ROI, data outside ROI preserved
• Same test matrix for images (minus interpolation method)

---

Phase 2 — DataLab: GUI Integration

2.1 Processor Registration

Files:

• datalab/gui/processor/signal.py → register_processing()
• datalab/gui/processor/image.py → register_processing()

Register under the "Level adjustment" subsection (alongside normalize and clip):

# Signal processor
self.register_1_to_1(
    sips.replace_nan,
    _("Replace NaN values"),
    paramclass=sigima_base.ReplaceNaNParam,
    icon_name="replace_nan.svg",
)

# Image processor
self.register_1_to_1(
    sipi.replace_nan,
    _("Replace NaN values"),
    paramclass=sigima_base.ReplaceNaNParam,
    icon_name="replace_nan.svg",
)

2.2 Action Handler (Menu Placement)

File: datalab/gui/actionhandler.py

Add self.action_for("replace_nan") in both Signal and Image action handlers, inside the "Level adjustment" submenu, after clip:

with self.new_menu(_("Level adjustment"), icon_name="level_adjustment.svg"):
    self.action_for("normalize")
    self.action_for("clip")
    self.action_for("replace_nan")  # NEW
    ...

2.3 NaN Visualization (Preview Before Replacement)

This is the most significant GUI addition. Two possible approaches:

Option A — Lightweight: Highlight in existing plot (recommended for v1)

• Before applying the replacement, show a dialog or overlay that:
  • For signals: plots the original curve and overlays markers (e.g., red circles) at NaN positions on the X axis
  • For images: displays the image with NaN pixels highlighted (e.g., as a colored overlay mask)
• Display a summary: "Found N NaN values out of M total data points (X%)"
• User confirms or cancels the replacement

This could be implemented as a custom dialog using PlotPy widgets, similar to existing preview dialogs in DataLab (e.g., baseline correction dialog).

Option B — Analysis function: detect_nan (1-to-0 computation)

Add a separate analysis function that produces a result without modifying data:

# Register as 1-to-0 (analysis producing metadata, no output object)
self.register_1_to_0(
    sips.detect_nan,
    _("Detect NaN values"),
    icon_name="detect_nan.svg",
)

This would add a result entry in the metadata table showing NaN count, positions, and percentage. The user can then decide whether to run replace_nan.

2.4 Icon

Create replace_nan.svg icon in DataLab's icon resources. Style should be consistent with existing icons (e.g., similar to clip.svg but with a NaN symbol or a "fill gap" visual metaphor).

2.5 DataLab Tests

File: datalab/tests/features/signal/ and datalab/tests/features/image/

• Integration test creating a signal/image with NaN, running replace_nan through the processor, verifying the result
• Test with ROI: NaN replacement only within selected region
• Test parameter dialog interaction (if applicable)

---

Phase 3 — Documentation and Translations

3.1 Sphinx Documentation

File	Content
doc/features/signal/menu_processing.rst	Document "Replace NaN values" under Processing > Level adjustment
doc/features/image/menu_processing.rst	Same for image side

Include:

• Description of the feature and each replacement method
• Screenshot of the parameter dialog
• Screenshot of the NaN visualization preview
• Example use case (e.g., sensor data with dropouts)

3.2 UI Translations (gettext)

Run translation scan and update French .po files:

Replace NaN values → Remplacer les valeurs NaN
Replacement method → Méthode de remplacement
Zero → Zéro
Mean → Moyenne
Median → Médiane
Interpolation → Interpolation
Constant → Constante
Constant value → Valeur constante
Found {n} NaN values out of {m} data points → {n} valeurs NaN trouvées sur {m} points de données

3.3 Documentation Translations

Update French .po files under doc/locale/fr/ for the new documentation sections.

3.4 Release Notes

Add entry in doc/release_notes/release_X.YY.md:

**Signal/Image processing:**

* Added "Replace NaN values" processing function to detect and replace NaN
  values using configurable methods (zero, mean, median, interpolation,
  constant) — available for both signals and images
* Added NaN visualization preview showing affected data points before
  applying the replacement

Release Classification

Feature (minor release) — adds entirely new processing capability with dedicated UI integration.

Open Questions

1. Should "interpolation" be available for images? 2D interpolation (e.g., using scipy.interpolate.griddata) is more complex and slower. Recommendation: defer to a follow-up with an "inpaint" method using OpenCV.
2. Should NaN detection also check for ±inf? Many processing pipelines produce infinity values alongside NaN. Consider adding an option to treat inf as NaN (or a separate replace_inf function).
3. Preview dialog scope: Should the preview be a modal dialog with a PlotPy widget, or a simpler message box with statistics only? The former is more useful but requires more development effort.
4. ROI behavior: When ROI is defined, should NaN replacement apply only within the ROI (consistent with other processing functions), or should it optionally apply to the entire dataset regardless of ROI?