Anti-Degeneracy Defence

Fitting the homodyne laminar-flow or the heterodyne two-component kernel simultaneously across \(N_\phi\) azimuthal angles is mathematically ill-conditioned without additional structure. The per-angle scaling parameters \((\beta(\phi_k), c_\mathrm{offset}(\phi_k))\) are degenerate with the physical parameters — principally \(D_0\) and \(\dot{\gamma}_0\) — and the shear gradient cancels when summed over angles, producing a flat optimisation landscape that collapses \(\dot{\gamma}_0\) to a non-physical value. The xpcsjax xpcsjax.optimization.nlsq.anti_degeneracy_controller orchestrates a five-layer defence that breaks the degeneracy, addresses the gradient cancellation, and monitors the optimisation in real time.

This page explains the degeneracy mechanism, walks through each of the five layers, and points to the implementing module. The implementation closely follows the strategy first introduced in the upstream homodyne package and ported into xpcsjax for v0.1.

The parameter absorption degeneracy

At a single angle \(\phi_k\) the laminar-flow kernel ((7)) is

(1)\[c_2(\phi_k, t_1, t_2; \theta) \;=\; c_\mathrm{offset}(\phi_k) \;+\; \beta(\phi_k)\, \exp\!\left(-q^2\!\int_{t_1}^{t_2} J(t')\,dt'\right) \mathrm{sinc}^2\!\left(\tfrac{q h \cos(\phi - \phi_0)\,\Gamma(t_1, t_2)}{2\pi}\right).\]

If \(\beta(\phi_k)\) and \(c_\mathrm{offset}(\phi_k)\) are treated as independent free parameters per angle, the optimisation landscape has a flat direction:

Increasing \(D_0 \to D_0 + \delta\) and simultaneously rescaling \(\beta(\phi_k) \to \beta(\phi_k)\, e^{q^2 \delta\, t_\mathrm{ref}}\) produces identical \(c_2\) values across all angles. The physical parameters are not identifiable from the per-angle contrasts without a constraint.

This degeneracy is generic whenever:

  1. Per-angle \(\beta(\phi_k)\) and \(c_\mathrm{offset}(\phi_k)\) are freely optimised;

  2. The number of angle-specific parameters exceeds the information content per angle;

  3. The diffusion contribution and the contrast contribution share the same functional form.

The gradient cancellation problem

The shear term in \(c_2\) introduces an angle-dependent piece whose gradient with respect to \(\dot{\gamma}_0\) is proportional to \(\cos(\phi - \phi_0)\). Summed over angles that span \([0, 2\pi)\), positive and negative contributions partially cancel:

Example for 8 equally spaced angles, phi_0 = 0:

    phi = 0   :  cos = +1.00 ----+
    phi = 45  :  cos = +0.71     | partially cancel when summed
    phi = 90  :  cos =  0.00     |
    phi = 135 :  cos = -0.71     |
    phi = 180 :  cos = -1.00 ----+
    ...

The net gradient on \(\dot{\gamma}_0\) is weak, and the optimiser finds it easier to absorb the angle dependence into per-angle \((\beta(\phi_k), c_\mathrm{offset}(\phi_k))\) than to drive \(\dot{\gamma}_0\) toward its true value. The result is parameter collapse: \(\dot{\gamma}_0\) floats to its lower bound.

The five-layer defence

The AntiDegeneracyController orchestrates five complementary mechanisms. The layers are not redundant: each addresses a different root cause and they compose.

Layer 1 – Per-angle reparameterisation

Module: xpcsjax.optimization.nlsq.per_angle_mode. Class: PerAngleScalingPlan (resolved by resolve_per_angle_mode()).

This layer attacks the structural degeneracy by reducing the dimension of the per-angle scaling space. Three resolved modes are available, selected through the per_angle_mode setting (the auto token resolves to one of them):

  • constant – per-angle \((\beta(\phi_k), c_\mathrm{offset}(\phi_k))\) are estimated from data quantiles and held fixed during the fit. Only the physical parameters are optimised. Total parameters: physical only.

  • averaged – computes the quantile estimates, averages them to a single \((\bar{\beta}, \bar{c}_\mathrm{offset})\), and optimises these two averaged scalars together with the physical parameters. This is what auto (the default) resolves to for \(N_\phi \geq 3\).

  • individual – each angle has independent \((\beta(\phi_k), c_\mathrm{offset}(\phi_k))\), adding \(2 N_\phi\) free parameters. This is what auto resolves to for \(N_\phi < 3\); also usable as a post-hoc refinement of an averaged-mode fit.

The quantile estimation underlying constant and averaged exploits the Siegert plateau:

  • At small lags (\(\Delta t \to 0\)), \(c_2 \to \beta + c_\mathrm{offset}\) (the ceiling).

  • At large lags (\(\Delta t \to \infty\)), \(c_2 \to c_\mathrm{offset}\) (the floor).

The 90th percentile of small-lag values gives a robust ceiling, the 10th percentile of large-lag values gives a robust floor, and \(\beta = \text{ceiling} - \text{floor}\) follows. Quantiles are used instead of min / max for outlier robustness.

Parameter count for a 23-angle laminar-flow fit:

Mode

Parameters

Notes

constant

7

Scaling fixed from quantiles; fastest convergence.

averaged

9

7 physical + 2 averaged scaling; what auto (the recommended default) resolves to for \(N_\phi \geq 3\).

individual

53

7 physical + 46 per-angle; high degeneracy risk. auto resolves to this for \(N_\phi < 3\).

Layer 2 – Hierarchical two-stage optimisation

Module: xpcsjax.optimization.nlsq.hierarchical. Class: HierarchicalOptimizer.

This layer breaks gradient cancellation by alternating between two optimisation stages that operate on disjoint parameter blocks.

Stage 1 — physical parameters only. Per-angle scaling parameters are frozen at their current values. The trust-region solver receives the full gradient signal on \((D_0, \alpha, D_\mathrm{offset}, \dot{\gamma}_0, \beta_\gamma, \dot{\gamma}_\mathrm{offset}, \phi_0)\) without dilution from the scaling block.

Stage 2 — per-angle parameters only. Physical parameters are frozen at the Stage 1 result. The per-angle parameters adjust to match the fixed physics model.

The two stages alternate until the change in the physical parameter block falls below the outer tolerance or the maximum outer iteration count is reached. The alternation prevents either block from absorbing signal that properly belongs to the other.

Layer 3 – Adaptive CV-based regularisation

Module: xpcsjax.optimization.nlsq.adaptive_regularization. Class: AdaptiveRegularizer.

Classical variance-penalty regularisation \(L_\mathrm{reg} = \lambda\,\mathrm{Var}(\text{params})\cdot N\) is typically swamped by the data loss and contributes a negligible fraction (\(\sim 0.01\%\)) of the total objective. Layer 3 replaces it with a relative penalty based on the coefficient of variation,

(2)\[\mathrm{CV} \;=\; \frac{\mathrm{std}(\text{params})}{|\mathrm{mean}(\text{params})|}, \qquad L_\mathrm{reg} \;=\; \lambda \cdot \mathrm{CV}^2 \cdot \mathrm{MSE} \cdot N.\]

With \(\lambda\) auto-tuned so that the penalty contributes a target fraction (typically \(10\%\)) of MSE at a target CV (typically \(0.10\)), the regularisation becomes scale-invariant, physically interpretable, and large enough to actually constrain the optimisation.

Layer 4 – Gradient collapse monitor

Module: xpcsjax.optimization.nlsq.gradient_monitor. Class: GradientCollapseMonitor.

The fourth layer monitors the optimisation in real time and detects gradient collapse — the state in which physical-parameter gradients become negligible compared to per-angle gradients. The detection criterion is

(3)\[\mathrm{ratio} \;=\; \frac{\|\nabla_\mathrm{physical} L\|}{\|\nabla_\mathrm{per\text{-}angle} L\|}.\]

When \(\mathrm{ratio} < \tau\) (default \(10^{-2}\)) for \(N_c\) consecutive iterations (default \(5\)), collapse is declared and recorded.

Important

Layer 4 is strictly observational in the wired solve path. The per-iteration callback actually passed to the NLSQ solver (build_gradient_collapse_callback()) feeds the monitor and always returns None – monitor-on and monitor-off produce a bit-identical fit trajectory. response_mode ("warn" / "hierarchical" / "reset" / "abort", default "hierarchical") configures what GradientCollapseMonitor.get_response() would recommend, and it is surfaced in diagnostics for a human or a post-hoc caller to act on – but no production call site currently invokes get_response() to change the running solve. Treat collapse detection as a diagnostic signal (log it, inspect nlsq_diagnostics["gradient_monitor"]), not as an active intervention.

Layer 5 – Shear-sensitivity weighting

Module: xpcsjax.optimization.nlsq.shear_weighting. Class: ShearSensitivityWeighting.

The fifth layer addresses gradient cancellation directly by weighting residuals according to their sensitivity to the shear parameter. The shear-term gradient at angle \(\phi\) scales with \(|\cos(\phi - \phi_0)|\), so the weight assigned to angle \(\phi\) is

(4)\[w(\phi) \;=\; w_\mathrm{min} + (1 - w_\mathrm{min})\, \bigl|\cos(\phi - \phi_0)\bigr|^{\,a},\]

with defaults \(w_\mathrm{min} = 0.3\) and exponent \(a = 1.0\). The weights are normalised so that their mean equals one, preserving the loss scale.

The effect is to amplify residuals at shear-sensitive angles (\(\phi \approx \phi_0\) or \(\phi \approx \phi_0 + \pi\)) and attenuate residuals at shear-insensitive angles (\(\phi \approx \phi_0 \pm \pi/2\)). The asymmetric weighting breaks the gradient cancellation symmetry and produces a net signal on \(\dot{\gamma}_0\).

Note

Layer 5 is gated to laminar_flow only. The \(|\cos(\phi - \phi_0)|\) weighting is derived from the laminar shear gradient \(\partial g_1 / \partial \dot{\gamma}_0 \propto \cos(\phi - \phi_0)\), so it is meaningful only where a shear rate appears in the kernel:

  • static_isotropic / static_anisotropic — no flow direction and no shear-sensitivity peak, so L5 is gated off (the static degeneracy is handled structurally by Layers 1–4).

  • two_component (heterodyne) — has its own velocity/flow term (\(v_0\), \(v_\mathrm{offset}\), \(\phi_{0,\mathrm{het}}\)), but it is structurally different from laminar_flow’s shear rate, so the laminar weighting does not transfer; the angular information is already well distributed across \(\phi\).

The gating is declared in _LAYER_GATES inside xpcsjax.optimization.nlsq.anti_degeneracy_controller. Note that is_layer_active() returns True for every layer when analysis_mode is None (the homodyne characterisation gate’s path), so the gating does not affect the rtol=1e-10 parity baselines.

Layer-by-layer coverage by optimisation path

The five layers compose differently depending on the optimisation path.

Layer coverage by optimisation path

Path

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Local NLSQ (gradient)

yes

yes

yes

yes

yes

Multistart (LHS)

yes

yes

yes

yes

yes

CMA-ES escape

yes

–

–

–

–

Layers 2–5 are specific to gradient-based optimisation; CMA-ES uses fitness ranking rather than gradients, so the hierarchical alternation, the gradient monitor, and the gradient-cancellation weighting do not apply. Layer 1 (parameter-space reduction) is, however, essential for CMA-ES too — it reduces the search dimension from \(53\) to \(9\) for a 23-angle laminar-flow fit and is the difference between a tractable and an intractable global search.

Layer activation by analysis mode

Only Layer 5 is gated by analysis_mode; Layers 1–4 are available in every mode (their effect still depends on configuration and per-angle mode). The following table shows which layers a mode can run (the hard gate), independent of the config-level enable flags:

Layer availability by analysis mode (the gate)

Mode

L1

L2

L3

L4

L5

static_isotropic

yes

yes

yes

yes

no

static_anisotropic

yes

yes

yes

yes

no

laminar_flow

yes

yes

yes

yes

yes

two_component

yes

yes

yes

yes

no

Configuration

The controller is configured by an AntiDegeneracyConfig dataclass, built via AntiDegeneracyConfig.from_dict() from the mode YAML’s anti_degeneracy: block. The YAML shape is nested (a sub-mapping per layer); the flat layer_field names below are the Python dataclass attribute, not the YAML key:

anti_degeneracy:
  enable: true                         # -> AntiDegeneracyConfig.enable
  per_angle_mode: "auto"               # "individual" | "constant" | "averaged" | "auto"
  constant_scaling_threshold: 3        # Nphi cutover: auto -> "averaged" at n_phi >= threshold, else "individual"
  execute_layers: false                # opt-in L2/L3 escape gate on the >=1M stratified-LS path only

  hierarchical:                        # -> hierarchical_* fields
    enable: true                       # hierarchical_enable
    max_outer_iterations: 5            # hierarchical_max_outer_iterations
    outer_tolerance: 1.0e-6            # hierarchical_outer_tolerance
    physical_max_iterations: 100       # hierarchical_physical_max_iterations
    per_angle_max_iterations: 50       # hierarchical_per_angle_max_iterations

  regularization:                      # -> regularization_* fields
    enable: false
    mode: "relative"                   # regularization_mode: "absolute" | "relative" | "auto"
    lambda: 1.0                        # regularization_lambda
    target_cv: 0.10                    # regularization_target_cv
    target_contribution: 0.10          # regularization_target_contribution
    max_cv: 0.20                       # regularization_max_cv
    auto_tune_lambda: true             # regularization_auto_tune_lambda

  gradient_monitoring:                 # -> gradient_* fields
    enable: true                       # gradient_monitoring_enable
    ratio_threshold: 0.01              # gradient_ratio_threshold
    consecutive_triggers: 5            # gradient_consecutive_triggers
    response: "hierarchical"           # gradient_response_mode -- see the Layer 4 note above: not currently wired to change the running solve

  shear_weighting:                     # -> shear_weighting_* fields (laminar_flow only)
    enable: true
    min_weight: 0.3
    alpha: 1.0

enable, per_angle_mode, constant_scaling_threshold, and execute_layers are the only top-level (non-nested) keys; every other layer’s settings live under its own sub-mapping. The full set of fields and their nested-key mapping is enumerated in AntiDegeneracyConfig.from_dict().

Usage

The defence is wired into xpcsjax.optimization.nlsq.fit_nlsq() and activates automatically when per_angle_mode is non-individual and \(N_\phi \geq 3\). A typical invocation is:

from xpcsjax import fit_nlsq, load_xpcs_data

data = load_xpcs_data("experiment.hdf5")
# analysis_mode ("laminar_flow") and per_angle_mode ("auto", the default)
# are set in the config (ConfigManager / YAML), not passed as kwargs.
result = fit_nlsq(data, config)

The fitted parameter vector and the per-angle scaling are stored on OptimizationResult. Diagnostics including the gradient-monitor decisions and the per-angle CV are exposed through the nlsq_diagnostics attribute (streaming / out-of-core and stratified paths additionally populate streaming_diagnostics / stratification_diagnostics).

When to use which mode

Mode

Recommended when

auto

Default for all production runs (\(N_\phi \geq 3\)).

constant

Debugging, or when the quantile estimate is known to be reliable and speed matters most.

averaged

A single shared contrast/offset is adequate and you want it optimised rather than frozen; what auto resolves to for \(N_\phi \geq 3\).

individual

Post-hoc refinement only, initialised from an auto result. Never as a first attempt for \(N_\phi > 6\).

See also