Remove of the off-features?

[axiomcura]

Recent analysis shows that morphological signatures are capturing a disproportionate amount of noise, which is likely driving the scoring inconsistencies we are currently observing. In particular, off-features contribute unexpectedly high scores (e.g., 12–18), overwhelming the signal of interest and obscuring meaningful biological or experimental patterns.

Our initial hypothesis was that these off-features might encode unrelated morphological structure. However, we are currently unable to reliably separate true unaffected signal from pure noise within this feature set. Standard statistical approaches does not do that.

Given these limitations, continuing to include off-features in clustering and downstream analyses is likely degrading both interpretability and metric stability.

I think restricting clustering and downstream analysis exclusively to the on-morphological signature, where features are directly associated with the state of interest.

Expected Benefits

• Improved Signal-to-Noise Ratio: Removing off-features should amplify the biological or experimental signals we aim to capture.
• Increased Precision: Metrics will reflect only features meaningfully associated with the target state.
• Computational Efficiency: A reduced feature space will lower clustering and processing costs.
• Improved Metric Stability: Eliminating high-noise off-features should resolve the current scoring anomalies.

implementation Ideas

• Early Pipeline Filtering: Drop off-signature columns immediately after initial feature extraction.
• On-Signal Re-clustering: Re-run existing clustering models using only on-morphological features to validate improvements in cluster separation and score behavior.