Metadata-Version: 2.4
Name: actirhythm-toolkit
Version: 0.1.0
Summary: Reproducible accelerometer analysis pipeline for circadian and behavioral rhythm studies
Home-page: https://github.com/nerminjukan/masters-thesis
Author: Nermin Jukan
License: MIT
Project-URL: Source, https://github.com/nerminjukan/masters-thesis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.21.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: pyarrow>=5.0.0
Requires-Dist: matplotlib>=3.4.0
Requires-Dist: seaborn>=0.11.0
Requires-Dist: statsmodels>=0.13.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: hmmlearn>=0.2.7
Requires-Dist: pyyaml>=5.4.0
Requires-Dist: loguru>=0.5.3
Provides-Extra: ml
Requires-Dist: xgboost>=1.5.0; extra == "ml"
Requires-Dist: pomegranate>=0.14.8; extra == "ml"
Requires-Dist: CosinorPy>=1.1; extra == "ml"
Provides-Extra: glmm
Requires-Dist: pymer4>=0.7.0; extra == "glmm"
Requires-Dist: polars>=1.0.0; extra == "glmm"
Requires-Dist: rpy2>=3.6.0; extra == "glmm"
Requires-Dist: great-tables>=0.23.0; extra == "glmm"
Provides-Extra: notebooks
Requires-Dist: jupyter>=1.0.0; extra == "notebooks"
Requires-Dist: ipykernel>=6.0.0; extra == "notebooks"
Requires-Dist: nbformat>=5.1.0; extra == "notebooks"
Requires-Dist: tqdm>=4.62.0; extra == "notebooks"
Provides-Extra: dev
Requires-Dist: pytest>=6.2.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
Requires-Dist: python-dotenv>=0.19.0; extra == "dev"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Masters Thesis Analysis Pipeline

A reproducible Python-based data analysis pipeline for thesis research, focusing on time series analysis, hidden Markov models, cosinor analysis, and machine learning.

**Designed for accelerometer data analysis** with support for 3-axis accelerometer readings and activity metrics.

## ⚠️ Important: Methodological Corrections

**This pipeline has been updated with critical methodological corrections** for biological and statistical validity. If you're using this for animal activity rhythm analysis:

1. **READ FIRST**: [`METHODOLOGY_CORRECTIONS.md`](METHODOLOGY_CORRECTIONS.md) - Detailed explanation of all corrections
2. **QUICK START**: [`QUICK_START.md`](QUICK_START.md) - Quick reference guide for the corrected pipeline

### Key Corrections Implemented
- ✅ Feature standardization to prevent HMM state flickering
- ✅ Biological state validation (replaces unreliable AIC/BIC selection)
- ✅ Minimum dwell-time filtering for realistic behavioral dynamics
- ✅ State-based cosinor analysis (correct approach - applied AFTER state identification)
- ✅ Per-individual feature computation for multi-subject studies
- ✅ Complete documentation of methodological choices

**Previous pipeline order (incorrect):** Raw → Features → Cosinor → HMM  
**Corrected pipeline order:** Raw → Features → Standardization → HMM → Dwell-time Filter → Biological Validation → State-Based Cosinor

See [`METHODOLOGY_CORRECTIONS.md`](METHODOLOGY_CORRECTIONS.md) for full details.

---

## Data Format

The pipeline is configured to work with accelerometer data in CSV format with the following columns:

- **ActMindata**: Tag-derived activity metric (PRIMARY ACTIVITY SIGNAL)
- **XAccel, YAccel, ZAccel**: 3-axis accelerometer readings (UNSIGNED RELATIVE VALUES, 0-255)
- **PostChg**: Posture change indicator
- **PostCt**: Posture count
- **ACTEndTimeAllS**: Timestamp of the reading (e.g., "7/8/23 10:46")
- **serial**: Device serial number
- **subject**: Subject identifier

### IMPORTANT: Accelerometer Data Assumptions

**CRITICAL**: XAccel/YAccel/ZAccel are **UNSIGNED RELATIVE VALUES** (typically 0-255), NOT calibrated acceleration in physical units.

**What this means**:
- Absolute axis values encode device orientation + gravity (NOT activity intensity)
- DO NOT compute VeDBA, ODBA, or raw magnitude: `sqrt(X² + Y² + Z²)` is INCORRECT
- DO NOT square or directly combine axes assuming zero-centered data
- USE ActMindata as your primary activity signal (validated by tag manufacturer)
- DERIVE relative movement features from axes (variance, std, absolute differences)

**Correct workflow**:
```python
# CORRECT: Use ActMindata as primary signal
df['activity'] = df['ActMindata']

# CORRECT: Add relative movement features from axes
df = features.add_accelerometer_movement_features(df, subject_column='subject')

# INCORRECT: Do not compute magnitude from unsigned axes
# df['activity'] = sqrt(XAccel² + YAccel² + ZAccel²)  # WRONG!
```

A sample dataset is provided in `data/raw/sample_accelerometer_data.csv`.

## Project Structure

```
masters-thesis/
├── data/
│   ├── raw/              # Raw data files (not committed to git)
│   └── processed/        # Processed data files (not committed to git)
├── notebooks/            # Jupyter notebooks for analysis
│   ├── 00_intake.ipynb           # Data intake
│   ├── 01_qc_eda.ipynb          # Quality control and EDA
│   ├── 02_features.ipynb         # Feature engineering (⚠️ WITH STANDARDIZATION)
│   ├── 03_hmm_hsmm.ipynb        # HMM/HSMM modeling (⚠️ BIOLOGICAL VALIDATION)
│   ├── 04_cosinor.ipynb         # Cosinor analysis (⚠️ STATE-BASED)
│   └── 05_glmm_ml.ipynb         # GLMM and ML models
├── src/                  # Source code modules
│   ├── __init__.py
│   ├── io.py            # Data I/O functions
│   ├── qc.py            # Quality control
│   ├── features.py      # Feature engineering (⚠️ NEW: standardization, log transform)
│   ├── models_hmm.py    # HMM/HSMM (⚠️ NEW: dwell-time filter, biological validation)
│   ├── cosinor.py       # Cosinor analysis (⚠️ NEW: state-based methods)
│   ├── effects.py       # GLMM and ML models
│   └── eval.py          # Evaluation utilities
├── outputs/             # Generated outputs (plots, results)
├── logs/                # Log files
├── docs/                # Documentation
├── METHODOLOGY_CORRECTIONS.md  # ⚠️ CRITICAL: Read this for methodological details
├── QUICK_START.md              # Quick reference guide
├── requirements.txt     # Python dependencies
├── setup.py            # Package setup
└── README.md           # This file
```

⚠️ = Contains critical methodological corrections

## Features

### Reproducible Pipeline

The pipeline implements the following workflow:

1. **Data Intake** (`00_intake.ipynb`)
   - Load accelerometer data from CSV files
   - Use ActMindata as primary activity signal (NOT magnitude from axes)
   - Parse timestamps
   - Initial data exploration
   - Logging of parameters and random seeds

2. **Quality Control** (`01_qc_eda.ipynb`)
   - Detection of missing values
   - Detection and removal of duplicates
   - Time gap analysis
   - Exploratory data analysis with visualizations

3. **Feature Engineering** (`02_features.ipynb`)
   - Use ActMindata as primary activity signal (NOT axis magnitude)
   - Add relative movement features from accelerometer axes (variance, std, changes)
   - Rolling statistics (mean, std, min, max) per individual
   - Time-based features (hour, day of week, etc.)
   - Log transformation and z-score standardization per individual
   - Lag and difference features

4. **HMM/HSMM Analysis** (`03_hmm_hsmm.ipynb`)
   - Input: ActMin + relative movement features (standardized)
   - Gaussian Hidden Markov Models with K=2..5 states
   - Minimum dwell-time filtering to prevent state flickering
   - Biological state validation (not just IC-based selection)
   - State labeling and interpretation
   - State sequence prediction (Viterbi algorithm)
   - Posterior probability estimation

5. **Cosinor Analysis** (`04_cosinor.ipynb`)
   - Applied AFTER behavioral state identification (CORRECT approach)
   - State-based cosinor: probability of active states
   - Per-state cosinor: state-conditioned activity rhythms
   - MESOR (mean level) estimation per state
   - Amplitude and acrophase extraction per state
   - Multi-period analysis
   - Statistical significance testing

6. **GLMM and Machine Learning** (`05_glmm_ml.ipynb`)
   - Generalized Linear Mixed Models (GLMM)
   - Random Forest regression/classification
   - XGBoost models
   - Model comparison and evaluation
   - Feature importance analysis

### Key Modules

- **io.py**: Functions for loading and saving data in multiple formats
  - `load_accelerometer_data()`: Load accelerometer CSV data with format validation
  - `load_raw_data()`: Generic loader for CSV, Excel, Parquet, JSON, Feather
  - `save_processed_data()`: Save processed data
- **qc.py**: Quality control checks (gaps, duplicates, validation)
- **features.py**: Feature engineering utilities
  - `add_accelerometer_movement_features()`: Compute relative movement features (variance, std, changes) from XAccel/YAccel/ZAccel
  - `add_rolling_statistics_per_individual()`: Rolling window features per subject
  - `standardize_features()`: Z-score normalization per individual (CRITICAL for HMM)
  - `log_transform_activity()`: Log transform for count data
  - `add_time_features()`: Extract time-based features
  - ~~`calculate_accelerometer_activity()`~~: DEPRECATED - do not use for unsigned axes
- **models_hmm.py**: HMM/HSMM implementation with model selection
  - `fit_hmm_pipeline()`: Complete HMM workflow
  - `apply_minimum_dwell_time()`: Post-process to prevent state flickering
  - `analyze_state_characteristics()`: Biological validation of states
  - `select_states_biologically()`: State selection based on interpretability
- **cosinor.py**: Cosinor analysis for circadian rhythms
  - `fit_cosinor_per_state()`: State-based cosinor (CORRECT approach)
  - `fit_cosinor_per_individual_per_state()`: Per-individual state-based analysis
  - `fit_cosinor()`: Basic cosinor fitting
- **effects.py**: Mixed effects models and machine learning
- **eval.py**: Evaluation metrics and visualization utilities

## Installation

1. Clone the repository:
```bash
git clone https://github.com/nerminjukan/masters-thesis.git
cd masters-thesis
```

2. Create a virtual environment:
```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

3. Install dependencies:
```bash
pip install -r requirements.txt
```

Or install as a package:
```bash
pip install -e .
```

Optional extras:

```bash
pip install -e .[dev]
pip install -e .[notebooks]
pip install -e .[ml]
pip install -e .[glmm]
```

Validated packaging states:

```bash
# Base CLI/runtime only
pip install -e .

# Add optional machine-learning/research extras
pip install -e .[ml]

# Add GLMM support used by optional pymer4 workflows
pip install -e .[glmm]
```

GLMM note:

```text
The Python extra installs the pymer4-side Python dependencies, but a working R installation
and required R packages are still needed for pymer4-backed models. In a clean Windows test
environment, pymer4 additionally required R packages such as tibble.
```

You can diagnose the GLMM environment with:

```bash
actirhythm glmm-doctor
```

### Run as an Installed Package (CLI)

After installation, run the full workflow from the project root with a single command:

```bash
actirhythm
```

Equivalent explicit commands:

```bash
actirhythm full --run-version v3
actirhythm run --run-version v3
```

Run stages separately:

```bash
actirhythm preprocess --run-version v3
actirhythm analytics --run-version v3
```

Advanced options:

```bash
# Show what will run and where outputs will go (no execution)
actirhythm full --run-version v4 --dry-run

# Resume safely: skip stages whose expected outputs already exist
actirhythm full --run-version v4 --skip-existing

# Override input/output locations
actirhythm preprocess \
   --data-revised-dir ./my-revised-data \
   --config ./config.yaml \
   --raw-data-file ./data/raw/fallback.csv \
   --processed-base-dir ./custom-processed \
   --output-base-dir ./custom-runs \
   --run-version experiment-01

# Run analytics from an explicit HMM parquet file
actirhythm analytics \
   --hmm-input-file ./custom-processed/experiment-01/hmm_results_improved.parquet \
   --output-base-dir ./custom-runs \
   --run-version experiment-01

# Use a non-default config file
actirhythm full --project-root /path/to/masters-thesis --config ./config.custom.yaml --dry-run

# Check optional GLMM prerequisites on the current machine
actirhythm glmm-doctor
```

Release validation:

```bash
python -m build
python -m twine check dist/*
```

Run from another folder by passing the project root:

```bash
actirhythm full --project-root /path/to/masters-thesis --run-version v3
```

## Usage

### Running the Pipeline

Execute notebooks in sequence:

```bash
cd notebooks
jupyter notebook
```

Run notebooks in order (00 → 01 → 02 → 03 → 04 → 05).

### Using Modules in Code

```python
from src import io, qc, features, models_hmm, cosinor, effects, eval

# Load accelerometer data
df = io.load_accelerometer_data('data/raw/sample_accelerometer_data.csv')

# CORRECT: Use ActMindata as primary activity signal
df['activity'] = df['ActMindata']

# CORRECT: Add relative movement features from accelerometer axes
# These capture variance and changes, NOT absolute magnitude
df = features.add_accelerometer_movement_features(
    df, 
    subject_column='subject',
    windows=[6, 12, 24],
    standardize=True
)

# Quality control
qc_report = qc.qc_report(df, time_column='timestamp')

# Feature engineering - use ActMin, NOT magnitude
df_features = features.add_rolling_statistics_per_individual(
    df, 
    'activity',  # ActMin
    subject_column='subject',
    windows=[6, 12, 24]
)
df_features = features.add_time_features(df_features, 'timestamp')

# Standardize features before HMM (CRITICAL)
df_features = features.log_transform_activity(df_features, 'activity', offset=1.0)
df_features = features.standardize_features(
    df_features, 
    ['activity_log'],
    group_by='subject'
)

# HMM analysis with standardized features
feature_cols = ['activity_log_standardized']
# Add standardized movement features
movement_features = [col for col in df_features.columns if 'movement_var' in col and '_standardized' in col]
feature_cols.extend(movement_features[:3])

X = df_features[feature_cols].dropna().values
hmm_results = models_hmm.fit_hmm_pipeline(X, state_range=range(2, 6))

# Apply dwell-time filter
states_filtered = models_hmm.apply_minimum_dwell_time(
    hmm_results['states'], 
    min_dwell=4
)

# State-based cosinor (CORRECT - applied AFTER state identification)
df_with_states = df.copy()
df_with_states['hmm_state'] = states_filtered
cosinor_results = cosinor.fit_cosinor_per_state(
    df_with_states,
    time_column='timestamp',
    state_column='hmm_state',
    activity_column='activity'
)

# Machine learning
rf_results = effects.fit_random_forest(X_train, y_train, X_test, y_test)
```

## Reproducibility

All analyses use fixed random seeds (default: 42) for reproducibility. Parameters and seeds are logged in each notebook for full traceability.

## Dependencies

Core dependencies:
- numpy, pandas, scipy
- matplotlib, seaborn
- scikit-learn, xgboost
- hmmlearn, pomegranate
- statsmodels, pymer4
- jupyter, loguru

See `requirements.txt` for complete list.

## Contributing

This is a thesis project. For questions or suggestions, please open an issue.

## License

This project is part of a master's thesis research.

## Author

Masters Thesis Author

## Citation

If you use this pipeline in your research, please cite appropriately.
