# progress.md

## Purpose

This file is the running record of project execution for **covid data analysis**. Every phase must update this document with what was done, what was discovered, what decisions were made, and what changes are required in the specs or downstream work. No transformation, assumption change, or modeling decision should exist only in code or notebooks without also being reflected here.

---

## Project Metadata

* **Project:** covid data analysis
* **Domain:** Healthcare
* **Primary goal:** Produce machine-learning based models to help evaluate when patients should be admitted and when they may be appropriate for discharge when presenting with COVID-like symptoms
* **Tech stack:** Python 3.12+, Pandas, scikit-learn
* **Status:** Phase 4 EDA in progress
* **Current phase:** Exploratory data analysis
* **Last updated:** 2026-04-22

---

## Workflow Rules

* Before starting a phase, read the relevant spec in `specs/`
* After completing a phase, update this file immediately
* Do not move to the next phase until the current phase acceptance criteria are met
* If EDA or data validation changes the problem definition, update `specs/01-problem.md`
* Every data transformation must be documented here
* Every modeling assumption, exclusion rule, and leakage concern must be documented here

---

## Phase Status Summary

| Phase | Spec                              | Status      | Owner | Notes                        |
| ----- | --------------------------------- | ----------- | ----- | ---------------------------- |
| 1     | `specs/01-problem.md`             | In progress | TBD   | Initial problem spec drafted |
| 2     | `specs/02-data.md`                | Complete    | TBD   | Ingestion and validation baseline implemented and documented |
| 3     | `specs/03-data-cleaning.md`       | Complete    | TBD   | Initial cleaning spec, cleaned dataset, and cleaning report implemented |
| 4     | `specs/04-eda.md`                 | In progress | TBD   | EDA notebook built and executed; findings under review |
| 5     | `specs/05-model-development.md`   | Not started | TBD   | Deferred until EDA review and later feature design |
| 6     | `specs/06-evaluation.md`          | Not started | TBD   | Not started                  |
| 7     | `specs/07-reporting.md`           | Not started | TBD   | Not started                  |

---

## Decision Log

Use this section to capture decisions that affect scope, data definitions, modeling, evaluation, or deployment assumptions.

| Date       | Phase | Decision                                                                              | Reason                                                                                                                        | Impact                                                                 |
| ---------- | ----- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| YYYY-MM-DD | 1     | Project split into two primary prediction problems: admission and discharge readiness | The business problem contains two distinct clinical decision points with different units of analysis and feature availability | Separate labels, datasets, and evaluation logic will be required       |
| YYYY-MM-DD | 1     | Added exploratory analysis as a formal analytical question                            | EDA is needed to validate assumptions, data quality, and target feasibility before modeling                                   | Phase 2 and Phase 3 must explicitly test assumptions and leakage risks |
| 2026-04-21 | 2     | Phase 2 spec will use `specs/02-data.md` instead of `specs/02-data-ingestion.md`     | The repository currently uses a simplified spec layout and the user requested `specs/02-data.md`                             | Progress tracking and downstream references should use the new filename |
| 2026-04-21 | 2     | Treat `visit_occurrence_id` as the provisional encounter identifier                   | The source CSV has one unique `visit_occurrence_id` per row across 156,030 rows                                               | Ingestion validation and later cohort logic can begin from encounter-level assumptions |
| 2026-04-21 | 2     | Defer cohort filtering until after raw ingestion                                      | The source file includes many non-COVID conditions and implausible early dates that require validation before filtering       | Raw ingestion remains lossless; cohort definition moves to cleaning and EDA |
| 2026-04-21 | 2     | Copy both the source CSV and metadata file into `data/raw/` without renaming          | The metadata file is part of the source provenance and should travel with the raw dataset                                     | Raw ingestion now preserves both tabular data and source-side schema summary |
| 2026-04-22 | 2     | Keep raw ingestion immutable on repeat runs                                           | Re-running ingestion should not overwrite files in `data/raw/` because that area is the project’s auditable baseline         | Existing identical files are left untouched; conflicting files now raise an error |
| 2026-04-22 | 2     | Use a JSON validation report as the Phase 2 executable artifact                       | Phase 2 needs a durable, machine-readable record of ingestion-time checks and warnings                                         | Validation findings can feed tests, progress updates, and later cleaning work |
| 2026-04-22 | 2     | Standardize local setup around `venv/` and pinned `requirements.txt`                 | The repository needs a reproducible developer environment before further pipeline and notebook work                           | Dependency installation, commands, and ignore rules now target a single local environment |
| 2026-04-22 | 2     | Validate the current Phase 2 workflow inside `venv/`                                 | Environment setup is only useful if the actual pipeline and tests run under the managed interpreter                           | Subsequent work can use `venv/bin/python` and `venv/bin/pytest` as the default execution path |
| 2026-04-22 | 2     | Rebuild `venv/` with Python 3.14 and refresh dependency pins                         | The earlier Python 3.11 environment did not match the requested interpreter, and the previous package pins were not compatible with Python 3.14 | The local environment now matches the requested interpreter and uses dependency versions with Python 3.14 wheels |
| 2026-04-22 | 3     | Keep Phase 3 cleaning row-preserving and use flags instead of hidden cohort drops    | The dataset scope is still under review, so cleaning should improve structure without silently changing the population         | Downstream modeling can choose filters explicitly from documented quality flags |
| 2026-04-22 | 3     | Normalize `condition` into a derived de-duplicated term field while retaining source text | The raw condition field contains colon-delimited strings and 2,263 rows with repeated terms                                   | Exploratory analysis and feature engineering can use a cleaner representation without losing provenance |
| 2026-04-22 | 4     | Execute EDA before feature engineering                                                | The cleaned data still has unresolved cohort, timing, and target-definition risks that should be tested before deriving features | Phase 4 now focuses on EDA readiness checks and hypothesis testing rather than direct feature construction |
| 2026-04-22 | 4     | Treat EDA patterns as hypotheses rather than conclusions                              | The current file still has cohort and timing ambiguities, so descriptive patterns should not be over-interpreted              | Notebook outputs and progress updates use exploratory language and competing explanations |

---

## Change Log

Use this section to record changes to specs, code, notebooks, data definitions, or scope.

| Date       | Changed artifact      | Change summary                                                                                                                       | Trigger                |
| ---------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
| YYYY-MM-DD | `AGENTS.md`           | Created steering file with architecture, standards, data conventions, and workflow rules                                             | Project initialization |
| YYYY-MM-DD | `specs/01-problem.md` | Created initial problem definition spec with business context, analytical questions, scope, assumptions, constraints, and phase gate | Project initialization |
| 2026-04-21 | `specs/02-data.md`    | Added the Phase 2 data-ingestion spec for `final_with_deceased.csv` and `metadata.json`, including schema findings, risks, and acceptance criteria | Phase 2 planning |
| 2026-04-21 | `progress.md`         | Updated project status to Phase 2 planning and recorded source-file findings, schema mismatch, and cohort-scope risks              | Phase 2 planning |
| 2026-04-21 | `data/raw/`           | Created raw-data directories and copied immutable baseline versions of `final_with_deceased.csv` and `metadata.json`               | Phase 2 execution |
| 2026-04-22 | `src/config.py` and `src/logging_config.py` | Added project path configuration and standard logging setup for executable data workflows                                        | Phase 2 implementation |
| 2026-04-22 | `src/data/ingest.py` and `src/data/validate.py` | Implemented raw ingestion, immutability safeguards, schema validation, date checks, cohort-scope checks, and JSON report generation | Phase 2 implementation |
| 2026-04-22 | `src/pipeline.py` and `tests/test_data.py` | Added a Phase 2 pipeline entrypoint and automated dataset validation tests                                                        | Phase 2 implementation |
| 2026-04-22 | `data/output/ingestion_validation_report.json` | Generated the first machine-readable validation report from the raw dataset                                                       | Phase 2 execution |
| 2026-04-22 | `requirements.txt`, `.gitignore`, and `AGENTS.md` | Added pinned dependencies, local environment ignore rules, and updated project instructions for `venv/` and `specs/02-data.md` | Environment setup |
| 2026-04-22 | `venv/` | Created a local virtual environment and installed the pinned dependencies from `requirements.txt`                                              | Environment setup |
| 2026-04-22 | `venv/`, `requirements.txt`, `AGENTS.md`, and `progress.md` | Recreated the virtual environment with Python 3.14.3 and updated dependency pins to a Python 3.14 compatible baseline | Environment refresh |
| 2026-04-22 | `specs/03-data-cleaning.md` | Added the Phase 3 data-cleaning spec with conservative row-preserving rules, derived flags, and cleaned-output acceptance criteria | Phase 3 planning |
| 2026-04-22 | `src/data/clean.py`, `src/pipeline.py`, and `tests/test_data.py` | Implemented the first cleaning workflow, integrated it into the pipeline, and added cleaned-data tests | Phase 3 implementation |
| 2026-04-22 | `data/cleaned/final_with_deceased_cleaned.csv` and `data/output/cleaning_report.json` | Generated the first cleaned encounter-level base table and machine-readable cleaning report | Phase 3 execution |
| 2026-04-22 | `specs/04-eda.md` | Added the EDA planning spec covering trust checks, four-dimensional exploration, hypotheses, and assumption validation before modeling | Phase 4 planning |
| 2026-04-22 | `src/eda/build_notebook.py` and `notebooks/01_eda.ipynb` | Added a reproducible notebook builder and executed EDA notebook covering the full exploration plan from `specs/04-eda.md` | Phase 4 execution |

---

## Risks and Open Questions

Track anything unresolved that could block progress or invalidate the current plan.

| Date       | Category | Risk or question                                                                                | Severity | Owner | Mitigation / next step                                                   |
| ---------- | -------- | ----------------------------------------------------------------------------------------------- | -------- | ----- | ------------------------------------------------------------------------ |
| YYYY-MM-DD | Data     | It is not yet confirmed how patients with COVID-like symptoms will be identified in source data | High     | TBD   | Validate cohort definition during data ingestion and EDA                 |
| YYYY-MM-DD | Labels   | “Appropriate discharge” is not yet operationally defined                                        | High     | TBD   | Work with stakeholders to define discharge outcome and safe proxy labels |
| YYYY-MM-DD | Leakage  | Future clinical events may leak into admission or discharge features                            | High     | TBD   | Audit feature timestamps before any modeling                             |
| YYYY-MM-DD | Access   | Availability of longitudinal inpatient data is not yet confirmed                                | High     | TBD   | Confirm data sources and completeness in Phase 2                         |
| 2026-04-21 | Schema   | CSV header uses `measurement_Date` while `metadata.json` uses `measurement_date`               | Medium   | TBD   | Validate the authoritative schema during ingestion and document mismatch |
| 2026-04-21 | Cohort   | The source file includes many visits before 2020 and many apparently non-COVID conditions      | High     | TBD   | Determine whether the file is a broad source extract or a mislabeled cohort |
| 2026-04-21 | Time     | The dataset appears to use date-only fields rather than timezone-aware timestamps               | Medium   | TBD   | Record this limitation and assess whether available granularity is sufficient for leakage control |
| 2026-04-21 | Labels   | The `deceased` field may not map cleanly to admission or discharge targets                      | High     | TBD   | Treat as a candidate outcome-related field only until target definitions are validated |
| 2026-04-22 | Proxy    | `visit_type` may be the only available admission proxy, but its suitability as a true label remains unconfirmed | High | TBD | Review with stakeholders and test whether alternative admission labels exist |
| 2026-04-22 | Discharge | The current extract does not show clear discharge-readiness labels or inpatient decision-time snapshots | High | TBD | Likely defer Question 2 or request additional inpatient/discharge data |
| 2026-04-22 | Measurement | Oxygen saturation appears tightly compressed near the upper range, suggesting clipping or recording artifacts | Medium | TBD | Compare against source-system measurement definitions before using it heavily in models |

---

## Assumption Validation Tracker

Each assumption from `specs/01-problem.md` must be tested during EDA or data validation.

| Assumption                                                       | Validation method                                                           | Status      | Findings |
| ---------------------------------------------------------------- | --------------------------------------------------------------------------- | ----------- | -------- |
| Patients with COVID-like symptoms can be identified consistently | Review diagnosis codes, symptom fields, encounter reasons, and cohort logic | In progress | `is_candidate_covid_like_visit` captures 79,944 rows, but heavy `observation_source` missingness and pre-2020 contamination mean the cohort is still ambiguous |
| Admission decisions are recorded clearly enough for labeling     | Inspect encounter outcomes and admission flags                              | In progress | `visit_type` behaves like a severity proxy, but no explicit admission label has been confirmed |
| Discharge decisions and timestamps are reliable                  | Compare discharge fields, timestamps, and disposition values                | Failed / unsupported | Current extract does not expose a clear discharge-readiness label or inpatient decision-time snapshots |
| Timestamped data is accurate enough to prevent leakage           | Audit event times relative to decision points                               | In progress | Dates are parseable, but only at date granularity; 2,790 measurements occur after visit start |
| Sufficient pre-admission predictors exist                        | Profile completeness of candidate admission-time variables                  | In progress | Demographics and some vitals exist, but vital missingness is substantial and likely setting-dependent |
| Sufficient inpatient longitudinal data exists                    | Inspect repeated vitals, labs, and state changes over time                  | Failed / unsupported | Current file is encounter-level and does not show clear within-stay longitudinal snapshots |
| Sample size is adequate                                          | Measure cohort size, class balance, and subgroup counts                     | In progress | Overall and candidate cohort sizes are large enough for EDA and likely for at least admission-focused baseline modeling |
| Data quality is adequate across key variables                    | Run missingness, range, and consistency checks                              | In progress | Many checks pass, but seven notebook validation checks failed, largely around cohort scope, representativeness, and possible artifacts |
| Stakeholders can agree on operational definitions                | Review with doctors and administrators                                      | Not started | EDA now isolates the main ambiguity points for stakeholder review |
| Unsafe-discharge proxies are available                           | Check for readmission, ICU transfer, mortality, or deterioration outcomes   | Failed / unsupported | No clear readmission, ICU transfer, or post-discharge safety proxy is visible in the current extract |

---

## Phase 1: Problem Definition

### Planned Work

* Document the business context and decision points
* Define precise analytical questions
* Establish scope and out-of-scope boundaries
* Record assumptions and constraints
* Confirm phase gate before moving to data work

### Findings

* The business problem naturally separates into two distinct prediction tasks:

  1. admission decision support at presentation
  2. discharge readiness decision support during or near end of stay
* These two questions likely require different units of analysis, feature sets, and labels
* A formal exploratory question is needed to validate assumptions before any modeling begins

### Decisions

* Treat admission and discharge as separate modeling problems
* Require time-aware feature eligibility rules to reduce leakage risk
* Keep the study period provisional until data inventory confirms available historical coverage

### Blockers

* Stakeholder review of scope has not yet occurred
* Operational definition of “appropriate discharge” remains unresolved
* Available data sources and label quality are not yet validated

### Exit Check

* [x] Business context documented
* [x] At least one analytical question is precise and classified
* [x] Success criteria defined for each question
* [ ] Stakeholders reviewed and agreed on scope
* [x] Assumptions documented and testable
* [ ] Phase approved for transition to Phase 2

---

## Phase 2: Data Ingestion

### Planned Work

* Inventory all data sources
* Confirm access, schema, and extract method
* Create immutable raw datasets in `data/raw/`
* Record source provenance and refresh cadence

### Findings

* `final_with_deceased.csv` and `metadata.json` are present as the current source artifacts for Phase 2
* Immutable copies of both source artifacts were created in `data/raw/`
* `data/raw/final_with_deceased.csv` matches the source file byte-for-byte with SHA-256 `a488c2bec4a3543b08ca4c81f3abd84a8f7a21e0b64a554f242274bfbb0f9ea7`
* `data/raw/metadata.json` matches the source file byte-for-byte with SHA-256 `b5d8242baed5c80805e4f6cb34da0eac18b798132b6f7ddfbd3f7e4626a8193f`
* The source CSV contains 156,030 rows and 27 columns
* `visit_occurrence_id` is unique on every row, supporting a provisional encounter-level interpretation
* The CSV header contains `measurement_Date`, while the metadata file refers to `measurement_date`
* Missingness is concentrated in measurement and observation fields; `body_height_cm` and `bmi` are missing for more than 98% of rows
* `visit_type` is heavily dominated by outpatient visits, with only 21,056 inpatient visits and 20 emergency room visits
* The dataset includes many non-COVID conditions and many rows with missing `observation_source`, so the COVID-like cohort is not yet confirmed
* Metadata reports implausibly early dates such as `1909-09-17` for `visit_start_date` and `1919-05-11` for `measurement_date`, which must be investigated before downstream filtering or modeling
* `src.data.validate` now generates `data/output/ingestion_validation_report.json` with executable checks for schema alignment, identifier uniqueness, required columns, date ordering, date-only fields, and cohort-scope warnings
* The current validation report confirms 69,324 visits before `2020-01-01`, 8,144 measurements before `2020-01-01`, 77,833 rows with missing `observation_source`, and 4,893 non-missing observations other than `Suspected COVID-19`
* Automated tests pass for row and column counts, required-column presence, visit identifier uniqueness, date ordering, header mismatch detection, and broad-cohort warning detection
* A local `venv/` environment is being added for reproducible dependency management, with pinned versions recorded in `requirements.txt`
* The local `venv/` was recreated successfully with Python 3.14.3
* The original pins were not fully compatible with Python 3.14, so `requirements.txt` was updated to `pandas==3.0.2`, `scikit-learn==1.8.0`, `numpy==2.4.4`, `pytest==9.0.3`, and `jupyter==1.1.1`
* `venv/bin/pytest tests/ -x -q` passes and `venv/bin/python -m src.pipeline` runs successfully against the current raw dataset under Python 3.14.3

### Decisions

* Use `specs/02-data.md` as the Phase 2 spec filename
* Treat `final_with_deceased.csv` as the current authoritative source file for planning purposes
* Preserve raw ingestion as lossless and defer any filtering, renaming, or cleaning to later phases
* Use `visit_occurrence_id` as the provisional encounter identifier pending deeper validation
* Keep ingestion and validation dependency-light with Python standard-library modules so Phase 2 can run before the full data-science environment is provisioned
* Use `venv/` plus pinned dependencies as the default local execution environment for subsequent work

### Data Transformations

* None yet
* Raw ingestion only: copied `final_with_deceased.csv` to `data/raw/final_with_deceased.csv` without modification
* Raw ingestion only: copied `metadata.json` to `data/raw/metadata.json` without modification
* Generated `data/output/ingestion_validation_report.json` as a derived validation artifact from the raw source files

### Blockers

* Phase 1 has not yet recorded stakeholder review or formal phase-gate approval
* The source provenance beyond the local files is still undocumented
* Cohort scope is unclear because the file appears broader than a clean COVID-only encounter set
* Timestamp granularity appears limited to dates, not timezone-aware timestamps
* The system default `python3` still points to Python 3.11.9, so project commands should continue to use `venv/bin/python` or an activated `venv`

### Exit Check

* [x] Relevant data sources identified
* [x] Raw data ingested and stored immutably
* [x] Source provenance documented
* [x] Initial data quality review completed

---

## Phase 3: Data Cleaning

### Planned Work

* Standardize formats and column names
* Resolve missing values explicitly
* Validate types, ranges, and consistency
* Produce cleaned datasets in `data/cleaned/`

### Findings

* The raw/header mismatch was resolved in the cleaned dataset by renaming `measurement_Date` to `measurement_date`
* The cleaned dataset preserves all 156,030 raw rows and expands the schema from 27 to 35 columns through derived cleaning fields and flags
* `age_at_visit_years` is internally consistent with `birth_datetime` and `visit_start_date`, so the source age value can be retained
* 2,263 rows contain repeated `condition` terms; the cleaner now preserves the source string and adds a de-duplicated `condition_terms_cleaned` field
* 69,324 visits occur before `2020-01-01`, so cohort scope remains unresolved and is represented with `is_pre_covid_era_visit` rather than implicit row removal
* 8,144 measurements occur before `2020-01-01`, and 2,790 measurements occur after visit start; both are now explicit flags in the cleaned output
* A broad candidate COVID-like subset can be approximated with `has_suspected_covid_observation` or respiratory symptom terms in `condition`; this captures 79,944 visits
* The cleaned dataset keeps missing values explicit and does not impute vitals, labs, or categorical fields in this phase
* Automated tests now pass for both raw validation and cleaned-data invariants under `venv/bin/pytest`

### Decisions

* Keep Phase 3 cleaning row-preserving and defer any cohort exclusion to explicit downstream logic
* Add derived quality flags rather than dropping pre-2020 or non-COVID-like rows
* Retain original source columns and add derived cleaned columns for interpretability and reproducibility

### Data Transformations

* Renamed `measurement_Date` to `measurement_date` in the cleaned dataset
* Standardized blank strings to explicit missing values before writing cleaned outputs
* Parsed date fields and rewrote them in stable `YYYY-MM-DD` format in the cleaned CSV
* Added `condition_terms_cleaned`, `condition_terms_repeated`, and `condition_term_count`
* Added `has_suspected_covid_observation`, `is_candidate_covid_like_visit`, `is_pre_covid_era_visit`, `is_pre_covid_era_measurement`, and `measurement_after_visit_start`
* Wrote `data/cleaned/final_with_deceased_cleaned.csv`
* Wrote `data/output/cleaning_report.json`

### Blockers

* Phase 1 still lacks stakeholder scope approval and a finalized operational definition of the target cohorts
* The cleaned dataset still includes a broad non-COVID and pre-2020 population because cohort restrictions are not yet approved as cleaning-time exclusions
* Source dates remain date-only rather than timezone-aware timestamps, which limits leakage-sensitive temporal modeling later

### Exit Check

* [x] Cleaning rules documented
* [x] Cleaned datasets created
* [x] Data validation tests added or updated
* [x] All transformations recorded in this file

---

## Phase 4: Exploratory Data Analysis

### Planned Work

* Execute the trust, provenance, and field-validation checks from `specs/04-eda.md`
* Explore the cleaned dataset across distributional, relational, comparative, and structural dimensions
* Test the initial problem-spec assumptions and document whether they are supported, unsupported, or still ambiguous
* Record revised hypotheses before any feature engineering work begins

### Findings

* The EDA notebook was built and executed at `notebooks/01_eda.ipynb`
* Stage 1-2 validation produced 15 passes and 7 documented failures; the failures are material but not immediate blockers for exploratory admission-focused work
* The major validation failures were pre-2020 visit contamination, possible date artifacts, weak cohort-definition support from `observation_source`, unproven sample representativeness, suspicious vital-sign compression, and limited interpretability of immunization fields
* The data suggests a strong regime change in early 2020: the largest monthly encounter-count breaks occur at `2020-03-01` and `2020-04-01`
* The data suggests the candidate COVID-like flag is informative but not definitive: candidate visits have an inpatient rate of 25.94% versus 0.42% for non-candidate visits, and the candidate share is 89.18% in 2020+ visits versus 3.78% before 2020
* The data suggests `visit_type` behaves like a severity proxy rather than a random administrative label: inpatient mortality is 30.26% versus 7.45% for outpatient visits
* The data suggests temperature differs meaningfully by suspected COVID observation status: median temperature is 40.0 for suspected-COVID rows versus 37.8 otherwise
* The data suggests condition complexity differs by setting: median `condition_term_count` is 4 for inpatient visits and 1 for outpatient visits
* The data does not support discharge-readiness modeling from the current extract because no clear inpatient decision-time snapshots or discharge-safety proxies are visible
* The data suggests oxygen saturation may have limited discriminative value in its current form: the median is 82.0 for both inpatient and outpatient groups, and the recorded range is unusually compressed near the upper bound

### Revised Hypotheses

* Pattern observed: a large pre-2020 segment coexists with a concentrated early-2020 respiratory-like surge
  Explanation A: the file combines historical baseline utilization with a later COVID-era respiratory wave
  Explanation B: extraction logic changed over time or non-COVID encounters were included unevenly
  What would distinguish them: stable source documentation and coding logic would support A; abrupt source-rule changes would support B
* Pattern observed: inpatient visits show higher mortality and respiratory-severity patterns than outpatient visits
  Explanation A: `visit_type` may approximate admission/severity well enough for exploratory Question 1 work
  Explanation B: downstream documentation and measurement availability may exaggerate the separation and create label-leakage risk
  What would distinguish them: explicit admission labels and stricter timing review would clarify whether the pattern is operational or artifact-driven
* Pattern observed: candidate COVID-like visits concentrate strongly in 2020+ and are far more likely to be inpatient
  Explanation A: the flag captures a real respiratory/COVID-like subgroup
  Explanation B: the condition-based logic may be over-inclusive because source condition strings are broad and concatenated
  What would distinguish them: review of discordant cases and access to richer diagnosis/context fields
* Pattern observed: oxygen saturation is tightly compressed and does not separate inpatient and outpatient medians
  Explanation A: most recorded values are clustered in a narrow clinical band
  Explanation B: the source may clip, round, or only record selected values
  What would distinguish them: source measurement definitions or raw measurement provenance
* Pattern observed: the current file supports encounter-level EDA but not discharge-time analysis
  Explanation A: the current extract is appropriate for admission-focused work before discharge modeling
  Explanation B: discharge-relevant fields may exist outside the current file rather than being truly absent
  What would distinguish them: a wider data inventory across additional inpatient and discharge tables

### Decisions

* Treat all strong EDA patterns as hypotheses, not conclusions
* Continue using a conservative row-preserving cleaned base table
* Defer feature engineering until the EDA review clarifies whether Question 1 is the only supportable near-term modeling target

### Data Transformations

* No new source-data transformations were added beyond the existing cleaned dataset
* Created and executed `notebooks/01_eda.ipynb`
* Generated exploratory figures, validation tables, decomposition outputs, and acceptance-check summaries inside the notebook

### Blockers

* The current file still lacks an explicit admission label and does not support discharge-readiness labeling
* Cohort definition remains partially ambiguous because `observation_source` is highly missing and the source includes many pre-2020 encounters
* Representativeness cannot be established from the current extract alone
* Temporal granularity remains date-only, which limits precise leakage assessment

### Exit Check

* [⚠️] All validation checks passed (or failures documented with impact assessment)
* [x] All four EDA dimensions explored for key variables
* [x] At least two competing explanations documented for each strong pattern
* [⚠️] Assumptions from problem spec validated or flagged
* [x] EDA findings do NOT overstate confidence
* [x] `progress.md` updated with key findings and revised hypotheses

---

## Phase 5: Model Development

### Planned Work

* Establish baselines
* Train candidate models
* Tune within defined scope
* Save reproducible model artifacts

### Findings

* Not started

### Decisions

* None yet

### Data Transformations

* None yet

### Blockers

* Not started

### Exit Check

* [ ] Baseline models created
* [ ] Candidate models trained reproducibly
* [ ] Random seeds fixed and documented
* [ ] No training-only evaluation reported as final performance

---

## Phase 6: Evaluation

### Planned Work

* Evaluate on holdout data
* Compare against baselines
* Assess subgroup behavior where possible
* Review error patterns and operational usefulness

### Findings

* Not started

### Decisions

* None yet

### Blockers

* Not started

### Exit Check

* [ ] Evaluation performed on non-training data
* [ ] Metrics documented for each task
* [ ] Limitations and risks documented
* [ ] Stakeholder review completed

---

## Phase 7: Reporting

### Planned Work

* Summarize methods, findings, and limitations
* Produce final outputs in `data/output/`
* Document recommendations and next steps

### Findings

* Not started

### Decisions

* None yet

### Blockers

* Not started

### Exit Check

* [ ] Final report completed
* [ ] Outputs stored in `data/output/`
* [ ] Recommendations documented
* [ ] Next steps agreed

---

## Transformation Log

Record every material transformation applied to data.

| Date       | Dataset | Step | Description                     | Reason                        | Code reference |
| ---------- | ------- | ---- | ------------------------------- | ----------------------------- | -------------- |
| YYYY-MM-DD | TBD     | TBD  | No transformations recorded yet | Project not yet in data phase | TBD            |

---

## Modeling Log

Record experiments, evaluation choices, and important modeling decisions.

| Date       | Task                  | Model | Data split | Metrics | Result summary              | Notes |
| ---------- | --------------------- | ----- | ---------- | ------- | --------------------------- | ----- |
| YYYY-MM-DD | Admission / Discharge | TBD   | TBD        | TBD     | No experiments recorded yet | TBD   |

---

## Next Actions

1. Review `specs/01-problem.md` with doctors and hospital administrators.
2. Finalize the operational definition of “appropriate discharge.”
3. Draft `specs/02-data-ingestion.md`.
4. Identify candidate source systems and required access.
