Chronic disease
Background[edit | edit source]
Chronic Disease is a contract managed by the Health team. This article describes a custom process to manage data for these clients during the transition phase between Aiga and Indici.
Rationale[edit | edit source]
As at June 2024 the health team are recording Chronic disease contract activity in Indici.
Because Aiga is still in use - and still acts as the key data source for referral and allocation data - it is necessary to merge and reconcile the two data sets.
The general workflow for this pipeline is to:
- Source a set of current Chronic disease clients from Aiga
- Push data set to S3[1]
- Write data set to RDS[2]
- Build reconciled patient cohort, and count activities for reporting.
Technical information[edit | edit source]
Sourcing cohort data from Aiga[edit | edit source]
A custom batch job using python is scheduled to run on the Windows server every day at 4pm.[3]
This executes a SQL query to find current clients allocated under the Chronic disease contract. The result set is then saved locally as parquet with a date key.
Push data to s3[edit | edit source]
The script then pushes the latest parquet file to s3://kpa-aiga.[4]
A lifecycle policy expires files after 30 days.
Load to RDS[edit | edit source]
PUT into S3 bucket triggers lambda function aigaLoadPatient. This function truncates target table aiga_staging.patient then inserts every record in new parquet data. A history is not required, so no redundancy is built into this process.
Reconcile data[edit | edit source]
Finally, database function rpt.fn_merge_patient_aiga_indici() runs on a cron schedule and performs the following actions:
- Gets patients flagged in Indici as Chronic disease patients.[5]
- Gets the Aiga records from
aiga_staging.patientthat already have an NHI. - Gets the remaining Aiga records that don't have an NHI available.
- These 3 sets are then reconciled and matched against each other to try and find an NHI identifier for all patients. Result is written to
rpt.patient_aiga_indici. - This reporting dataset can now be used as a Chronic Disease 'cohort', and identifiers used to look up consults/interactions for them in Indici.
- Sweet!