Simple ETL
The Simple ETL blueprint (pipeline v1) turns file drops into database
rows. One Lambda function watches a source bucket; when a file lands, it
parses the file, applies your schema mapping, and writes rows into the
linked database. There is no cluster, no scheduler, and no always-on
cost.
A pipeline connects two applied deployments: a source (a Secure Bucket, or an SFTP endpoint’s bucket) and a target (Database). Both links are wired: the platform injects the linked deployments’ outputs into the template, so you never copy identifiers by hand.
What it provisions
Section titled “What it provisions”- A Lambda function (
stacktakt-pipeline-<name>, Python 3.13, 300 second timeout, 512 MB; comfortable for files to roughly 50 MB). - A scoped execution role: reads only the source bucket’s configured prefix, writes only its own log group, and for DynamoDB targets writes only the one linked table.
- An S3 notification on the source bucket, filtered by your prefix and suffix, as the only thing allowed to invoke the function.
- A CloudWatch log group carrying every run’s per-file results, with 365-day retention.
- For PostgreSQL targets: VPC wiring (the function runs inside the database’s VPC, an S3 gateway endpoint, and one ingress rule admitting only the function’s security group on the database port).
Inputs
Section titled “Inputs”| Input | Required | Description |
|---|---|---|
| Source and Target | Yes | Two applied deployments to wire together. |
| Pipeline name | Yes | Resources are named stacktakt-pipeline-<name>. |
| Format | Yes | CSV, TSV, NDJSON, or JSON array. No format sniffing. |
| Key prefix / file suffix | No | Filters which object keys trigger the pipeline. |
| Schema fields | Yes | Up to 100 mappings: source field, target column, type (string, integer, number, boolean, timestamp, date), required flag. |
| Store unmatched fields | No | On by default: unmatched source fields land in an extra overflow column. |
How loads behave
Section titled “How loads behave”- A conversion failure on a required field rejects the row and logs
the file, row number, field, and reason. On an optional field, the
column becomes empty and the original value is preserved in
extra. - Loads are idempotent: rows carry a deterministic key from the source file and row number, so S3’s at-least-once event delivery never duplicates loaded rows.
- Failures are per-file and named in the log group. Nothing retries forever; re-upload a corrected file to reprocess it.
- Changing the schema mapping in place is out of scope: destroy and redeploy the pipeline.
Control mapping
Section titled “Control mapping”Generated from backend/templates/pipeline/v1/controls.yaml (framework: hipaa-security-rule). Run npm run generate:blueprints to refresh.
| HIPAA citation | Requirement | Status | Implemented by |
|---|---|---|---|
| §164.312(a)(1) | Access control | Implemented | aws_iam_role_policy.pipeline_scope, aws_lambda_permission.allow_source_bucket |
| §164.312(b) | Audit controls | Implemented | aws_cloudwatch_log_group.pipeline, platform.audit_timeline |
| §164.312(c)(1) | Integrity | Implemented | aws_lambda_function.pipeline |
| §164.312(e)(1) | Transmission security | Implemented | aws_lambda_function.pipeline, aws_vpc_security_group_ingress_rule.lambda_to_database |
Generated documents
Section titled “Generated documents”Once applied, the Documentation Center derives a System overview (including the data flow and operations notes) and a Control narrative for this deployment.