For data and IT teams
Data engineering and AWS consulting
Hands-on data engineering: designing and building pipelines, managing infrastructure with Terraform, improving access controls and reliability, and more.
We draw on years of enterprise data engineering on AWS, plus pipelines built for consulting clients with idempotent loads, replay, failure notifications and regional failover.
Describe what needs workWhat the work looks like
Design a data flow, or fix one that keeps breaking
Serverless ETL on AWS with Lambda, Step Functions, Glue and Spark, S3, EventBridge, SQS and SNS. How data arrives, what has to happen to it, what should happen when a step fails, and how a rerun avoids doing the work twice.
Make infrastructure repeatable
Terraform modules, remote state, and CI/CD for infrastructure changes in GitLab or GitHub. Existing resources brought under version control, with changes managed through code and reviewed before deployment.
Review access and security
IAM roles and policies checked against what each job actually needs, Lake Formation permissions, and security sweeps of data infrastructure. This is an engineering review, not a compliance audit or certification.
Make failures visible and reruns safe
Idempotent steps, replay of past runs, alerts that reach a person, searchable logging, data lifecycle rules, and regional failover where the design calls for it.
Plan or tune the data platform
Data lake and warehouse layout on S3, Athena and Redshift, with DynamoDB where a key-value store fits; retention and lifecycle; and tuning pipelines that have grown slow or expensive.
Connect reporting tools to dependable data
Getting dashboards and BI tools, including Power BI, onto well-structured, reliable data.
When a step fails
Designed for reliable operation, clear alerts and straightforward recovery.
- A step fails, for example because a source system times out.
- An alert reaches a person, with enough context to act on.
- The step is rerun. Because writes are idempotent, the rerun replaces the load instead of duplicating it.
- If an upstream correction arrives later, past runs can be replayed.
Tools
- AWS
- Lambda, Step Functions, Glue, S3, Lake Formation, Athena, Redshift, DynamoDB, EventBridge, SNS, SQS, CloudWatch, IAM, Amplify
- Code and processing
- Python and boto3, PySpark, Scala Spark
- Infrastructure and delivery
- Terraform, GitLab CI/CD, GitHub CI/CD, YAML
How technical engagements work
We can work alongside your engineers or take responsibility for a defined scope of work from start to finish: a design, a fix, a review or a build. You work directly with the engineer doing the work, and it’s documented as it goes, with diagrams, runbooks and code your team can maintain.
Scope and terms are agreed per engagement.
What needs work?
Describe your project or current system, what it should do, and where you need help. A few sentences are enough to start.