Data Platform Engineer

I help run a production data platform on Kubernetes: Airflow, Spark, Apache Iceberg and AWS. I find out why things break, prove it with evidence and write it down.

AWS, Kubernetes, Apache Airflow, Apache Spark, Apache Iceberg and Prometheus

Who I am and how I work

πŸ’Ό

Background & Current Role

Data Platform Engineer at COCUS Portugal since March 2026, on a small team that owns the infrastructure of a production analytics platform for a company in the energy sector. 24/7 on-call rotation since September 2026.


Before IT, 30 years as an officer in the SΓ£o Paulo Military Police: dental surgeon, then legal advisory, intelligence, procurement and personnel. Retired as a Major. Degrees in Dentistry (USP) and Law (UNIESP).

⚑

How I Work

Verify before concluding: my reports separate what I observed directly from what I infer. Command line first, so every investigation can be repeated step by step. In production, one change at a time, and only after the evidence is in. Everything written down: runbooks, protocols and incident reports.

πŸ› οΈ

Technical Skills

  • Kubernetes on Amazon EKS β€” daily operation and troubleshooting
  • Apache Airflow 2.8 and Spark on Kubernetes
  • Apache Iceberg and Hive Metastore
  • AWS (EKS, EMR, S3, IAM, CloudWatch Logs)
  • Prometheus and Grafana
  • Python, Bash and SQL for operations tooling

Current Tech Stack & Ongoing Learning

☸️
Container Orchestration

Kubernetes (EKS)

Daily operation in production: workloads, scheduling, autoscaling, persistent volumes and troubleshooting with kubectl. Preparing for the CKA.

πŸ”„
Workflow Orchestration

Apache Airflow

Airflow 2.8 in production: DAG failure triage, scheduler health and investigations in the metadata database.

βš™οΈ
Data Processing

Apache Spark

Spark on Kubernetes and Amazon EMR: driver and executor failures, right-sizing and cost analysis.

🧊
Lakehouse

Apache Iceberg

Iceberg tables on S3 with Hive Metastore: metadata growth, snapshot retention and lock issues.

☁️
Cloud Services

AWS

EKS, EMR, S3, IAM and CloudWatch Logs in production; S3, CloudFront and Terraform in personal projects.

πŸ“Š
Observability

Prometheus & Grafana

Platform dashboards and alerts in production, plus a self-built Airflow monitoring stack in the lab.

Production Work & Projects

Production cases are generalized and figures rounded to respect client confidentiality.


In production
🧊

Ingestion Tasks Taking Hours, With No Errors


A pipeline kept reporting success while tasks that should take minutes ran for over four hours. Two years of tiny commits without snapshot expiration had grown the table's Iceberg metadata to more than 5,000 times the size of its data.

Tools: AWS CLI, kubectl, Python (fastavro, psycopg2)

Result: Root cause confirmed, investigation report and remediation plan; a follow-up scan found dozens of tables with the same pattern

In production
🌐

Spark Jobs Failing on New Nodes


Spark drivers failed seconds after the cluster scaled up. New nodes reported Ready in about 13 seconds, but pod networking took 2 to 3.5 minutes to converge. Confirmed with live measurements and EKS API server audit logs.

Stack: Amazon EKS, Calico, Cluster Autoscaler, CloudWatch Logs

Result: Four mitigation options documented; fix in progress

In production
🧹

Daily Platform Audit


A written morning protocol of 21 checks, backed by a dependency-free Python CLI that finds waste and stuck work across environments: crash-looping and OOM-killed pods, zombie Spark pods, orphaned EMR clusters, stuck DAG runs and orphaned Hive Metastore locks.

Tools: Python standard library, kubectl, AWS CLI

Highlight: Found idle EMR clusters with 18 instances and no auto-termination

Personal project
πŸ“Š

Infrastructure Observability Stack


A monitoring stack for Apache Airflow. It uses Prometheus to scrape metrics via StatsD and Grafana for real-time visualization of heartbeat, throughput, and task latency.

Stack: Prometheus, Grafana, StatsD, PromQL, Docker

Highlight: PromQL rate() functions for instant performance analysis

View GitHub Repo β†’

Personal project
πŸ“ˆ

Automated Exchange Rate Pipeline


End-to-end data pipeline using Apache Airflow. It monitors EUR/BRL rates, processes metrics with Pandas, and automatically publishes a responsive dashboard via GitLab CI/CD & Pages.

Stack: Airflow, Docker, Python (Pandas), GitLab CI/CD

Highlight: Fully decoupled ETL architecture & automated SSH deployment

View Live Dashboard β†’
View Source Code (GitLab) β†’

Education & Continuous Learning

Ongoing Roadmap

CURRENT

Kubernetes: CKA exam preparation.

Apache Iceberg: internals and concurrency model.

dbt: fundamentals, toward analytics engineering.

Labs: Terraform, GitHub Actions and Linux.

FUTURE

Spark performance tuning and AWS Certified Data Engineer – Associate.

Goal: Deepen platform engineering for data workloads on Kubernetes.

Certifications

Certified Kubernetes Administrator (CKA) In Progress
AWS Certified Data Engineer – Associate Target

Technical Blog & Articles


Article

The Lock Held by a Process That No Longer Exists

An ingestion pipeline stopped for hours by an orphaned Hive Metastore lock, and why nothing ever expired it.

Read Article β†’
Article

Two Timeouts, One DAG, and the Detail Almost Nobody Explains

Why the Airflow task timeout must sit above the Spark job timeout, and how to size the gap between them.

Read Article β†’
Article

From Barracks to Cloud

How the principles of discipline and method from the Military Police apply to Cloud and DevOps management.

Read full article β†’
Article

Hosting a Static Website on AWS for Under $2/Month

S3, CloudFront and HTTPS, step by step: the setup behind this site.

Read Article β†’
Article

Anatomy of a CI/CD Pipeline with AWS

Orchestrating GitHub Actions, S3, and CloudFront for resilient global delivery. (In Portuguese)

Read Article β†’

Let's have a technical chat