The Lock Held by a Process That No Longer Exists
An ingestion pipeline stopped for hours by an orphaned Hive Metastore lock, and why nothing ever expired it.
Read Article βI help run a production data platform on Kubernetes: Airflow, Spark, Apache Iceberg and AWS. I find out why things break, prove it with evidence and write it down.
Data Platform Engineer at COCUS Portugal since March 2026, on a small team that owns the infrastructure of a production analytics platform for a company in the energy sector. 24/7 on-call rotation since September 2026.
Before IT, 30 years as an officer in the SΓ£o Paulo Military Police: dental surgeon, then legal advisory, intelligence, procurement and personnel. Retired as a Major. Degrees in Dentistry (USP) and Law (UNIESP).
Verify before concluding: my reports separate what I observed directly from what I infer. Command line first, so every investigation can be repeated step by step. In production, one change at a time, and only after the evidence is in. Everything written down: runbooks, protocols and incident reports.
Daily operation in production: workloads, scheduling, autoscaling, persistent volumes and troubleshooting with kubectl. Preparing for the CKA.
Airflow 2.8 in production: DAG failure triage, scheduler health and investigations in the metadata database.
Spark on Kubernetes and Amazon EMR: driver and executor failures, right-sizing and cost analysis.
Iceberg tables on S3 with Hive Metastore: metadata growth, snapshot retention and lock issues.
EKS, EMR, S3, IAM and CloudWatch Logs in production; S3, CloudFront and Terraform in personal projects.
Platform dashboards and alerts in production, plus a self-built Airflow monitoring stack in the lab.
Production cases are generalized and figures rounded to respect client confidentiality.
A pipeline kept reporting success while tasks that should take minutes ran for over four hours. Two years of tiny commits without snapshot expiration had grown the table's Iceberg metadata to more than 5,000 times the size of its data.
Tools: AWS CLI, kubectl, Python (fastavro, psycopg2)
Result: Root cause confirmed, investigation report and remediation plan; a follow-up scan found dozens of tables with the same pattern
Spark drivers failed seconds after the cluster scaled up. New nodes reported Ready in about 13 seconds, but pod networking took 2 to 3.5 minutes to converge. Confirmed with live measurements and EKS API server audit logs.
Stack: Amazon EKS, Calico, Cluster Autoscaler, CloudWatch Logs
Result: Four mitigation options documented; fix in progress
A written morning protocol of 21 checks, backed by a dependency-free Python CLI that finds waste and stuck work across environments: crash-looping and OOM-killed pods, zombie Spark pods, orphaned EMR clusters, stuck DAG runs and orphaned Hive Metastore locks.
Tools: Python standard library, kubectl, AWS CLI
Highlight: Found idle EMR clusters with 18 instances and no auto-termination
A monitoring stack for Apache Airflow. It uses Prometheus to scrape metrics via StatsD and Grafana for real-time visualization of heartbeat, throughput, and task latency.
Stack: Prometheus, Grafana, StatsD, PromQL, Docker
Highlight: PromQL rate() functions for instant performance analysis
End-to-end data pipeline using Apache Airflow. It monitors EUR/BRL rates, processes metrics with Pandas, and automatically publishes a responsive dashboard via GitLab CI/CD & Pages.
Stack: Airflow, Docker, Python (Pandas), GitLab CI/CD
Highlight: Fully decoupled ETL architecture & automated SSH deployment
Kubernetes: CKA exam preparation.
Apache Iceberg: internals and concurrency model.
dbt: fundamentals, toward analytics engineering.
Labs: Terraform, GitHub Actions and Linux.
Spark performance tuning and AWS Certified Data Engineer β Associate.
An ingestion pipeline stopped for hours by an orphaned Hive Metastore lock, and why nothing ever expired it.
Read Article βWhy the Airflow task timeout must sit above the Spark job timeout, and how to size the gap between them.
Read Article βHow the principles of discipline and method from the Military Police apply to Cloud and DevOps management.
Read full article βS3, CloudFront and HTTPS, step by step: the setup behind this site.
Read Article βOrchestrating GitHub Actions, S3, and CloudFront for resilient global delivery. (In Portuguese)
Read Article β