Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash Batch CAP Coders At Work Data data lakes Data Pipelines Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql PostHog Postman Power Bi Powershell Pyspark Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Understanding Distributed Systems whale-alert Yaml

Archives

Tag: Data Pipelines

Daily Failure Reporting in DynamoDB Using Lambda, EventBridge Scheduler, and SES

Operational monitoring requires structured visibility into failures. If your processes write execution logs to DynamoDB and mark failed executions with status = FAILED, you can implement a deterministic daily reporting pipeline using AWS Lambda, EventBridge Scheduler, and Amazon SES. This article describes a single, production-grade implementation. Objective Architecture This solution is fully serverless and horizontally
Read More

Understanding ip-api Batch Limits and Effective Throughput

When integrating IP geolocation into a data pipeline, understanding rate limits and batching constraints is essential. This post analyzes the practical limits of the ip-api free tier and how to compute effective throughput. 1. Free Tier Constraints The ip-api free plan imposes the following restrictions: These limits apply globally per source IP address. 2. Maximum
Read More

AWS Glue Workflow vs Apache Airflow: A Professional Comparison

While both serve the common purpose of managing and automating data workflows, they differ significantly in architecture, flexibility, integration capabilities, and operational control. This article offers a comprehensive and professional comparison of AWS Glue Workflow and Apache Airflow to help data engineers, architects, and decision-makers choose the most suitable tool for their use case. 1.
Read More