Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash bastion host Batch CAP Coders At Work Data data lakes Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql Power Bi Powershell Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Traversy Media Ubuntu Understanding Distributed Systems Virtual Box whale-alert Yaml

Archives

Category: Cloud

Controlling Branch Deployments and Redirects in Vercel: A Practical Guide

Continuous deployment platforms simplify the release process, but they can easily become noisy when every branch triggers a build. Teams working with multiple development environments often need finer control — building only when specific branches are updated and ignoring the rest. The Problem Imagine a development team maintaining three main branches: By default, Vercel automatically
Read More

Estimating the Cost of an AWS Glue Workflow

When working with AWS Glue, one of the most common questions data engineers ask is: How much will this job cost me? If you have a workflow that runs for 13 minutes, understanding the cost model of AWS Glue helps you avoid surprises on your AWS bill. How AWS Glue Pricing Works AWS Glue pricing
Read More

AWS EventBridge Rules vs EventBridge Scheduler: Which One Should You Use?

In the AWS ecosystem, there are two main ways to schedule and automate tasks: EventBridge Rules (scheduled rules) and the newer EventBridge Scheduler, which introduces Schedule Groups. While both can trigger actions at defined times, their design, scalability, and flexibility differ significantly. Choosing the right option depends on your workload requirements. 1. What Are EventBridge
Read More

Running Production Servers on AWS: EC2 vs RDS Cost Breakdown

When planning to run production workloads in the cloud, cost is one of the most important considerations. In this post, we will explore the monthly expenses of running two application servers and a database server on AWS, and compare two deployment approaches: EC2-only vs EC2 + RDS. Infrastructure Requirements Our baseline infrastructure looks like this:
Read More

Orchestrating Multiple AWS Glue Workflows: A Practical Guide

AWS Glue provides a robust environment for building and managing ETL pipelines, but many data engineers face the challenge of chaining or coordinating multiple workflows. This article explores practical approaches to relate two or more Glue workflows, covering both native features and complementary AWS services. Why You Might Need Multiple Workflows In many data engineering
Read More

Secure Ways to Share Private Data on AWS: Beyond Public Buckets

When building data platforms in the cloud, it is common to share data with partners, clients, or internal teams outside your own. AWS provides several mechanisms to grant secure, granular access — far beyond the simple (and risky) “make the bucket public” approach. In this post, we will explore the main strategies for sharing data
Read More

How to Set CloudWatch Log Retention Policies with Terraform

AWS CloudWatch is a powerful service for monitoring applications and infrastructure. However, by default, CloudWatch Logs are configured to never expire. This can lead to excessive storage costs and retention of data that you may not need. A better approach is to define a retention policy that aligns with your operational and compliance requirements. In
Read More

Orchestrating Multiple AWS Glue Workflows with Step Functions

In modern data architectures, it is common to manage multiple ETL pipelines that must run in sequence or in parallel. AWS Glue provides a robust framework for building workflows, but when we need to orchestrate two or more Glue Workflows together, AWS Step Functions becomes the natural choice. In this post, we will explain how
Read More

How to Disable an AWS Glue Trigger from the CLI

When working with AWS Glue, triggers are an important mechanism to orchestrate jobs or workflows. Sometimes, however, you may need to temporarily disable a trigger without deleting it—for example, to pause scheduled ingestions during maintenance or testing. This article explains how to disable a trigger using the AWS CLI. Understanding AWS Glue Triggers AWS Glue
Read More