Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash bastion host Batch CAP Coders At Work Data data lakes Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql Power Bi Powershell Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Traversy Media Ubuntu Understanding Distributed Systems Virtual Box whale-alert Yaml

Archives

Month: April 2025

What is HDFS and Why Was It Revolutionary for Big Data?

In the early 2000s, the world was generating data at a scale never seen before—web logs, social media, sensors, and more. Traditional storage systems simply couldn’t keep up with the volume, velocity, and variety of this data. Enter HDFS: the Hadoop Distributed File System, a cornerstone of the Apache Hadoop ecosystem. This blog post explains
Read More

What Is Serialization?

In the world of data engineering and software systems, serialization is a fundamental concept that allows you to efficiently store, transmit, and reconstruct data structures. If you’ve worked with formats like Parquet, Avro, JSON, or CSV, you’ve already interacted with serialization—whether you knew it or not. In this post, we’ll explore: What Is Serialization? Serialization
Read More

Is S3 the New HDFS? Comparisons and Use Cases in Big Data

Over the past decade, the way organizations store and manage big data has shifted dramatically. Once dominated by the Hadoop Distributed File System (HDFS), the field is now led by Amazon S3 and similar cloud object storage systems. This raises a compelling question in today’s data engineering world: Is Amazon S3 the new HDFS? Let’s
Read More

From HDFS to S3: The Evolution of Data Lakes in the Cloud

For years, HDFS (Hadoop Distributed File System) was the default choice for building data lakes in on-premises and Hadoop-based environments. But as cloud computing gained momentum, a new player took the lead: Amazon S3. Today, S3 is widely recognized as the de facto data lake storage layer in the AWS ecosystem. How did this shift
Read More

The History and Evolution of Amazon S3: Was It Ever Based on HDFS?

When discussing cloud storage today, Amazon S3 is almost synonymous with scalable, reliable object storage. However, a common question among those familiar with big data technologies like Hadoop is:Was Amazon S3 ever based on HDFS (Hadoop Distributed File System)? The short answer is: No. Amazon S3: Launched Before HDFS Amazon S3 was officially launched on
Read More

MapReduce: A Framework for Processing Unstructured Data

MapReduce is both a programming model and a framework designed to process massive volumes of data across distributed systems. It gained popularity primarily due to its efficiency in handling unstructured or semi-structured data, especially text. Key Concepts of MapReduce Strength in Text Processing MapReduce excels with text data for several reasons: Beyond Text: Processing Other
Read More

Understanding .master() in Apache Spark

In Apache Spark, the .master() method is used to specify how your application will run, either on your local machine or on a cluster. Choosing the correct option is essential depending on your environment. This post will explain the different .master() options in Spark and when to use them. Local Mode The local mode runs
Read More

How Joins Work in PostgreSQL

Joins are one of the most powerful features in SQL, allowing you to combine data from multiple tables in a single query. PostgreSQL, as a relational database system, provides robust support for different types of joins. Understanding how joins work under the hood helps you write more efficient queries and troubleshoot performance issues. What Is
Read More

How to Improve Query Performance in PostgreSQL

PostgreSQL is a powerful relational database, but even the most robust systems can suffer from slow queries without proper tuning. Optimizing query performance is crucial to ensure scalability, responsiveness, and efficient resource usage. In this post, we’ll explore actionable techniques to speed up your PostgreSQL queries. 1. Use Indexes Effectively Create Indexes on Filter and
Read More

Optimizing Joins in PostgreSQL: Practical Cases

Joins are essential for querying relational databases, but they can significantly impact performance if not optimized correctly. PostgreSQL provides several ways to improve join efficiency, from indexing strategies to query restructuring. In this post, we’ll explore different types of joins, performance considerations, and practical ways to optimize them. Types of Joins in PostgreSQL PostgreSQL supports
Read More