Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash Batch CAP Coders At Work Data data lakes Data Pipelines Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql PostHog Postman Power Bi Powershell Pyspark Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Understanding Distributed Systems whale-alert Yaml

Archives

Understanding Pagination vs. Batch Processing in Data Handling

When working with large datasets, developers often face the challenge of efficiently extracting, processing, and managing data. Two commonly used techniques for handling such data efficiently are pagination and batch processing. While both methods aim to optimize memory usage and performance, they serve different purposes and are implemented differently. What is Pagination? Pagination is a
Read More

Tracking Daily File Size Changes in SQL

When working with databases that store file metadata, it’s often useful to track how file sizes change over time. If you have a table with the following structure: You may want to analyze the day-to-day changes in file size. This can help in monitoring storage usage, detecting anomalies, or understanding file growth trends. SQL Query
Read More

Resolving ‘index.lock’ Issue in Git

When working with Git, you may encounter an error preventing you from switching branches or performing other operations. A common issue is the following: This typically happens when another Git process is running or if a previous operation was interrupted, leaving a stale index.lock file. How to Fix the ‘index.lock’ Error 1. Check for Running
Read More

Merging Data in PostgreSQL vs. MySQL: How to Handle Upserts

When working with databases, you often need to update existing records or insert new ones based on whether a match is found. In PostgreSQL, this is efficiently handled using the MERGE statement. However, MySQL does not support MERGE, so alternative approaches must be used. This post explores how to achieve the same functionality in MySQL.
Read More

Understanding the Differences Between Parquet, Avro, JSON, and CSV

When working with data, choosing the right file format can significantly impact performance, storage efficiency, and ease of use. In this post, we will compare four widely used data formats: Parquet, Avro, JSON, and CSV. Each has its strengths and weaknesses, making them suitable for different scenarios. 1. Parquet Overview: Parquet is a columnar storage
Read More

Understanding the CAP Theorem in NoSQL Databases

The CAP theorem (Consistency, Availability, and Partition Tolerance) plays a crucial role in designing and selecting NoSQL databases. This theorem states that in a distributed system, it is impossible to achieve all three properties simultaneously: How CAP Theorem Relates to NoSQL Databases NoSQL databases are designed for scalability and flexibility, often trading off one CAP
Read More

Optimizing Queries with Partitioning in Databricks

Partitioning is a crucial optimization technique in big data environments like Databricks. By partitioning datasets, we can significantly improve query performance and reduce computation time. This post will walk through an exercise on partitioning data in Databricks, using a real-world dataset. Exercise: Managing Partitions in Databricks Objective Step 1: Load Data into Databricks For this
Read More

Calculating Levenshtein Distance in Apache Spark Using a UDF

When working with text data in big data environments, measuring the similarity between strings can be essential. One of the most commonly used metrics for this is the Levenshtein distance, which calculates the number of insertions, deletions, and substitutions required to transform one string into another. In this post, we’ll demonstrate how to implement a
Read More

Creating a PySpark DataFrame for Sentiment Analysis

When working with sentiment analysis, having structured data in a PySpark DataFrame can be very useful for processing large datasets efficiently. In this post, we will create a PySpark DataFrame containing sample text opinions, which can then be analyzed using NLP techniques. Setting Up PySpark First, ensure you have PySpark installed. If not, install it
Read More