Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash Batch CAP Coders At Work Data data lakes Data Pipelines Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql PostHog Postman Power Bi Powershell Pyspark Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Understanding Distributed Systems whale-alert Yaml

Archives

Renaming Modules in Python for Clarity and Accuracy

Renaming modules in Python is an essential practice to improve code clarity and maintainability, especially as projects grow in complexity. Using intuitive and descriptive names helps in quickly understanding the purpose of each module, reducing confusion and potential errors during development. In this post, we’ll explore the process of renaming a module in Python and
Read More

Counting Covered Points on a Number Line

Introduction Algorithmic challenges often involve intervals and can initially seem complex. One such problem is determining how many unique points are covered by a set of intervals on a number line. In this post, we’ll examine a solution to this problem and explore potential improvements. Problem Statement Given a number line with several intervals representing
Read More

Handling shutil.SameFileError When Copying Files in Python

When using Python’s shutil.copy() or shutil.copy2() to copy files, you might run into a shutil.SameFileError if you mistakenly attempt to copy a file onto itself. This error occurs when the source and destination paths are the same, disrupting your script’s execution. In this post, we’ll explore how to prevent this error and ensure files are
Read More

Preserving Directory Structure While Copying Files in Python – version 2

When copying files from one directory to another in Python, it’s important to maintain the original directory structure, especially when dealing with nested directories. In this post, we’ll explore how to use Python’s shutil and os libraries to copy files while preserving the directory structure. Problem Imagine you have a source directory with nested folders
Read More

Avoiding Duplicate File Copies Based on Content in Python on AWS

When working with large file systems, copying files can often lead to unintentional duplication, especially if files with the same content are repeatedly copied into different directories. While filenames can vary, the underlying content might remain the same, leading to redundant data and wasted storage space. In this post, we’ll explore how to avoid copying
Read More

Handling NoneType Errors When Extending Lists in Python

When working with Python, especially with functions that return lists or other iterable objects, you might encounter a TypeError that says something like: This error occurs when you try to iterate over or extend a list using a value that turns out to be None. In Python, NoneType represents a null value, and it is
Read More

Tracking File Changes in S3 Using ETags

When working with AWS S3, tracking changes to files can be essential, especially when versioning is not enabled on the bucket. The ETag associated with each file in S3 can provide a simple way to detect changes. In this post, we’ll explore how to use ETags to monitor file modifications in an S3 bucket. What
Read More

Implementing Retries in Python

In many real-world applications, simply handling an error isn’t always enough. Sometimes, the failure is temporary, and retrying the operation can help resolve the issue. In this post, we’ll explore how to implement retries in Python, improving the robustness of our programs. Why Implement Retries? Let’s imagine you’re making a request to an external API
Read More