Geek Logbook

Tech sea log book

Recent Posts

Categories

#50projectsIn50days Airflow Amazon Web Services Apache Iceberg AWS Azure bash Batch CAP Coders At Work Data data lakes Data Pipelines Data Warehouse Docker Edsger W. Dijkstra Excel Git Hadoop HDFS Hellerstein Hive Javascript Networking NoSql PostHog Postman Power Bi Powershell Pyspark Python Redash S3 Social Skills Software Developer Spark Spark: The Definitive Guide SQL SSH terraform Testing The Elements of Programming Style Understanding Distributed Systems whale-alert Yaml

Archives

Using Presing in AWS

Presing is a command you can use in the AWS CLI that allows anyone to have the pre-signed URL to make and HTTP get request to retrieve the data that is inside the bucket pre-signed. In the CLI you will execute: aws s3 presign s3://DOC-EXAMPLE-BUCKET/test2.txt And the output you’re going to receive is the following:
Read More

Load CSV file from S3 to NEO4J

If you try to load data from S3 to NEO4J you are going to need to presing the file. So you need to expose the data to somebody that have the file. So, first you need to presing the file: aws s3 presign s3://rohank/actors.csv And then you can load the data on your NEO4J Instance:
Read More

UNIX: A History and a Memoir

In the era of bright consultancy, where all things are opinionated, it’s difficult to find some refreshing ideas. For real it exists, but it is difficult to find. We are talking also about some people, who have the expertise of a field in Computer Science take time to say something about the route they have,
Read More

Enviroments in Virtual Env

The importance of using environments As was said in Setting environments in Python it’s important to use environments for your deployment, even if these are side projects or wild repositories. But at first, I used Conda to create the environment. However, a while ago, I made the switch to Virtual Env As a point of
Read More

Event Driven Architectures

In the Gartner Submit of 2006 Mani Chandy talked about the existence of a misconception of Event Driven Architecture (EDA). So, he proposed to talk about the understanding of EDA and its Return of Investment. So he can evaluate the true costs and benefits. In this talk an event was defined as:  “An event is
Read More

Matei Zaharia – Spark: The Definitive Guide. Common Operations

Define Schemas manually When using Spark for production Extract, Transform, and Load (ETL), it is often a good idea to define your schemas manually, especially when working with untyped data sources like CSV and JSON, because schema inference can vary depending on the type of data that you read in. (Chambers, 2017, 66) SQL Expressions
Read More

Kleppmann – Designing Data Intensive Applications

A data-intensive application is typically built from standard building blocks that provide commonly needed functionality. For example, many applications need to: • Store data so that they, or another application, can find it again later (databases) • Remember the result of an expensive operation, to speed up reads (caches) • Allow users to search data
Read More