Frameworks Archives - Geek Logbook

By - Jino Michel Aque
Posted on 2023-02-162023-02-16
Posted in Frameworks

Create Unique Random Id using faker

You have to use: A full example where you can see the creation of a persona is the following:

By - Jino Michel Aque
Posted on 2023-02-032023-02-03
Posted in Frameworks

Install Astronomer on Windows OS

Install Astronomer is quite simple In windows you have to follow the following steps: 1 – Install docker Go to the docker’s webpage, download the executable and install it. Installation succeeded 2 – Install Linux on Windows with WSL As you can see on the Microsoft documentation you have to run on PowerShell the following

By - Jino Michel Aque
Posted on 2023-01-132023-01-13
Posted in Frameworks

SODA: A way to make data quality check

With the open-source library SODA, you can make different operations you need to know when you are going to make some transformation to a data set. Source:

By - Jino Michel Aque
Posted on 2022-12-292022-12-29
Posted in Frameworks

Set secrets in Databricks

If you add as plain text the user and password of your connections you are making a mistake that it’s easy to solve. In order to solve Now you can use the CLI and it should be connected to your instances. So, now we can run the commands to set the keys. For example, if

By - Jino Michel Aque
Posted on 2022-10-232022-12-09
Posted in Frameworks

Matei Zaharia – Spark: The Definitive Guide – Life Cycle of a Spark Application

The Life Cycle of a Spark Application (Inside Spark) The SparkSession The first step of any Spark Application is creating a SparkSession. In many interactive modes, this is done for you, but in an application, you must do it manually. (Chambers, 2017, 259) Some of your legacy code might use the new SparkContext pattern. This

By - Jino Michel Aque
Posted on 2022-10-232022-12-09
Posted in Frameworks

Matei Zaharia – Spark: The Definitive Guide – Architecture of a Spark Application

The Architecture of a Spark Application The Spark driver The driver is the process “in the driver seat” of your Spark Application. It is the controller of the execution of a Spark Application and maintains all of the state of the Spark cluster (the state and tasks of the executors). It must interface with the

By - Jino Michel Aque
Posted on 2022-10-222022-12-09
Posted in Frameworks

Matei Zaharia – Spark: The Definitive Guide. Common Operations

Define Schemas manually When using Spark for production Extract, Transform, and Load (ETL), it is often a good idea to define your schemas manually, especially when working with untyped data sources like CSV and JSON, because schema inference can vary depending on the type of data that you read in. (Chambers, 2017, 66) SQL Expressions