PySpark Project: Using PySpark for some basic ETL process
While doing a Tableau course, I came across this ETL process where I had to combine multiple CSV files and pivot them. Unfortunately, Tableau doesn’t …
Category: PySpark
User-Defined Functions (UDF) in PySpark
The general idea of a UDF is to use a regular python function and translate that to a PySpark function which can be applied to …
Handling Null Values in PySpark: Using dropna() and fillna() methods
In this article, I will go over two different methods for handling null values in PySpark dropna() -> used for dropping the null values fillna() …
The Basics of Data Manipulation in PySpark
In this article, I will go over the following topics: Viewing Schema Selecting column/s Showing rows Dropping column/s Renaming column/s Importing Data The files used …
Grouping, Aggregating, and Ordering our Data Frame in PySpark
Grouping, Aggregating, and Ordering are the most commonly used functions in PySpark. With the help of this article, I will try to explain how we …