Recently I built a data pipeline in Pandas that would be run on a weekly basis.

Python data wrangling pipelines reproducible

PySpark is the Python API wrapper for Apache Spark, a big data processing framework.

Python PySpark data wrangling