Spark and Sedona processing of crime, census, income and geospatial data, with DataFrame, UDF and RDD implementations compared.
The problem
Large, heterogeneous datasets need spatial joins, cleaning and choices about distributed execution that keep results correct and the performance trade-offs explainable.
What was built
A Dockerised PySpark environment for Los Angeles crime, census, police-station and income datasets, using a Hadoop-oriented configuration and Apache Sedona.
Technical approach
- Ingest tabular and GeoJSON sources
- Clean and align heterogeneous records
- Run spatial operations with Apache Sedona
- Compare DataFrame, Python UDF and RDD strategies
- Render analytical output with pandas and matplotlib where appropriate
Engineering decisions
- Prefer native DataFrame expressions when Catalyst can optimise the plan.
- Use the UDF and RDD variants to expose the cost of Python/JVM serialisation and other lower-level execution trade-offs.
- Keep data access compatible with local and HDFS-oriented environments.
- Package the infrastructure and queries in a reproducible container setup.
How it was checked
- The repository includes q1.py to q5.py and reusable Spark utilities.
- One analysis is implemented with DataFrame expressions, a Python UDF and the RDD API.
- Execution time is measured in that comparative implementation.
- The dataset references cover LA crime data for 2010–2019 and 2020–2025, plus census and supporting data.
What it shows
Engineering of distributed data pipelines, geospatial processing and performance-aware implementation, beyond small in-memory analytics.
About the source
Academic engineering project. Claims about datasets and architecture are limited to what the repository shows.