An incremental data engineering pipeline built with Apache Spark, PySpark, and Delta Lake, featuring data validation, quarantine, idempotent batch processing, and recovery from partial failures. This ...