Spark errors are long, layered, and often reported far from the cause. Debugging well means recognising the family of an error from a few keywords, knowing where the real message hides, and having a reproduction ritual. This week is a taxonomy plus drills.

Where the message is

Python shows Py4JJavaError (or SparkConnectGrpcException) wrapping a Java stack. Read the last Caused by:. For task failures the message says Task 3 in stage 7.0 failed 4 times, most recent failure: ... (executor 2): the cause is in that executor's stderr, not the driver's. A Python UDF failure appears as PythonException with your traceback embedded, after the Java noise.

The families

Keyword	Family	First move
AnalysisException: cannot resolve / ambiguous / not found	Plan-time: names, types, views	Print df.columns, alias the sides, check the session
ParseException	SQL syntax	The caret shows the position; check quotes and reserved words
OutOfMemoryError / Container killed / ExecutorLostFailure	Memory, executor side	Which stage? skew, spill, cache, pyspark memory
OutOfMemoryError on driver / Total size of serialized results ... bigger than spark.driver.maxResultSize	Memory, driver side	collect/toPandas/broadcast too big; huge plan
Task not serializable / PicklingError	A closure captured something unpicklable (the SparkSession, a DB connection, a lock)	Create the object inside the function or use mapPartitions
FileNotFoundException / FileAlreadyExists / PathNotFound	Storage: stale listing, overwrite of an input, concurrent writes	refreshTable; never overwrite the path you read from in the same job
SparkUpgradeException / DateTimeParse / ANSI cast	Semantics: legacy vs new parser, ANSI mode	Fix the format; set legacy policy only as a bridge
Job aborted due to stage failure ... FetchFailed	Shuffle files lost (executor died, disk full, node lost)	Look one stage earlier; check disk; enable shuffle service

The reproduction ritual

Cut the pipeline at the failing action; give each intermediate a name.
Replace the input with .limit(1000) or a filter on one key; does it still fail? (No → data-dependent: memory or a bad row.)
Bisect the transformations; explain() at the cut.
For bad rows: use PERMISSIVE modes, try_cast/try_to_timestamp (3.x/4.x), and quarantine rather than crash.
For non-determinism: look for rand(), monotonically_increasing_id(), first(), dropDuplicates, Python set ordering and time zones.

Correctness bugs that never throw

Silent null from cast, three-valued !=, duplicate join keys inflating counts, timestamp shifted by the session time zone, integer division, count vs count(col) (ignores nulls), dropDuplicates keeping a random row. Every pipeline needs row-count and null-count assertions (Week 17) precisely because these are silent.