Partitions, repartition and coalesce

A DataFrame is split into partitions, and each partition is processed by a task on a core. With 8 cores and 200 partitions, the stage runs in waves of up to 8 tasks at a time.

Where the number of partitions comes from

• When reading files: by size, with spark.sql.files.maxPartitionBytes (default 128 MB) per partition, roughly.
• After a shuffle (groupBy, join, distinct, orderBy): spark.sql.shuffle.partitions (default 200). With adaptive execution (AQE, on by default since Spark 3.2), small partitions are merged after the shuffle.

df = spark.read.parquet("/mnt/raw/orders")
print(df.rdd.getNumPartitions())
print(spark.conf.get("spark.sql.shuffle.partitions"))

16
200

(The first number depends on the size of the files and the number of cores: the example assumes about 2 GB on a small cluster, 2 GB ÷ 128 MB. The second is the default.)

repartition: full shuffle, balanced partitions

repartition(n) redistributes all rows over the network into n partitions of the same size (or by column, repartition(n, "order_date"), bringing together the rows with the same key). It is used to increase parallelism or to prepare a partitioned write.

coalesce: no shuffle, just merges

coalesce(n) only decreases: it merges neighboring partitions on the same executor, without moving data over the network. It is cheap, but it inherits the imbalance: if one partition was huge, it stays huge.

large = df.repartition(64)       # shuffle: 64 similar partitions
print(large.rdd.getNumPartitions())
few = large.coalesce(8)          # no shuffle: 8 partitions
print(few.rdd.getNumPartitions())
print(few.coalesce(100).rdd.getNumPartitions())  # coalesce does not increase

64
8
8

Skew: one partition that makes everyone wait

The stage only finishes when the last task finishes. If one key concentrates half of the rows, its task becomes the bottleneck, and the rest of the cluster waits idle. Signs: in the Spark UI, one task with a duration far above the median.

• AQE handles skew in joins (spark.sql.adaptive.skewJoin.enabled), splitting the large partition.
• Without AQE, the classic way out is *salting*: adding a random suffix to the hot key to spread it out.
• coalesce(1) before writing a single file puts all the previous work in one task, if Spark manages to push it into the same stage. For a single file, prefer repartition(1).