> For the complete documentation index, see [llms.txt](https://nag-9-s.gitbook.io/spark-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nag-9-s.gitbook.io/spark-notes/spark-architecture/stages.md).

# Stages

Recall that Spark lazily evaluates transformations; transformations are not executed until an action is called. As mentioned previously, a job is defined by calling an action. The action may include one or several transformations, and wide transformations define the breakdown of jobs into*stages*.

Each stage corresponds to a shuffle dependency created by a wide transformation in the Spark program. At a high level, one stage can be thought of as the set of computations (tasks) that can each be computed on one executor without communication with other executors or with the driver. In other words, a new stage begins whenever network communication between workers is required; for instance, in a shuffle.

These dependencies that create stage boundaries are called`ShuffleDependencies`. As we discussed in [“Wide Versus Narrow Dependencies”](file:///G:/ScrapBook%20Data/Hadoop/data/20170703002220/index.html#narrow_wide_intro), shuffles are caused by those wide transformations, such as`sort`or`groupByKey`, which require the data to be redistributed across the partitions.Several transformations with narrow dependencies can be grouped into one stage.

As we saw in the word count example where we filtered stop words ([Example 2-2](file:///G:/ScrapBook%20Data/Hadoop/data/20170703002220/index.html#word_count_example)), Spark can combine the`flatMap`,`map`, and

`filter`steps into one stage since none of those transformations requires a shuffle. Thus, each executor can apply the`flatMap`,

`map`, and`filter`steps consecutively in one pass of the data.