Which transformation would you use to remove duplicates in an RDD?

The distinct transformation is specifically designed to remove duplicates from an RDD (Resilient Distributed Dataset). When you apply this transformation, it ensures that each element in the resulting RDD occurs only once, effectively filtering out all duplicate entries. This operation is particularly useful when you want to obtain a unique set of values from a dataset, enhancing data integrity and simplifying subsequent analyses.

In contrast, the other transformations have different purposes. The map transformation is used to apply a function to each element of the RDD, thereby transforming the data but not addressing duplication. groupByKey organizes data by keys but does not inherently remove duplicates, as it can create a collection of values for each key, which may still include duplicates. The combine transformation is focused on reducing data by combining elements through a function, but it does not explicitly target the removal of duplicates from the dataset.

Thus, distinct is the optimal choice for removing duplicates in an RDD.