Question

How can you set up automatic cleanups in Spark to deal with metadata that has built up?

Answer 1

You can start the cleanups by splitting long-running jobs into batches and writing the intermediate results to disc.