Question

How can you trigger automatic clean-ups in Spark to handle accumulated metadata?

Answer 1

You can trigger the clean-ups by setting the parameter ' Spark.cleaner.ttl' or dividing the long-running jobs into different batches and writing the intermediary results to the disk.