in Spark Sql by
Q:
Load a csv/textFile and remove header and footer in Spark?

► Click here to show 1 Answer

0 votes
by

valheaderFooterRemovedDF = loadDF.take(loadDF.count.toInt).drop(1).dropRight(1)

valschemaDefine = “id|name|date|type|status”.split(‘|’).map(col =>StructField(col.toString,StringType,true))

val schema = StructType(schemaDefine)

valfinalDF = spark.createDataFrame(spark.sparkContext.parallelize(headerFooterRemovedDF),schema)

Loan/Mortgage
Learn More with Madanswer

Related questions

+1 vote
asked Mar 9, 2020 in Spark Sql by SakshiSharma
...