+1 vote
in Spark Sql by
Load a csv/textFile and remove header and footer in Spark?

1 Answer

0 votes
by

valheaderFooterRemovedDF = loadDF.take(loadDF.count.toInt).drop(1).dropRight(1)

valschemaDefine = “id|name|date|type|status”.split(‘|’).map(col =>StructField(col.toString,StringType,true))

val schema = StructType(schemaDefine)

valfinalDF = spark.createDataFrame(spark.sparkContext.parallelize(headerFooterRemovedDF),schema)

Related questions

+1 vote
asked Mar 9, 2020 in Spark Sql by SakshiSharma
0 votes
asked Feb 12, 2023 in Spark Sql by sharadyadav1986
...