in Spark Sql by
Q:
Load a csv/textFile and remove header and footer in Spark?

1 Answer

0 votes
by

valheaderFooterRemovedDF = loadDF.take(loadDF.count.toInt).drop(1).dropRight(1)

valschemaDefine = “id|name|date|type|status”.split(‘|’).map(col =>StructField(col.toString,StringType,true))

val schema = StructType(schemaDefine)

valfinalDF = spark.createDataFrame(spark.sparkContext.parallelize(headerFooterRemovedDF),schema)

Click here to read more about Loan/Mortgage
Click here to read more about Insurance

Related questions

+1 vote
asked Mar 9, 2020 in Spark Sql by SakshiSharma
+1 vote
asked Mar 9, 2020 in Spark Sql by SakshiSharma
0 votes
asked Jun 19, 2020 in Bootstrap by Robindeniel
0 votes
asked Mar 14, 2020 in Spark Sql by rajeshsharma
0 votes
asked Mar 14, 2020 in Spark Sql by rajeshsharma
...