in Spark Sql by
Q:
Take first 10 record and last 10 record of file and combine both using DataFrame in Spark?

1 Answer

0 votes
by

val loadDF = spark.read.format(“csv”).option(“path”,”file:///home/maria_dev/Files/assignment_table.csv”).load()

val combineDF = loadDF.take(10) ++ loadDF.take(loadDF.count.toInt).takeRight(10)

val schemaDesign = loadDF.first.toSeq.map(c => c.toString.trim).map(col => StructField(col,StringType,true))

val schema = StructType(schemaDesign)

val createDF = spark.createDataFrame(spark.sparkContext.parallelize(combineDF),schema)

createDF.show()

Click here to read more about Loan/Mortgage
Click here to read more about Insurance

Related questions

+1 vote
asked Mar 9, 2020 in Spark Sql by SakshiSharma
0 votes
asked Mar 14, 2020 in Spark Sql by rajeshsharma
0 votes
asked Nov 29, 2020 in Tableau by SakshiSharma
0 votes
asked Feb 11 in Python by SakshiSharma
0 votes
asked Mar 7, 2020 in Spark Sql by rahuljain1
...