Add Support for delta lake v0.7 and spark 3.0 in spark pool
Delta lake v0.6.1 doesn't support much of the ACID functionality. It would be great to upgrade the delta lake version and Spark version to utilize functionality as supported by data bricks.
Miron Berlin (People Tech Group) commented
We need Spark 3.0+ to use to_csv Spark sql function to produce CSV formatted strings as query product for our usage.
Currently we are forced to use hand written parser that miraculously works, but having something more substantial is really nice
Euan Garden commented
0.6.1 supports full ACID but it has to be done from PySpark or Scala, whats missing is the Spark SQL support and thats one of the big investments for 0.7.
We are working on support for Delta 0.7 which as you say also requires support for Spark 3.0