Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.
This PyPi package contains the Python APIs for using Delta Lake with Apache Spark.
- Install using
pip install delta-spark - To use the Delta Lake with Apache Spark, you have to set additional configurations when creating the SparkSession. See the online project web page for details.
This README file only contains basic information related to pip installed Delta Lake. You can find the full documentation on the project web page