👀 Views: 72 💬 Answers: 1 📅 Created: 2025-06-14

apache-spark delta-lake dataframe Python

I'm working through a tutorial and I'm upgrading from an older version and I'm wondering if anyone has experience with I'm currently working on a Spark job using Spark 3.4.1 with Delta Lake to write data into a Delta table in append mode... I've set up my DataFrame and I'm trying to append new records, but I'm working with the following behavior: ``` org.apache.spark.sql.AnalysisException: Table already exists: my_database.my_delta_table ``` This seems to happen intermittently and is particularly troublesome when running this job multiple times in a streaming context. My DataFrame is defined as follows: ```python from pyspark.sql import SparkSession from delta.tables import DeltaTable spark = SparkSession.builder .appName("DeltaLakeExample") .config("spark.sql.extensions", "delta.sql.DeltaSparkSessionExtensions") .getOrCreate() data = [("Alice", 1), ("Bob", 2)] columns = ["name", "value"] new_df = spark.createDataFrame(data, columns) ``` When I try to write to the Delta table with the following command: ```python new_df.write.format("delta") \ .mode("append") \ .save("/path/to/my_delta_table") ``` I expect it to append the data, but it seems to throw the table already exists behavior, suggesting that Spark might be trying to recreate the table instead of appending. I've ensured that the Delta Lake library is properly included in my Spark session. To troubleshoot, I added the following configurations to my Spark session, hoping to resolve potential issues with concurrent writes: ```python spark.conf.set("spark.sql.extensions", "delta.sql.DeltaSparkSessionExtensions") spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false") ``` Despite these configurations, the scenario continues. I've also checked the Delta table's metadata in the specified path, and it appears to be intact. Has anyone experienced similar issues with appending to Delta tables and found a resolution? Any advice on how to avoid this 'Table already exists' behavior would be greatly appreciated. This is my first time working with Python 3.11. Thanks for your help in advance! My team is using Python for this CLI tool. What's the best practice here? My team is using Python for this CLI tool. I appreciate any insights!