Databricks is introducing a new data type for semi-structured data (JSON-only to start). The data type will be open source, provide more flexibility, and improve performance for working with complex JSON.

databricks

Introducing the open variant data type for semi-structured data in Delta Lake and Apache Spark, providing performance improvements compared to storing data as JSON strings. It maintains flexibility for supporting highly nested and evolving schema.

Introducing the Open Variant Data Type in Delta Lake and Apache Spark