You don’t need to me to tell you that there’s a plethora of different data quality frameworks and tools out there. Databricks Labs announced their venture into the space with dqx the other day. Dqx…

Data Engineer Things

Databricks Labs has announced dqx, a data quality tool designed for PySpark workloads. Dqx simplifies data quality checks for both normal and streaming DataFrames, providing features like check failure info, support for various Spark workloads, different reactions to failures, row and column level application, profiling, and a data quality dashboard. Quality rules can be configured using JSON or YAML, and the tool supports custom SQL checks. The interoperability with other Databricks features remains an area of interest.

Dqx: Databricks Data Quality