pandera is a framework for validating dataframe-like objects against explicit schemas. It helps data scientists, analysts and engineers detect problems in datasets before those problems affect downstream analysis, machine learning or production data pipelines.
A dataframe may have the expected columns but still contain invalid values, unexpected types or inconsistent relationships between fields. pandera addresses these issues by allowing developers to describe the structure and properties that acceptable data must satisfy. Validation then checks actual datasets against those requirements.
The framework is designed for integration with existing data processing workflows rather than requiring a separate validation service. It supports several dataframe libraries, although the precise validation capabilities vary between backends. Schemas can be expressed declaratively, making data requirements easier to review, reuse and test.
This is free and open source software.
Key Features
- Validate dataframes using DataFrameSchema and class-based DataFrameModel definitions.
- Define required columns, data types, nullable values and uniqueness constraints.
- Apply built-in checks for ranges, membership, string patterns and other conditions.
- Create custom validation checks for dataset-specific requirements.
- Validate pandas, Polars, PySpark, Ibis and PyArrow data structures through supported backends.
- Use statistical hypothesis checks where supported by the selected backend.
- Run lazy validation to collect multiple failures in a single report.
- Identify invalid rows and provide details of failed constraints.
- Validate function inputs and outputs using decorators.
- Define typed dataframe models for integration with application code.
- Generate example data from schemas for property-based testing where supported.
- Configure validation behaviour and reuse schemas across processing stages.
Website: github.com/unionai-oss/pandera
Support:
Developer: Niels Bantilan and contributors
License: MIT License
pandera is written in Python. Learn Python with our recommended free books and free tutorials.
Related Software
| Python Data Validation | |
|---|---|
| Pydantic | Data validation using Python type hints |
| pandera | Framework for precision data testing |
| jsonschema | Implementation of JSON Schema for Python |
| Cerberus | Lightweight and extensible data validation library |
| schema | Library for validating Python data structures |
| GX | Validating, documenting, and profiling data |
| marshmallow | ORM/ODM/framework-agnostic library |
| Voluptuous | Python data validation library |
| Schematics | Combine types into structures, validate , and transform the shapes of data |
| Colander | Serialization / deserialization / validation library |
| Valideer | Lightweight data validation and adaptation Python library |
| OpenRefine | Desktop program for data cleanup and transformation |
| Soda Core | Data quality and data contract verification engine |
| OpenMetadata | Unified metadata platform |
| Elementary OSS | dbt-native data observability command-line tool |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |