Vaex is a toolkit for exploring and analysing large tabular datasets without requiring every row to be loaded into memory. It uses a column-oriented DataFrame model and memory-mapped data access to support analytical operations on datasets that may be substantially larger than available RAM.
The software was originally motivated by the need to explore large astronomical catalogues, but its analytical capabilities are useful in other fields that work with substantial quantities of structured data.
Vaex evaluates expressions lazily, allowing calculated columns to be defined without immediately materialising their values. Combined with efficient filtering and statistical aggregation, this makes it useful for interactive exploration and large-scale numerical analysis.
This is free and open source software.
Key Features
- Out-of-core DataFrames for working with datasets larger than available system memory.
- Memory-mapped data access to reduce unnecessary reading and copying of data.
- Lazy expressions and virtual columns evaluated when their values are required.
- Fast statistical operations including counts, sums, means, variance and standard deviation.
- Multidimensional binning and aggregation for analysing distributions across numerical variables.
- Filtering and Boolean selections for examining subsets of large datasets.
- Histograms, heatmaps and other statistical visualisations.
- Group-by operations for aggregating data by categorical or numerical variables.
- Efficient transformations using mathematical and string expressions.
- Support for HDF5, Arrow and other supported columnar data formats.
- Integration with Jupyter environments for exploratory analysis.
- Interoperability with pandas, NumPy and Arrow-based analytical workflows.
- Support for processing data through local and supported remote data sources.
Website: vaex.io
Support:
Developer: Maarten A. Breddels and contributors
License: MIT License

Vaex is written in Python. Learn Python with our recommended free books and free tutorials.
Related Software
| Python Data Analysis | |
|---|---|
| pandas | High-level building block for doing practical, real world data analysis |
| NumPy | Core package for scientific computing with Python |
| SciPy | Ecosystem for mathematics, science, and engineering |
| Polars | DataFrame interface on top of an OLAP Query Engine |
| statsmodels | Statistical modeling and econometrics in Python |
| Dask | Advanced parallelism for analytics |
| Orange | Component-based framework for machine learning and data mining |
| Modin | Drop-in replacement for pandas |
| Vaex | Fast visualization of big data |
| AWS DW | Extends the power of pandas library |
| yt | Multi-code Toolkit for Analyzing and Visualizing Volumetric Data |
| HoloViews | Make Data Analysis and Visualization Seamless |
| datatable | Manipulate 2-dimensional tabular data structures |
| xarray | Work with labelled multi-dimensional arrays and datasets |
| pyjanitor | Extend pandas with readable data-cleaning functions |
| Optimus | Agile Data Preparation Workflows |
Read our verdict in the software roundup.
| Python Visualization Packages | |
|---|---|
| matplotlib | Python 2D plotting library which produces publication quality figures |
| Diagrams | Draw the cloud system architecture in Python code |
| Bokeh | Elegant, concise construction of versatile graphics |
| Dash | Python framework for building analytical web applications |
| seaborn | Python visualization library based on matplotlib |
| Plotly | Interactive, browser-based graphing library for Python |
| VisPy | Visualize massive datasets in real time |
| Vega-Altair | Declarative Visualization in Python |
| plotnine | Grammar of graphics for Python |
| PyQtGraph | Python graphics and GUI library built on PyQt4 / PySide and numpy |
| bqplot | Interactive Plotting Framework for the Jupyter Notebook |
| Vaex | Fast visualization of big data |
| PyVista | 3D plotting and mesh analysis |
| folium | Visualize data in a Leaflet map |
| HoloViews | Make Data Analysis and Visualization Seamless |
| Datashader | Generates aggregate arrays and representations of them as images |
| yt | Multi-code Toolkit for Analyzing and Visualizing Volumetric Data |
| Glumpy | Intuitive interface between NumPy and modern OpenGL |
| GeoViews | Explore and visualize geographical, meteorological, and oceanographic datasets |
| Pygal | Dynamic SVG charting library |
| Glue | Multi-dimensional linked-data exploration |
Read our verdict in the software roundup.
LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |


Please read our Comment Policy before commenting.