Data Analysis

Vaex – out-of-core DataFrame analysis for large datasets

Vaex is a toolkit for exploring and analysing large tabular datasets without requiring every row to be loaded into memory. It uses a column-oriented DataFrame model and memory-mapped data access to support analytical operations on datasets that may be substantially larger than available RAM.

The software was originally motivated by the need to explore large astronomical catalogues, but its analytical capabilities are useful in other fields that work with substantial quantities of structured data.

Vaex evaluates expressions lazily, allowing calculated columns to be defined without immediately materialising their values. Combined with efficient filtering and statistical aggregation, this makes it useful for interactive exploration and large-scale numerical analysis.

This is free and open source software.

Key Features

  • Out-of-core DataFrames for working with datasets larger than available system memory.
  • Memory-mapped data access to reduce unnecessary reading and copying of data.
  • Lazy expressions and virtual columns evaluated when their values are required.
  • Fast statistical operations including counts, sums, means, variance and standard deviation.
  • Multidimensional binning and aggregation for analysing distributions across numerical variables.
  • Filtering and Boolean selections for examining subsets of large datasets.
  • Histograms, heatmaps and other statistical visualisations.
  • Group-by operations for aggregating data by categorical or numerical variables.
  • Efficient transformations using mathematical and string expressions.
  • Support for HDF5, Arrow and other supported columnar data formats.
  • Integration with Jupyter environments for exploratory analysis.
  • Interoperability with pandas, NumPy and Arrow-based analytical workflows.
  • Support for processing data through local and supported remote data sources.

Website: vaex.io
Support:
Developer: Maarten A. Breddels and contributors
License: MIT License

Vaex

Vaex is written in Python. Learn Python with our recommended free books and free tutorials.


Related Software

Python Data Analysis
pandasHigh-level building block for doing practical, real world data analysis
NumPyCore package for scientific computing with Python
SciPyEcosystem for mathematics, science, and engineering
PolarsDataFrame interface on top of an OLAP Query Engine
statsmodelsStatistical modeling and econometrics in Python
DaskAdvanced parallelism for analytics
OrangeComponent-based framework for machine learning and data mining
ModinDrop-in replacement for pandas
VaexFast visualization of big data
AWS DWExtends the power of pandas library
ytMulti-code Toolkit for Analyzing and Visualizing Volumetric Data
HoloViewsMake Data Analysis and Visualization Seamless
datatableManipulate 2-dimensional tabular data structures
xarrayWork with labelled multi-dimensional arrays and datasets
pyjanitorExtend pandas with readable data-cleaning functions
OptimusAgile Data Preparation Workflows

Read our verdict in the software roundup.

Python Visualization Packages
matplotlibPython 2D plotting library which produces publication quality figures
DiagramsDraw the cloud system architecture in Python code
BokehElegant, concise construction of versatile graphics
DashPython framework for building analytical web applications
seabornPython visualization library based on matplotlib
PlotlyInteractive, browser-based graphing library for Python
VisPyVisualize massive datasets in real time
Vega-AltairDeclarative Visualization in Python
plotnineGrammar of graphics for Python
PyQtGraphPython graphics and GUI library built on PyQt4 / PySide and numpy
bqplotInteractive Plotting Framework for the Jupyter Notebook
VaexFast visualization of big data
PyVista3D plotting and mesh analysis
foliumVisualize data in a Leaflet map
HoloViewsMake Data Analysis and Visualization Seamless
DatashaderGenerates aggregate arrays and representations of them as images
ytMulti-code Toolkit for Analyzing and Visualizing Volumetric Data
GlumpyIntuitive interface between NumPy and modern OpenGL
GeoViewsExplore and visualize geographical, meteorological, and oceanographic datasets
PygalDynamic SVG charting library
GlueMulti-dimensional linked-data exploration

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted