Big Data

HPCC Systems – distributed data processing platform

HPCC Systems is a distributed computing platform for processing, transforming and querying large datasets. It combines data storage, parallel processing and query-serving components within a single architecture. The platform is particularly suited to workloads involving large collections of records that need to be cleaned, linked, analysed and made available to other applications.

Two major processing engines provide different capabilities. Thor handles data preparation and large-scale computational jobs, while Roxie is designed to respond efficiently to concurrent queries against prepared and indexed data. Both engines use Enterprise Control Language (ECL), a declarative language for describing data operations. This allows data-processing logic to be reused across different stages of a workflow. The platform also includes services for managing distributed files, publishing queries and monitoring cluster activity.

This is free and open source software.

Key Features

  • Thor provides massively parallel data preparation, transformation and analysis.
  • Roxie serves concurrent queries against distributed and indexed datasets.
  • Supports declarative data processing using Enterprise Control Language (ECL).
  • Compiles ECL queries into executable processing operations.
  • Provides a distributed filesystem for storing data across cluster nodes.
  • Supports indexed data storage using B+ tree structures.
  • Allows prepared datasets and indexes to be deployed from Thor to Roxie.
  • Provides distributed joins, aggregations, sorting and data transformations.
  • Supports record linking and analysis of relationships between datasets.
  • Includes the Distributed File Utility for loading and managing data.
  • Provides ECL Watch for web-based administration and monitoring.
  • Supports query submission, execution monitoring and workunit management.
  • Includes facilities for managing distributed files and processing queues.
  • Offers ECL development tools with syntax checking and query execution.
  • Provides Enterprise Services Platform interfaces for web services and authentication.
  • Includes Dali services for metadata management and distributed filesystem naming.
  • Supports publishing analytical queries as services for external applications.
  • Provides interfaces for integrating external data sources and processing systems.
  • Includes machine learning libraries accessible through ECL.
  • Supports containerised deployments with Kubernetes orchestration.
  • Offers configurable security, permissions and authentication controls.

Website: github.com/hpcc-systems
Support:
Developer: HPCC Systems, LexisNexis Risk Solutions
License: Apache License 2.0

HPCC is written in C++. Learn C++ with our recommended free books and free tutorials.


Related Software

Data Analysis Tools
HadoopDistributed processing of large data sets across clusters of computers
StormDistributed and fault-tolerant realtime computation
DrillDistributed system for interactive analysis of large-scale datasets
FlinkFramework and distributed processing engine
SparkUnified analytics engine for large-scale data processing
PentahoEnterprise reporting, analysis, dashboard, data mining, workflow and more
HPCC SystemsDesigned for the enterprise to resolve Big Data challenges
DaftHigh-performance distributed data processing engine
Apache BeamUnified programming model for batch and streaming data processing

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted