Logo

Hortonworks and Hewlett Packard Enterprise accelerate Apache Spark

Hortonworks and Hewlett Packard Labs are working together to enhance Apache Spark, one of the most active Apache big data projects. The collaboration will center around an entirely new class of analytic workloads that benefit from large pools of shared memory.

Early results of the collaboration include the following:
  • Enhanced shuffle engine technologies: Faster sorting and in-memory computations, which has the potential to dramatically improve Spark performance.   
  • Better memory utilization: Improved performance and usage for broader scalability, which will help enable new large-scale use cases.

 

“This collaboration indicates our mutual support of and commitment to the growing Spark community and its solutions,” said Scott Gnau, chief technology officer, Hortonworks.  “We will continue to focus on the integration of Spark into broad data architectures supported by Apache YARN as well as enhancements for performance and functionality and better access points for applications like Apache Zeppelin.” 

 

“We’re hoping to enable the Spark community to derive insight more rapidly from much larger data sets without having to change a single line of code,” said Martin Fink, EVP and CTO, Hewlett Packard Enterprise and Hortonworks Board Member. “We’re very pleased to be able to work with Hortonworks to broaden the range of challenges that Spark can address.”

 

Hortonworks and Hewlett Packard Enterprise plan to contribute the new technologies to the Apache Spark community.

Ocorian unlocks 50 percent more data efficiency with Fivetran

Fivetran accelerates global financial services provider’s move to the cloud; transforms reporting...

Majority of UK IT decision makers fear exponential rise of data within their businesses

As demand for data-driven insights increases, data overload could render UK businesses...

Fivetran report finds enterprises racing toward AI without the data to support it

Survey of 500 tech leaders shows companies are moving fast on AI plans while struggling with scale,...

Seamless data integration for smarter AI

Reducing cost and complexity of distributed data landscapes, Hitachi Vantara offers a modern data...

Data silos prevent healthcare organisations from capitalising on the AI opportunity

Only 54% of organisations have robust systems for moving data internally, while only 56% have...

Skills shortages top the list of the UK’s data challenges

Real-time data gap risks stalling AI progress.

Data streaming enables AI product innovation

In the largest global report on data streaming, 89% say DSPs make AI adoption easier.

Nearly half of enterprises report failed AI projects due to poor data readiness

New global survey reveals rising business costs from failed AI projects, as integration complexity...

© 2025 - Angel Business Communications Limited

Made with by Angels