Cloudera to lower cloud compute spend and accelerate Apache Spark pipelines with NVIDIA

Cloudera has announced native GPU acceleration for Apache Spark 4.1 in Cloudera Data Engineering, enabled by the NVIDIA CUDA-X library, cuDF. The NVIDIA cuDF plug-in for Apache Spark will support the just-announced Cloudera Anywhere Cloud, which is designed to enable organizations to accelerate Spark workloads without rewriting PySpark or SQL code. This helps data teams prepare AI-ready data faster while reducing cloud infrastructure costs across hybrid environments.

As organizations expand AI initiatives, the speed of data preparation has become a critical challenge. Large-scale Spark workloads often take hours to complete, delaying analytics and AI applications while driving up cloud compute costs. By embedding GPU acceleration directly into Cloudera Data Engineering, organizations can dramatically reduce processing times using their existing Spark applications, without changing code or operational workflows.

Accelerating Spark for Enterprise AI

Apache Spark powers many of today's enterprise data pipelines. With native GPU acceleration built into Cloudera Data Engineering, organizations can improve performance while maintaining the security and governance required for production workloads. Leveraging NVIDIA cuDF for Spark workloads, Cloudera will provide up to 4x workload acceleration on NVIDIA GPUs compared to traditional CPU infrastructure, enabling enterprises to drastically shorten data processing cycles. 

Together, Cloudera Data Engineering accelerated by NVIDIA CUDA-X libraries will deliver:

●      Zero-code GPU acceleration for Apache Spark 4.1 workloads  

●      Faster ETL and data preparation for analytics and AI  

●      Lower cloud infrastructure costs through shorter compute runtimes  

●      Built-in deployment with no manual driver configuration  

●      Enterprise security and governance through the Cloudera Unified Data Fabric  

●      Consistent performance across public cloud, private cloud, sovereign cloud, and on-premises environments  

Unlike GPU acceleration offerings limited to a single cloud provider, Cloudera extends these capabilities across hybrid environments while preserving consistent governance and operations. Organizations can accelerate Spark workloads wherever their data resides— whether that is on premises, in the clouds, or at the edge— without sacrificing security or flexibility.

"For many organizations, AI isn't limited by models. It's limited by how quickly they can turn raw data into trusted, usable insights," said Leo Brunnick, Chief Product Officer at Cloudera. "Accelerating Spark inside Cloudera Data Engineering helps remove that bottleneck, allowing customers to move from data preparation to analytics and AI faster while keeping governance, security, and operational consistency at the center of their strategy."

"The fastest path to accelerating AI deployments is the one that aligns with how enterprises already operate today," said Pat Lee, vice president, Strategic Enterprise Partnerships at NVIDIA. "With NVIDIA AI infrastructure and CUDA-X libraries now native to Cloudera Data Engineering, enterprises can lower costs and dramatically speed up Apache Spark pipelines without changing a single line of PySpark or SQL code, turning business data into a foundation for AI."