Apache Spark is likely one of the most generally used instruments within the massive information area. It excels at processing huge datasets for predictive modeling, fraud detection, and real-time analytics. Because the demand for processing and understanding information continues to develop, enterprises are in search of extra environment friendly methods to deal with ever-increasing workloads.
A number of the largest firms on the planet have turned to NVIDIA RAPIDS Accelerator for Apache Spark to handle the rising challenges of processing huge datasets effectively. The open-source plug-in, constructed on NVIDIA’s accelerated computing platform, is designed to make the info science and analytics course of quicker and simpler. Nvidia claims the software permits customers to handle full information pipelines with out requiring any modifications to their current Spark code.
This week on the GTC 2025, Nvidia launched Undertaking Aether to make it even simpler for firms to get worth out of NVIDIA-accelerated Spark. Undertaking Aether is a set of instruments and processes created by the chip producer to streamline information processing, providing substantial time and value financial savings, in keeping with the corporate.
Supply: Shutterstock
In a weblog put up introducing the brand new innovation, Nvidia shared, “Undertaking Aether automates the myriad steps that firms beforehand have carried out manually, together with analyzing all of their Spark jobs to establish one of the best candidates for GPU acceleration, in addition to staging and performing check runs of every job. It makes use of AI to fine-tune the configuration of every job to acquire the utmost efficiency.”
Undertaking Aether simplifies what was as soon as a tedious, guide technique of transitioning from CPU-based programs to GPU-powered computing. By using AI, it analyzes and adjusts Spark job configurations to maximise efficiency. Nvidia claims that the software permits customers to do “yr’s value of labor in lower than per week”.
Migrating Apache workloads has historically been a extremely guide course of. Customers usually needed to analyze Spark jobs individually, decide which workloads would profit from GPU acceleration, after which configure and run assessments to optimize efficiency. Staging the chosen workloads or adjusting the configuration additional added to the complexity.
Now, with Undertaking Ather, customers can automate a number of steps of the method. In accordance with Nvidia, if 100 Spark jobs require an engineer to work all the yr, Undertaking Aether can full every of the roles inside 4 days. This consists of fine-tuning the configuration of the roles for max Nvidia GPU acceleration.
How is that this attainable? Nvidia shared a case research the place Australia’s largest monetary establishment, the Commonwealth Financial institution of Australia (CBA), benefitted considerably from utilizing NVIDIA-Accelerated Apache Spark.
CBA, accountable for processing 60% of the continent’s monetary transactions, confronted challenges associated to latency and prices working its Spark workloads. The financial institution was utilizing CPU-only computing clusters and confronted nearly 9 years of processing time by way of coaching backlog, not together with the time wanted to deal with each day information calls for, which is estimated to be round 40 million transactions.
Supply: Shutterstock
By using RAPIDS Accelerator for Apache Spark on GPU-powered programs, CBA achieved a big 640x enchancment in efficiency. Nvidia shared that the financial institution accomplished the processing of 6.3 billion transactions for coaching in solely 5 days. Moreover, CBA can now conduct inference in as little as 46 minutes and is ready to scale back its prices by 80%. These outcomes could possibly be much more spectacular with Undertaking Aether in play.
In accordance with McMullan, one of many benefits of utilizing NVIDIA-accelerated Apache Spark is the power to cut back computation time, which permits his staff to create fashions extra effectively and at a decrease price. Which means that CBA can improve its customer support by predicting when clients could require assist with its services.
The financial institution plans on taking this additional by analyzing the shopper’s digital journey and figuring out the place they have an inclination to desert the digital course of.
A number of different firms are additionally leveraging NVIDIA RAPIDS Accelerator for Apache Spark to reinforce information processing effectivity and scale back prices. Dell Applied sciences has introduced that it’s incorporating the RAPIDS Accelerator for Apache Spark into its Dell Knowledge Lakehouse platform.
In accordance with Dell, the core advantages of utilizing NVIDIA RAPIDS Accelerator for Apache Spark embrace a large improve in speeds, price financial savings, scalability, and a unified acceleration that mixes CPU and GPU processes.
“The mixing of NVIDIA RAPIDS Accelerator for Apache Spark into Dell Knowledge Lakehouse isn’t simply an incremental enchancment — it’s a forward-looking development for companies prepared to satisfy as we speak’s calls for and tomorrow’s scale,” shared Dell. “By decreasing information complexity and accelerating AI workflows, firms can gasoline development and drive success in more and more data-driven markets.”
Associated Objects
From Monolith to Microservices: The Way forward for Apache Spark
Apache Spark Is Nice, However It’s Not Good
The Rise of Clever Machines: Nvidia Accelerates Bodily AI Progress