Accelerating research at Oak Ridge National Laboratory

Thanks to the unique features of the new NVIDIA DGX-2 AI systems and their rapid and successful installation, ORNL research teams have been able to expand existing and enable new innovative projects focusing on machine learning and AI with advanced architectures throughout the Lab.

The DGX-2 systems feature a unique density of 16 NVIDIA V100 GPUs plus innovative NVIDIA NVSwitch technology to fully interconnect all GPUs. Since their delivery, they have proven extraordinarily complementary to ORNL’s record-breaking 200 petaflop “Summit” supercomputer.

Enabling innovative projects

These specialized systems allow users to work on a uniquely large and complex set of problems that other large GPU solutions are incapable of tackling.

“I get requests for access to these systems often, from both researchers and students from all over the lab who want to learn on the best hardware around. These requests cover the full range of use cases and the DGX-2s never fail to impress. As word of mouth, combined with outreach, ramps up, I only see usage of these systems increasing,” said Chris Layton, Linux Systems Engineer for ORNL’s Compute and Data Environment for Science (CADES) team.

Heng Ma, a postdoctoral research associate in the Center for Molecular Biophysics, shared that the DGX-2 systems have made scaling projects up to the Summit system easier and more successful. “We use machine learning algorithms to control Molecular Dynamics simulations… For my current projects, I use the DGX-2 to produce a prototype of data, which later on we are trying to move to Summit. So, this prototype is like the proof of concept that it actually works before we actually put it on Summit.”

The Compute and Data Environment for Science (CADES) team at ORNL sought this groundbreaking new architecture to help advance their research. The ORNL team then decided to trust AI & HPC specialist Microway with their deployment. They were rewarded with the two DGX-2 systems physically installed, up and running, and doing benchmark testing within 4 hours of the first crate being opened.

Deployment: running benchmarks 4 hours after the crates were opened

As an experienced cluster integrator and NVIDIA Partner Network Elite DGX partner, Microway’s role was essential to delivering a complete solution that was operational as soon as it was installed.

In the weeks before delivery Microway experts performed a careful system, storage, and network architecture design and design review with ORNL personnel and NVIDIA solutions architects—enabling rapid installation and setup once Microway personnel arrived onsite.

Delivery of the two new machines also required careful advanced logistical preparation to ensure that the room, network, contacts, cooling, and system admins were all ready for the installation and launch of the DGX-2 systems. The ORNL admin and Microway teams constantly collaborated via, phone, email, and web prior to the installation to ensure a smooth deployment.

“Microway was able to, via their installation crew, make the integration of the DGX-2 into the CADES environment a smooth process... This was done under a deadline and they met all the timelines flawlessly. Microway was able to update the DGX-2s to a point where they were ready for the CADES team to hit the ground running in configuring them for end users,” Layton shared.

Upon arriving onsite, the Microway team uncrated the systems, readied racks, installed the systems into the racks, ran power and network cabling, updated all firmware, deployed the complete DGX software stack, and readied the systems for benchmark testing.

Into the Future

The new DGX-2 systems have already provided unexpected capacities and insights to the ORNL team, and the researchers expect that this will continue into the future.

Groups in such diverse fields as Molecular Biophysics, geographic data sciences, and AI-Driven Biosystems modeling have all utilized the new hardware deployment to drive their science & research since delivery.

The systems have attracted attention across the lab. Additional users at ORNL have selected a third DGX-2 for a recent deployment. As with the initial systems, the Microway team has ensured a smooth delivery experience and rapid bringup.