High-Performance and Batch Computing
High-performance computing (HPC) systems are built to handle complex computational tasks and large-scale data processing needs. These systems are designed to execute scripts that can run without supervision when the resources required are available. This is usually referred to as batch computing.
HPC systems support a wide range of computational workflows, from data analysis scripts that require more memory than is available on a personal computer to large scale simulations that utilize many cores over multiple nodes.
Please include the Acknowledgment of Use when publishing research done using Quest.
Key Components of HPC and Batch Systems
- Computing Hardware: Multiple interconnected high-performance nodes featuring powerful processors, including GPUs, and large memory capacity.
- Parallel Computing Support: Hardware and software to support running many processes in parallel or complex processes across multiple cores and nodes.
- Job Scheduling: Advanced workload management systems that efficiently distribute computing tasks across available resources while maintaining fair use policies across users.
- Storage Infrastructure: High-speed storage systems optimized for parallel access and data-intensive operations.
- Network Architecture: High-bandwidth, low-latency intra-cluster connections enabling efficient communication between compute nodes.
Available Resources
Quest
Quest is Northwestern’s high-performance computing (HPC) cluster. Northwestern provides free access to Quest for all Northwestern researchers, with options for researchers with significant computing needs to purchase dedicated resources.
Quest comprises over 1200 nodes (80,000 cores) running Red Hat Enterprise Linux and interconnected with an InfiniBand HDR network. The newest Quest nodes have Intel Emerald Rapids Xeon Platinum 8592+ 1.9 GHz processors with 128 cores and 512 GB RAM per node. Quest uses the Slurm workload manager to schedule jobs across the compute nodes. To store research data, Quest has a General Parallel File System (GPFS) backed by a 12 PB capacity IBM Elastic Storage Server (ESS) system. It also includes a 500 TB flash scratch storage for accelerated computing and each user has an 80 GB home directory for storing code files and applications. Quest is connected to Globus for managing file transfers.
Learn More and Get Started with Quest
The Genomics Compute Cluster (GCC) resides within Quest infrastructure and provides high-performance computing resources for Northwestern researchers from any school or center who are engaged in genomics research. This cluster comprises over 10,000 cores across 150 compute nodes.
External Resources
Beyond Northwestern, HPC systems funded by the National Science Foundation (NSF) and Department of Energy (DOE) provide options for researchers needing specialized hardware, container orchestration, or systems that can support very large-scale workflows.
External resources supported by Research Computing and Data Services include:
- ACCESS: This NSF-funded system supports researchers across domains, with or without NSF grants, with free access to computational resources. Research Computing and Data Services computational specialists can help you evaluate ACCESS resources and submit an application.
- Open Science Grid: This system supports high-throughput workflows consisting of many small processes. Research Computing and Data Services computational specialists can help you assess whether the Open Science Grid is a good fit for your needs.
- Frontera: NSF-funded system run by the Texas Advanced Computing Center (TACC) for researchers requiring multi-petascale computing resources. Frontera is an option for large workflows that would utilize a significant share of Quest’s resources.
- INCITE: Access to the DOE Leadership Computing Facilities is allocated through a competitive application process; researchers who are not working on Department of Energy projects but whose projects would benefit from the unique computing resources available can apply.
Additional systems may be available to researchers working on Department of Energy-funded projects: Oak Ridge National Laboratory Leadership Computing Facility, Argonne Leadership Computing Facility, and NERSC.
Discuss Options with a Computational Specialist
Cloud Computing
Public cloud services, such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform, can be a good option for computational workflows requiring resources not available on Quest. Use cases include databases, archival storage, secure computing, and application hosting.
Research Computing and Data Services staff can assist you with setting up cloud accounts under Northwestern contracts with the providers, estimate cloud computing costs, design cloud computing workflows, and advise on security and best practices.