High Performance Computer Cluster (HPC)

The MRI and MEG scanners in CUBRIC have demanding computing requirements, in terms of data storage, data backup and processing capabilities. Typically, data analysis times are at least 10 times greater than the acquisition time, so provision of an appropriate and scalable IT environment for CUBRIC is essential for delivering optimal research productivity. This is provided by a high-performance compute cluster coupled to a large storage array and fast quad-core workstations.

Most of the software that CUBRIC uses is based on the Linux/Unix platform, so we chose a uniform computing structure, for both cluster and workstations, based on SUSE Linux Entreprise 11.

The software used in CUBRIC is a mixture of freeware, commercial packages and in-house developed tools. Some of the software is based on the mathematics toolkit, MATLAB. The core research applications are:

  • Analysis and visualisation of high-resolution structural MR images of the brain.
  • Analysis and visualisation of functional MR images of the brain. A typical experiment involves the application of univariate time-series statistics to tens of thousands of voxels in tens of subjects and several experimental conditions. Real-time analysis of data, multivariate statistics and exploratory techniques such as Independent Components Analysis (ICA) will benefit from cluster acceleration.
  • Diffusion-tensor imaging (DTI) and tractography.
  • Magnetic Resonance Spectroscopy (MRS).
  • Analysis and visualisation of MEG data. Here, the 4-dimensional nature of the data (mm in space, millisecond in time) leads to high computational demands, which again benefit from cluster acceleration.
  • Multi-modal fusion of MEG and fMRI datasets.

Although some of these analyses can typically be run in a reasonable time (tens of minutes) on a single desktop workstation, research productivity at CUBRIC will benefit enormously from the compute cluster system, which dynamically and transparently allocates jobs to each processing-core in the cluster. This is especially important as multiple researchers begin analysing data simultaneously or real-time analysis is desired.

The CUBRIC cluster is the largest in the UK that is completely dedicated to Neuroimaging applications. It consists of 108 compute nodes, each with 12-core Intel Westmere, giving a total of 1296 processing cores. A conservative estimate puts the performance of the CUBRIC cluster at 14 trillion operations per second. The CUBRIC cluster is provided by Bull.

In addition to the CUBRIC cluster, we provide a remote desktop interface to allow researchers and collaborators to connect to our system from any internet connected computer. This allow us to keep all data within CUBRIC and to provide a coherent working environment. The service is hosted on a dedicated pool of 40 virtual machines (8-cores, 16GB of RAM) running NoMachine.

Together with the cluster and the workstations, CUBRIC also houses high-performance network and storage solutions for the large amount of data generated by the scanners and subsequent analyses. These include:

  • 125TB storage configured in RAID-Z2 and using a ZFS file system.
  • Multisite redundant storage
  • 70TB data museum
  • 40Gb QDR Infiniband connection between the cluster nodes and the storage system
  • 1Gb ethernet with 10Gb stacked switches.
  • A tape-library for data archiving.