High Performance Computer Cluster (HPC)
The MRI and MEG scanners in CUBRIC have demanding computing requirements, in terms of data storage, data backup and processing capabilities. Typically, data analysis times are at least 10 times greater than the acquisition time, so provision of an appropriate and scalable IT environment for CUBRIC is essential for delivering optimal research productivity. This is provided by a high-performance compute cluster coupled to a large storage array and fast quad-core workstations.
Most of the software that CUBRIC uses is based on the Linux/Unix platform, so we chose a uniform computing structure, for both cluster and workstations, based on RedHat Enterprise 4. In order to ensure binary compatability, the workstations and the cluster use the same processor – dual core AMD Opterons, which allow fast execution of both 32- and 64-bit code.
The software used in CUBRIC is a mixture of freeware, commercial packages and in-house developed tools. Some of the software is based on the mathematics toolkit, Matlab. The core research applications are:
- Analysis and visualisation of high-resolution structural MR images of the brain.
- Analysis and visualisation of functional MR images of the brain. A typical experiment involves the application of univariate time-series statistics to tens of thousands of voxels in tens of subjects and several experimental conditions. Real-time analysis of data, multivariate statistics and exploratory techniques such as Independent Components Analysis (ICA) will benefit from cluster acceleration.
- Diffusion-tensor imaging (DTI) and tractography.
- Magnetic Resonance Spectroscopy (MRS).
- Analysis and visualisation of MEG data. Here, the 4-dimensional nature of the data (mm in space, millisecond in time) leads to high computational demands, which again benefit from cluster acceleration.
- Multi-modal fusion of MEG and fMRI datasets.
Although some of these analyses can typically be run in a reasonable time (tens of minutes) on a single desktop workstation, research productivity at CUBRIC will benefit enormously from the compute cluster system, which dynamically and transparently allocates jobs to each processing-core in the cluster. This is especially important as multiple researchers begin analysing data simultaneously or real-time analysis is desired.
All the computing resources whithin CUBRIC are provided by IBM, through their UK Partner, OCF.
The CUBRIC cluster is the largest in the UK that is completely dedicated to Neuroimaging applications. It consists of 75 compute nodes, each with 2 dual-core Opterons, giving a total of 300 processing cores. Throughout the building, there are 40 desktop workstations, each with 2 dual-core Opterons. These workstations can be either completely stand-alone processing engines, or can be recruited into the compute cluster to give a total of 460 processing cores. A conservative estimate puts the performance of the CUBRIC cluster at 530 GFLOPS.
Together with the cluster and the workstation, CUBRIC also houses high-performance network and storage solutions for the large amount of data generated by the scanners and subsequent analyses. These include:
- 40 TB storage configured in RAID5 and using a GPFS file system.
- 1GB ethernet with 10GB stacked switches.
- 2GB bandwidth to the storage array via fibre channel.
- A tape-library for data archiving.