The scientific and technical goal for Hilado is to create optimized prototype software and demonstrator processing pipelines that improve the capabilities of currently planned software packages for existing and emerging radio telescopes. These developments are essential to increase the potential of the
RadioNet user community in opening up those facilities for the more demanding scientific applications. Three examples may serve to illustrate this:
- Given the current limitations of processing platforms, the standard LOFAR imaging pipeline will not be able to process in a realistic timescale data for all 80 km baselines at full FoV, 30MHz bandwidth at the lowest frequencies. By applying the optimised software developed in Task 1 of this JRA, and deploying time critical functions (Solvers, Gridding) on specialised platforms as studied in Task 3, such extreme cases will become feasible.
- The most demanding ALMA imaging cases are multi-field mosaics of complex Galactic fields with multiple arrays, combined with single dish and with large numbers of spectral channels. Performing these on the computing hardware currently projected for ALMA will take an unrealistically long time. The development of prototype software in this JRA that can handle these data rates reliably (Task 1) and can run on relatively cheap specialised platforms (Task 3) is needed to enable these observations.
- Fast transient imaging is currently limited for LOFAR and other RadioNet facilities, the main reason being that the current processing pipelines are not capable of continuously handling the short integration times required for surveying Rotating Radio Transients (RRATS) or studying the nature of extragalactic millisecond phenomena like the "Lorimer-transient". The developments of a demonstrator Fast Transient Imager (Task 2) will bring such options to the user.
Far from being limited to the two facilities mentioned, LOFAR and ALMA, these developments will readily apply to enable faster and thus deeper processing of data from other RadioNet facilities (e.g. e-Merlin, WSRT and EVN) as well as increasing the capabilities of the RadioNet community in engaging with other instruments, in particular EVLA, MeerKAT and ASKAP. The work in this JRA will also prepare for new facilities including the SKA.
It should be noted that none of the on-going SKA studies in software and computing is addressing the topics covered by this JRA, and thus will not help current RadioNet users in addressing the subjects illustrated above. Also the forthcoming Pre-Construction Phase for the SKA will not work on specific software developments, but rather on architectural studies, planning and costing of the software effort. However, the knowledge and prototype software developed in this JRA will form a knowledge base for the SKA Construction Phase (planned start 2016) when production software for the SKA will start to be developed.
Hilado is well aware that parallel computing initiatives for radio astronomy are taking place in various contexts (ALMA, LOFAR, EVLA, ASKAP, MeerKAT). Far from duplicating these activities, Hilado builds on these results to make a significant impact by targeted studies in a number of well-defined areas. Hilado can make a significant impact with a modest amount of resources exactly because of these links with existing project, where the basis infrastructure and development is done. Through specific and targeted optimisations, Hilado will enable specialised applications (like the Fast Transient Imager), open up new platforms (like fast solvers on GPU based clusters) and boost performance (through improved robustness and data formats). The common denominator in Hilado is to address the issues related to the size of the datasets produced by the more extreme observations possible with telescopes like ALMA and LOFAR.
Hilado also explores a new development model for astronomical processing software. Expertise from a variety of disciplines needs to be tapped: from mathematics for the foundations of new algorithms, from computer science for optimizations for high-performance computer platforms and from industry to explore the space of novel architectures. Talented people around the world should be enabled and encouraged to contribute. This collaborative project will facilitate the adoption of the new developments, technologies and insights required by huge datasets, and will tap expertise beyond the boundaries of the radio astronomy community. The success of this project depends critically on placing existing as well as novel HPC technologies at the centre of the development of algorithms, software and procedures. The JRA aims at optimized prototype software that can be re-used in a variety of contexts, including both automated pipelines and user-adaptable scripts. This brings maximum value to a broad group of RadioNet users by allowing them to balance between highly optimized automated processing and highly flexible interactive application without sacrificing efficiency and performance.
Software will be developed in publicly available Open Source repositories like the CASACore library and Python scripting environment. This approach gives good opportunities for dissemination and training. It should be noted that there is currently no funded R&D effort on CASACore. The CASACore libraries are maintained by voluntary contributions from ASTRON, NRAO and CSIRO. Therefore no significant effort in improving the robustness and optimizing the performance of these libraries can be expected without additional effort. Especially on the reliable handling of huge astronomical datasets, significant gains can be obtained through a modest effort. Hilado will allow CASACore to be further optimized in those areas where most impact is expected. For ALMA and LOFAR this will be through performance optimizations, farming out of critical algorithms to optimized processing platforms and application of optimized data models for the various phases in the data processing. Hilado builds on the earlier RadioNet joint research activities by reusing ParselTongue (ALBUS) and the insights from interoperability studies (ALBiUS). Collaborations are being formed with MeerKAT and ASKAP (on optimization of the CASACore library and benchmarking) and NRAO (extending their algorithm development and HPC for both ALMA and EVLA).
The leader of this activity is Marco de Vos
