Figure 6 - uploaded by Aske Plaat
Content may be subject to copyright.
Source publication
Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each st...
Contexts in source publication
Context 1
... system parameters studied here are the CPU speed, memory throughput, cache size and disk speed. Modern computers can have a complex memory hierarchy as demonstrated in Figure 6 ( Katz and Patterson, 2001). This is due to the cost trade-off between memory size and memory speed. Because of this trade-off, the full dataset is stored on disk, while the working set is placed in RAM. This is the data that the processor needs to access at the current time (Denning, 1968). The most frequently ac- cessed parts of the data are stored in the CPU cache, which evicts the oldest data when full ( Hazelwood and Smith, 2004). The CPU processing speed is faster than the RAM la- tency, so a hierarchy of caches exist. Caches store small subsets of the working set and have a fast connection to the processor. The fastest data link is between the CPU and the L1 Cache, with the link to RAM being slower and the disk read speed slower still. The limited memory capac- ity of the different levels of the memory hierarchy as well as the throughput between them will lead to performance bottlenecks. These bottlenecks will lead to the processor waiting on memory. Such stalls lead to longer processing times. ...
Context 2
... slowest processing steps for the prefactor pipeline were identified as the calib cal and gsmcal solve steps. While the data can fit into the RAM for all of the pro- cessing machines, it is much larger than the processor's internal cache ( Figure 6). The discoveries made concerned the memory hierarchy in Figure 6. Results labeled R2, R8 and R9 related to the CPU performance; R2, R6 and R7 related to the Cache performance; R3 and R5 related to the Memory usage and R4 discussed the Disk ...
Context 3
... slowest processing steps for the prefactor pipeline were identified as the calib cal and gsmcal solve steps. While the data can fit into the RAM for all of the pro- cessing machines, it is much larger than the processor's internal cache ( Figure 6). The discoveries made concerned the memory hierarchy in Figure 6. Results labeled R2, R8 and R9 related to the CPU performance; R2, R6 and R7 related to the Cache performance; R3 and R5 related to the Memory usage and R4 discussed the Disk ...
Context 4
... CPU has a hierarchy of caches consisting of Level 1, Level 2 Cache and LLC Cache. For the four processors tested, the Level 1 and 2 caches were all the same size, thus the only difference is the Last Level Cache (LLC or just Cache in Figure 6). This cache stores data needed by the CPU, so the larger it is, the less the processor needs to wait for RAM to return ...
Context 5
... system parameters studied here are the CPU speed, memory throughput, cache size and disk speed. Modern computers can have a complex memory hierarchy as demonstrated in Figure 6 ( Katz and Patterson, 2001). This is due to the cost trade-off between memory size and memory speed. ...
Context 6
... CPU has a hierarchy of caches consisting of Level 1, Level 2 Cache and LLC Cache. For the four processors tested, the Level 1 and 2 caches were all the same size, thus the only difference is the Last Level Cache (LLC or just Cache in Figure 6). This cache stores data needed by the CPU, so the larger it is, the less the processor needs to wait for RAM to return data. ...
Context 7
... slowest processing steps for the prefactor pipeline were identified as the calib cal and gsmcal solve steps. While the data can fit into the RAM for all of the processing machines, it is much larger than the processor's internal cache ( Figure 6). The discoveries made concerned the memory hierarchy in Figure 6. ...
Context 8
... the data can fit into the RAM for all of the processing machines, it is much larger than the processor's internal cache ( Figure 6). The discoveries made concerned the memory hierarchy in Figure 6. Results labeled R2, R8 and R9 related to the CPU performance; R2, R6 and R7 related to the Cache performance; R3 and R5 related to the Memory usage and R4 discussed the Disk speed. ...
Similar publications
We present a post-training weight pruning method for deep neural networks that achieves accuracy levels tolerable for the production setting and that is sufficiently fast to be run on commodity hardware such as desktop CPUs or edge devices. We propose a data-free extension of the approach for computer vision models based on automatically-generated...
Citations
... In the framework of LoTSS the LOFAR Surveys Key Science Project (SKSP) developed an automized job management and execution system for the grid-computing facilities at SURFsara in Amsterdam. These tools are described in Mechev et al. (2017) 16 and Mechev et al. (2018ab) 17,18 . Since the biggest fraction of LOFAR data observed in the framework of the LoTSS is stored in the Jülich LTA, an efficient and fast processing of the data is crucial. ...
The new generation of high-resolution broad-band radio telescopes, like the Low Frequency Array (LOFAR), produces, depending on the level of compression, between 1 to 10 TB of data per hour after correlation. Such a large amount of scientific data demand powerful computing resources and efficient data handling strategies to be mastered. The LOFAR Two-metre Sky Survey (LoTSS) is a Key Science Project (KSP) of the LOFAR telescope. It aims to map the entire northern hemisphere at unprecedented sensitivity and resolution. The survey consist of 3 168 pointings, requiring about 30 PBytes of storage space. As a member of the German Long Wavelength Consortioum (GLOW) the Forschungszentrum J\"ulich (FSZ) stores in the Long Term Archive (LTA) about 50% of all LoTSS observations conducted to date. In collaboration with SURFsara in Amsterdam we developed service tools that enable the KSP to process LOFAR data stored in the J\"ulich LTA at the supercomputer JUWELS in an automated and robust fashion. Through our system more than 500 out of 800 existing LoTSS observations have already been processed with the prefactor pipeline. This pipeline calibrates the direction-independent instrumental and ionospheric effects and furthermore reduces the data size significantly. For continuum imaging, this processing pipeline is the standard pipeline that is executed before more advanced processing and image reconstruction methods are applied.
With the advent of new generation low-frequency telescopes, such as the LOw Frequency ARray (LOFAR), and improved calibration techniques, we have now started to unveil the subgigahertz radio sky with unprecedented depth and sensitivity. The LOFAR Two Meter Sky Survey (LoTSS) is an ongoing project in which the whole northern radio sky will be observed at 150 MHz with a sensitivity better than 100 μ Jy beam ⁻¹ at a resolution of 6′′. Additionally, deeper observations are planned to cover smaller areas with higher sensitivity. The Lockman Hole, the Boötes, and the Elais-N1 regions are among the most well known northern extra-galactic fields and the deepest of the LoTSS Deep Fields so far. We exploited these deep observations to derive the deepest radio source counts at 150 MHz to date. Our counts are in broad agreement with those from the literature and show the well known upturn at ≤1 mJy, mainly associated with the emergence of the star-forming galaxy population. More interestingly, our counts show, for the first time a very pronounced drop around S ~ 2 mJy, which results in a prominent “bump” at sub-mJy flux densities. Such a feature was not observed in previous counts’ determinations (neither at 150 MHz nor at a higher frequency). While sample variance can play a role in explaining the observed discrepancies, we believe this is mostly the result of a careful analysis aimed at deblending confused sources and removing spurious sources and artifacts from the radio catalogs. This “drop and bump” feature cannot be reproduced by any of the existing state-of-the-art evolutionary models, and it appears to be associated with a deficiency of active galactic nuclei (AGN) at an intermediate redshift (1 < z < 2) and an excess of low-redshift ( z < 1) galaxies and/or AGN.
It is well established that particle acceleration by shocks and turbulence in the intra-cluster medium can produce cluster-scale synchrotron emitting sources. However, the detailed physics of these particle acceleration processes is still not well understood. One of the main open questions is the role of fossil relativistic electrons that have been deposited in the intracluster medium (ICM) by radio galaxies. These synchrotron-emitting electrons are very difficult to study as their radiative lifetime is only tens of Myr at gigahertz frequencies, and they are therefore a relatively unexplored population. Despite the typical steep radio spectrum due to synchrotron losses, these fossil electrons are barely visible even at radio frequencies well below the gigahertz level. However, when a pocket of fossil radio plasma is compressed, it boosts the visibility at sub-gigahertz frequencies, creating what are known as radio phoenices. This compression can be the result of bulk motion and shocks in the ICM due to merger activity. In this paper we demonstrate the discovery potential of low-frequency radio sky surveys to find and study revived fossil plasma sources in galaxy clusters. We used the 150 MHz TIFR GMRT Sky Survey and the 1.4 GHz NVSS sky survey to identify candidate radio phoenices. A subset of three candidates was studied in detail using deep multi-band radio observations (LOFAR and GMRT), X-ray obserations ( Chandra or XMM-Newton ), and archival optical observations. Two of the three sources are new discoveries. Using these observations, we identified common observational properties (radio morphology, ultra-steep spectrum, X-ray luminosity, dynamical state) that will enable us to identify this class of sources more easily, and will help us to understand the physical origin of these sources.
Context. Radio recombination lines (RRLs) at frequencies ν < 250 MHz trace the cold, diffuse phase of the interstellar medium, and yet, RRLs have been largely unexplored outside of our Galaxy. Next-generation low-frequency interferometers such as LOFAR, MWA, and the future SKA will, with unprecedented sensitivity, resolution, and large fractional bandwidths, enable the exploration of the extragalactic RRL universe.
Aims. We describe methods used to (1) process LOFAR high band antenna (HBA) observations for RRL analysis, and (2) search spectra for RRLs blindly in redshift space.
Methods. We observed the radio quasar 3C 190 ( z ≈ 1.2) with the LOFAR HBA. In reducing these data for spectroscopic analysis, we placed special emphasis on bandpass calibration. We devised cross-correlation techniques that utilize the unique frequency spacing between RRLs to significantly identify RRLs in a low-frequency spectrum. We demonstrate the utility of this method by applying it to existing low-frequency spectra of Cassiopeia A and M 82, and to the new observations of 3C 190.
Results. Radio recombination lines have been detected in the foreground of 3C 190 at z = 1.12355 (assuming a carbon origin) owing to the first detection of RRLs outside of the local universe (first reported in A&A, 622, A7). Toward the Galactic supernova remnant Cassiopeia A, we uncover three new detections: (1) stimulated C ϵ transitions (Δn = 5) for the first time at low radio frequencies, (2) H α transitions at 64 MHz with a full width at half-maximum of 3.1 km s ⁻¹ the most narrow and one of the lowest frequency detections of hydrogen to date, and (3) C α at vLSR ≈ 0 km s ⁻¹ in the frequency range 55–78 MHz for the first time. Additionally, we recover C α , C β , C γ , and C δ from the −47 km s ⁻¹ and −38 km s ⁻¹ components. In the nearby starburst galaxy M 82, we do not find a significant feature. With previously used techniques, we reproduce the previously reported line properties.
Conclusions. RRLs have been blindly searched and successfully identified in Galactic (to high-order transitions) and extragalactic (to high redshift) observations with our spectral searching method. Our current searches for RRLs in LOFAR observations are limited to narrow (<100 km s ⁻¹ ) features, owing to the relatively small number of channels available for continuum estimation. Future strategies making use of a wider band (covering multiple LOFAR subbands) or designs with larger contiguous frequency chunks would aid calibration to deeper sensitivities and broader features.
Context. Recombination lines involving high principal quantum numbers (n ∼ 50 − 1000) populate the radio spectrum in large numbers. Low-frequency (< 1 GHz) observations of radio recombination lines (RRLs) primarily from carbon and hydrogen offer a new, if not unique, way to probe cold, largely atomic gas and warm, ionised gas in other galaxies. Furthermore, RRLs can be used to determine the physical state of the emitting regions, such as temperature and density. These properties make RRLs, potentially, a powerful tool of extragalactic interstellar medium (ISM) physics. At low radio frequencies, it is conceivable to detect RRLs out to cosmological distances when illuminated by a strong radio continuum. However, they are extremely faint ( τ peak ∼ 10 ⁻³ − 10 ⁻⁴ ) and have so far eluded detection outside of the local universe.
Aims. With observations of the radio quasar 3C 190 ( z = 1.1946), we aim to demonstrate that the ISM can be explored out to great distances through low-frequency RRLs.
Methods . 3C 190 was observed with the LOw Frequency ARray (LOFAR) and processed using newly developed techniques for spectral analysis.
Results. We report the detection of RRLs in the frequency range 112 MHz–163 MHz in the spectrum of 3C 190. Stacking 13 α -transitions with principal quantum numbers n = 266 − 301, a peak 6 σ feature of optical depth τ peak = (1.0 ± 0.2)×10 ⁻³ and FWHM = 31.2 ± 8.3 km s ⁻¹ was found at z = 1.124. This corresponds to a velocity offset of −9965 km s ⁻¹ with respect to the systemic redshift of 3C 190.
Conclusions. We consider three interpretations of the origin of the RRL emission: an intervening dwarf-like galaxy, an active galactic nucleus (AGN) driven outflow, and the inter-galactic medium. We argue that the recombination lines most likely originate in a dwarf-like galaxy ( M ∼ 10 ⁹ M ⊙ ) along the line of sight, although we cannot rule out an AGN-driven outflow. We do find the RRLs to be inconsistent with an inter-galactic medium origin. With this detection, we have opened up a new way to study the physical properties of cool, diffuse gas out to cosmological distances.
