Skip to content

Commit 00cac67

Browse files
committed
Merge branch 'ADC-500344-convolve-beams' into pawsey_testing
2 parents 816304c + 3b0a5a6 commit 00cac67

7 files changed

+252
-13
lines changed

docs/source/introduction.md

+14-12
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,23 @@
22

33
Robbie automates the process of cataloguing sources, finding variables, and identifying transients.
44

5-
The workflow is described in [Hancock et al. 2018](https://ui.adsabs.harvard.edu/abs/2019A%26C....27...23H/abstract) and carries out the following steps:
5+
The workflow was described initially in [Hancock et al. 2018](https://ui.adsabs.harvard.edu/abs/2019A%26C....27...23H/abstract) however the current workflow is shown below:
66
- Preprocessing:
7-
- Find sources in images
8-
- Compare these catalogues to a reference catalogue
9-
- Use the offsets to model image based distortions
10-
- Make warped/corrected images
11-
- Persistent source catalogue creation:
12-
- Stack the warped images into a cube and form a mean image
13-
- Source find on the mean image to make a master catalogue
7+
- Convolve all images to a common psf (optional)
8+
- Create background and noise maps (if they are not found)
9+
- Correct astrometry using fitswarp (optional)
10+
- Variabile/persistent source detection:
11+
- Stack the warped images to form a mean image
12+
- Source find on the mean image to make a reference catalogue
1413
- Priorized fit this catalogue into each of the individual images
15-
- Join the catalogues into a single table and calculate variability stats
14+
- Join the epoch catalogues to make a persistent source catalogue
15+
- Calculate variability stats and generate a light curve for each source
1616
- Transient candidate identification:
17-
- Use the persistent source to mask known sources from the individual images
18-
- Source find on the masked images to look for transients
19-
- Combine transients tables into a single catalogue, identifying the epoch of each detection
17+
- Use the persistent source to mask known sources from the individual epochs
18+
- Source find on the masked images to find transients
19+
- Concatenate transients into a single catalogue, identifying the epoch of each detection
20+
21+
See [workflow](workflow) for a diagram of how Robbie works.
2022

2123
## Dependencies
2224
Robbie relies on the following software:

docs/source/workflow.md

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
(workflow)=
2+
## Workflow
3+
The workflow was described initially in [Hancock et al. 2018](https://ui.adsabs.harvard.edu/abs/2019A%26C....27...23H/abstract) however the current workflow is shown below:
4+
- Preprocessing:
5+
- Convolve all images to a common psf (optional)
6+
- Create background and noise maps (if they are not found)
7+
- Correct astrometry using fitswarp (optional)
8+
- Variabile/persistent source detection:
9+
- Stack the warped images to form a mean image
10+
- Source find on the mean image to make a reference catalogue
11+
- Priorized fit this catalogue into each of the individual images
12+
- Join the epoch catalogues to make a persistent source catalogue
13+
- Calculate variability stats and generate a light curve for each source
14+
- Transient candidate identification:
15+
- Use the persistent source to mask known sources from the individual epochs
16+
- Source find on the masked images to find transients
17+
- Concatenate transients into a single catalogue, identifying the epoch of each detection
18+
19+
20+
```mermaid
21+
flowchart TD
22+
subgraph Preprocessing
23+
direction LR
24+
img[Raw images] --> psf["PSF correction (?)"]
25+
psf --> bkg[Create background/noise maps]
26+
bkg --> warp["Fits warping (?)"]
27+
warp --> epim[Epoch images]
28+
end
29+
subgraph Persistent sources
30+
epim --> mean[Mean image]
31+
mean --> pscat[Persistent sources]
32+
epim --> pscat
33+
pscat --> vout[(Flux table, variability table, light curves, variabiltiy plot)]
34+
end
35+
subgraph Transients
36+
mean --> mask[Masked epoch images]
37+
epim --> mask
38+
pscat --> mask
39+
mask --> trans[Transient candidates]
40+
trans --> tout[(Filtered candidates, transients plot)]
41+
end
42+
```
43+
44+
Items with a (?) are optional processing steps that can be turned on/off.
45+
Items in a cylinder are the final outputs of the processing workflow.

nextflow.config

+1
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ params.image_file = "images.txt"
77

88
// Warping stage
99
params.warp = false
10+
params.convolve = false
1011
params.ref_catalogue = null
1112
params.refcat_ra = 'RAJ2000'
1213
params.refcat_dec = 'DEJ2000'

requirements.txt

+2-1
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@ git+https://github.com/PaulHancock/Aegean.git@v2.2.5
99
git+https://github.com/nhurleywalker/fits_warp.git
1010
git+https://gitlab.com/Sunmish/flux_warp.git
1111
bokeh>=2.4.2
12-
reproject>=0.7.1
12+
reproject>=0.7.1
13+
radio-beam>=0.3.3

robbie.nf

+26
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ if ( params.help ) {
3838
| Use a source finding file. [default: ${params.use_region_file}]
3939
| --region_file The location of the source finding file. [default: ${params.region_file}]
4040
|
41+
|Convolution arguments:
42+
| --convolve
43+
| Determine the smallest psf common to all input images and then convolve all images
44+
| to this psf prior to any other processing [default: ${params.convol}]
45+
|
4146
|Directory arguments:
4247
| --output_dir The directory to output the results to.
4348
| [default: ${params.output_dir}]
@@ -101,6 +106,22 @@ process get_version {
101106
}
102107

103108

109+
process convolve_beams {
110+
input:
111+
path(image)
112+
113+
output:
114+
path("*_convolved.fits")
115+
116+
script:
117+
"""
118+
echo ${task.process} on \${HOSTNAME}
119+
120+
convol_common_resolution.py --in ${image}
121+
"""
122+
}
123+
124+
104125
process bane_raw {
105126
label 'bane'
106127

@@ -542,6 +563,11 @@ process reproject_images {
542563
workflow {
543564
get_version( )
544565
// image_ch = epoch_label, image_fits
566+
if (params.convolve) {
567+
convolve_beams(image_ch.map{it->it[1]}.collect())
568+
image_ch = convolve_beams.out.flatten().map{it -> tuple(it.baseName.split('_')[0], it)}
569+
//image_ch.view()
570+
}
545571
bane_raw( image_ch )
546572
// image_bkg_rms = epoch_label, image_fits, [bkg_fits, rms_fits]
547573
image_bkg_rms = bane_raw.out

scripts/convol_common_resolution.py

+163
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
#! /usr/bin/env python3
2+
3+
__author__ = 'Paul Hancock'
4+
__date__ = '2022/07/08'
5+
6+
import argparse
7+
import os
8+
import sys
9+
10+
import AegeanTools.wcs_helpers as wcsh
11+
import numpy as np
12+
from astropy.convolution import Gaussian2DKernel
13+
from astropy.convolution.kernels import Gaussian2DKernel
14+
from astropy.io import fits
15+
from radio_beam import Beam, Beams
16+
from scipy import ndimage, optimize, signal
17+
18+
SIGMA_TO_FWHM = np.sqrt(8*np.log(2))
19+
20+
21+
def get_common_beam(fnames):
22+
"""
23+
Read headers from a list of fits files and determine the smallest
24+
common beam to which they could be convolved
25+
26+
parameters
27+
----------
28+
fnames : list([str, ...])
29+
List of filenames
30+
31+
return
32+
------
33+
cb : `radio_beam.Beam`
34+
The smallest common beam
35+
"""
36+
headers = [fits.getheader(f) for f in fnames]
37+
beams = [Beam.from_fits_header(h) for h in headers]
38+
beams=Beams(beams=beams)
39+
cb = beams.common_beam()
40+
return cb
41+
42+
43+
def beam_to_kernel(beam,
44+
ra, dec,
45+
wcshelper):
46+
"""
47+
Convert a beam object into a Gaussian kernel.
48+
Since the sky->pix mapping may not be affine the sky location is required for this conversion.
49+
50+
parameters
51+
----------
52+
beam : `radio_beam.Beam`
53+
The beam object
54+
55+
ra, dec : float, float
56+
The sky location
57+
58+
wcshelper: `AegeanTools.wcs_helpers.wcshelper`
59+
An augmented WCS object.
60+
61+
returns
62+
-------
63+
kern : `astropy.convolution.Gaussian2DKernel`
64+
The kernel
65+
"""
66+
_, _, a, b, pa = wcshelper.sky2pix_ellipse((ra,dec), beam.major.value, beam.minor.value, beam.pa.value)
67+
kern = Gaussian2DKernel(a/SIGMA_TO_FWHM, y_stddev=b/SIGMA_TO_FWHM, theta=pa)
68+
return kern
69+
70+
71+
def find_gaussian_kernel(image_kern, target_kern):
72+
"""
73+
Find the kernel B which when convolved with image_kern will produce target_kern
74+
This is solving A*B = C for B, where A and C are known and * is the 2d convolution.
75+
76+
parameters
77+
----------
78+
image_kern, target_kern : `astropy.convolution.Gaussian2DKernel`
79+
The kernels A and C
80+
81+
return
82+
------
83+
B : `astropy.convolution.Gaussian2DKernel`
84+
"""
85+
def diff(kern):
86+
return np.sum(np.abs(signal.convolve2d(kern.array, image_kern.array, mode='same')- target_kern.array))
87+
88+
89+
def func(x):
90+
a,b,pa = x
91+
B = Gaussian2DKernel(a/SIGMA_TO_FWHM, y_stddev=b/SIGMA_TO_FWHM, theta=pa,
92+
x_size=image_kern.shape[0], y_size=image_kern.shape[1])
93+
return diff(B)
94+
95+
res = optimize.minimize(func, x0=[1,1,0])
96+
97+
B_fit = Gaussian2DKernel(res.x[0]/SIGMA_TO_FWHM, y_stddev=res.x[1]/SIGMA_TO_FWHM, theta=res.x[2],
98+
x_size=image_kern.shape[0], y_size=image_kern.shape[1])
99+
return B_fit
100+
101+
102+
def convolve_to(psf, infile, outfile):
103+
"""
104+
Convolved an image to the target psf.
105+
Save the resulting image.
106+
107+
parameters
108+
----------
109+
psf : `radio_beam.Beam`
110+
The target psf (must be same/larger than that of the input file)
111+
112+
infile : str
113+
Path to read input image
114+
115+
outfile : str
116+
Path to save output
117+
"""
118+
hdu = fits.open(infile)
119+
120+
# determine the current psf
121+
wcshelper = wcsh.WCSHelper.from_header(hdu[0].header)
122+
beam = Beam.from_fits_header(hdu[0].header)
123+
im_kern = beam_to_kernel(beam, hdu[0].header['CRVAL1'], hdu[0].header['CRVAL2'], wcshelper)
124+
t_kern = beam_to_kernel(psf, hdu[0].header['CRVAL1'], hdu[0].header['CRVAL2'], wcshelper)
125+
126+
# find the kernel that will get us from im_kern -> t_kern
127+
fit_kern = find_gaussian_kernel(im_kern, t_kern)
128+
129+
# convolve and update the image data
130+
data = signal.convolve2d(hdu[0].data, fit_kern, mode='same')
131+
hdu[0].data = data
132+
133+
# update the header info
134+
header = hdu[0].header
135+
header['BMAJ'] = psf.major.value
136+
header['BMIN'] = psf.minor.value
137+
header['BPA'] = psf.pa.value
138+
139+
# save the file
140+
hdu.writeto(outfile, overwrite=True)
141+
print("Wrote {0}".format(outfile))
142+
return
143+
144+
145+
if __name__ == "__main__":
146+
parser = argparse.ArgumentParser()
147+
group1 = parser.add_argument_group("Convolve images")
148+
group1.add_argument("--in", dest='files', type=str, default=None, nargs='+',
149+
help="List of input images")
150+
group1.add_argument("--suffix", dest='suffix', type=str, default="convolved",
151+
help="Output suffix for filenames.")
152+
results = parser.parse_args()
153+
154+
if results.files is None or len(results.files) <2:
155+
parser.print_help()
156+
sys.exit(0)
157+
158+
psf = get_common_beam(results.files)
159+
for f in results.files:
160+
base, ext = os.path.splitext(f)
161+
outfile = "{0}_{1}{2}".format(base, results.suffix, ext)
162+
convolve_to(psf, f, outfile)
163+

setup.py

+1
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ def get_git_version():
5151
"scripts/plot_variables.py",
5252
"scripts/make_weights.py",
5353
"scripts/reprojection.py",
54+
"scripts/convol_common_resolution.py",
5455
# nextflow
5556
"robbie.nf",
5657
"nextflow.config",

0 commit comments

Comments
 (0)