Processor tools

Clustering of AMBER candidates

class darc.processor_tools.clustering.Clustering(obs_config, output_dir, log_queue, input_queue, output_queue, ncluster, config_file='/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/darc/config.yaml', obs_name='')

Bases: Process

Clustering and thresholding of AMBER triggers

Parameters

  • obs_config (dict) – Observation settings

  • output_dir (str) – Output directory for data products

  • log_queue (Queue) – Queue to use for logging

  • input_queue (Queue) – Input queue for triggers

  • output_queue (Queue) – Output queue for clusters

  • ncluster (mp.Value) – 0

  • config_file (str) – Path to config file

  • obs_name (str) – Observation name to use in log messages

_cluster(triggers)

Execute trigger clustering

Parameters

triggers (np.ndarray) – Input triggers, columns: DM, S/N, time, integration_step, sb

_load_config()

Load configuration

run()

Main loop

stop()

Stop this thread

Filterbank reader

class darc.processor_tools.filterbank_reader.ARTSFilterbankReader(fname, cb, ntab=12, config_file='/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/darc/config.yaml')

Bases: object

Filterbank reader for ARTS data, one file per TAB

Parameters

  • fname (str) – path to filterbank files, with {cb:02d} and {tab:02d} for CB and TAB indices

  • cb (int) – CB index

  • ntab (int) – Number of TABs (Default: NTAB from constants)

  • config_file (str) – Path to config file, passed to SBGenerator

get_header(tab=0)

Read filterbank parameters to self.header

Parameters

tab (int) – TAB index (Default: 0)

get_sb(sb)

Construct an SB. TAB data must be read before calling this method

Parameters

sb (int) – SB index

Returns

Spectra object with SB data

load_single_sb(sb, startbin, chunksize)

Convenience tool to read only a single SB and its associated TABs. Note: Any internal TAB data is cleared after calling this method

Parameters

  • sb (int) – SB index

  • startbin (int) – Index of first time sample to read

  • chunksize (int) – Number of time samples to read

Returns

Spectra object with SB data

read_filterbank(tab, startbin, chunksize)

Read a chunk of filterbank data

Parameters

  • tab (int) – TAB index

  • startbin (int) – Index of first time sample to read

  • chunksize (int) – Number of time samples to read

Returns

chunk of data with shape (nfreq, chunksize)

read_tabs(startbin, chunksize, tabs=None)

Read TAB data

Parameters

  • startbin (int) – Index of first time sample to read

  • chunksize (int) – Number of time samples to read

  • tabs (list) – which TABs to read (Default: all)

exception darc.processor_tools.filterbank_reader.ARTSFilterbankReaderError

Bases: Exception

class darc.processor_tools.spectra.Spectra(freqs, dt, data, starttime=0, dm=0)

Bases: object

A class to store spectra. This is mainly to provide reusable functionality.

Spectra constructor

Parameters

  • freqs (list) – Observing frequencies for each channel.

  • dt (float) – Sampling time (seconds)

  • data (array) – A 2D numpy array containing pulsar data. Axis 0 should contain channels. (e.g. data[0,:]) Axis 1 should contain spectra. (e.g. data[:,0])

  • starttime (float) – Start time (in seconds) of the spectra with respect to the start of the observation. (Default: 0).

  • dm (float) – Dispersion measure (in pc/cm^3). (Default: 0)

Returns

Spectra object

dedisperse(dm=0, padval=0)

Shift channels according to the delays predicted by the given DM. * Dedispersion happens in place *

Parameters

  • dm (float) – The DM (in pc/cm^3) to use.

  • padval (float/str) – The padding value to use when shifting channels during dedispersion. See documentation of Spectra.shift_channels. (Default: 0)

downsample(factor=1, trim=True)

Downsample the spectra by co-adding ‘factor’ adjacent bins. * Downsampling is done in place *

Parameters

  • factor (int) – Reduce the number of spectra by this factor. Must be a factor of the number of spectra if ‘trim’ is False.

  • trim (bool) – Trim off excess bins.

get_chan(channum)
get_spectrum(specnum)
masked(mask, maskval='median-mid80')

Replace masked data with ‘maskval’. Returns a masked copy of the Spectra object.

Parameters

  • mask (array) – An array of boolean values of the same size and shape as self.data. True represents an entry to be masked.

  • maskval (str) –

    Value to use when masking. This can be a numeric value, ‘median’, ‘mean’, or ‘median-mid80’.

    The values ‘median’ and ‘mean’ refer to the median and mean of the channel, respectively. The value ‘median-mid80’ refers to the median of the channel after the top and bottom 10% of the sorted channel is removed. (Default: ‘median-mid80’)

Returns

maskedspec: A masked version of the Spectra object.

scaled(indep=False)

Return a scaled version of the Spectra object. When scaling subtract the median from each channel, and divide by global std deviation (if indep==False), or divide by std deviation of each row (if indep==True).

Parameters

indep (bool) – If True, scale each row independently (Default: False).

Returns

scaled_spectra: A scaled version of the Spectra object.

scaled2(indep=False)

Return a scaled version of the Spectra object. When scaling subtract the min from each channel, and divide by global max (if indep==False), or divide by max of each row (if indep==True).

Parameters

indep (bool) – If True, scale each row independently (Default: False).

Returns

scaled_spectra: A scaled version of the Spectra object.

shift_channels(bins, padval=0)

Shift each channel to the left by the corresponding value in bins, an array. * Shifting happens in-place *

Parameters

  • bins (array) – An array containing the number of bins to shift each channel by.

  • padval (float/str) –

    Value to use when shifting near the edge of a channel. This can be a numeric value, ‘median’, ‘mean’, or ‘rotate’.

    The values ‘median’ and ‘mean’ refer to the median and mean of the channel. The value ‘rotate’ takes values from one end of the channel and shifts them to the other.

smooth(width=1, padval=0)

Smooth each channel by convolving with a top hat of given width. The height of the top had is chosen shuch that RMS=1 after smoothing. Overlap values are determined by ‘padval’. This bit of code is taken from Scott Ransom’s PRESTO’s single_pulse_search.py (line ~ 423). * Smoothing is done in place. *

Parameters

  • width (int) – Number of bins to smooth by (Default: no smoothing)

  • padval (float/str) – Padding value to use. Possible values are float-value, ‘mean’, ‘median’, ‘wrap’. (Default: 0).

subband(nsub, subdm=None, padval=0)

Reduce the number of channels to ‘nsub’ by subbanding. The channels within a subband are combined using the DM ‘subdm’. ‘padval’ is passed to the call to ‘Spectra.shift_channels’. * Subbanding happens in-place *

Parameters

  • nsub (int) – Number of subbands. Must be a factor of the number of channels.

  • subdm (float) – The DM with which to combine channels within each subband (Default: don’t shift channels within each subband)

  • padval (float/str) – The padding value to use when shifting channels during dedispersion. See documentation of Spectra.shift_channels. (Default: 0)

trim(bins=0)

Trim the end of the data by ‘bins’ spectra. * Trimming is done in place *

Parameters

bins (int) – Number of spectra to trim off the end of the observation. If bins is negative trim spectra off the beginning of the observation.

darc.processor_tools.spectra.delay_from_DM(DM, freq_emitted)

Return the delay in seconds caused by dispersion, given a Dispersion Measure in cm-3 pc, and the emitted frequency of the pulsar in MHz.

Parameters

  • DM (float) – dispersion measure

  • freq_emitted (float) – frequency

darc.processor_tools.spectra.rotate(arr, bins)

Return an array rotated by ‘bins’ places to the left

Parameters

  • arr (list) – Input data

  • bins (int) – Number of bins to rotate by

Data extraction

class darc.processor_tools.extractor.Extractor(obs_config, output_dir, log_queue, input_queue, output_queue, ncand_above_threshold, config_file='/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/darc/config.yaml', obs_name='')

Bases: Process

Extract data from filterbank files

Parameters

  • obs_config (dict) – Observation settings

  • output_dir (str) – Output directory for data products

  • log_queue (Queue) – Queue to use for logging

  • input_queue (Queue) – Input queue for clusters

  • output_queue (Queue) – Output queue for classifier

  • ncand_above_threshold (mp.Value) – 0

  • config_file (str) – Path to config file

  • obs_name (str) – Observation name to use in log messages

_extract(dm, snr, toa, downsamp, sb)

Execute data extraction

Parameters

  • dm (float) – Dispersion Measure (pc/cc)

  • snr (float) – AMBER S/N

  • toa (float) – Arrival time at top of band (s)

  • downsamp (int) – Downsampling factor

  • sb (int) – Synthesized beam index

_load_config()

Load configuration

_rficlean()

Clean data of RFI

_store_data(fname, sb, tsamp, dms, params_amber, params_opt)

Save candidate to HDF5 file

Parameters

  • fname (str) – Output file

  • sb (int) – Synthesized beam index

  • tsamp (float) – Sampling time (s)

  • dms (Quantity) – array of DMs used in DM-time array

  • params_amber (tuple) – AMBER parameters: dm, snr, toa

  • params_opt (tuple) – Optimized parameters: snr, toa

init_filterbank_reader()

Initialize the ARTS filterbank reader

run()

Main loop

stop()

Stop this thread

Neural network classification of candidates

class darc.processor_tools.classifier.Classifier(log_queue, input_queue, conn, config_file='/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/darc/config.yaml', obs_name='')

Bases: Process

Classify candidates from HDF5 files produced by Extractor

Parameters

  • log_queue (Queue) – Queue to use for logging

  • input_queue (Queue) – Input queue for triggers

  • conn (Connection) – Pipe connection to send output to

  • config_file (str) – Path to config file

  • obs_name (str) – Observation name to use in log messages

_classify(fname)

Classify a candidate

Parameters

fname (str) – Path to HDF5 file containing candidate data and metadata

_init_models()

Load the keras models

_load_config()

Load configuration

_load_tensorflow()

Load tensorflow into local namespace

_prepare_data()

Verify data shape and downsampled as needed

Returns

success (bool)

run()

Main loop

stop()

Stop this thread

Candidate visualization

exception darc.processor_tools.visualizer.ProcessorException

Bases: Exception

class darc.processor_tools.visualizer.Visualizer(output_dir, result_dir, log_queue, obs_config, files, config_file='/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/darc/config.yaml', obs_name='')

Bases: object

Visualize candidates

Parameters

  • output_dir (str) – Output directory for data products

  • result_dir (str) – central directory to copy output PDF to

  • log_queue (Queue) – Queue to use for logging

  • obs_config (dict) – Observations settings

  • files (list) – HDF5 files to visualize

  • config_file (str) – Path to config file

  • obs_name (str) – Observation name to use in log messages

_get_plot_order()

Get the order of files to plot them in descending freq-time probability order, then by S/N if probabilities are equal

Returns

file order (np.ndarray)

_load_config()

Load configuration

Returns

config (Namespace)

_load_data(fname, data_type)

Load HDF5 data

Parameters

  • fname (str) – Path to HDF5 file

  • data_type – which data type to get, options: freq_time, dm_time, 1d_time

Returns

data (np.ndarray), params (dict with tsamp, dm, snr, toa, downsamp, sb, dms)

_visualize()

Run the visualization of candidates