FLEET: A Redshift-agnostic Machine Learning Pipeline to Rapidly Identify Hydrogen-poor Superluminous Supernovae

Type I superluminous supernovae (hereafter SLSN-I) are a class of astrophysical transients that exceed the luminosity of normal SNe by up to two orders of magnitude. They were originally classified based on their luminosity, since most have typical peak absolute magnitudes of ≲−21 (Chomiuk et al. 2011; Quimby et al. 2011). However, events with spectroscopic signatures that match those of SLSN-I have been discovered at lower luminosities (e.g., Lunnan et al. 2013) and they are now classified based on their hydrogen-free spectra, strong O ii absorption lines at early time, and a blue continuum (Angus et al. 2019). At present, about 150 SLSN-I have been spectroscopically classified; see Table A1 for a listing and references.

While the energy source of SLSN-I was intensely debated for a few years following their discovery, it now appears that radioactive decay of ⁵⁶Ni (as in normal Type I SNe) and circumstellar interaction (as in Type IIn SNe) cannot explain the bulk of the population. Instead, the most likely energy source appears to be the spindown of a millisecond magnetar produced in the explosion (Kasen & Bildsten 2010; Metzger et al. 2015). This model can explain the diverse light-curve behavior (Nicholl et al. 2017c), the early-time UV spectra (Mazzali et al. 2016), the late-time light-curve flattening (Blanchard et al. 2018; Nicholl et al. 2018), and the nebular spectra (Dessart et al. 2012; Nicholl et al. 2019) of SLSN-I. Still, the nature of SLSN-I progenitors, their environments, and their relation to those of other stripped-envelope explosions remain areas of active investigation (e.g., Blanchard et al. 2020). Similarly, the ubiquity and origin of unusual light-curve and spectroscopic features seen in some SLSN-I, such as late-time "bumps" (Nicholl et al. 2016; Inserra et al. 2017; Blanchard et al. 2018; Lunnan et al. 2019), double-peaked light curves (Nicholl et al. 2015), or potential helium lines (Yan et al. 2020) remain unclear.

Making progress on these open questions Requires a substantial increase in the identification rate of SLSN-I, preferably at early times to enable spectroscopic follow-up. A significant challenge is that SLSN-I are intrinsically rare; at a volumetric rate of ∼90 SNe yr⁻¹ Gpc⁻³ at a weighted redshift of z = 1.13, they represent <0.1% of the core-collapse SN rate (Prajs et al. 2017). Even accounting for their larger discovery volume they represent only $\sim 1.5 \%$ of the detection rate in magnitude-limited surveys (Villar et al. 2019; Fremling et al. 2020). Currently, only $\sim 10 \%$ of all optical transients are classified spectroscopically, and with the Legacy Survey of Space and Time (LSST) on the Vera C. Rubin Observatory, this will decline to ≲0.1%. Thus, efficient and rapid selection of SLSN-I candidates is essential.

One approach to identifying SLSN-I candidates is to use general purpose machine learning (ML) classification algorithms that attempt to sort optical transients into various spectroscopic classes. Some of these (e.g., RAPID: Muthukrishna et al. 2019, Avocado: Boone 2019) have been trained on synthetic data, such as the Photometric LSST Astronomical Time-series Classification project (PLAsTiCC; Kessler et al. 2019), but their performance with real data remains untested. Other classifiers such as SuperRAENN (Villar et al. 2020) or Superphot (Villar et al. 2019; Hosseinzadeh et al. 2020) have been trained on real survey data from the Pan-STARRS1 Medium Deep Survey (PS1/MDS). Overall, these classifiers have a fairly high success rate and recover ∼80% of SLSN-I, but only when using redshift information and fairly complete light curves. Additionally, the Automatic Learning for the Rapid Classification of Events (ALeRCE) broker, which is currently providing real-time classifications for transients from ZTF (Sánchez-Sáez et al. 2020), is able to recover up to 100% of the SLSN-I in their training sample, but with a large standard deviation of $\sim 26 \%$ for the predicted classification, which they estimate by running 20 versions of their classifier.

An alternative approach, which we develop and use in this paper, is to devise a classification algorithm that is optimized specifically for SLSN-I. In Blanchard (2019) we introduced an initial simple algorithm that improved SLSN-I selection from the random $\sim 1.5 \%$ to $\sim 20 \%$, using the brightness contrast between a transient and its host galaxy. This approach yielded other unusual transients as well (Blanchard et al. 2017; Gomez et al. 2019; Nicholl et al. 2020). Here, we describe a more sophisticated machine learning algorithm that utilizes light-curve and contextual information to enable efficient real-time SLSN-I selection without the need for redshift information. This classifier is the core of our Finding Luminous and Exotic Extragalactic Transients (FLEET) observational program. We find that this targeted approach achieves an overall higher success rate than all-encompassing classifiers.

The structure of the paper is as follows. In Section 2 we introduce and motivate the philosophy behind our approach. In Section 3 we present the data set used to train our algorithm. In Section 4 and Section 5 we outline the contextual and light-curve information used for classification, respectively. In Section 6 we describe the ML algorithm and the classification results. In Section 7 we present our alternative classifiers that use redshifts and full light curves as additional information. Finally, we summarize our conclusions in Section 8. FLEET is provided as a Python package on Github⁵ and Zenodo (Gomez et al. 2020), as well as included in the Python Package Index with the name fleet-pipe.

As discussed above, there are several efforts aimed at ML classification of astronomical transients, mainly based on light-curve information from wide-field surveys. By design, some classifiers make choices that tend to optimize their overall classification success rate across a range of astronomical transients (e.g., Boone 2019; Muthukrishna et al. 2019; Gagliano et al. 2020; Hosseinzadeh et al. 2020; Villar et al. 2020). Here, we take a distinct approach by focusing on optimized classification of a single class of transients. Our algorithm is based on the following guiding principles:

• 1.  
  Classifying only SLSN-I with no regard for the classification success of other transients.
• 2.  
  Obtaining the purest possible sample of SLSN-I, at the expense of sample completeness.
• 3.  
  Prioritizing speed and computational resources over model complexity to allow for rapid classification.
• 4.  
  Finding SLSN-I at early times to enable real-time follow-up.

This approach enables us to make efficient use of large-aperture telescopes for spectroscopic classification, as well as perform later follow-up studies.

At the present, most transients are reported to the Transient Name Server (TNS), a repository for transient discoveries and classifications. FLEET is designed to assign any transient reported to the TNS a classification probability of being a SLSN-I. The current rate of ∼1500 transients per month reported to the TNS (and $\gtrsim {10}^{5}$ per month expected from LSST) motivates our emphasis on computational speed, as well as purity at the expense of completeness. In particular, even if we manage to identify less than half of the SLSN-I in the data stream, but with a high success rate, then we can double the existing sample of SLSN-I by the time LSST commences.

We provide a main rapid version of the classifier in addition to two additional classifiers with somewhat different motivations: first, a classifier that uses redshift information for higher purity classification, mainly in anticipation of robust photometric redshifts that will be provided by LSST; and second, a full light-curve classifier that can more confidently classify SLSN-I, mainly aimed at constructing large samples with only photometric data.

To train our classifier we obtained all spectroscopically classified transients from the TNS: SNe, tidal disruption events (TDEs), active galactic nuclei (AGN) flares, and Galactic transients (e.g., cataclysmic variables and variable stars). In addition to those, we included the TDEs published in van Velzen et al. (2020), which are not yet reported to the TNS, and every unambiguous SLSN-I from the literature; see Table A1. We also obtained all of the available photometry for each transient, from the Open Supernova Catalog (OSC; Guillochon et al. 2017) or the Zwicky Transient Facility (ZTF; Bellm et al. 2019). We require each transient to have at least 2 g-band and 2 r-band measurements to model their light curves. We restrict the list to transients within the footprint of the Pan-STARRS1 3π (PS1/3π) survey (Chambers & Pan-STARRS Team 2018) for the purpose of identifying host galaxies. Finally, we removed from the training set 44 transients with ambiguous host galaxy identifications or spurious data in order to have the cleanest data set possible; however, we kept these events in our test set to analyze any resulting biases. The resulting sample is composed of 1813 transients, with the following distinct labels from the TNS: 800 SN Ia, 381 SN II, 156 SLSN-I, 95 CV, 71 SN IIn, 63 SN IIP, 59 SN Ic, 43 SLSN-II, 37 SN Ib, 33 SN IIb, 19 TDE, 16 SN Ic-BL, 13 SN Ibc, 12 AGN, 8 SN Ibn, and 7 Varstar (variable stars).

Since the number of events per class varies substantially, making the training set unbalanced, the classification would be biased toward the more common classes. To mitigate this bias we oversample each class to have a total of 800 events, using the Synthetic Minority Over-sampling Technique (SMOTE; Chawla et al. 2002). This algorithm draws random samples along vectors joining every pair of objects in feature space until all classes have the same number of events. We tested an alternative multivariate-Gaussian (MVG) oversampling technique, as implemented in Villar et al. (2019), but find that when sampling features that are close to zero and constrained to be positive (e.g., redshift), SMOTE performs significantly better, even when imposing a >0 threshold for the samples, or sampling in log-space.

Since some of the classes in our sample are too small to be properly oversampled, we experiment by grouping different sets of transients together, not only to allow for oversampling but to attempt to optimize the success of the classifier at finding SLSN-I and to improve computational efficiency. We find that the best performing grouping is Varstar+CV, TDE+AGN, SN II+SN IIP, SN Ib+SN Ibn+SN Ibc+SN Ic+SN Ic-BL, SLSN-I, SLSN-II, SN IIb, SN IIn, and SN Ia. We stress that since our interest is in classifying SLSN-I with high purity, the grouping and classification success of the other classes are not critical. Still, it is interesting to note that the optimized groupings are indeed related in terms of underlying physics.

3.1. Test Set

We test the efficacy of our classifier on all of the events from the training set. In addition to all the events from the training set, we include in the test set the 44 transients that were removed in Section 3 to avoid introducing possible biases. We implement a leave-one-out cross-validation method, allowing us to train the classifier on every event except for one, and then predict the classification of that one event, cycling through all events. This allows us to robustly test our classifier without having to divide the data set into a training and test set, which would compromise the sample size.

We define completeness, classifier purity, and observed purity as useful metrics to test the efficacy of our algorithm:

$$\begin{eqnarray}\begin{array}{rcl}\mathrm{Completeness} & = & \displaystyle \frac{{\mathrm{SN}}_{{\rm{T}}}}{{{\rm{N}}}_{\mathrm{SLSN}}}\\ \mathrm{Classifier}\ \mathrm{Purity} & = & \displaystyle \frac{{\mathrm{SN}}_{{\rm{T}}}}{{\mathrm{SN}}_{{\rm{T}}}+{\mathrm{SN}}_{{\rm{F}}}}\\ \mathrm{Observed}\ \mathrm{Purity} & = & \displaystyle \frac{{\mathrm{SN}}_{{\rm{T}}}}{{\mathrm{SN}}_{{\rm{T}}}+{\displaystyle \sum }_{i}{\eta }_{i}{\mathrm{SN}}_{{\rm{F}},{\rm{i}}}}\\ {\eta }_{i} & = & \displaystyle \frac{{{\rm{N}}}_{\mathrm{SLSN}}\times {f}_{i}}{{N}_{i}\times {f}_{\mathrm{SLSN}}},\end{array}\end{eqnarray} \tag{ 1 }$$

where ${{\rm{N}}}_{\mathrm{SLSN}}$ is the total number of SLSN-I in the test set, ${\mathrm{SN}}_{{\rm{T}}}$ is the total number of true positive SLSN-I recovered, and ${\mathrm{SN}}_{{\rm{F}}}$ is the total number of false-positive SLSN-I. The relative fractions of each transient class in our test set, which we obtained directly from the TNS, do not reflect the true fractions of these transients in a magnitude-limited survey. To determine a purity that is representative of ongoing and future surveys, we renormalize the classifier purity into an observed purity, which more accurately represents the outcome of our pipeline in a real survey. Here, ${\mathrm{SN}}_{F,i}$ is the false-positive rate for an individual transient class i, and f_i is the corresponding true observational rate for that class, listed in Table 1. We use the observational rates of SNe from Fremling et al. (2020) to estimate the expected Target Rate, f, for any magnitude-limited survey. We then include nuclear transients (TDEs + AGN) and Galactic transients (CVs + variable stars) from the TNS, normalizing by the total number of classified transients from the TNS to the total number of SNe in the Fremling et al. (2020) sample.

Table 1.  Observational Rates of Transients

Transient	Fremling	TNS	Target f
SNI	587 (77.1%)	6500 (70.8%)	73.9%
SNII	155 (20.4%)	2109 (23.0%)	19.6%
SLSN-I	12 (1.6%)	123 (1.3%)	1.5%
SLSN-II	7 (0.9%)	45 (0.5%)	0.9%
Nuclear	⋯	58 (0.6%)	0.6%
Star	⋯	340 (3.7%)	3.5%

Note. Total number of each of the relevant classes of transients from the Fremling et al. (2020) sample and the TNS sample. In parentheses is the percent of the total sample each class represents. We normalize the rate of events in our test set to an expected Target rate f calculated from the Fremling et al. (2020) sample and the TNS sample, used for Equation (1).

Download table as:  ASCIITypeset image

Given that SLSN-I are over-represented in our test set compared to the rate they would have in a magnitude limited survey, observed purity will be lower than the classifier purity. For example, our test set has 800 SN Ia and 156 SLSN-I, or 0.20 SLSN-I for each SN Ia. But in a magnitude-limited survey, there is typically only 0.02 SLSN-I for each SN Ia. Therefore, if we wanted to predict how many SLSN-I we would be able to find in a real survey, we need to normalize the classifier purity by, in this example, multiplying the false-positive rate by a factor of ∼10.

SLSN-I are known to prefer low-luminosity galaxies (Lunnan et al. 2014), and it is therefore advantageous to use contextual information in their classification. Here we describe our method of assigning a host galaxy to each transient, while in Section 6.1 we explore which host galaxy properties are the most useful features in the SLSN-I classification. For each transient in our training set we obtain PS1/3π grizy (Chambers & Pan-STARRS Team 2018) and SDSS ugriz (Alam et al. 2015; Ahumada et al. 2020) PSF and Kron magnitudes of every cataloged source in a $1^{\prime}$ radius region around the transient location. We use this information both to separate galaxies from stars and to identify the most likely host galaxy.

4.1. Star–Galaxy Separation

The first step to identifying the host galaxy of each transient is to separate stars from galaxies. SDSS provides a classification for every object in their catalog, but since SDSS is shallower than PS1/3π and has a smaller footprint, this is not sufficient for our purposes. Instead, we develop a method to assign a probabilistic value (between 0 and 1) of how likely every object in SDSS and PS1/3π is to be a galaxy.

To train our star–galaxy separation algorithm we use data from the Canada–France–Hawaii Telescope Legacy Survey (CFHTLS; Hudelot et al. 2012), which provides magnitudes and star–galaxy classifications down to $\approx 26$ mag, significantly deeper than SDSS and PS1/3π. We specifically use the D1 field (1 deg²) and cross match with every overlapping object in SDSS and PS1/3π, for a total of ∼23,000 objects. Galaxies tend to have a larger difference between their PSF and Kron magnitudes than stars, so we use this specific feature (PSF-Kron) to separate them; see Figure 1 for an example in the i-band. The CFHTLS uses the CLASS_STAR classifier flag in SExtractor to separate stars from galaxies, which relies on a multilayer feed forward neural network (Bertin & Arnouts 1996).

Zoom In Zoom Out Reset image size

In our galaxy-star separator we assign a probability of being a galaxy to any object in SDSS or PS1/3π by using a custom k-nearest-neighbors algorithm. Given an object's PSF and Kron magnitude, we find the 20 nearest objects in the PSF versus PSF-Kron phase-space (Figure 1) to calculate its probability of being a galaxy based on the fraction of those 20 neighbors from the CFHTLS training set that are galaxies. Experimenting with different numbers of neighbors, we find that at least 10 neighbors are required to produce robust estimates, with only marginal improvement in accuracy beyond 20 neighbors. For every object we calculate its probability of being a galaxy in every available filter, and adopt the average probability among all filters.

An alternative star–galaxy separator for objects in PS1/3π is presented in Tachibana & Miller (2018). Although this latest one has a very high accuracy, it does not include objects from SDSS, for which we also require a classification when they are not in the PS1/3π catalog. We note that if we label objects with a probability of being a galaxy of ${P}_{G}\leqslant 10 \%$ as stars, our classifier agrees with the classification from Tachibana & Miller (2018) at the 90% level. In Figure 2 we show an example of our star–galaxy separator applied on a field from PS1/3π centered on the location of the SLSN-I SN 2013hy.

Zoom In Zoom Out Reset image size

We opt to only label objects with a galaxy probability of ${P}_{G}\lt 10 \%$ as stars to avoid missing a possible host galaxy identification. While this conservative cut retains more stars in the sample, these are rarely predicted to be the most likely host galaxy of a SN due to the small size of their PSF. We find that a more strict threshold results in a large number of host galaxies being rejected as stars. In the top panel of Figure 1 we show that using the classification from the CFHTLS as a reference, our threshold for labeling stars yields a successful galaxy classification for essentially all objects with ≳22 mag and ≈65% down to 23 mag.

4.2. Host Identification

Once we have identified which objects in the field are likely to be galaxies we can determine which galaxy is the most likely host for a given transient. First, we label stellar transients, using the criterion of a star (i.e., ${P}_{G}\lt 10$%) being located <1'' from a transient's position. Then, for the nonstellar transients we determine the probability of chance coincidence for each galaxy in the field relative to the transient's position. We follow the method of Bloom et al. (2002) and Berger (2010) using the measured number density of galaxies, ${\rm{\Sigma }}(\leqslant m)$, brighter than a magnitude m, to calculate the probability of chance coincidence:

$$\begin{eqnarray}\begin{array}{rcl}{P}_{\mathrm{cc}} & = & 1-{e}^{-\pi \left({d}^{2}+4{R}^{2}\right){\rm{\Sigma }}(\leqslant m)}\\ {\rm{\Sigma }}(\leqslant m) & = & \displaystyle \frac{{10}^{0.33(m-24)-2.44}}{0.33\,\mathrm{ln}(10)},\end{array}\end{eqnarray} \tag{ 2 }$$

where d is the angular separation between the center of a galaxy and the transient, and R is the half-light radius of the galaxy obtained from the SDSS catalog, or from the PS1/3π catalog if the object is not in the SDSS catalog. We consider the galaxy with the lowest value of ${P}_{\mathrm{cc}}$ to be the host, as long as ${P}_{{cc}}\leqslant 0.1$. Otherwise, we designate the transient as "hostless" given the more likely situation that its host galaxy is fainter than the magnitude limit of SDSS and PS1/3π.

In addition to contextual information, we use the light curves of each transient to predict which transients are most likely SLSN-I. We obtain photometric data from the OSC, as well as from ZTF using the Make Alerts Really Simple (MARS)⁶ broker. We correct all the photometry for Galactic extinction using the Schlafly & Finkbeiner (2011) dust maps assuming R_V = 3.1.

Since we are interested in gross features of the light curves, rapid classification, and identifying only SLSN-I (rather than robustly classifying all transient classes), we use a simple exponential light-curve model:

$$\begin{eqnarray}&&m={e}^{W(t-\phi )}-A\times W(t-\phi )+{m}_{0},\end{eqnarray} \tag{ 3 }$$

where W is the effective width of the light curve, A modifies the decline time relative to the rise time, m₀ is the peak magnitude, and ϕ is a phase offset relative to the time of the first observation. An example of this function fit to a SLSN-I (SN 2011ke) is shown in Figure 3. We fit this model independently to the g- and r-band light curves using the emcee implementation of the Goodman and Weare (Goodman & Weare 2010) Markov chain Monte Carlo algorithm (Foreman-Mackey et al. 2013) and adopt the median of the posterior as the best estimate for each parameter. We use flat priors for all parameters: $W=[0,10]$, ${W}_{2}=[0.01,1.0]$, $\phi =[-50,50]$, and ${m}_{0}=[-30,30];$ but initiate the walkers' position at a value of m₀ equal to the brightest observed magnitude, and a value of ϕ that corresponds to the time of that measurement. We find that a model with 50 walkers and 500 steps converges and provides good results for the majority of transients, with a typical autocorrelation time of ∼30 steps.

Zoom In Zoom Out Reset image size

We use two versions of Equation (3) to test and evaluate the classifier. One version has a fixed value of A = 0.6 (the mean value from fitting all of the SLSN-I light curves up to a timescale of 70 days postdiscovery), and is used for the rapid version of FLEET, which only uses the first 20 days of data (described in Section 6). We note that the actual choice of A has only a marginal effect on the results, since this model only uses data up to 20 days after detection, which do not encompass a decline phase. The second version of the model uses data up to 70 days after discovery and has A as a free parameter to fit the light-curve decline. This model is used for the full light-curve classifier described in Section 7.2. In Figure 3 we show both versions of the model, using only the first 20 days of data (fixed A) and 70 days of data (A as a free parameter).

To classify the transients we use the contextual and light-curve information described in Section 4 and Section 5, respectively, with an implementation of the random forest (RF) algorithm in the scikit-learn Python package (Pedregosa et al. 2012). In this manner, we assign to each transient a classification probability of being an SLSN-I. This algorithm takes various subsamples of the training set and forms a number of decision tree classifiers to classify each object. The output classification probability is the result of averaging the output of all the trees in the forest. We run the classifier with 100 estimators to mitigate overfitting and improve predictive accuracy. We also run each version of the model 25 times using different initial random seeds to estimate the classifier's uncertainties. We run the classifier using the Gini index as the criterion that minimizes the probability of misclassification. We optimize the depth of the trees in each RF by running a grid of models from a depth of 3 to 12 in steps of 1 and find a depth of 7 performs best (a depth of 6 and 8 performed similarly well, within a 1σ uncertainty derived from the different random seed iterations).

Additionally, we optimize the grouping of transient classes into different sets, described in Section 3.1. We find that for the most part grouping different classes of SNe together (e.g., SN IIn and SN II) or separating SNe into distinct classes (e.g., SN Ib and SN Ic) provides very similar results, with the one exception of grouping all SNe that are not SLSNe into one group, which produces a much lower purity.

6.1. Feature Selection

Unlike newly discovered transients, the transients in our training set have full light curves. Since a goal of FLEET is to find SLSN-I in real time we test the algorithm using a varying cutoff time for the light-curve data. Naturally, with more data the light-curve models are better constrained, but this delays the identification and spectroscopic follow-up into a later phase when the SN is fainter. We find that using the first 20 days of data provides the best results, where using less data significantly reduces the classifier purity, but including more data provides no improvement (within 1σ uncertainties). After the first 20 days of data most SLSN-I have not reached their peak luminosity.

For the rapid classifier we have six light-curve parameters (three in each filter) that can be used as input features: the widths of the light curve, W, the phase offsets ϕ, and the peak magnitudes m₀. In addition to these we explore the use of two additional features: (i) Δt, which is the time difference from first detection of a transient to its observed light-curve peak in either g- or r-band, whichever one is brightest; and (ii) the g − r color at peak, using the model fits, where the time of peak is the one with the brightest observed magnitude in either the g- or r-band.

For the contextual information features we test the use of several host galaxy parameters: the apparent magnitude of the host, m_h, its half-light radius in the r-band, R, the projected angular separation between the transient and its host center D, the projected angular separation normalized by the galaxy radius R_n, and the difference between m₀ and m_h in the r-band, ${\rm{\Delta }}m$. For hostless transients we use the limiting magnitude of PS1/3π of r = 23.2 as an upper limit on m_h, and set all other galaxy parameters to 0 (since those cannot be measured for a nondetected host).

We tested several combinations of the available light-curve and contextual features in order to determine which combination set yields the highest purity of SLSN-I, while maintaining reasonable completeness; see Table 2. We find that the most relevant features that help separate SLSN-I from other transients are W_g and W_r, Δt, R_n, ${\rm{\Delta }}m$, and (g−r). In Figure 4 we show how the different classes of transients lie in feature space. In Table 2 we list the highest purity and associated uncertainty achieved for each feature set, as well as the corresponding completeness and classification confidence, P(SLSN-I), at which this highest purity is achieved.

Zoom In Zoom Out Reset image size

Table 2.  Feature Sets

#	Features	Peak Classifier Purity	Completeness	P(SLSN-I)
1	W+Δt+Rn+ ${\rm{\Delta }}m$	83.0 ± 2.0%	16.3 ± 1.8%	0.73
2	W+Δt+ ${\rm{\Delta }}m$	73.3 ± 40.8%	2.6 ± 1.7%	0.81
3	W+Δt+Rn	81.5 ± 2.5%	20.4 ± 1.3%	0.84
4	Rn+${\rm{\Delta }}m$	98.7 ± 6.5%	4.1 ± 1.4%	0.89
5	W+Rn	75.5 ± 4.9%	16.1 ± 1.8%	0.74
6	W+Δt+Rn+(g−r)	91.2 ± 2.4%	17.9 ± 3.3%	0.80
7	W+Δt+Rn+ ${\rm{\Delta }}m$+(g−r)	82.0 ± 25.4%	2.2 ± 0.9%	0.88
8	W+Rn+ ${\rm{\Delta }}m$+(g−r)	94.7 ± 12.2%	3.0 ± 0.7%	0.87

Note. Different sets of light-curve and contextual features used to train our classifier. We list the highest classifier purity that each set of features achieves, as well as the corresponding completeness and classification probability P(SLSN-I) that correspond to that peak purity. W is the width of the light curve, R_n is the normalized host separation, ${\rm{\Delta }}m$ is the peak transient magnitude minus the host magnitude, Δt is the time of peak magnitude minus the time of discovery, and (g−r) is the light-curve color at peak.

Download table as:  ASCIITypeset image

We find that for the rapid classifier, feature set #6 from Table 2 performs best in terms of purity, while retaining a reasonable completeness. This set contains the width of the light curve W in g- and r-band, the normalized host separation R_n, the time of peak magnitude minus the time of discovery in either band, Δt, and the light-curve color at peak, (g−r).

The importance of each feature used is not defined independently of other features; if two features are correlated then their relative importance might be affected. In the bottom panel of Figure 5 we show the correlation between features, and find that with the exception of W_g and W_r, the features are mostly independent. In order to calculate the correlated importance we use the permutation importance method described in Breiman (2001). The correlated importance of each feature is shown in the top panel of Figure 5.

Zoom In Zoom Out Reset image size

In Figure 6 we show how the rapid version of the classifier (trained on the first 20 days of light-curve data) performs as a function of days of light-curve data used, and include the contaminating classes of transients. When considering the top 20 transients with the highest predicted confidence P(SLSN-I), we find that the classifier performance rises for the first ∼20 days, and then plateaus to a peak classifier purity of about 90% (i.e., we correctly identify about 18 of the top 20 transients classified as SLSN-I). This purity is relevant for the training set, without normalizing for the observational rates described in Section 3.1. The remaining 10% of misclassified events are SLSN-II and SNII. We are generally less concerned about misclassifying SLSN-II as SLSN-I, since the former are still of scientific interest. The performance of the classifier degrades slightly beyond 70 days, since it is only trained on the rising part of the light curve. If we instead consider the top 40 events predicted to be SLSN-I, we find that the fraction of correctly identified SLSN-I goes down to about 75% (Figure 6).

Zoom In Zoom Out Reset image size

6.2. Model Validation

We use three different methods to evaluate the performance of our classifier: a confusion matrix, a purity/completeness curve, and the fraction of SLSN-I recovered. Unless otherwise stated, in this section the values being reported have been corrected for the observational rates expected in a magnitude-limited survey as described in Section 3.1; i.e., we use the observed purity. Since we are not concerned with the classification of transients other than SLSN-I, we collapse the individual transient classifications into a binary SLSN-I versus nonSLSN-I classification. To calculate the predicted probability of nonSLSN-I for each transient we sum the probabilities of all other transient classes.

In Figure 7 we show how the rapid classifier performs at classifying SLSN-I and not misclassifying other objects, as a function of classification confidence level. We find that most of the misclassified SLSN-I are at P(SLSN-I) ≲ 0.6, with only four misclassified SLSN-I at higher values of P(SLSN-I). The few objects that are true SLSN-I but were misclassified as something else with high confidence are usually SLSN-I with relatively bright host galaxies that got misclassified as Type-II SNe, which have light curves that might also appear broad due to their late-time plateau.

Zoom In Zoom Out Reset image size

The completeness and purity of the rapid classifier for the three top performing feature sets are shown in Figure 8. As expected, the purity increases and the completeness declines as we restrict the sample to events with progressively higher values of classification confidence. For P(SLSN-I) > 0.5, the observed purity is $\approx 50 \%$, with a completeness of ≈30–40%. This represents about a factor of 30 times improvement over a random selection of SLSN-I, which would yield a $\approx 1.5 \%$ success rate in a magnitude-limited survey (Fremling et al. 2020; Villar et al. 2019). The peak observed purity achieved by our classifier is even higher, $\approx 85 \% ;$ however, we note that the completeness achieved at peak observed purity is about 20%. This low completeness level may still be acceptable given that current surveys are reporting ∼18,000 transients a year, assuming an observational rate of 1.5% for SLSN-I (Table 1); a 20% completeness corresponds to ∼55 SLSN-I candidates that could be targeted for follow-up, 85% of which would be expected to be true SLSN-I.

Zoom In Zoom Out Reset image size

In Figure 9 we show the confusion matrix, namely, the label predicted by our classifier compared to the true label of the transient. We impose a confidence cut of $P\gt 0.75$ for either the SLSN-I or nonSLSN-I classes, corresponding to the peak classifier purity (Figure 8); this leads to a sample of 1438 events. We see that 14 out of the 18 transients predicted to be SLSN-I are correctly labeled, indicating a classifier purity of 80%.

Zoom In Zoom Out Reset image size

We run an additional model validation to test for overfitting. Given the relatively small sample size of our data set we cannot split the data set into the traditional training/validation/test sets. Instead, we split the entire data set into two independent sets, a training set (with 1209 objects) and a test set (with 604 objects). We optimize the combination of transient class grouping, depth or the RF trees, and included features using a leave-one-out cross-validation method only on the training set. We find that the best results (in terms of purity and completeness) are consistent with those from the rapid classifier presented in this section, with the exception that a depth of 5 is slightly preferred over a depth of 7 for the RF trees. We then test this classifier on the 604 object test set and find it performs as expected with a maximum classifier purity of 75% and a corresponding completeness of 15% for objects with P(SLSN-I) $\gt \,0.75$.

Running FLEET to classify a new transient takes in the order of 10–20 on a personal computer, and about half the time to rerun on an existing transient once the required catalog data has been downloaded and stored locally. We note that since FLEET is designed to rapidly select the most promising SLSN-I candidates for follow-up, manual vetting of the top candidate events can further increase the sample purity. This is because some candidates might be due to obvious failure modes; for example, an AGN with a highly variable light curve might be classified as a SLSN-I due to its "broad" light curve, but manual inspection will reveal a variable nuclear source that is not SN-like. Another potential failure mode that can be mitigated with manual inspection, is when SDSS and/or PS1/3π report large galaxies as multiple individual sources, causing the classifier to associate the transient with a small dim source, instead of the main galaxy.

To summarize, our rapid classifier, using basic light-curve and contextual information (and no redshift information) can achieve a factor of 30–60 times improvement over random selection for SLSN-I, with a completeness of $\sim 20 \%$.

The rapid version of the FLEET classifier presented above is tailored to find a pure sample of SLSN-I before or near peak, as to enable real-time follow-up. In this section we explore two alternative classifiers that utilize additional information: (i) using redshift as a feature, based on the expectation that LSST will provide photometric redshifts with $\sim 5 \%$ uncertainty for galaxies down to $i\approx 25$ mag (Graham et al. 2018); and (ii) using more complete light-curve information, including the decline phase, which may hinder spectroscopic classification, but will provide samples of SLSN-I for pure photometric population studies. We optimize these alternative classifiers in terms of feature selection, depth of the classifier's trees, and time span of the light curve used in the same manner as for the main rapid classifier, as described in Section 6.

7.1. Redshift Classifier

A key advantage of our rapid classifier is that it does not rely on redshift information. However, with the advent of LSST it is expected that robust photometric redshifts will be available for galaxies down to $i\approx 25$ mag. Since SLSN-I are generally more luminous than other SN classes, redshift information is certain to aid in the classification confidence. In Figure 10 we plot the peak absolute r-band magnitude as a function of redshift for all of the extragalactic transients in our training set, indicating how well SLSN-I can be separated when redshift information is available.

Zoom In Zoom Out Reset image size

To test this effect, we use the known spectroscopic redshift of each transient in our training set (assigning Galactic transients a redshift of 0). As in the rapid classifier, we only use the first 20 days of data (designed to enable rapid follow-up) and optimize for RF depth and features. We find that feature set #6 (Table 2) performs best, with an optimal depth of 9 for the RF trees.

In Figure 11 we show the observed purity and completeness of this classifier as a function of classification confidence. We find that the redshift classifier performs better than the main rapid classifier, for essentially all values of P(SLSN-I), with an observed purity of about 60% and completeness of about 60% at P(SLSN-I) > 0.5 (compared to 50% and 30%, respectively, for the main rapid classifier). The peak purity is effectively 100% with a corresponding completeness of about 15% at P(SLSN-I) > 0.85. We run an additional test on this classifier by adding a random 5% scatter to the redshift values from the testing set and find no measurable difference in the output purity and completeness. This scatter is meant to be representative of the uncertainty of the photometric redshift measurements expected from LSST (Graham et al. 2018).

Zoom In Zoom Out Reset image size

In Figure 12 we show the classifier's performance in terms of the number of top SLSN-I candidates selected. The redshift classifier achieves a purity of 50% for the top $\approx 65$ candidate SLSN-I, significantly higher than the $\approx 27$ candidate SLSN-I at 50% purity for the main classifier. Stated differently, the redshift classifier achieves 80% observed purity for the 27 top candidate SLSN-I, compared to the 50% observed purity for the main classifier. We therefore conclude that when robust redshift information is available it can significantly aid in the purity and completeness of the classifier.

Zoom In Zoom Out Reset image size

7.2. Full Light-curve Classifier

We have presented a random forest classifier, FLEET, designed specifically to rapidly identify SLSN-I with a high purity, without the need for redshift information. We trained this classifier on a sample of about 1800 classified transients reported to the TNS, including 156 SLSN-I (i.e., 8.6% of the total sample). The classifier uses both light-curve and contextual host galaxy information. We assess the observed purity achieved by FLEET for the actual rate of SLSN-I in a magnitude-limited survey of $\approx 1.5 \%$. Our key findings for the rapid FLEET classifier are as follows:

• 1.  
  We find that the most important features are the light-curve width, g − r color at peak, and the projected angular separation between the transient and host galaxy normalized by host radius.
• 2.  
  We find an observed purity of about 50% for events classified as SLSN-I, with a probability confidence of P(SLSN-I) > 0.5. This is a factor of 33 times improvement compared to a random selection (i.e., compared to the fraction of 1.5% of SLSN-I in a magnitude-limited survey). The completeness for this classification confidence threshold is about 30%.
• 3.  
  We find a peak observed purity of about 85% for SLSN-I, corresponding to a classification probability threshold of P(SLSN-I) > 0.80 and a total of ∼15 objects. The completeness for this classification confidence threshold is about 20%.

In addition to the main rapid classifier we also explored two alternative classifiers that use redshift information and full light curves, respectively. As expected, we find that these classifiers achieve better results, with a significant increase in completeness by about a factor of 2, for an observed purity that matches the peak performance of the main rapid classifier.

Placing our results in context we note that at present, current surveys are reporting ∼18,000 transients a year, out of which ∼6000 transients per year have the requisite photometry (minimum of 2 data points in the g- and r-bands) and localization (within the footprint of PS1/3π) to be classified by our algorithm. For an observational SLSN-I fraction of 1.5%, this sample contains about 90 SLSN-I per year. Our rapid classifier can therefore recover about 30 SLSN-I with a purity of 50%, thereby requiring about 60 follow-up spectra per year; or alternatively, about 18 SLSN-I per year with a purity of about 85%, requiring about 21 follow-up spectra. Looking forward to LSST, which is expected to have $\sim {10}^{4}$ SLSN-I in its data stream (Villar et al. 2018), our classifier could discover ∼140 SLSN-I a month, with ∼170 follow-up spectra. This would increase the existing sample by two orders of magnitude over the lifetime of LSST.

The Berger Time-Domain Group is supported in part by NSF grant AST-1714498. V.A.V. acknowledges support from a Ford Foundation Dissertation Fellowship. Operation of the Pan-STARRS1 telescope is supported by the National Aeronautics and Space Administration under grant No. NNX12AR65G and grant No. NNX14AM74G issued through the NEO Observation Program. This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement. This research has made use of NASAs Astrophysics Data System. This research has made use of the SIMBAD database, operated at CDS, Strasbourg, France. Based on observations obtained with MegaPrime/MegaCam, a joint project of CFHT and CEA/IRFU, at the Canada–France–Hawaii Telescope (CFHT) which is operated by the National Research Council (NRC) of Canada, the Institut National des Science de l'Univers of the Centre National de la Recherche Scientifique (CNRS) of France, and the University of Hawaii. This work is based in part on data products produced at Terapix available at the Canadian Astronomy Data Centre as part of the Canada–France–Hawaii Telescope Legacy Survey, a collaborative project of NRC and CNRS. This research has made use of the NASA/IPAC Extragalactic Database, which is funded by the National Aeronautics and Space Administration and operated by the California Institute of Technology.

Facilities: ADS - , TNS - .

Software: Astropy (Astropy Collaboration 2018), extinction((Barbary 2016)), Matplotlib (Hunter 2007), emcee (Foreman-Mackey et al. 2013), NumPy (van der Walt et al. 2011), scikit-learn (Pedregosa et al. 2012), SMOTE Chawla et al. 2002.

We show in Table A1 the sample of all the SLSN-I used for this classifier, sorted by redshift.

Table A1.  Type I SLSNe

Name	Redshift	Reference	Name	Redshift	Reference	Name	Redshift	Reference
SN2017egm	0.0307	49	PS1-5cjz	0.2200	2	DES14C1rhg	0.4810	2
PTF11hrq	0.0571	21	SN2016wi	0.2240	27	SN2019itq	0.4810	this work
SN2018hti	0.0600	62	SN2018gft	0.2300	56	SN2016aj	0.4850	60
SN2019unb	0.0635	1	SN2010gx	0.2301	30	SN2019kwq	0.5000	53
SN2018bgv	0.0795	23	SN2018ffj	0.2340	this work	PTF09atu	0.5015	10
SN2012aa	0.0830	48	SN2018gkz	0.2400	29	PS1-14bj	0.5125	4
SN2019hge	0.0866	61	SN2011kf	0.2450	28	SN2019otl	0.5140	this work
SN2017gci	0.0900	47	iPTF16bad	0.2467	27	PS1-12bqf	0.5220	4
SN2010md	0.0987	10	SN2019enz	0.2550	26a	PS1-11ap	0.5240	4
SN2010kd	0.1010	64	LSQ12dlf	0.2550	25	DES16C3dmp	0.5620	2
SN2016eay	0.1013	46	LSQ14mo	0.2560	24	DES15S1nog	0.5650	2
PTF12hni	0.1056	30	PTF09cnd	0.2584	10	SN2019sgg	0.5726	54
PTF12dam	0.1070	41	SN2019dlr	0.2600	53	SN2019kwu	0.6000	53
SN2019neq	0.1075	45	SN2019hno	0.2600	53	DES14X3taz	0.6080	2
SN2018kyt	0.1080	44	SN2018fd	0.2630	this work	PS1-10bzj	0.6500	4
SN2017ens	0.1086	43	SN2013dg	0.2650	25	SN2013hy	0.6630	9,2
SN2015bn	0.1136	42	SN2018lfd	0.2700	55	SN2019fiy	0.6700	53
PTF10nmn	0.1237	$10,21$	iPTF13bjz	0.2712	30	PS1-12zn	0.6740	52
SN2007bi	0.1279	41	SN2018bym	0.2740	23	DES17X1blv	0.6900	2
SN2017dwh	0.1300	40	SN2011ep	0.2800	16	DES16C3cv	0.7270	2
SN2018avk	0.1320	23	SN2005ap	0.2832	22	PS1-11bdn	0.7380	4
SN2020exj	0.1330	59	PTF10uhf	0.2879	21	iPTF13ajg	0.7403	8
SN2019lsq	0.1400	39	SN2016inl	0.2980	this work	SNLS07D3bs	0.7570	51
SN2018ffs	0.1420	this work	MLS121104	0.3030	52	DES15X3hm	0.8600	2
SN2011ke	0.1429	21	SN2019eot	0.3057	20	DES14X2byo	0.8680	2
SN2019bgu	0.1480	58	SN2017beq	0.3100	19	PS1-13gt	0.8840	4
SN2019cdt	0.1530	38	PS1-12cil	0.3200	4	PS1-10awh	0.9084	7
LSQ14an	0.1630	37	SN2019cwu	0.3200	53	DES17X1amf	0.9200	2
SN2019ujb	0.1647	this work	PTF12mxx	0.3296	10	DES16C3ggu	0.9490	2
SN2019obk	0.1656	61	iPTF13ehe	0.3434	18	PS1-10ky	0.9558	7
SN2018ibb	0.1660	57	SN2019sgh	0.3440	this work	PS1-11aib	0.9970	4
SN2019pvs	0.1670	this work	LSQ14bdq	0.3450	17	DES16C2aix	1.0680	2
PTF10bfz	0.1701	10	SN2018lfe	0.3500	63	PS1-10ahf	1.1000	4
SN2012il	0.1750	28	SN2019kwt	0.3562	53	DES15X1noe	1.1880	2
PTF12gty	0.1768	21	PTF10bjp	0.3584	10	SCP06F6	1.1890	6
CSS160710	0.1800	36	LSQ14fxj	0.3600	16	PS1-10pm	1.2060	5
SN2019gfm	0.1816	35	SN2019zbv	0.3700	this work	PS1-11tt	1.2830	4
SN2009cb	0.1864	21	SN2006oz	0.3760	15	DES14C1fi	1.3020	2
SN2009jh	0.1867	10, 21	SN2019zeu	0.3900	this work	PS1-11afv	1.4070	4
iPTF16asu	0.1870	34	DES15C3hav	0.3920	2	SNLS07d2bv	1.5000	3
SN2019nhs	0.1900	33	iPTF13cjq	0.3962	30	DES14S2qri	1.5000	2
SN2018cxa	0.1900	this work	SN2019kcy	0.4000	53	PS1-13or	1.5200	4
SN2010hy	0.1901	10, 30	iPTF13bdl	0.4030	30	PS1-11bam	1.5650	4
SN2011kg	0.1924	10	SN2019cca	0.4103	14	PS1-12bmy	1.5720	4
SN2019kws	0.1977	53,61	iPTF16eh	0.4270	13	SNLS06d4eu	1.5881	3
SN2019xaq	0.2000	this work	CSS130912	0.4305	11, 12	DES16C2nm	1.9980	2
SN2016ard	0.2025	32	PTF10vqv	0.4518	10	SN2213	2.0500	50
PTF10aagc	0.2060	10	CSS140925	0.4600	16
SN2016els	0.2170	31	DES17C3gyp	0.4700	2

Note. All the SLSN-I used to train our classifier. Note there are more SLSNe candidates in the literature, but we keep only the unambiguous ones to avoid polluting the sample. 1:Prentice et al. (2019); 2:Angus et al. (2019); 3:Howell et al. (2013); 4:Lunnan et al. (2018a); 5:McCrum et al. (2015); 6:Quimby et al. (2011); 7:Chomiuk et al. (2011); 8:Vreeswijk et al. (2014); 9:Papadopoulos et al. (2015); 10:Perley et al. (2016); 11:Vreeswijk et al. (2017); 12:Liu et al. (2018); 13:Lunnan et al. (2018b); 14:Perley et al. (2019b); 15:Leloudas et al. (2012); 16:Schulze et al. (2018); 17:Nicholl et al. (2015); 18:Yan et al. (2015); 19:Kasliwal & Cao (2019); 20:Fremling et al. (2019e); 21:Quimby et al. (2018); 22:Quimby et al. (2007); 23:Lunnan et al. (2019); 24:Chen et al. (2017); 25:Nicholl et al. (2014); 26:Short et al. (2019); 27:Yan et al. (2017); 28:Inserra et al. (2013); 29:Fremling et al. (2018b); 30:de Cia et al. (2018); 31:Fraser et al. (2016); 32:Blanchard et al. (2018); 33:Perley et al. (2019a); 34:Whitesides et al. (2017); 35:Chen (2019); 36:Drake et al. (2009); 37: Inserra et al. (2017); 38:Fremling et al. (2019d); 39:Fremling & Dahiwale (2019); 40:Blanchard et al. (2019); 41:Nicholl et al. (2013); 42:Nicholl et al. (2016); 43:Chen et al. (2018); 44:Fremling et al. (2019b); 45:Perley et al. (2019c); 46:Nicholl et al. (2017b); 47:Lyman et al. (2017); 48:Roy et al. (2016); 49:Nicholl et al. (2017a); 50:Cooke et al. (2012); 51:Prajs et al. (2017); 52:Lunnan et al. (2014); 53:Yan et al. (2019b); 54:Yan et al. (2019a); 55:Fremling et al. (2019a); 56:Fremling et al. (2018a); 57:Fremling et al. (2018c); 58:Fremling et al. (2019c); 59:Dahiwale & Fremling (2020); 60:Young (2016); 61:Yan et al. (2020); 62:Lin et al. (2020); 63: Y. Yin et al. 2020, in preparation; 64:Kumar et al. (2020).

^aWe find that a redshift of z = 0.255 is a better match to the SNe spectral features than the z = 0.22 reported in Short et al. (2019).

Download table as:  ASCIITypeset image