Solar Flare Forecasting Using Time Series and Extreme Gradient Boosting Ensembles

Abstract

Space weather events may cause damage to several types of technologies, including aviation, satellites, oil and gas industries, and electrical systems, leading to economic and commercial losses. Solar flares belong to the most significant events, and refer to sudden radiation releases that can affect the Earth’s atmosphere within a few hours or minutes. Therefore, it is worth designing high-performance systems for forecasting such events. Although in the literature there are many approaches for flare forecasting, there is still a lack of consensus concerning the techniques used for designing these systems. Seeking to establish some standardization while designing flare predictors, in this study we propose a novel methodology for designing such predictors, further validated with extreme gradient boosting tree classifiers and time series. This methodology relies on the following well-defined machine learning based pipeline: (i) univariate feature selection provided with the F-score, (ii) randomized hyperparameters search and optimization, (iii) imbalanced data treatment through cost function analysis of classifiers, (iv) adjustment of cut-off point of classifiers seeking to find the optimal relationship between hit rate and precision, and (v) evaluation under operational settings. To verify our methodology effectiveness, we designed and evaluated three proof-of-concept models for forecasting flares with an X class larger than C up to 72 hours ahead. Compared to baseline models, those models were able to significantly increase their scores of true skill statistics (TSS) under operational forecasting scenarios by 0.37 (predicting flares in the next 24 hours), 0.13 (predicting flares within 24 – 48 hours), and 0.36 (predicting flares within 48 – 72 hours). Besides increasing the TSS, the methodology also led to significant increases in the area under the receiver operating characteristic (ROC) curve, corroborating that we improved the positive and negative recalls of classifiers while decreasing the number of false alarms.

Appendix

In this article, we use some well-defined scores according to the proposed methodology to design the forecasting models. Here, we present all of them, namely the true skill statistics (TSS) (Jolliffe and Stephenson, 2003), true positive rate (TPR) (Han and Kamber, 2006), true negative rate (TNR) (Han and Kamber, 2006), area under curve (AUC) (Witten, Frank, and Hall, 2011), false positive rate (FPR) (Witten, Frank, and Hall, 2011), positive predictive value (PPV) (Zaki and Junior, 2013), negative predictive value (NPV) (Zaki and Junior, 2013), overall accuracy (ACC) (Han and Kamber, 2006), and false alarm ratio (FAR) (Jolliffe and Stephenson, 2003).

1.1 A.1 TSS

The TSS score ranks a model performance over a scale lying on the interval [$-1,1$], according to which results close to −1 mean all incorrect predictions and those close to 1 mean all correct predictions. The TSS was proposed as a quality measure that combines both class-specific hit rates without being affected by imbalanced data scenarios (Bloomfield et al., 2012). In Equation 6 we show how to calculate this score:

$$ \mathrm{TSS} = \mbox{TPR} + \mbox{TNR} - 1, $$

(6)

where the TPR refers to the true positive rate and the TNR is the true negative rate.

1.2 A.2 TPR

The TPR – also known as the positive recall or probability of detection (POD) – accounts for the number of positive samples correctly predicted (Han and Kamber, 2006). In Equation 7 we show how to calculate the TPR:

$$ \mathrm{TPR} = \frac{\text{TP}}{\text{TP}+\text{FN}}, $$

(7)

where the TP accounts for the true positives (positive samples predicted as positive) and the FN are the false negatives (positive samples predicted as negative) (Han and Kamber, 2006). The TPR lies around the interval [$0,1$], in which higher values are better.

1.3 A.3 TNR

The TNR refers to the true negative rate – also known as the negative recall – and accounts for the number of negative samples correctly predicted (Han and Kamber, 2006). Similarly to the TPR, the TNR is scored in the same interval, as we show in Equation 8:

$$ \mathrm{TNR} = \frac{\text{TN}}{\text{TN}+\text{FP}}, $$

(8)

where the TN accounts for the true negatives (negative samples predicted as negative) and the FP are the false positives (negative samples predicted as positive) (Han and Kamber, 2006).

1.4 A.4 AUC

The AUC measures the two-dimensional area underneath the receiver operating characteristic (ROC) curve, which graphically analyzes the TPR score (y-axis) against the false positive rate (FPR) (x-axis) for a set of increasing probability thresholds to make the yes/no decisions, i.e. 0.1, 0.2, 0.3, etc. (see Figure 8). Also known as the probability of a false detection (POFD) or the false alarm rate, the FPR calculates the probability of detecting false alarms among the negative predictions, as we show in Equation 9 (Witten, Frank, and Hall, 2011):

$$ \mathrm{FPR} = \frac{\text{FP}}{\text{TN} + \text{FP}}, $$

(9)

where the FP are the false positives and the TN are the true negatives.

Figure 8

Example of a ROC curve plot.

Best classifiers score the AUC next to the graph left-hand corner ($\mbox{FPR} = 0$ and $\mbox{TPR} = 1$). On the other hand, worst classifiers score next to the graph bottom right-hand corner ($\mbox{FPR} = 1$ and $\mbox{TPR} = 0$). The AUC is always positive and ideally should be greater than 0.5.

1.5 A.5 PPV

Also known as the class-specific accuracy score of a classifier, both PPV and NPV account for the classifier precision regarding individual classes. The positive precision measures the fraction of correctly predicted positive samples over all samples marked to be positive, as we show in Equation 10 (Zaki and Junior, 2013):

$$ \mathrm{PPV} = \frac{\text{TP}}{\text{TP}+\text{FP}}, $$

(10)

where the TP are the true positives and the FP are the false positives. The PPV is measured on the scale [$0,1$], in which the higher the values, the better the classifier.

1.6 A.6 NPV

The negative precision, in turn, measures the fraction of correctly predicted negative samples over all samples marked to be negative, as we show in Equation 11 (Zaki and Junior, 2013):

$$ \mathrm{NPV} = \frac{\text{TN}}{\text{TN}+\text{FN}}, $$

(11)

where the TN are true negatives and the FN are the false negatives. Similarly to the TPR, the NPV is also measured on the scale [$0,1$], where the higher the values, the better the classifier.

1.7 A.7 ACC

The overall accuracy, in turn, is a global score for classifier performance and also lies around the interval [$0,1$], in which the higher the score, the better the classifier. Accordingly, the overall accuracy accounts for the weighted mean of the class-specific accuracy scores, as we show in Equation 12 (Han and Kamber, 2006).

$$ \mathrm{ACC} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{FN} + \text{FP} + \text{TN}}, $$

(12)

where the TP is the number of true positives, TN is the number of true negatives, FN is the number of false negatives, and FP of false positives. It is worth mentioning that the ACC should not be used alone, since it can mask the real hit rates of individual classes in imbalanced data scenarios; however, we keep it in our study for completeness purposes, since most authors use it.

1.8 A.8 FAR

Finally, the false alarm ratio represents the number of times a method forecasts events that are not observed. It is a complementary metric, and thus should be used with the positive recall to allow a better understanding of the observed false alarms. The FAR also lies around the interval [$0,1$]; however, the lower the score, the better the classifier. In Equation 13 we show how to calculate the FAR (Jolliffe and Stephenson, 2003):

$$ \mathrm{FAR} = \frac{\text{FP}}{\text{TP} + \text{FP}}, $$

(13)

where the FP is the number of false positives and the TP are the true positives.