Ex) Article Title, Author, Keywords
Current Optics
and Photonics
Ex) Article Title, Author, Keywords
Curr. Opt. Photon. 2024; 8(3): 282-299
Published online June 25, 2024 https://doi.org/10.3807/COPP.2024.8.3.282
Copyright © Optical Society of Korea.
Qianghui Wang^{1}, Bing Zhou^{1} , Wenshen Hua^{1}, Jiaju Ying^{1}, Xun Liu^{2}, Lei Deng^{1}
Corresponding author: ^{*}zhbgxgc@163.com, ORCID 0000-0002-1932-6540
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Target detection (TD) is a research hotspot in the field of hyperspectral imaging (HSI). Traditional TD methods often mine targets from HSIs under a single imaging condition, without considering the influence of imaging conditions. In fact, the spectra of ground objects in HSIs are uncertain and affected by the imaging conditions (weather, atmospheric, light, time, and other angle conditions including zenith angle). Hyperspectral data changes under different imaging conditions. Therefore, the detection result for a single imaging condition cannot accurately reflect the effectiveness of the detection method used. It is necessary to analyze the performance of various detection methods under different imaging conditions, to find a more applicable detection method. In this paper, we study the performance of TD methods under various land-based imaging conditions. We first summarize classical TD methods and evaluation methods. Then, the detection effects under various imaging conditions are analyzed. Finally, the concepts of the stability coefficient (SC) and effective area under the curve (EAUC) are proposed to comprehensively evaluate the applicability of detection methods under land-based imaging conditions, in terms of both detection accuracy and stability. This is conducive to our selection of detection methods with better applicability in land-based contexts, to improve detection accuracy and stability.
Keywords: Effective area under the curve, Imaging condition, Land-based applications, Stability coefficient, Target detection
OCIS codes: (100.4145) Motion, hyperspectral image processing; (300.6320) Spectroscopy, high-resolution
Before the advent of spectral imaging, the images obtained by traditional imaging technology were usually grayscale or color images. In such cases the amount of information was relatively limited, and the related processing or analysis tasks were mostly based on spatial information such as grayscale and texture distribution, thus making it impossible to distinguish ground-object categories from images [1]. With the development of imaging techniques that integrate spectral information into image data on the basis of traditional optical imaging techniques, spectral imaging has aroused much interest. Spectral imaging solves the “only the image without spectrum” or “only the spectrum without image” problems of traditional optical imaging technology, and makes it possible to analyze the object categories in an image [2, 3].
Hyperspectral imaging technology is an advanced image-data acquisition technology that records the spatial and spectral information of ground objects through an imaging spectrometer [4]. Hyperspectral images contain two-dimensional spatial and one-dimensional spectral information; Thus hyperspectral data can be considered three-dimensional data space. The spectral resolution of hyperspectral imaging technology is usually below 10 nm, and its number of spectral bands can reach the hundreds. Such technology can obtain the diagnostic spectral features of ground objects, thus achieving accurate classification. In recent years, hyperspectral imaging (HSI) has rapidly developed in various fields, due to the development of science and technology and the practical needs in the information field.
By extracting spectral and spatial information contained in HSIs, researchers obtained many research results regarding spectral demixing [5], feature extraction [6], and target detection (TD) [7]. The spectra of different ground objects in HSIs are often quite different; Therefore, different types of ground objects can be diagnosed in HSIs. The field of TD has received a great deal of attention. TD is often based on similarity measurements, establishing a similarity relation between the test spectrum vector (the spectral vector of unknown pixels in HSI) and the reference spectrum vector (the spectral vector of the known target). With that, the target distributions with high levels of similarity to the reference spectrum can be obtained [8].
In the ideal state, the category and spectrum of the target have a one-to-one correspondence; That is, the spectra should be the same if the target types are the same, while the spectra are different when the target types are different. However, achieving this state is impossible in practice. Traditional space-based imaging methods have high altitudes and fixed times, and their detection directions are basically perpendicular to the ground. Thus hyperspectral data obtained are less affected by imaging conditions [9], and the spectrum is relatively stable. Land-based imaging methods typically use small unmanned aerial vehicles, or human-powered methods. Imaging time and direction are arbitrary, and the imaging conditions clearly affect the hyperspectral data. As a result, the uncertain characteristics of the spectra of targets are significant, usually manifesting as same object, different spectrum.
The spectral curves of targets show uncertainty when imaging conditions change. On the other hand, the spatial distribution of targets is usually not uniform or regular, and obvious uncertainties exist when taking the whole large-area average spectrum as the spectrum of the targets. This approach weakens TD accuracy, in terms of the similarity between the test and reference spectrum vectors. Therefore, it is worth discussing the effectiveness of the TD method under various land-based imaging conditions. Traditional detection evaluations are often based on detection results for a single imaging condition, without taking into account the influence of imaging conditions on hyperspectral data, which results in TD methods that may not be directly applied to land-based imaging conditions. Therefore, from the perspective of applicability, detection effectiveness should be analyzed to find a new evaluation method that can comprehensively evaluate performance under land-based conditions. Subsequently, a universal detection method that can be adapted to different imaging conditions with better results will be needed.
To solve this problem, on the premise of fully analyzing the advantages and disadvantages of detection and evaluation methods, this paper proposes the concepts of the stability coefficient (SC) and effective area under the curve (EAUC) to comprehensively evaluate the TD effectiveness under land-based imaging conditions, while taking into account both detection accuracy and stability. The experimental results show that the evaluation results of this method are consistent with those of the qualitative analysis, and have good universality.
Section 2 summarizes the principles of classical TD, and analyzes the strengths and weaknesses of each method. Section 3 introduces the principles of the classic evaluation methods. Section 4 analyzes the performance of the classical detection methods under various imaging conditions, and analyzes their applicability using the EAUC value. Section 5 summarizes the content of this paper.
The basic form of TD in HSI is a binary hypothesis-testing problem, formulated as Eq. (1):
where x represents the known reference-spectrum vector, y represents the unknown test-spectrum vector, η represents the threshold, and D(x, y) represents the detection function. The TD methods work by comparing the known target spectrum to the test pixel spectrum in an HSI. The pixels with high levels of similarity are regarded as target pixels, and the pixels with low levels of similarity are regarded as background pixels.
In general, TD methods can be categorized into projection-based, distance-based, information-based, and statistics-based methods. In addition, the constrained-energy-minimization (CEM) method is also a classical TD method [10] and is widely used. The idea behind this method is to enhance the information in the direction of interest and suppress the information in the other direction, thus highlighting the target.
There are various methods through which to improve classical TD methods. However, most of these methods are based on the similarity across spectral vectors. This paper takes traditional TD methods as examples, and analyzes their applicability under land-based imaging conditions. A brief introduction to classical TD methods is presented following section.
A projection-based TD method mainly exploits the shape differences between the reference-spectrum vector and the test-spectrum vector. This method is insensitive to differences in spectral amplitude, can overcome the influence on TD of the amplitude differences caused by light intensity, terrain, shadows, particle size, and other factors, and has the characteristic of multiplier-factor invariance. The disadvantage of this method is that it can measure only the difference in overall shape between the reference-spectrum vector and test-spectrum vector, and cannot reflect local differences; Usually the contrast of the detection results is not at a high level.
Projection-based TD methods include the spectral-angle-metric (SAM) [11] method, the spectral-angle-cosine (SAC) method, the spectral-gradient-angle (SGA) method, the normalized SGA (NSGA) method, the kernel-spectral-angle (KSA) method [12], and the orthogonal-projection-divergence (OPD) method [13]. Among them, the SAM, SAC, SGA, and NSGA methods are the most classic. The SAM method reflects the shape differences between spectral vectors and measures the inclusive angle between reference-spectrum vector and test-spectrum vector. The SAM method is defined as Eq. (2):
where x^{T} and y^{T} represent the transpositions of x and y respectively. The SAM values fall in the range [0, π]. The smaller the value is, the higher the degree of similarity. Similarly the SAC method converts the range to [−1, 1]. The higher the value is, the higher the degree of similarity between reference-spectrum vector and test-spectrum vector. The SAC method is defined as follows:
The SGA method measures the similarity between spectral gradient vectors [14], as defined in Eq. (4). This method takes into account the tilt of the vector, and is robust to geometric distortions and intensity variations. As mentioned, the range of the SGA value is [−1, 1]; To convert its range to [0, 1], the NSGA method is defined as in Eq. (5):
where x′ = [x_{2} − x_{1}, x_{3} − x_{2}, ..., x_{n} − x_{n−1}], y′ = [y_{2} − y_{1}, y_{3} − y_{2}, ..., y_{n} − y_{n−1}]. x_{i} and y_{i} represent the reflectivity of the i^{th} spectral band of x and y, respectively. n represents the number of spectral bands. The spectral gradient reflects the change in the slope of the spectral vector, and describes the morphological features of the spectral vector. Typically these changes are associated with the absorption features of the targets, which are essential features of an image.
Distance-based TD methods assume that the spectrum is a higher-dimensional vector in Euclidean space, where the dimension is denoted as the number of spectral bands. The TD problem is transformed into a similarity-measurement problem for distance: the smaller the distance, the greater the degree of spectral similarity. The principle of this method is simple, and there are many correlation-distance indices with which to describe measurement for distance. In fact, other types of detection methods are similar to distance-based detection methods, and their results can be transformed into one another. Compared to projection-based methods, a distance-based method has difficulty overcoming the influence on TD of light intensity, terrain, shadows, particle size, and other factors, especially when the target is in a shadow.
Distance-based TD methods include the Euclidean-distance (ED) method, the normalized ED (NED) [15] method, the Mahalanobis-distance (MaD) method [16], the Tchebyshev-distance (TcD) method [17], and the Hamming-distance (HD) method [18]. The ED method is a frequently used distance-based TD method, but its results are largely dependent on spectral amplitude and are insensitive to differences in shape between test-spectrum vector and reference-spectrum vector. To solve this problem, the NED method is defined in Eq. (6):
where
Information-based TD methods are based on evaluating the information-entropy (IE) characteristics of the spectral vectors. This method analyzes the similarity between vectors based on the amount of information provided, and is not affected by amplitude or shape. However, the false-alarm rate is higher in the detection results of this method, compared to that for other methods.
IE describes the uncertainty of a signal and reflects the amount of information it carries. Mutual information (MI) is used to describe the correlation between two systems; that is, the amount of information contained in both systems. The MI between the test-spectrum vector and the reference-spectrum vector is defined as Eq. (7):
where
Statistics-based TD methods measure similarity by calculating the correlation between vectors [19]. These methods are not affected by light intensity, shadows, or other conditions. Representative methods include the normalized-correlation-coefficient (NCC) and spectral-correlation-angle (SCA) methods. The NCC and SCA methods are defined as follows:
where
where cov(x, y) represents the covariance between x and y. R_{xy} represents the correlation coefficient between x and y. σ_{x} and σ_{y} represent the standard deviations of x and y respectively. The range of the NCC value is [0, 1]; The larger the value is, the higher the degree of similarity. The range of the SCA value is [0, π / 2]; The smaller the value is, the higher the degree of similarity.
The principle of the CEM method is to design a linear filter vector w = [w_{1}, w_{2}, ..., w_{n}]^{T} that minimizes the energy output of the hyperspectral data and satisfies Eq. (12):
The output of test spectrum y after passing through the filter vector is given by Eq. (13):
Therefore, the average output energy of all pixels in HSI is given by Eq. (14):
where
Eq. (15) can be solved by the Lagrange-multiplier method:
The CEM method is applied to each pixel to achieve TD.
To successfully evaluate the detection effectiveness of TD methods, the detection results are analyzed qualitatively and quantitatively. Qualitative analysis is used mainly to visually measure the number, contour, and integrity of targets in the detection results with the naked eye. The closer the detection results are to the real target distribution, the clearer the contour and the higher the integrity, and exhibit better detection.
In fact, a quantitative analysis of the detection results is more convincing. The receiver-operating-characteristic (ROC) curves [20] and area-under-the-curve (AUC) values are used to measure the detection results. The ROC curve reflects the relationship between the detection rate and the false-alarm rate; The abscissa indicates the false-alarm rate P_{f} [21], while the ordinate indicates the detection rate P_{d}. This two-dimensional relationship is based on the three-dimensional ROC curve containing a common threshold τ (the number of equal intervals between 0 and 1). The three-dimensional ROC curve and its projection are shown in Fig. 1.
The three-dimensional ROC curve is projected into three vertical directions to obtain the τ − P_{f} (false-alarm rate) curve, τ − P_{d} (detection rate) curve, and P_{d} − P_{f} (ROC) curve. The τ − P_{f} curve reflects the ability of the detector to suppress the background. The τ − P_{d} curve reflects the ability of the detector to enhance the target. The P_{d} − P_{f} curve reflects the overall performance of the detector.
The two-dimensional ROC curve is essentially composed of τ − P_{f} and τ − P_{d} curves. Multiple groups of P_{f} and P_{d} values can be obtained by setting different thresholds τ. The ROC curve can be obtained by combining the values of P_{f} and P_{d} corresponding to each threshold. P_{f} and P_{d} are defined as follows:
where N_{d} represents the number of detected true target pixels, which is the number of pixels that actually belong to the target and are considered as such by the detector. N_{t} represents the total number of target pixels in the image, N_{f} represents the number of false-alarm pixels detected, and N_{tot} represents the total number of pixels in the image.
In fact the ROC curve is not continuous, but consists of discrete points, each of which represents the false-alarm rate and detection rate corresponding to a threshold. The AUC value quantitatively describes the degree of inclination of the ROC curve toward the upper left. The AUC value is the area surrounded by the ROC curve and the abscissa. The more the ROC curve is bent toward the upper left, the larger the AUC value, the better detection of the detection method, and the higher the detection reliability. The smaller the AUC value, the worse the detection results and the lower the detection reliability.
It is not entirely accurate to determine the merits of a detection method based on the ROC curve or AUC value. Especially in land-based applications, spectral data show a certain degree of uncertainty depending on the imaging conditions. The performance of TD methods under multiple imaging conditions is variable. It is obviously inappropriate to consider only TD performance under a single imaging condition. For example, a detection method can have a better detection effect under some imaging conditions yet face difficulty in detecting the target under other conditions, which cannot confirm the performance of the detection method and cannot describe the applicability of the method under land-based imaging conditions. Therefore, we need to comprehensively consider the average value and degree of dispersion of the detection results. The larger the average value and the smaller the degree of dispersion, the better the detection effect and the better applicability under land-based imaging conditions.
Detection results are diverse depending on land-based imaging conditions. Therefore, the analysis of detection results under land-based imaging conditions is based on the comprehensive analysis of multiple detection results. This approach requires that the detection method meet not only the requirements of detection accuracy, but also those of stability; That is, the detection method with a higher level of detection accuracy under each imaging condition has a greater degree of applicability. Based on this idea, this paper proposes the SC and effective AUC (EAUC) concepts. The calculation process is as follows.
First, the τ − P_{f} and τ − P_{d} curves are analyzed. The step size of the threshold value is constant, and thus the τ − P_{f} and τ − P_{d} curves are evenly distributed on the horizontal axis. Therefore, the average false-alarm and detection curves can be obtained by taking both the τ − P_{f} and τ − P_{d} curves under various imaging conditions:
where K represents the number of imaging conditions. P_{fi} and P_{di} represent the false-alarm rate and detection rate curves under the i^{th} imaging condition respectively.
Then the SC value is calculated to combine the stability characteristics of the detection results, and the AUC value under different imaging conditions is weighted according to the SC value to calculate the EAUC value. The SC value combines the average value and degree of dispersion, which more comprehensively reflects the performance of the detection methods under various imaging conditions. The larger the SC value, the more similar the ROC curve of the detection results to the average ROC curve, and the greater the weight involved in calculating the EAUC value. The SC value is calculated as follows:
where σ_{i} represents the standard deviation for the i^{th} imaging condition. P_{fik} and P_{dik} represent the k^{th} element of false-alarm rate P_{fi} and detection rate P_{di} for the i^{th} imaging condition respectively. Q represents the number of the threshold τ (equal intervals between 0 and 1).
The EAUC values of all detection methods are calculated by combining the SC and AUC values under all imaging conditions:
By analyzing the calculation process for the EAUC, it is found that the EAUC can reflect both the average value and degree of dispersion of the detection results. Therefore, in theory this approach can comprehensively evaluate the applicability of a detection method under land-based imaging conditions.
The experimental data are obtained by an HSI-300 hyperspectral imaging system based on acousto-optic tunable filter (Gooch & Housego Co., Ilminster, England). The size of each hyperspectral image data is 712 × 1,002 pixels. The spectral resolution (band interval) of the HSI-300 spectrometer is 4 nm, its bandwidth is 2.3 nm, and its band range is 449–801 nm. A total of 89 bands are obtained. The experiment takes place in Shijiazhuang, Hebei, China, on June 1, 2023. The geographical coordinates are 38° 27′ N, 114° 30′ E. The weather is clear during the experiment. The simulation platform consists of an Intel Core i7-7700HQ CPU with a dominant frequency of 2.80 GHz, 8 GB RAM, and MATLAB R2017a software.
The targets include two pieces of camouflage clothing, sample 1, and a brown plate, sample 2. The background is composed mainly of green leaves, soil, leaves of Chinese ilex, and weeds. Samples 1 and 2 are almost integrated into the background, and it is difficult to detect their specific positions with the naked eye. The distributions and spectral curves of the two samples are shown in Fig. 2, and their real locations are in Fig. 3.
Imaging conditions are arbitrary in land-based applications, meaning that the use of any imaging condition is possible; Therefore, the more imaging conditions, the better the analysis of the detection results. However, it is impossible to obtain all imaging conditions completely, and only representative groups of imaging conditions can be selected. Experiments are conducted at 10:00, 10:30, 11:00, 11:30, 12:00, 12:30, 13:00, 13:30, 14:00, and 14:30, and ten groups of hyperspectral data are obtained. The imaging conditions are listed in Table 1.
TABLE 1 Imaging conditions
Group No. | Time | Solar Zeinth Angle (˚) | View Zeinth Angle (˚) | Relative Azimuth Angle (˚) |
---|---|---|---|---|
1 | 10:00 | 34.5 | 62 | 148.5 |
2 | 10:30 | 29.1 | 62 | 139.2 |
3 | 11:00 | 24.2 | 62 | 130.1 |
4 | 11:30 | 20.0 | 62 | 107.6 |
5 | 12:00 | 17.2 | 62 | 85.3 |
6 | 12:30 | 16.6 | 62 | 66.4 |
7 | 13:00 | 18.5 | 62 | 35.5 |
8 | 13:30 | 22.1 | 62 | 20.2 |
9 | 14:00 | 26.7 | 62 | 13.7 |
10 | 14:30 | 32.0 | 62 | 6.1 |
Since the obtained hyperspectral data denote the reflected intensity values, radiometric calibration of the data is needed; That is, the normalization of the obtained hyperspectral data to the form of reflectance. On the one hand, this approach can overcome the effect of radiation intensity, and on the other hand, it can also reduce the effect of noise. We use a standard whiteboard to normalize the obtained hyperspectral data, as shown in Fig. 4. The standard whiteboard used in this experiment is a polytetrafluoroethylene (PTFE) board that has been calibrated. Its reflection properties are uniform in all directions, and it can be approximated as a Lambertian reflector. The reflection intensity of this board is higher than that of the other pixels in each band. The spectral reflectance r(λ) of each pixel in this experiment can be calculated by Eq. (22):
where r_{s}(λ) represents the reflectance of the standard whiteboard and value of the reflectance was 0.98. I_{s}(λ) represents the obtained spectral intensity value of the standard whiteboard. I(λ) represents the obtained spectral intensity value of each pixel. Then the spectral of each pixel in the hyperspectral data is transformed into reflectance, and the spectral curve represents the spectral reflectance as determined by the standard whiteboard. The spectral curves of two samples under all imaging conditions are calculated, as shown in Fig. 5.
We take two samples as the desired targets, and the applicability of each is analyzed. The mean spectra of sample 1 and sample 2 under all conditions are taken as known reference spectra. The detection results of the SAC, NED, NCC, CEM, MI, and NSGA methods under all conditions are analyzed. Since the measured spectral data consists solely of positive values, the ranges of the six detection methods are all in [0, 1]. Thus it is unnecessary to unify the dimensions of the different detection methods. The closer the detection results are to 1 for the SAC, CEM, NCC, MI, and NSGA methods, the more similar the test spectrum is to the reference spectrum. The NED method has a larger calculation result when the test spectrum is more similar to the reference spectrum. Therefore we reversed the calculation result of the NED method using D_{new} = 1 − D_{origin}, where D_{origin} represents the calculation result before reversal and D_{new} represents the calculation result after reversal. The detection results for two samples under all imaging conditions are shown in Figs. 6 and 7.
First, we qualitatively analyze the detection results. As shown in Fig. 6, the NSGA method hardly detects the target. The NSGA method measures the change rate of the spectra, and the rate of sample 1 under visible light is very similar to those of the other objects in the background. As a result, the NSGA method detection results are not proper in distinguishing different ground objects. The results of the MI method under some conditions are clearly visible and show a large contrast with the background. However, under conditions 7 and 10, the target (sample 1) is obliterated by the background and it is almost impossible to detect the target, indicating that the stability of the MI method is low and that the detection results are greatly affected by the imaging conditions. The SAC, NED, NCC, and CEM methods can detect the target under all conditions. The gray value of the target position is the largest, and its detection results are less affected by the imaging conditions and are more stable. Therefore, for the detection of sample 1 against this background, the SAC, NED, NCC, and CEM methods exhibit better detection effects, followed by the MI method, and the NSGA method exhibits the worst detection effects.
Similarly, the detection results for the target (sample 2) with the NSGA method shown in Fig. 7 are worse than those of other methods. The MI method can detect the target under conditions 1, 3, 4, 5, 6, and 8, but the detection results are poor under other conditions, which also indicates that the degree of stability of the MI method is low. The SAC and NED methods can detect the target under all conditions, but there is an obviously serious false alarm rate in the detection results. The CEM method can detect the target under all conditions, and the contrast between the target and the background is obvious. However, there are also some false alarms in the detection results. The results of the NCC method under the first nine conditions are clearly visible, but under condition 10 the target is obliterated by the background and is almost impossible to detect.
According to the comprehensive analysis, the reason for the poor detection results of the NCC method under condition 10 can be caused by errors in the data. However, this method shows significantly better detection results than those of the other methods under the first nine groups of conditions, and the false alarm rate is low. The CEM method can detect the target under almost all imaging conditions, but it is also accompanied by a certain degree of false alarms. Although the SAC and NED methods also show detection effects, false alarms are more obvious when these methods are used. The MI method has a detection effect only under a few conditions, while the NSGA method has no detection effect at all. In summary, the detection methods for sample 2 can be listed in order of effectiveness: CC, CEM, SAC, NED, MI, and NSGA methods.
Next, we quantitatively analyze the results. The step-in threshold is set to 0.001, and the number of steps is 1,000. Figure 8 shows the ROC curves for all detection methods under all imaging conditions.
By analyzing the trend of the ROC curves in Fig. 8(a), we find that the ROC curves for the SAC, NED, NCC and CEM methods are significantly curved toward the upper left, proportional to the increasing detection performance. As seen from Fig. 8(b), the ROC curves for the MI method are greatly affected by the imaging conditions, and the detection performance is unstable. The ROC curves for the SAC, NED, NCC, and CEM methods are significantly curved toward the upper left, compared to those for the NSGA and MI methods. This finding is consistent with that of the previous qualitative analysis. The AUC values are shown in Table 2.
TABLE 2 Area-under-the-curve (AUC) values for different methods
AUC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.9959 | 0.9959 | 0.9998 | 0.9999 | 0.9878 | 0.5914 |
Condition 2 | 0.9899 | 0.9899 | 0.9989 | 0.9997 | 0.9868 | 0.5719 | |
Condition 3 | 0.9796 | 0.9796 | 0.9992 | 0.9995 | 0.9875 | 0.5661 | |
Condition 4 | 0.9645 | 0.9645 | 0.9985 | 0.9974 | 0.9795 | 0.5403 | |
Condition 5 | 0.9616 | 0.9616 | 0.9983 | 0.9976 | 0.9843 | 0.5754 | |
Condition 6 | 0.9565 | 0.9565 | 0.9985 | 0.9967 | 0.9819 | 0.5319 | |
Condition 7 | 0.9834 | 0.9834 | 0.9974 | 0.9959 | 0.9698 | 0.5200 | |
Condition 8 | 0.9578 | 0.9578 | 0.9981 | 0.9932 | 0.9734 | 0.5296 | |
Condition 9 | 0.9776 | 0.9776 | 0.9970 | 0.9904 | 0.9618 | 0.5350 | |
Condition10 | 0.9965 | 0.9965 | 0.9891 | 0.9897 | 0.8959 | 0.4609 | |
Mean | 0.9763 | 0.9763 | 0.9975 | 0.9960 | 0.9709 | 0.5423 | |
Sample 2 | Condition 1 | 0.9717 | 0.9718 | 0.9980 | 0.9909 | 0.9389 | 0.2829 |
Condition 2 | 0.9579 | 0.9578 | 0.9969 | 0.9898 | 0.9281 | 0.2409 | |
Condition 3 | 0.9495 | 0.9495 | 0.9965 | 0.9903 | 0.9221 | 0.2212 | |
Condition 4 | 0.9385 | 0.9385 | 0.9960 | 0.9876 | 0.8791 | 0.2466 | |
Condition 5 | 0.9320 | 0.9320 | 0.9961 | 0.9845 | 0.8527 | 0.2451 | |
Condition 6 | 0.9275 | 0.9275 | 0.9926 | 0.9845 | 0.9037 | 0.2489 | |
Condition 7 | 0.9416 | 0.9416 | 0.9914 | 0.9799 | 0.7585 | 0.2693 | |
Condition 8 | 0.9320 | 0.9320 | 0.9923 | 0.9790 | 0.8371 | 0.2916 | |
Condition 9 | 0.9506 | 0.9506 | 0.9895 | 0.9744 | 0.7023 | 0.3174 | |
Condition10 | 0.9669 | 0.9670 | 0.7583 | 0.9686 | 0.3787 | 0.3077 | |
Mean | 0.9468 | 0.9468 | 0.9708 | 0.9832 | 0.8101 | 0.2672 |
Table 2 shows that the AUC values of the NCC and CEM methods are almost always the largest among all imaging conditions, with a small range of variation. This finding shows that the NCC and CEM methods perform better and have the best level of applicability under various land-based imaging conditions, compared to other methods. When detecting sample 2, the average AUC value of the NCC method is smaller than that of the CEM method. This is not consistent with that of our previous qualitative analysis, because only the average AUC value is calculated without combining the stability characteristics of all detection results. Consequentially, the comprehensive evaluation result is greatly affected by a certain result. Thus large deviation occurs in the comprehensive evaluation result, which calculates only the average AUC value.
Notably, the source of the average AUC values is obtained only by averaging the AUC values in Table 2, not by averaging the ROC curves. Since the distribution of the points in the ROC curve on the coordinate axis is not continuous but discrete, the ROC curve cannot be averaged directly. To comprehensively evaluate the detection effect under land-based imaging conditions, this study uses the EAUC method to comprehensively analyze and evaluate the applicability of the detection methods.
First, the average false alarm and detection curves are shown in Fig. 9. Since the average false-alarm curves and average detection curves have the same threshold value τ, the average false-alarm-rate curves and average detection-rate curves can be combined into the average ROC curves according to Eq. (22), as shown in Fig. 10. Then, the SC values of all detection methods under various imaging conditions are calculated according to Eq. (23), as shown in Table 3. Finally, the calculation results for the EAUC values are shown in Table 4.
TABLE 3 Stability coefficient (SC) values
SC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.0396 | 0.0421 | 0.0461 | 0.0626 | 0.0489 | 0.0834 |
Condition 2 | 0.0630 | 0.0657 | 0.0693 | 0.0537 | 0.0727 | 0.1016 | |
Condition 3 | 0.1405 | 0.1485 | 0.1588 | 0.0737 | 0.1065 | 0.1829 | |
Condition 4 | 0.1773 | 0.1579 | 0.2257 | 0.1391 | 0.2922 | 0.1116 | |
Condition 5 | 0.1292 | 0.1224 | 0.1213 | 0.1434 | 0.0579 | 0.1230 | |
Condition 6 | 0.0865 | 0.0978 | 0.0807 | 0.1797 | 0.0971 | 0.2064 | |
Condition 7 | 0.1397 | 0.1324 | 0.1309 | 0.0987 | 0.1200 | 0.0569 | |
Condition 8 | 0.0876 | 0.0865 | 0.0424 | 0.0777 | 0.0896 | 0.0474 | |
Condition 9 | 0.0849 | 0.0822 | 0.0886 | 0.0554 | 0.0618 | 0.0643 | |
Condition 10 | 0.0518 | 0.0645 | 0.0363 | 0.1160 | 0.0533 | 0.0226 | |
Sample 2 | Condition 1 | 0.0513 | 0.0534 | 0.0538 | 0.0489 | 0.0671 | 0.1333 |
Condition 2 | 0.0689 | 0.0666 | 0.0732 | 0.0608 | 0.0469 | 0.0809 | |
Condition 3 | 0.0683 | 0.0830 | 0.1027 | 0.0601 | 0.0859 | 0.0381 | |
Condition 4 | 0.1473 | 0.1619 | 0.1575 | 0.0713 | 0.1576 | 0.0927 | |
Condition 5 | 0.1615 | 0.1469 | 0.1673 | 0.2311 | 0.1380 | 0.1393 | |
Condition 6 | 0.1209 | 0.0961 | 0.1022 | 0.1745 | 0.1039 | 0.1552 | |
Condition 7 | 0.1499 | 0.1418 | 0.1899 | 0.0917 | 0.1529 | 0.0823 | |
Condition 8 | 0.0925 | 0.0932 | 0.0451 | 0.1281 | 0.1243 | 0.0774 | |
Condition 9 | 0.1074 | 0.1210 | 0.0803 | 0.0670 | 0.0935 | 0.0557 | |
Condition 10 | 0.0321 | 0.0362 | 0.0281 | 0.0665 | 0.0300 | 0.1451 |
TABLE 4 Effective area under the curve (EAUC) values
EAUC | SAC | NED | NCC | CEM | MI | NSGA |
---|---|---|---|---|---|---|
Sample 1 | 0.9732 | 0.9737 | 0.9980 | 0.9960 | 0.9745 | 0.5513 |
Sample 2 | 0.9425 | 0.9432 | 0.9870 | 0.9834 | 0.8309 | 0.2635 |
As seen from Table 4, the EAUC value is higher in order of NCC, CEM, MI, NED, SAC, and NSGA when the target is sample 1. This is consistent with the previous qualitative analysis. However, it is inconsistent with the sequence of average AUC values, because the EAUC method combines the stability characteristics of the detection results rather than considering only the average value of the detection, which can reflect the detection effect more objectively. At the same time, the detection results under all imaging conditions are relatively stable, and there is no large degree of variation. The use of the EAUC-value method does not affect the average size of the detection results. When sample 2 is the target, the EAUC value is higher in order of NCC, CEM, NED, SAC, MI, and NSGA. This sequence is inconsistent with the average AUC value but consistent with the qualitative analysis. This also proves that the EAUC-value method is more reasonable for the evaluation of detection results and can effectively realize the evaluation of the detection method under land-based imaging conditions, compared to the other methods considered.
The results of the above analysis of the test results based on the EAUC value are consistent with those of the qualitative analysis, indicating that the EAUC value is more reasonable than other values for measuring detection effectiveness. Compared to the direct calculation of the average AUC value, the calculation of the EAUC value combined with the SC value gives different weights to the AUC value under different imaging conditions. Thus the value of EAUC is combined with the evaluation results under various imaging conditions, reducing the contingency of the evaluation, which is conducive to our selection of TD methods with high accuracy and strong stability under land-based imaging conditions.
From the perspective of land-based applications, this paper first introduces several different types of detection methods and classical evaluation methods, and then analyzes the detection results of TD methods and their applicability under land-based imaging conditions. The levels of detection performance of TD methods under different imaging conditions are not constant. Therefore, a traditional evaluation method that measures only the detection performance of the TD method under one condition is not suitable under land-based imaging conditions. Finally, the proposed EAUC-value method is used to comprehensively analyze the applicability of each detection method under land-based imaging conditions. This method fully combines detection accuracy and stability; That is, a method with high levels of accuracy and stability has a larger EAUC value and can realize the evaluation of TD methods under various imaging conditions. This method is suitable for the evaluation of detection performance under land-based imaging conditions. However, due to the limited number of imaging conditions, many groups of imaging conditions are selected based only on subjective experience. The more imaging conditions there are, the more conducive the method is to the more comprehensive calculation of EAUC values; The more able it is to reflect the applicability of the detection method under land-based conditions; And the more conducive it is to selecting appropriate detection methods for land-based applications. This paper also proposes objective and rational detection methods through the EAUC-value method, considering the uncertainty characteristics of the spectra into account.
We would like to thank the Army Engineering University of PLA, Electronic and Optical Engineering Department for financial and equipment support in developing this work. We would also like to thank the anonymous reviewer for their helpful and insightful comments, which significantly improved the quality of the manuscript.
National Natural Science Foundation of China (Grant no. 62005319).
The authors declare that there are no conflicts of interest related to this article.
Data underlying the results presented in this paper are not publicly available at this time, but may be obtained from the authors upon reasonable request.
Curr. Opt. Photon. 2024; 8(3): 282-299
Published online June 25, 2024 https://doi.org/10.3807/COPP.2024.8.3.282
Copyright © Optical Society of Korea.
Qianghui Wang^{1}, Bing Zhou^{1} , Wenshen Hua^{1}, Jiaju Ying^{1}, Xun Liu^{2}, Lei Deng^{1}
^{1}Army Engineering University of PLA (Shijiazhuang Campus), Shijiazhuang, Hebei 050000, China
^{2}Army Engineering University of PLA (Nanjing Campus), Nanjing, Jiangsu 210000, China
Correspondence to:^{*}zhbgxgc@163.com, ORCID 0000-0002-1932-6540
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Target detection (TD) is a research hotspot in the field of hyperspectral imaging (HSI). Traditional TD methods often mine targets from HSIs under a single imaging condition, without considering the influence of imaging conditions. In fact, the spectra of ground objects in HSIs are uncertain and affected by the imaging conditions (weather, atmospheric, light, time, and other angle conditions including zenith angle). Hyperspectral data changes under different imaging conditions. Therefore, the detection result for a single imaging condition cannot accurately reflect the effectiveness of the detection method used. It is necessary to analyze the performance of various detection methods under different imaging conditions, to find a more applicable detection method. In this paper, we study the performance of TD methods under various land-based imaging conditions. We first summarize classical TD methods and evaluation methods. Then, the detection effects under various imaging conditions are analyzed. Finally, the concepts of the stability coefficient (SC) and effective area under the curve (EAUC) are proposed to comprehensively evaluate the applicability of detection methods under land-based imaging conditions, in terms of both detection accuracy and stability. This is conducive to our selection of detection methods with better applicability in land-based contexts, to improve detection accuracy and stability.
Keywords: Effective area under the curve, Imaging condition, Land-based applications, Stability coefficient, Target detection
Before the advent of spectral imaging, the images obtained by traditional imaging technology were usually grayscale or color images. In such cases the amount of information was relatively limited, and the related processing or analysis tasks were mostly based on spatial information such as grayscale and texture distribution, thus making it impossible to distinguish ground-object categories from images [1]. With the development of imaging techniques that integrate spectral information into image data on the basis of traditional optical imaging techniques, spectral imaging has aroused much interest. Spectral imaging solves the “only the image without spectrum” or “only the spectrum without image” problems of traditional optical imaging technology, and makes it possible to analyze the object categories in an image [2, 3].
Hyperspectral imaging technology is an advanced image-data acquisition technology that records the spatial and spectral information of ground objects through an imaging spectrometer [4]. Hyperspectral images contain two-dimensional spatial and one-dimensional spectral information; Thus hyperspectral data can be considered three-dimensional data space. The spectral resolution of hyperspectral imaging technology is usually below 10 nm, and its number of spectral bands can reach the hundreds. Such technology can obtain the diagnostic spectral features of ground objects, thus achieving accurate classification. In recent years, hyperspectral imaging (HSI) has rapidly developed in various fields, due to the development of science and technology and the practical needs in the information field.
By extracting spectral and spatial information contained in HSIs, researchers obtained many research results regarding spectral demixing [5], feature extraction [6], and target detection (TD) [7]. The spectra of different ground objects in HSIs are often quite different; Therefore, different types of ground objects can be diagnosed in HSIs. The field of TD has received a great deal of attention. TD is often based on similarity measurements, establishing a similarity relation between the test spectrum vector (the spectral vector of unknown pixels in HSI) and the reference spectrum vector (the spectral vector of the known target). With that, the target distributions with high levels of similarity to the reference spectrum can be obtained [8].
In the ideal state, the category and spectrum of the target have a one-to-one correspondence; That is, the spectra should be the same if the target types are the same, while the spectra are different when the target types are different. However, achieving this state is impossible in practice. Traditional space-based imaging methods have high altitudes and fixed times, and their detection directions are basically perpendicular to the ground. Thus hyperspectral data obtained are less affected by imaging conditions [9], and the spectrum is relatively stable. Land-based imaging methods typically use small unmanned aerial vehicles, or human-powered methods. Imaging time and direction are arbitrary, and the imaging conditions clearly affect the hyperspectral data. As a result, the uncertain characteristics of the spectra of targets are significant, usually manifesting as same object, different spectrum.
The spectral curves of targets show uncertainty when imaging conditions change. On the other hand, the spatial distribution of targets is usually not uniform or regular, and obvious uncertainties exist when taking the whole large-area average spectrum as the spectrum of the targets. This approach weakens TD accuracy, in terms of the similarity between the test and reference spectrum vectors. Therefore, it is worth discussing the effectiveness of the TD method under various land-based imaging conditions. Traditional detection evaluations are often based on detection results for a single imaging condition, without taking into account the influence of imaging conditions on hyperspectral data, which results in TD methods that may not be directly applied to land-based imaging conditions. Therefore, from the perspective of applicability, detection effectiveness should be analyzed to find a new evaluation method that can comprehensively evaluate performance under land-based conditions. Subsequently, a universal detection method that can be adapted to different imaging conditions with better results will be needed.
To solve this problem, on the premise of fully analyzing the advantages and disadvantages of detection and evaluation methods, this paper proposes the concepts of the stability coefficient (SC) and effective area under the curve (EAUC) to comprehensively evaluate the TD effectiveness under land-based imaging conditions, while taking into account both detection accuracy and stability. The experimental results show that the evaluation results of this method are consistent with those of the qualitative analysis, and have good universality.
Section 2 summarizes the principles of classical TD, and analyzes the strengths and weaknesses of each method. Section 3 introduces the principles of the classic evaluation methods. Section 4 analyzes the performance of the classical detection methods under various imaging conditions, and analyzes their applicability using the EAUC value. Section 5 summarizes the content of this paper.
The basic form of TD in HSI is a binary hypothesis-testing problem, formulated as Eq. (1):
where x represents the known reference-spectrum vector, y represents the unknown test-spectrum vector, η represents the threshold, and D(x, y) represents the detection function. The TD methods work by comparing the known target spectrum to the test pixel spectrum in an HSI. The pixels with high levels of similarity are regarded as target pixels, and the pixels with low levels of similarity are regarded as background pixels.
In general, TD methods can be categorized into projection-based, distance-based, information-based, and statistics-based methods. In addition, the constrained-energy-minimization (CEM) method is also a classical TD method [10] and is widely used. The idea behind this method is to enhance the information in the direction of interest and suppress the information in the other direction, thus highlighting the target.
There are various methods through which to improve classical TD methods. However, most of these methods are based on the similarity across spectral vectors. This paper takes traditional TD methods as examples, and analyzes their applicability under land-based imaging conditions. A brief introduction to classical TD methods is presented following section.
A projection-based TD method mainly exploits the shape differences between the reference-spectrum vector and the test-spectrum vector. This method is insensitive to differences in spectral amplitude, can overcome the influence on TD of the amplitude differences caused by light intensity, terrain, shadows, particle size, and other factors, and has the characteristic of multiplier-factor invariance. The disadvantage of this method is that it can measure only the difference in overall shape between the reference-spectrum vector and test-spectrum vector, and cannot reflect local differences; Usually the contrast of the detection results is not at a high level.
Projection-based TD methods include the spectral-angle-metric (SAM) [11] method, the spectral-angle-cosine (SAC) method, the spectral-gradient-angle (SGA) method, the normalized SGA (NSGA) method, the kernel-spectral-angle (KSA) method [12], and the orthogonal-projection-divergence (OPD) method [13]. Among them, the SAM, SAC, SGA, and NSGA methods are the most classic. The SAM method reflects the shape differences between spectral vectors and measures the inclusive angle between reference-spectrum vector and test-spectrum vector. The SAM method is defined as Eq. (2):
where x^{T} and y^{T} represent the transpositions of x and y respectively. The SAM values fall in the range [0, π]. The smaller the value is, the higher the degree of similarity. Similarly the SAC method converts the range to [−1, 1]. The higher the value is, the higher the degree of similarity between reference-spectrum vector and test-spectrum vector. The SAC method is defined as follows:
The SGA method measures the similarity between spectral gradient vectors [14], as defined in Eq. (4). This method takes into account the tilt of the vector, and is robust to geometric distortions and intensity variations. As mentioned, the range of the SGA value is [−1, 1]; To convert its range to [0, 1], the NSGA method is defined as in Eq. (5):
where x′ = [x_{2} − x_{1}, x_{3} − x_{2}, ..., x_{n} − x_{n−1}], y′ = [y_{2} − y_{1}, y_{3} − y_{2}, ..., y_{n} − y_{n−1}]. x_{i} and y_{i} represent the reflectivity of the i^{th} spectral band of x and y, respectively. n represents the number of spectral bands. The spectral gradient reflects the change in the slope of the spectral vector, and describes the morphological features of the spectral vector. Typically these changes are associated with the absorption features of the targets, which are essential features of an image.
Distance-based TD methods assume that the spectrum is a higher-dimensional vector in Euclidean space, where the dimension is denoted as the number of spectral bands. The TD problem is transformed into a similarity-measurement problem for distance: the smaller the distance, the greater the degree of spectral similarity. The principle of this method is simple, and there are many correlation-distance indices with which to describe measurement for distance. In fact, other types of detection methods are similar to distance-based detection methods, and their results can be transformed into one another. Compared to projection-based methods, a distance-based method has difficulty overcoming the influence on TD of light intensity, terrain, shadows, particle size, and other factors, especially when the target is in a shadow.
Distance-based TD methods include the Euclidean-distance (ED) method, the normalized ED (NED) [15] method, the Mahalanobis-distance (MaD) method [16], the Tchebyshev-distance (TcD) method [17], and the Hamming-distance (HD) method [18]. The ED method is a frequently used distance-based TD method, but its results are largely dependent on spectral amplitude and are insensitive to differences in shape between test-spectrum vector and reference-spectrum vector. To solve this problem, the NED method is defined in Eq. (6):
where
Information-based TD methods are based on evaluating the information-entropy (IE) characteristics of the spectral vectors. This method analyzes the similarity between vectors based on the amount of information provided, and is not affected by amplitude or shape. However, the false-alarm rate is higher in the detection results of this method, compared to that for other methods.
IE describes the uncertainty of a signal and reflects the amount of information it carries. Mutual information (MI) is used to describe the correlation between two systems; that is, the amount of information contained in both systems. The MI between the test-spectrum vector and the reference-spectrum vector is defined as Eq. (7):
where
Statistics-based TD methods measure similarity by calculating the correlation between vectors [19]. These methods are not affected by light intensity, shadows, or other conditions. Representative methods include the normalized-correlation-coefficient (NCC) and spectral-correlation-angle (SCA) methods. The NCC and SCA methods are defined as follows:
where
where cov(x, y) represents the covariance between x and y. R_{xy} represents the correlation coefficient between x and y. σ_{x} and σ_{y} represent the standard deviations of x and y respectively. The range of the NCC value is [0, 1]; The larger the value is, the higher the degree of similarity. The range of the SCA value is [0, π / 2]; The smaller the value is, the higher the degree of similarity.
The principle of the CEM method is to design a linear filter vector w = [w_{1}, w_{2}, ..., w_{n}]^{T} that minimizes the energy output of the hyperspectral data and satisfies Eq. (12):
The output of test spectrum y after passing through the filter vector is given by Eq. (13):
Therefore, the average output energy of all pixels in HSI is given by Eq. (14):
where
Eq. (15) can be solved by the Lagrange-multiplier method:
The CEM method is applied to each pixel to achieve TD.
To successfully evaluate the detection effectiveness of TD methods, the detection results are analyzed qualitatively and quantitatively. Qualitative analysis is used mainly to visually measure the number, contour, and integrity of targets in the detection results with the naked eye. The closer the detection results are to the real target distribution, the clearer the contour and the higher the integrity, and exhibit better detection.
In fact, a quantitative analysis of the detection results is more convincing. The receiver-operating-characteristic (ROC) curves [20] and area-under-the-curve (AUC) values are used to measure the detection results. The ROC curve reflects the relationship between the detection rate and the false-alarm rate; The abscissa indicates the false-alarm rate P_{f} [21], while the ordinate indicates the detection rate P_{d}. This two-dimensional relationship is based on the three-dimensional ROC curve containing a common threshold τ (the number of equal intervals between 0 and 1). The three-dimensional ROC curve and its projection are shown in Fig. 1.
The three-dimensional ROC curve is projected into three vertical directions to obtain the τ − P_{f} (false-alarm rate) curve, τ − P_{d} (detection rate) curve, and P_{d} − P_{f} (ROC) curve. The τ − P_{f} curve reflects the ability of the detector to suppress the background. The τ − P_{d} curve reflects the ability of the detector to enhance the target. The P_{d} − P_{f} curve reflects the overall performance of the detector.
The two-dimensional ROC curve is essentially composed of τ − P_{f} and τ − P_{d} curves. Multiple groups of P_{f} and P_{d} values can be obtained by setting different thresholds τ. The ROC curve can be obtained by combining the values of P_{f} and P_{d} corresponding to each threshold. P_{f} and P_{d} are defined as follows:
where N_{d} represents the number of detected true target pixels, which is the number of pixels that actually belong to the target and are considered as such by the detector. N_{t} represents the total number of target pixels in the image, N_{f} represents the number of false-alarm pixels detected, and N_{tot} represents the total number of pixels in the image.
In fact the ROC curve is not continuous, but consists of discrete points, each of which represents the false-alarm rate and detection rate corresponding to a threshold. The AUC value quantitatively describes the degree of inclination of the ROC curve toward the upper left. The AUC value is the area surrounded by the ROC curve and the abscissa. The more the ROC curve is bent toward the upper left, the larger the AUC value, the better detection of the detection method, and the higher the detection reliability. The smaller the AUC value, the worse the detection results and the lower the detection reliability.
It is not entirely accurate to determine the merits of a detection method based on the ROC curve or AUC value. Especially in land-based applications, spectral data show a certain degree of uncertainty depending on the imaging conditions. The performance of TD methods under multiple imaging conditions is variable. It is obviously inappropriate to consider only TD performance under a single imaging condition. For example, a detection method can have a better detection effect under some imaging conditions yet face difficulty in detecting the target under other conditions, which cannot confirm the performance of the detection method and cannot describe the applicability of the method under land-based imaging conditions. Therefore, we need to comprehensively consider the average value and degree of dispersion of the detection results. The larger the average value and the smaller the degree of dispersion, the better the detection effect and the better applicability under land-based imaging conditions.
Detection results are diverse depending on land-based imaging conditions. Therefore, the analysis of detection results under land-based imaging conditions is based on the comprehensive analysis of multiple detection results. This approach requires that the detection method meet not only the requirements of detection accuracy, but also those of stability; That is, the detection method with a higher level of detection accuracy under each imaging condition has a greater degree of applicability. Based on this idea, this paper proposes the SC and effective AUC (EAUC) concepts. The calculation process is as follows.
First, the τ − P_{f} and τ − P_{d} curves are analyzed. The step size of the threshold value is constant, and thus the τ − P_{f} and τ − P_{d} curves are evenly distributed on the horizontal axis. Therefore, the average false-alarm and detection curves can be obtained by taking both the τ − P_{f} and τ − P_{d} curves under various imaging conditions:
where K represents the number of imaging conditions. P_{fi} and P_{di} represent the false-alarm rate and detection rate curves under the i^{th} imaging condition respectively.
Then the SC value is calculated to combine the stability characteristics of the detection results, and the AUC value under different imaging conditions is weighted according to the SC value to calculate the EAUC value. The SC value combines the average value and degree of dispersion, which more comprehensively reflects the performance of the detection methods under various imaging conditions. The larger the SC value, the more similar the ROC curve of the detection results to the average ROC curve, and the greater the weight involved in calculating the EAUC value. The SC value is calculated as follows:
where σ_{i} represents the standard deviation for the i^{th} imaging condition. P_{fik} and P_{dik} represent the k^{th} element of false-alarm rate P_{fi} and detection rate P_{di} for the i^{th} imaging condition respectively. Q represents the number of the threshold τ (equal intervals between 0 and 1).
The EAUC values of all detection methods are calculated by combining the SC and AUC values under all imaging conditions:
By analyzing the calculation process for the EAUC, it is found that the EAUC can reflect both the average value and degree of dispersion of the detection results. Therefore, in theory this approach can comprehensively evaluate the applicability of a detection method under land-based imaging conditions.
The experimental data are obtained by an HSI-300 hyperspectral imaging system based on acousto-optic tunable filter (Gooch & Housego Co., Ilminster, England). The size of each hyperspectral image data is 712 × 1,002 pixels. The spectral resolution (band interval) of the HSI-300 spectrometer is 4 nm, its bandwidth is 2.3 nm, and its band range is 449–801 nm. A total of 89 bands are obtained. The experiment takes place in Shijiazhuang, Hebei, China, on June 1, 2023. The geographical coordinates are 38° 27′ N, 114° 30′ E. The weather is clear during the experiment. The simulation platform consists of an Intel Core i7-7700HQ CPU with a dominant frequency of 2.80 GHz, 8 GB RAM, and MATLAB R2017a software.
The targets include two pieces of camouflage clothing, sample 1, and a brown plate, sample 2. The background is composed mainly of green leaves, soil, leaves of Chinese ilex, and weeds. Samples 1 and 2 are almost integrated into the background, and it is difficult to detect their specific positions with the naked eye. The distributions and spectral curves of the two samples are shown in Fig. 2, and their real locations are in Fig. 3.
Imaging conditions are arbitrary in land-based applications, meaning that the use of any imaging condition is possible; Therefore, the more imaging conditions, the better the analysis of the detection results. However, it is impossible to obtain all imaging conditions completely, and only representative groups of imaging conditions can be selected. Experiments are conducted at 10:00, 10:30, 11:00, 11:30, 12:00, 12:30, 13:00, 13:30, 14:00, and 14:30, and ten groups of hyperspectral data are obtained. The imaging conditions are listed in Table 1.
TABLE 1. Imaging conditions.
Group No. | Time | Solar Zeinth Angle (˚) | View Zeinth Angle (˚) | Relative Azimuth Angle (˚) |
---|---|---|---|---|
1 | 10:00 | 34.5 | 62 | 148.5 |
2 | 10:30 | 29.1 | 62 | 139.2 |
3 | 11:00 | 24.2 | 62 | 130.1 |
4 | 11:30 | 20.0 | 62 | 107.6 |
5 | 12:00 | 17.2 | 62 | 85.3 |
6 | 12:30 | 16.6 | 62 | 66.4 |
7 | 13:00 | 18.5 | 62 | 35.5 |
8 | 13:30 | 22.1 | 62 | 20.2 |
9 | 14:00 | 26.7 | 62 | 13.7 |
10 | 14:30 | 32.0 | 62 | 6.1 |
Since the obtained hyperspectral data denote the reflected intensity values, radiometric calibration of the data is needed; That is, the normalization of the obtained hyperspectral data to the form of reflectance. On the one hand, this approach can overcome the effect of radiation intensity, and on the other hand, it can also reduce the effect of noise. We use a standard whiteboard to normalize the obtained hyperspectral data, as shown in Fig. 4. The standard whiteboard used in this experiment is a polytetrafluoroethylene (PTFE) board that has been calibrated. Its reflection properties are uniform in all directions, and it can be approximated as a Lambertian reflector. The reflection intensity of this board is higher than that of the other pixels in each band. The spectral reflectance r(λ) of each pixel in this experiment can be calculated by Eq. (22):
where r_{s}(λ) represents the reflectance of the standard whiteboard and value of the reflectance was 0.98. I_{s}(λ) represents the obtained spectral intensity value of the standard whiteboard. I(λ) represents the obtained spectral intensity value of each pixel. Then the spectral of each pixel in the hyperspectral data is transformed into reflectance, and the spectral curve represents the spectral reflectance as determined by the standard whiteboard. The spectral curves of two samples under all imaging conditions are calculated, as shown in Fig. 5.
We take two samples as the desired targets, and the applicability of each is analyzed. The mean spectra of sample 1 and sample 2 under all conditions are taken as known reference spectra. The detection results of the SAC, NED, NCC, CEM, MI, and NSGA methods under all conditions are analyzed. Since the measured spectral data consists solely of positive values, the ranges of the six detection methods are all in [0, 1]. Thus it is unnecessary to unify the dimensions of the different detection methods. The closer the detection results are to 1 for the SAC, CEM, NCC, MI, and NSGA methods, the more similar the test spectrum is to the reference spectrum. The NED method has a larger calculation result when the test spectrum is more similar to the reference spectrum. Therefore we reversed the calculation result of the NED method using D_{new} = 1 − D_{origin}, where D_{origin} represents the calculation result before reversal and D_{new} represents the calculation result after reversal. The detection results for two samples under all imaging conditions are shown in Figs. 6 and 7.
First, we qualitatively analyze the detection results. As shown in Fig. 6, the NSGA method hardly detects the target. The NSGA method measures the change rate of the spectra, and the rate of sample 1 under visible light is very similar to those of the other objects in the background. As a result, the NSGA method detection results are not proper in distinguishing different ground objects. The results of the MI method under some conditions are clearly visible and show a large contrast with the background. However, under conditions 7 and 10, the target (sample 1) is obliterated by the background and it is almost impossible to detect the target, indicating that the stability of the MI method is low and that the detection results are greatly affected by the imaging conditions. The SAC, NED, NCC, and CEM methods can detect the target under all conditions. The gray value of the target position is the largest, and its detection results are less affected by the imaging conditions and are more stable. Therefore, for the detection of sample 1 against this background, the SAC, NED, NCC, and CEM methods exhibit better detection effects, followed by the MI method, and the NSGA method exhibits the worst detection effects.
Similarly, the detection results for the target (sample 2) with the NSGA method shown in Fig. 7 are worse than those of other methods. The MI method can detect the target under conditions 1, 3, 4, 5, 6, and 8, but the detection results are poor under other conditions, which also indicates that the degree of stability of the MI method is low. The SAC and NED methods can detect the target under all conditions, but there is an obviously serious false alarm rate in the detection results. The CEM method can detect the target under all conditions, and the contrast between the target and the background is obvious. However, there are also some false alarms in the detection results. The results of the NCC method under the first nine conditions are clearly visible, but under condition 10 the target is obliterated by the background and is almost impossible to detect.
According to the comprehensive analysis, the reason for the poor detection results of the NCC method under condition 10 can be caused by errors in the data. However, this method shows significantly better detection results than those of the other methods under the first nine groups of conditions, and the false alarm rate is low. The CEM method can detect the target under almost all imaging conditions, but it is also accompanied by a certain degree of false alarms. Although the SAC and NED methods also show detection effects, false alarms are more obvious when these methods are used. The MI method has a detection effect only under a few conditions, while the NSGA method has no detection effect at all. In summary, the detection methods for sample 2 can be listed in order of effectiveness: CC, CEM, SAC, NED, MI, and NSGA methods.
Next, we quantitatively analyze the results. The step-in threshold is set to 0.001, and the number of steps is 1,000. Figure 8 shows the ROC curves for all detection methods under all imaging conditions.
By analyzing the trend of the ROC curves in Fig. 8(a), we find that the ROC curves for the SAC, NED, NCC and CEM methods are significantly curved toward the upper left, proportional to the increasing detection performance. As seen from Fig. 8(b), the ROC curves for the MI method are greatly affected by the imaging conditions, and the detection performance is unstable. The ROC curves for the SAC, NED, NCC, and CEM methods are significantly curved toward the upper left, compared to those for the NSGA and MI methods. This finding is consistent with that of the previous qualitative analysis. The AUC values are shown in Table 2.
TABLE 2. Area-under-the-curve (AUC) values for different methods.
AUC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.9959 | 0.9959 | 0.9998 | 0.9999 | 0.9878 | 0.5914 |
Condition 2 | 0.9899 | 0.9899 | 0.9989 | 0.9997 | 0.9868 | 0.5719 | |
Condition 3 | 0.9796 | 0.9796 | 0.9992 | 0.9995 | 0.9875 | 0.5661 | |
Condition 4 | 0.9645 | 0.9645 | 0.9985 | 0.9974 | 0.9795 | 0.5403 | |
Condition 5 | 0.9616 | 0.9616 | 0.9983 | 0.9976 | 0.9843 | 0.5754 | |
Condition 6 | 0.9565 | 0.9565 | 0.9985 | 0.9967 | 0.9819 | 0.5319 | |
Condition 7 | 0.9834 | 0.9834 | 0.9974 | 0.9959 | 0.9698 | 0.5200 | |
Condition 8 | 0.9578 | 0.9578 | 0.9981 | 0.9932 | 0.9734 | 0.5296 | |
Condition 9 | 0.9776 | 0.9776 | 0.9970 | 0.9904 | 0.9618 | 0.5350 | |
Condition10 | 0.9965 | 0.9965 | 0.9891 | 0.9897 | 0.8959 | 0.4609 | |
Mean | 0.9763 | 0.9763 | 0.9975 | 0.9960 | 0.9709 | 0.5423 | |
Sample 2 | Condition 1 | 0.9717 | 0.9718 | 0.9980 | 0.9909 | 0.9389 | 0.2829 |
Condition 2 | 0.9579 | 0.9578 | 0.9969 | 0.9898 | 0.9281 | 0.2409 | |
Condition 3 | 0.9495 | 0.9495 | 0.9965 | 0.9903 | 0.9221 | 0.2212 | |
Condition 4 | 0.9385 | 0.9385 | 0.9960 | 0.9876 | 0.8791 | 0.2466 | |
Condition 5 | 0.9320 | 0.9320 | 0.9961 | 0.9845 | 0.8527 | 0.2451 | |
Condition 6 | 0.9275 | 0.9275 | 0.9926 | 0.9845 | 0.9037 | 0.2489 | |
Condition 7 | 0.9416 | 0.9416 | 0.9914 | 0.9799 | 0.7585 | 0.2693 | |
Condition 8 | 0.9320 | 0.9320 | 0.9923 | 0.9790 | 0.8371 | 0.2916 | |
Condition 9 | 0.9506 | 0.9506 | 0.9895 | 0.9744 | 0.7023 | 0.3174 | |
Condition10 | 0.9669 | 0.9670 | 0.7583 | 0.9686 | 0.3787 | 0.3077 | |
Mean | 0.9468 | 0.9468 | 0.9708 | 0.9832 | 0.8101 | 0.2672 |
Table 2 shows that the AUC values of the NCC and CEM methods are almost always the largest among all imaging conditions, with a small range of variation. This finding shows that the NCC and CEM methods perform better and have the best level of applicability under various land-based imaging conditions, compared to other methods. When detecting sample 2, the average AUC value of the NCC method is smaller than that of the CEM method. This is not consistent with that of our previous qualitative analysis, because only the average AUC value is calculated without combining the stability characteristics of all detection results. Consequentially, the comprehensive evaluation result is greatly affected by a certain result. Thus large deviation occurs in the comprehensive evaluation result, which calculates only the average AUC value.
Notably, the source of the average AUC values is obtained only by averaging the AUC values in Table 2, not by averaging the ROC curves. Since the distribution of the points in the ROC curve on the coordinate axis is not continuous but discrete, the ROC curve cannot be averaged directly. To comprehensively evaluate the detection effect under land-based imaging conditions, this study uses the EAUC method to comprehensively analyze and evaluate the applicability of the detection methods.
First, the average false alarm and detection curves are shown in Fig. 9. Since the average false-alarm curves and average detection curves have the same threshold value τ, the average false-alarm-rate curves and average detection-rate curves can be combined into the average ROC curves according to Eq. (22), as shown in Fig. 10. Then, the SC values of all detection methods under various imaging conditions are calculated according to Eq. (23), as shown in Table 3. Finally, the calculation results for the EAUC values are shown in Table 4.
TABLE 3. Stability coefficient (SC) values.
SC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.0396 | 0.0421 | 0.0461 | 0.0626 | 0.0489 | 0.0834 |
Condition 2 | 0.0630 | 0.0657 | 0.0693 | 0.0537 | 0.0727 | 0.1016 | |
Condition 3 | 0.1405 | 0.1485 | 0.1588 | 0.0737 | 0.1065 | 0.1829 | |
Condition 4 | 0.1773 | 0.1579 | 0.2257 | 0.1391 | 0.2922 | 0.1116 | |
Condition 5 | 0.1292 | 0.1224 | 0.1213 | 0.1434 | 0.0579 | 0.1230 | |
Condition 6 | 0.0865 | 0.0978 | 0.0807 | 0.1797 | 0.0971 | 0.2064 | |
Condition 7 | 0.1397 | 0.1324 | 0.1309 | 0.0987 | 0.1200 | 0.0569 | |
Condition 8 | 0.0876 | 0.0865 | 0.0424 | 0.0777 | 0.0896 | 0.0474 | |
Condition 9 | 0.0849 | 0.0822 | 0.0886 | 0.0554 | 0.0618 | 0.0643 | |
Condition 10 | 0.0518 | 0.0645 | 0.0363 | 0.1160 | 0.0533 | 0.0226 | |
Sample 2 | Condition 1 | 0.0513 | 0.0534 | 0.0538 | 0.0489 | 0.0671 | 0.1333 |
Condition 2 | 0.0689 | 0.0666 | 0.0732 | 0.0608 | 0.0469 | 0.0809 | |
Condition 3 | 0.0683 | 0.0830 | 0.1027 | 0.0601 | 0.0859 | 0.0381 | |
Condition 4 | 0.1473 | 0.1619 | 0.1575 | 0.0713 | 0.1576 | 0.0927 | |
Condition 5 | 0.1615 | 0.1469 | 0.1673 | 0.2311 | 0.1380 | 0.1393 | |
Condition 6 | 0.1209 | 0.0961 | 0.1022 | 0.1745 | 0.1039 | 0.1552 | |
Condition 7 | 0.1499 | 0.1418 | 0.1899 | 0.0917 | 0.1529 | 0.0823 | |
Condition 8 | 0.0925 | 0.0932 | 0.0451 | 0.1281 | 0.1243 | 0.0774 | |
Condition 9 | 0.1074 | 0.1210 | 0.0803 | 0.0670 | 0.0935 | 0.0557 | |
Condition 10 | 0.0321 | 0.0362 | 0.0281 | 0.0665 | 0.0300 | 0.1451 |
TABLE 4. Effective area under the curve (EAUC) values.
EAUC | SAC | NED | NCC | CEM | MI | NSGA |
---|---|---|---|---|---|---|
Sample 1 | 0.9732 | 0.9737 | 0.9980 | 0.9960 | 0.9745 | 0.5513 |
Sample 2 | 0.9425 | 0.9432 | 0.9870 | 0.9834 | 0.8309 | 0.2635 |
As seen from Table 4, the EAUC value is higher in order of NCC, CEM, MI, NED, SAC, and NSGA when the target is sample 1. This is consistent with the previous qualitative analysis. However, it is inconsistent with the sequence of average AUC values, because the EAUC method combines the stability characteristics of the detection results rather than considering only the average value of the detection, which can reflect the detection effect more objectively. At the same time, the detection results under all imaging conditions are relatively stable, and there is no large degree of variation. The use of the EAUC-value method does not affect the average size of the detection results. When sample 2 is the target, the EAUC value is higher in order of NCC, CEM, NED, SAC, MI, and NSGA. This sequence is inconsistent with the average AUC value but consistent with the qualitative analysis. This also proves that the EAUC-value method is more reasonable for the evaluation of detection results and can effectively realize the evaluation of the detection method under land-based imaging conditions, compared to the other methods considered.
The results of the above analysis of the test results based on the EAUC value are consistent with those of the qualitative analysis, indicating that the EAUC value is more reasonable than other values for measuring detection effectiveness. Compared to the direct calculation of the average AUC value, the calculation of the EAUC value combined with the SC value gives different weights to the AUC value under different imaging conditions. Thus the value of EAUC is combined with the evaluation results under various imaging conditions, reducing the contingency of the evaluation, which is conducive to our selection of TD methods with high accuracy and strong stability under land-based imaging conditions.
From the perspective of land-based applications, this paper first introduces several different types of detection methods and classical evaluation methods, and then analyzes the detection results of TD methods and their applicability under land-based imaging conditions. The levels of detection performance of TD methods under different imaging conditions are not constant. Therefore, a traditional evaluation method that measures only the detection performance of the TD method under one condition is not suitable under land-based imaging conditions. Finally, the proposed EAUC-value method is used to comprehensively analyze the applicability of each detection method under land-based imaging conditions. This method fully combines detection accuracy and stability; That is, a method with high levels of accuracy and stability has a larger EAUC value and can realize the evaluation of TD methods under various imaging conditions. This method is suitable for the evaluation of detection performance under land-based imaging conditions. However, due to the limited number of imaging conditions, many groups of imaging conditions are selected based only on subjective experience. The more imaging conditions there are, the more conducive the method is to the more comprehensive calculation of EAUC values; The more able it is to reflect the applicability of the detection method under land-based conditions; And the more conducive it is to selecting appropriate detection methods for land-based applications. This paper also proposes objective and rational detection methods through the EAUC-value method, considering the uncertainty characteristics of the spectra into account.
We would like to thank the Army Engineering University of PLA, Electronic and Optical Engineering Department for financial and equipment support in developing this work. We would also like to thank the anonymous reviewer for their helpful and insightful comments, which significantly improved the quality of the manuscript.
National Natural Science Foundation of China (Grant no. 62005319).
The authors declare that there are no conflicts of interest related to this article.
Data underlying the results presented in this paper are not publicly available at this time, but may be obtained from the authors upon reasonable request.
TABLE 1 Imaging conditions
Group No. | Time | Solar Zeinth Angle (˚) | View Zeinth Angle (˚) | Relative Azimuth Angle (˚) |
---|---|---|---|---|
1 | 10:00 | 34.5 | 62 | 148.5 |
2 | 10:30 | 29.1 | 62 | 139.2 |
3 | 11:00 | 24.2 | 62 | 130.1 |
4 | 11:30 | 20.0 | 62 | 107.6 |
5 | 12:00 | 17.2 | 62 | 85.3 |
6 | 12:30 | 16.6 | 62 | 66.4 |
7 | 13:00 | 18.5 | 62 | 35.5 |
8 | 13:30 | 22.1 | 62 | 20.2 |
9 | 14:00 | 26.7 | 62 | 13.7 |
10 | 14:30 | 32.0 | 62 | 6.1 |
TABLE 2 Area-under-the-curve (AUC) values for different methods
AUC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.9959 | 0.9959 | 0.9998 | 0.9999 | 0.9878 | 0.5914 |
Condition 2 | 0.9899 | 0.9899 | 0.9989 | 0.9997 | 0.9868 | 0.5719 | |
Condition 3 | 0.9796 | 0.9796 | 0.9992 | 0.9995 | 0.9875 | 0.5661 | |
Condition 4 | 0.9645 | 0.9645 | 0.9985 | 0.9974 | 0.9795 | 0.5403 | |
Condition 5 | 0.9616 | 0.9616 | 0.9983 | 0.9976 | 0.9843 | 0.5754 | |
Condition 6 | 0.9565 | 0.9565 | 0.9985 | 0.9967 | 0.9819 | 0.5319 | |
Condition 7 | 0.9834 | 0.9834 | 0.9974 | 0.9959 | 0.9698 | 0.5200 | |
Condition 8 | 0.9578 | 0.9578 | 0.9981 | 0.9932 | 0.9734 | 0.5296 | |
Condition 9 | 0.9776 | 0.9776 | 0.9970 | 0.9904 | 0.9618 | 0.5350 | |
Condition10 | 0.9965 | 0.9965 | 0.9891 | 0.9897 | 0.8959 | 0.4609 | |
Mean | 0.9763 | 0.9763 | 0.9975 | 0.9960 | 0.9709 | 0.5423 | |
Sample 2 | Condition 1 | 0.9717 | 0.9718 | 0.9980 | 0.9909 | 0.9389 | 0.2829 |
Condition 2 | 0.9579 | 0.9578 | 0.9969 | 0.9898 | 0.9281 | 0.2409 | |
Condition 3 | 0.9495 | 0.9495 | 0.9965 | 0.9903 | 0.9221 | 0.2212 | |
Condition 4 | 0.9385 | 0.9385 | 0.9960 | 0.9876 | 0.8791 | 0.2466 | |
Condition 5 | 0.9320 | 0.9320 | 0.9961 | 0.9845 | 0.8527 | 0.2451 | |
Condition 6 | 0.9275 | 0.9275 | 0.9926 | 0.9845 | 0.9037 | 0.2489 | |
Condition 7 | 0.9416 | 0.9416 | 0.9914 | 0.9799 | 0.7585 | 0.2693 | |
Condition 8 | 0.9320 | 0.9320 | 0.9923 | 0.9790 | 0.8371 | 0.2916 | |
Condition 9 | 0.9506 | 0.9506 | 0.9895 | 0.9744 | 0.7023 | 0.3174 | |
Condition10 | 0.9669 | 0.9670 | 0.7583 | 0.9686 | 0.3787 | 0.3077 | |
Mean | 0.9468 | 0.9468 | 0.9708 | 0.9832 | 0.8101 | 0.2672 |
TABLE 3 Stability coefficient (SC) values
SC | SAC | NED | NCC | CEM | MI | NSGA | |
---|---|---|---|---|---|---|---|
Sample 1 | Condition 1 | 0.0396 | 0.0421 | 0.0461 | 0.0626 | 0.0489 | 0.0834 |
Condition 2 | 0.0630 | 0.0657 | 0.0693 | 0.0537 | 0.0727 | 0.1016 | |
Condition 3 | 0.1405 | 0.1485 | 0.1588 | 0.0737 | 0.1065 | 0.1829 | |
Condition 4 | 0.1773 | 0.1579 | 0.2257 | 0.1391 | 0.2922 | 0.1116 | |
Condition 5 | 0.1292 | 0.1224 | 0.1213 | 0.1434 | 0.0579 | 0.1230 | |
Condition 6 | 0.0865 | 0.0978 | 0.0807 | 0.1797 | 0.0971 | 0.2064 | |
Condition 7 | 0.1397 | 0.1324 | 0.1309 | 0.0987 | 0.1200 | 0.0569 | |
Condition 8 | 0.0876 | 0.0865 | 0.0424 | 0.0777 | 0.0896 | 0.0474 | |
Condition 9 | 0.0849 | 0.0822 | 0.0886 | 0.0554 | 0.0618 | 0.0643 | |
Condition 10 | 0.0518 | 0.0645 | 0.0363 | 0.1160 | 0.0533 | 0.0226 | |
Sample 2 | Condition 1 | 0.0513 | 0.0534 | 0.0538 | 0.0489 | 0.0671 | 0.1333 |
Condition 2 | 0.0689 | 0.0666 | 0.0732 | 0.0608 | 0.0469 | 0.0809 | |
Condition 3 | 0.0683 | 0.0830 | 0.1027 | 0.0601 | 0.0859 | 0.0381 | |
Condition 4 | 0.1473 | 0.1619 | 0.1575 | 0.0713 | 0.1576 | 0.0927 | |
Condition 5 | 0.1615 | 0.1469 | 0.1673 | 0.2311 | 0.1380 | 0.1393 | |
Condition 6 | 0.1209 | 0.0961 | 0.1022 | 0.1745 | 0.1039 | 0.1552 | |
Condition 7 | 0.1499 | 0.1418 | 0.1899 | 0.0917 | 0.1529 | 0.0823 | |
Condition 8 | 0.0925 | 0.0932 | 0.0451 | 0.1281 | 0.1243 | 0.0774 | |
Condition 9 | 0.1074 | 0.1210 | 0.0803 | 0.0670 | 0.0935 | 0.0557 | |
Condition 10 | 0.0321 | 0.0362 | 0.0281 | 0.0665 | 0.0300 | 0.1451 |
TABLE 4 Effective area under the curve (EAUC) values
EAUC | SAC | NED | NCC | CEM | MI | NSGA |
---|---|---|---|---|---|---|
Sample 1 | 0.9732 | 0.9737 | 0.9980 | 0.9960 | 0.9745 | 0.5513 |
Sample 2 | 0.9425 | 0.9432 | 0.9870 | 0.9834 | 0.8309 | 0.2635 |