Because many biomolecular materials have absorption peaks in the midinfrared (mid-IR) spectral range, mid-IR hyperspectral imaging enables the identification of sample structure as well as sample chemistry, without staining or prior information about sample materials [1–8]. With the development of quantum cascade lasers (QCLs), it is possible to measure the mid-IR absorption spectrum faster than with Fourier-transform infrared spectroscopy (FT-IR) [9–11]. Compared to the fastest high-definition FT-IR spectroscopy, QCL-based mid-IR spectroscopy measures a tissue microarray (TMA) much faster, enabling real-time imaging of Amoeba proteus [12, 13].

Pathological studies based on mid-IR hyperspectral analysis have been conducted through the analysis of various clinical samples including respiratory fluid, urine, blood, liver fluid, and biopsy samples [14]. Combined with the statistical pattern recognition of the mid-IR spectrum to reveal the pathological characteristics of prostate tissue, a study on the classification of benign and malignant prostate epithelium was proposed [15]. In addition, a study was proposed in which a label-free spectrum of a lung cancer pathology sample was measured and diagnosed using a multivariate analysis method [7]. By extracting features using various methods, such as principal-component analysis (PCA), linear discriminant analysis (LDA), and hierarchical cluster analysis (HCA), studies to analyze the spectrum of cancer tissue using machine-learning (ML) -based algorithms, such as support vector machine (SVM) and random forest (RF), have also been proposed [11, 16–22]. Owing to the development of a deep-learning (DL) -based algorithm, Liu et al. [23] reported results from a study to detect breast cancer metastasis using a convolutional-neural-network (CNN) [24–27] model using gigapixel pathological images, and Gao et al. [28] reported a study that classified cancer molecular subtypes. In addition, studies confirming the results of early lung cancer diagnosis using DL for spectroscopic analysis of circulating exosomes have also been reported [29], and DL studies to analyze gastric cancer tissues using the difference in spatial characteristics of the spectrum have also been reported [30]. Furthermore, a study was proposed in which electroencephalograms (EEGs) or electrocardiograms (ECGs) were converted into spectrograms and analyzed with CNN [31, 32]. In prior studies, to improve the absorption-spectrum deviation and resonant Mie scattering (RMieS) [33, 34] that occur in biological samples, complex preprocessing, such as PCA [35–41], RMieS correction [33, 34], and secondary differentiation, was required [11]. In addition, ML-based algorithms such as RF yield different results based on the initial value or threshold [42, 43].

In this study, a method of converting the mid-IR spectra of colon cancer tissues into images and classifying them into groups N and T using CNN is proposed. This method simplifies the preprocessing compared to previous studies, reduces the effect of the initial value, and eliminates the need to set a threshold.

For the sample, we use a TMA (US Biomax, Maryland, USA) and routine pathological slide for training and verification of the CNN model and the SVM model respectively.

The sample is sliced to minimize the effect of the peak absorption value, by maintaining the sample of the paraffin block at a constant thickness of 4–5 µm. Paraffin with a mid-IR absorption peak is removed by immersing the section in xylene. In addition, serial sections of the sample are compared using hematoxylin and eosin (H&E) staining, to confirm the location of the sample.

All TMA purchased from US Biomax are diagnosed pathologically, as shown in Fig. 1(a). Among the TMA, E (diagnosed as colon cancer tissue) and J (diagnosed as noncancerous colon tissue) are used as test data. The actually measured TMA is deparaffinized and confirmed using 1650-cm⁻¹ absorption images, which are shown in Fig. 1(b). Figure 1(c) is an H&E-stained image with pathological diagnosis of a routine pathological slide at Chosun University Hospital, and Fig. 1(d) shows the 1650-cm⁻¹ absorption image of the deparaffinized routine pathological slide.

Figure 1. Measurement samples: (a) H&E stained tissue micro array (TMA) (US Biomax, Maryland, USA). TMA is sliced to minimize the effect of the peak absorption value, by maintaining the sample of the paraffin block at a constant thickness of 4–5 µm. Paraffin with a mid-IR absorption peak is removed by immersing the section in xylene. In addition, serial sections of the sample are compared using H&E staining, to confirm the location of the sample. (b) 1650-cm⁻¹ images of colon cancer tissues E3, E10 and noncancerous colon tissues J3, J10 in deparaffinized TMA (scale bar: 250 µm). (c) Routine pathological slide diagnosed by pathologists at Chosun University Hospital. (d) 1650-cm⁻¹ image of the deparaffinized routine pathological slide (scale bar: 1 mm).

All methods and experimental protocols are approved by the Institutional Review Board of Chosun University Hospital (IRB no. CHOSUN 2021-09-016). Signed informed consent has been waived by the review board. All enrolled patients have provided informed consent. All procedures are conducted in accordance with the approved guidelines and regulations for human experimental research.

QCL-based mid-IR microscopy (Spero QT; Daylight Solutions Inc., San Diego, USA) is used for mid-IR hyper spectral imaging, as shown in Fig. 2. The hyperspectral image (λ = 5.5–10.5 µm) is measured in 480 × 480 pixels using a microbolometer-array detector within approximately 45 s. The scale can be measured with a low magnification of 4× [field of view (FOV) 2 × 2 mm, spatial resolution 12 µm] and a high magnification of 12.5× (FOV 650 × 650 µm, spatial resolution 5 µm).

Figure 2. Schematic of the experimental setup. The mid-IR hyperspectral image is measured at a wavelength from 5.5 to 10.5 µm in 480 × 480 pixels using a microbolometer-array detector within approximately 45 s. The sample is measured with a low magnification of 4× [field of view (FOV) 2 × 2 mm, spatial resolution 12 µm] in transmission mode.

The mid-IR image of the sample is measured by choosing low magnification in transmission mode. To minimize variation of the spectral values due to the environment, the chamber is purged with nitrogen for 10 minutes, and the background is measured and applied once every 20 minutes.

Due to the preparation of the sample while maintaining a constant thickness, the Beer-Lambert Law can be applied, and the influence of the sample thickness on the mid-IR absorption spectrum can be reduced. The Beer-Lambert Law shows the relationship between transmission and light absorption due to light passing through a substance, which causes light attenuation, such as reflection, diffraction, refraction, and scattering. This law states that the absorption of light is proportional to the thickness and the concentration of the sample. The formula used for the Beer-Lambert Law [44] is as follows (T: transmission rate, I₀: intensity of incident light, I_t: intensity of transmission light, A: absorption of light, α: absorption factor, d: thickness of substance):

(1)$$T=\frac{I_t}{I_0}$$

(2)$$A=\log \left(\frac{I_0}{I_t}\right)=-\log T=ad$$

At the time of measurement, the absorption peak’s position of Amide I (a type of protein) could be shifted by ±2–4 cm⁻¹. For better comparison of the characteristics of the spectra, the spectra of all pixels must be aligned so that Amide 1 peak is located at 1650 cm⁻¹. To remove substrate-only parts without tissue samples from 230,400 pixels, only pixels with a peak value at 1650 cm⁻¹ between 0 and 2 are included in the training and evaluation data. The target data required for training data is set to group T to 1 and group N to 0. Group T is set to the TMA that is pathologically diagnosed with colon cancer, and group N is set to the TMA that is pathologically diagnosed with noncancerous colon tissue.

The preprocessing images are divided into a training dataset, test dataset, and validation dataset. The training dataset is the dataset used to train the model; the test dataset is the dataset that extracts 20% of random data from the training dataset, to check whether the training is performing well; and the validation dataset, called test data, is a separate dataset from the training dataset.

In SVM [45–48] model training, 10 principal components (PCs) extracted from the preprocessed data through PCA [35–41] are input. The SVM model is classified into two groups: colon cancer tissue group T and noncancerous colon tissue group N. As a result, the classification vector of the SVM model appears as Fig. 3(a) in the feature graph of primary PC and secondary PC. PCA is performed using the sklearn.decomposition. PCA function provided by Python (Python Software Foundation, DE, USA), and the SVM model is created using the sklearn.svm.SVC function provided by Python.

Figure 3. Preprocessing and structure of the convolutional-neural-network (CNN) model: (a) the classification vector of the support vector machine (SVM) model in the feature graph of primary principal component (PC) and secondary PC, (b) To train the CNN model, which is an image-based deep-learning (DL), the mid-IR spectrum is converted into an image. The number of duplicate spectra is set to 426, which is the number of measured wave numbers, because the input data of the CNN model must be square, and (c) to reduce the image size and training time, the data size is converted from 426 × 426 to 28 × 28 and entered into a CNN model, consisting of a convolution and maxpooling repetitive layer and a fully connected layer.

To train the CNN model, which is an image-based DL, the mid-IR spectrum is converted into an image. To classify the hyperspectral mid-IR image of the tissue pixel by pixel, the mid-IR spectrum of each pixel is duplicated several times to create an image. The number of duplicate spectra is set to 426, which is the number of measured wave numbers, because the input data of the CNN model must be square.

To train with large amounts of image data, the image size is converted from 426 × 426 to 28 × 28 using the resize function of OpenCV. The layer of the CNN model is designed to repeat convolution and max-pooling, and the feature nodes are connected as a fully connected layer; it is constructed using the Conv2D, MaxPooling, Flatten, Dropout, and Dense functions provided by Keras (Keras, CA, USA). All convolution layers consist of 3 × 3 kernels [stride = 1, padding = 1, and Rectified Linear Unit (ReLu)], and all pooling layers consist of 2 × 2 kernels.

The SVM model and CNN model are trained with 2,521,326 input data. The trained SVM and CNN models are tested using TMA and a routine pathological slide. Among routine pathological slides, 1 and 2 of Figs. 4 and 5 (diagnosed as cancer) and 3 and 4 of Figs. 4 and 5 (diagnosed as normal mucosa) are tested. The formula used for the spectral classification result of the hyperspectral mid-IR image is as follows (R_c: ratio of cancer tissue, R_n: ratio of noncancerous tissue, N_c: number of pixels corresponding to cancer tissue, N_n: number of pixels corresponding to noncancerous tissue, N_t: number of total pixels):

Figure 4. Classification results for the support vector machine (SVM) model for colon cancer and noncancerous colon tissue in a routine pathological slide: (a) s 1 and 2 (diagnosed as cancer) and 3 and 4 (diagnosed as normal mucosa) in a routine pathological slide (scale bar: 1 mm), (b) classification result for the SVM model of 1 and 2, diagnosed as cancer (scale bar: 50 µm), and (c) classification result for the SVM model of 3 and 4, diagnosed as normal mucosa (scale bar: 50 µm).

Figure 5. Classification results for the convolutional-neural-network (CNN) model for colon cancer and noncancerous colon tissue in a routine pathological slide: (a) locations 1 and 2 (diagnosed as cancer) and 3 and 4 (diagnosed as normal mucosa) in a routine pathological slide (scale bar: 1 mm), (b) classification result for the CNN model of 1 and 2, diagnosed as cancer (scale bar: 50 µm), and (c) classification result for the CNN model of 3 and 4, diagnosed as normal mucosa (scale bar: 50 µm).

(3)$$R_c\left(R_n\right)=\frac{N_c\left(N_n\right)}{N_t}\times 100$$

In the SVM model, 89.27% of the 367,780 pixels of tissues diagnosed with colon cancer among TMA are classified into group T, and 18.95% of the 310,001 pixels of tissues diagnosed with noncancerous colon tissue among TMA are classified as group N (Fig. 6). In addition, of the 668,947 pixels in the routine pathological slide, 80.16% are classified as group T and 19.84% are classified as group N. As a result, the test results of the SVM model tend to be biased toward group T regardless of the pathology diagnosis, for both TMA and routine pathological slide. The SVM regularization parameter and the type and coefficient of the SVM kernel are not optimized, so it is predicted that the SVM model will have difficulty in classifying PCs.

Figure 6. Classification results for the support vector machine (SVM) model for colon cancer and noncancerous colon tissue in tissue micro array (TMA): left, colon cancer tissue (E1:10); right, noncancerous colon tissue (J1:10) (scale bar: 250 µm).

In the CNN model, 63.29% of the 367,780 images of tissues diagnosed with colon cancer among TMA are classified as group T. Of the 310,001 images of tissues diagnosed with noncancerous colon tissue among TMA, 80.17% are classified into group N (Fig. 7). In addition, of the 668,947 images of a routine pathological slide, 35.20% are classified as group T and 64.80% as group N. Unlike the test results for the ML-based SVM model, the CNN model confirms that colon cancer and noncancerous colon tissues can be classified, and there is no need to optimize parameters and kernels, so it is more effective in classifying.

Figure 7. Classification results for the convolutional-neural-network (CNN) model for colon cancer and noncancerous colon tissue in tissue micro array (TMA): left, colon cancer tissue (E1:10); right, noncancerous colon tissue (J1:10) (scale bar: 250 µm).

In prior studies, complex preprocessing such as PCA, RMieS [33, 34] calibration, and secondary differentiation were required to improve RMieS [33, 34] and absorption-spectrum deviations occurring in biological samples [11]. In addition, for ML-based algorithms such as RF, the resulting value is based on the initial value or threshold [42, 43]. Therefore, a method for converting the mid-IR spectrum of colon cancer into an image and classifying it into group N or T using CNN has been proposed. The preprocessing is simpler than that in previous studies, and the influences of the initial value and threshold are less. The CNN model was evaluated based on TMA and a routine pathological slide. As a result of the CNN model in this study classifying TMA images by pixel, 63.29% of pixels were classified as Group T in cancer tissue TMA and 80.27% as Group N in normal tissue TMA. Therefore, the CNN model of this study has demonstrated that both TMA and a routine pathological slide can be classified as colon cancer or noncancerous colon tissue. In addition, the ML-based SVM model showed results skewed toward group T, and most of the mid-IR spectra of J (diagnosed as noncancerous colon tissue) were classified as group T. On the contrary, it was confirmed that the CNN model has relatively high accuracy in the classification of colon cancer and noncancerous colon tissue, compared to the SVM model. However, the accuracy of the CNN model is still not sufficient, because the number of epochs based on the model training time is insufficient compared to the number of data, and the target data are comprehensive because there is no pixel-by-pixel diagnosis for colon cancer tissue. To increase the accuracy of the model, pixel-by-pixel annotation of the colon cancer tissue by the pathologist is required, and either the number of epochs or the depth of the model should be increased by using a deeper CNN algorithm.

In conclusion, a method for converting the mid-IR spectrum of colon cancer to an image and classifying it into group N or T using CNN was proposed, and the model was evaluated using TMA and a routine pathological slide. We visualized mid-IR spectra into a 2D image, for classification of colon cancer tissue using a CNN model. As a result, the complex preprocessing was simplified and the deviation in the results that was observed in previous studies was reduced. This study showed that the CNN model could classify colon cancer and noncancerous colon tissue. In addition, the ML-based SVM model showed results that were skewed toward group T, and it was confirmed that the mid-IR spectra of J (diagnosed with noncancerous colon tissue) were mostly classified as group T. Compared to the SVM model, the CNN model has relatively high accuracy in the classification of colon cancer and noncancerous colon tissue. We expect that in future works the sensitivity, specificity, and accuracy of the CNN model could be assessed through comparison with the pathologist’s decision data with pixel-by-pixel target data. Thus, it is expected that the CNN model of this study will be helpful in pathological diagnosis through the mid-IR absorption spectrum, which is an objective indicator.

The authors declare no conflicts of interest.

Data underlying the results presented in this paper are not publicly available at the time of publication, which may be obtained from the authors upon reasonable request.

This work was supported by the Korean Medical Device Development Fund grant funded by the Korean government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, and the Ministry of Food and Drug Safety) (Project Number: 1711137874, KMDF_PR_20200901_0008), and was also partially supported by a grant from the “HPC Support” Project, supported by the “Ministry of Science and ICT” and NIPA.

Korean Medical Device Development Fund grant funded by the Korean government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, and the Ministry of Food and Drug Safety) (Project Number: 1711137874, KMDF_PR_20200901_0008).