Digital holography has been applied in various fields such as 3D displays, cultural asset exhibitions and performances, lithography, and microscope technologies [1_–7]. Since digital holograms can reconstruct actual 3D objects from planar display panels without causing dizziness to the observer, it is often referred to as the ultimate technology for creating 3D scenes. In recent years, various kinds of visual systems that directly apply holographic technology such as holographic head-up-displays (HUD) for vehicles have been studied and demonstrated [8_–12]. In such applications, one of the most important features is real-time computation of computer-generated holograms (CGH) to provide the current status of a vehicle and environmental information to passengers in real time. However, real-time calculation of CGH patterns is still a major bottleneck to fully commercializing holographic HUD systems due to the large amount of computation time and memory usage required to calculate CGH.

To reduce the calculation time of CGH, various studies have been conducted [13_–15]. For example, using polygon-based computational methods instead of point cloud [16], algorithms for the efficient numerical propagation of wavefields [17] were improved using wavelet transform-based calculation [18], and further studies have been conducted for fast calculation of CGH in many different routes [19_, 20]. Moreover, the use of multiple GPUs to calculate holograms in real time [21], fast CGH calculation from 3D objects consisting of multiple layers of line-drawn objects [22], or fast CGH algorithms based on pinhole-type look-up tables have been proposed [23].

However, applying the above-mentioned methods still seems to be not good enough for the practical use of CGH calculation for real-time application since the major bottleneck of plane-by-plane type CGH calculation time, caused by conducting 2D fast Fourier transform (FFT), is too large. For example, with MATLAB 2021a, computation time of 2D FFT of random phase data still consumes more than 0.3 s with a high-end specification GPU (NVIDIA GeForce RTX3090; NVIDIA, CA, USA) for 4096 × 4096 resolution, which required full data of CGH for a commercialized ultra-high definition (UHD) resolution spatial light modulator (SLM). Since the 2D FFT calculation in CGH generation is a necessary part, a method for calculating high-resolution CGH from low-resolution 2D FFT calculation may be needed. Conventional upscaling algorithms such as bicubic interpolation do not work well in CGH patterns, so a unique approach for upscaling CGH patterns may be needed.

In this work, we proposed to dramatically reduce the computational time of CGH using a concave lens array multiplied to the array of down-scaled resolution calculated CGH. Using this approach, we propose a method to generate a 4096 × 4096 resolution CGH in real time in a relatively low-performance computational GPU environment. Since the shape of the concave lens functions was fixed by the distance between CGH and object plane, computing CGHs of different objects with the same distance is much faster by pre-computing concave lens functions. We also compare the quality of reconstructed holograms from the CGHs computed by the conventional angular spectrum method (ASM) and proposed work through simulations and experiments.

Figure 1 shows a schematic of the proposed CGH resolution upscaling method using a concave function array compared to conventional ASM. For simplicity, we will assume that the pixel number (N) of x- and y-directions for CGH are identical. As shown in Fig. 1(a), direct calculation of CGH through conventional ASM will need calculation of FFT or inverse FFT at least two times, and this calculation often accounts for the major portion of the overall calculation time. Therefore, to further reduce the overall calculation time, the proposed method performs three steps as shown in Fig. 1(b). At first, the reduced-size object based on the magnification factor M is prepared so that the resized object is scaled by 1/M with respect to the main object along the x- and y-directions. For the case of z-direction, the resized plane is assumed to be placed on the distance z_r from the CGH plane based on the simple image equation relation between the distance of initial depth (z_i) of the object and the focal length of the lens function array (f) as follows:

Figure 1. Schematics of computer-generated hologram (CGH) generation methods based on (a) conventional angular spectrum method (ASM) calculation and (b) the proposed method.

(1)$$\frac{1}{-z_r}+\frac{1}{z_i}=\frac{1}{f}$$

Considering the reconstruction process of CGH, the reduced image is our “input” and the magnified original image is our “output” for the imaging equation. Therefore, the minus sign in z_r is applied because the virtual object is placed in front of the lens plane (CGH plane). In our configuration, the relation between z_i and z_r is determined by the magnification factor, z_i = Mz_r; therefore, the focal length of the lens function array is determined as f = z_i / (1 − M), which has a negative value (concave lens function) for our desired image magnification condition (M > 1).

After preparing the resized image, the second step is a calculation of simple CGH with a reduced resolution condition. Since FFT calculations are conducted along N/M by N/M resolution, the calculation time is dramatically reduced during this stage, and M × M array patterning of a reduced resolution CGH is conducted.

The final step of the proposed method is the multiplication of the concave lens array function into the array of reduced resolution CGH. The concave lens array function needs to be appropriately shifted to provide the reconstructed object image placed with the same size and position. In other words, after being magnified by the corresponding concave lens array function, each reduced-resolution CGH will generate an identical reconstructed image observed by a different viewing angle. To provide such a condition, the center location of each concave lens function c(i, j) should be located in the x-y plane as follows:

(2)$$c\left(i,j\right)=\left(-\frac{Np}{2}+\frac{Np}{M-1}\left(i-1\right),-\frac{Np}{2}+\frac{Np}{M-1}\left(j-1\right)\right)i,j=1,2,\dots M$$

Here, i, j are the block number of the arranged CGH pattern, which has an integer value from 1 to M. N and p are the number of pixels and pixel pitch, respectively. We assumed that the CGH calculation is done for an identical condition of N and p along the x and y directions. It is noteworthy that the center position of the arrayed CGH is not the same as c(i, j), and each block of the arrayed CGH has their center position of h(i, j) in the x-y plane as follows:

(3)$$h\left(i,j\right)=\left(-\frac{Np}{2}+\frac{Np}{M}\left(i-\frac{1}{2}\right),-\frac{Np}{2}+\frac{Np}{M}\left(j-\frac{1}{2}\right)\right)i,j=1,2,\dots M$$

Therefore, each of the concave lens functions is relatively shifted with respect to the center of CGH blocks with the amount of

(4)$$c\left(i,j\right)-h\left(i,j\right)=\left(Np\left(\frac{2i-M-1}{2M\left(M-1\right)}\right),NP\left(\frac{2j-M-1}{2M(M-1)}\right)\right)i,j=1,2,\dots M$$

in order to provide the identical image location of the reconstructed hologram from each CGH block.

Since the proposed method is focused on a dramatic reduction of the calculation time of the CGH that is upscaled from a low-resolution CGH, there must be a trade-off relationship with fast calculation time. The loss of information compared to direct calculation of the full-resolution CGH might be observed as a degradation of image quality. Figure 2 shows the simulation results using the same object under conditions of wavelength at 627 nm and focal length (z_i) of 17 cm by the proposed method of M = 5 (z_r = 3.4 cm) with the calculation of a computing system with a CPU (Intel i5-9600K 3.70 GHz; Intel, CA, USA), 32GB RAM (Samsung Electronics, Suwon, Korea) and GPU (NVIDIA GeForce GT 1030; NVIDIA, CA, USA). Note that the performance of the GPU of our computing system is cost effective and can be widely used for normal consumers.

Figure 2. Comparison of degradation between the conventional angular spectrum method (ASM) and the proposed method: (a) computer-generated holograms (CGH) calculated by conventional ASM (3840 × 3840 resolution, 3.6 μm pixel pitch), (b) CGH calculated by the proposed method of M = 5, (c) low-resolution CGH calculated without arraying the pattern (768 × 768 resolution, 3.6 μm pixel pitch), and (d)–(f) reconstructed hologram image each of (a)–(c).

Figure 2(a) shows CGH calculated by conventional ASM as a control group (3840 × 3840 resolution, 3.6 μm pixel pitch) and Fig. 2(b) shows CGH calculated by the proposed method of M = 5. Figure 2(d) shows a reconstructed hologram image of Fig. 2(a) and Fig. 2(e) shows a reconstructed hologram image of Fig. 2(b). When comparing Fig. 2(d) and Fig. 2(e) by enlarging the character parts, although speckle noise is increased by applying the proposed method, the reconstructed object is still observable. It is noteworthy that the CGH calculation time in Fig. 2(d) required 3879 ms in our system, whereas calculation of Fig. 2(e) only required 96 ms. On the other hand, Fig. 2(c) shows a single portion of CGH calculated by reduced image without M × M arraying, which requires a similar calculation time as that of Fig. 2(e), and Fig. 2(f) shows a reconstructed hologram image of Fig. 2(c). The reconstructed image of Fig. 2(f) shows that speckle noise becomes more severe if a single ASM is used without M × M arraying. Therefore, we expect that the trade-off between image quality and calculation time can be further improved by applying our method.

Since a discretized lens function needs to be multiplied to the array of reduced-size CGH pattern, we need to investigate the possible range of image reconstruction without unwanted diffracted images occurring. When the proposed method is applied, the unwanted diffracted images occur when the focal length is too close, whereas speckle noises become severe when the focal length is too far.

Table 1 shows a comparison of the reconstructed hologram image’s degradation effect with respect to focal length and magnification factor, and Table 2 shows a reconstructed hologram image of Table 1. In Table 2, The images in the first row shows the reference, which conducts the full ASM simulation of the 4k × 4k resolution CGH, and the images in the second to fourth rows are reconstructed images when M = 2, M = 4, and M = 8, respectively. To selectively observe the degradation effect caused by diffracted image generation and speckle noises, the brightness of reconstructed images are normalized by the maximum value of light intensity.

TABLE 1. Comparison of computer-generated holograms (CGH) noise and blurring according to focal length and magnification factor.

Magnification Factor [M]	Focal Length (zi)
	5 cm	10 cm	17 cm	30 cm	50 cm
Original
2×
4×
8×

TABLE 2. Reconstructed hologram image of Table 1.

Magnification Factor [M]	Focal Length (zi)
	5 cm	10 cm	17 cm	30 cm	50 cm
Original
2×
4×
8×

Except for the case of the first row (conventional ASM), the diffracted image can be seen at a focal length of 5 cm (first column). Although such diffracted images disappear in the 10 cm distance condition, weakly blurred images remain as background noise can be seen (second column). However, from about 17 cm, such noises disappear so that the image quality of the proposed method is reasonably good compared to conventional ASM. When the focal length of the image plane is far away, speckle noise increases much faster when M is larger. The fourth and fifth columns of Table 2 show the reconstructed image when the focal lengths are 30 cm and 50 cm, respectively. It can be seen that the speckle noise becomes quite severe for the M = 8, 50 cm condition, and such restriction of the image plane is also caused by the trade-off relationship of our method instead of a much faster calculation speed.

As shown in Table 2, diffracted images up to 17 cm and increased speckle noises are major problems caused by CGH generation using lens functions. The main cause of noise is the empty space of CGH that occurs when calculating CGH using reduced images according to given parameters (resolution, pixel pitch, wavelength, size of object and focal length), which can also be confirmed in Table 1. Because other parameters are difficult to modify in a fixed environment, it is recommended to design an appropriate focal length to avoid these problems. Since it only shows a qualitative analysis in Table 2, we also extracted the quantitative figure of merit parameter from each of the reconstructed images. We extract the structural similarity index measurement (SSIM) value that can be used for comparing the degree of distortion of structural information in images. This method can precisely determine how much the image of the experimental group differs from its reference group, which can be expressed as [24]

(5)$$SSIM\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{({\mu }_x^2+{\mu }_y^2+C_1)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)}$$

In Eq. (5), μ_x, μ_y indicate the x- and y- directional luminance of each pixel, calculated by averaging the nearby intensity, and σ_x, σ_y, and σ_xy indicate the contrast coefficients, which are calculated by standard deviation of nearby intensity. C₁ and C₂ are small adjustment constants that are included to avoid instability when pixel luminance is close to zero, which are often defined as a very small value compared to μ_x, μ_y in general cases [24].

Based on these calculations, Fig. 3(a) shows the comparison results of SSIM graph according to the focal length and magnification factor M under the conditions of 3840 × 3840 resolution, 3.6 μm pixel pitch and 627 nm wavelength. The experimental group is a reconstructed hologram image, whereas reference image is given as the original 2D image of a vehicle instrument that is shown in Fig. 2 and Table 2. Since the diffracted images gradually disappeared as the focal length became farther, SSIM becomes higher until the focal length value. For all magnification factors except the case of conventional ASM (M = 1), SSIM was the highest at a point with a focal length of approximately 17 cm. After that, the blurring became worse as the focal length became farther, and the SSIM gradually decreased. As the value of the magnification factor became larger, slight degradation of SSIM was also observed and the fluctuation of SSIM also increased. The fluctuation of SSIM might be because the brightness of the reconstructed hologram image is not constant, which is caused by the speckle noise generation by random phase multiplication. However, using SSIM, we observed that generating CGH with the proposed method does not significantly degrade the quality of the reconstructed hologram image by increasing the value M. The focal length of 17 cm presented above may vary depending on the given parameter. When the proposed method is used under a condition of 1.8 μm pixel pitch with a larger diffraction angle characteristic, the empty space of CGH may be removed at a closer distance. As a result, the SSIM value is maximized at 4 cm, closer than 17 cm, as shown in Fig. 3(b).

Figure 3. Structural similarity index measurement (SSIM) comparison graph by focal length, magnification factor and pixel pitch of (a) 3.6 μm and (b) 1.8 μm.

Table 3 shows the exact time consumption of the proposed method according to the variation of the value M. As shown in Fig. 1, by applying the proposed method, we can dramatically reduce the calculation time of the ASM process, but there is concern about time consumption caused by other processes such as image resizing, lens function preparation, M by M arraying, and multiplications. Fortunately, among those additional calculations, the lens function preparation needs to be done only once when the center depth of hologram image is predetermined, and it can also be prepared as a lookup table designed for various depth conditions. The image resizing process can also be ignored if we start the calculation from the low-resolution images. Therefore, the major time consumption needs for each frame of the CGH image are M by M arraying, ASM calculation of low-resolution image, and lens function multiplications. The summation of these times can be defined as the actual calculation time for a single frame. According to the results shown in Table 3, we have shown that the actual calculation time for a single frame of the proposed method is dramatically reduced from six times for M = 2 to 39 times for M = 10 compared to conventional ASM. Since the calculation is done on a relatively low-cost personal computing system with the MATLAB program, the absolute time consumption can be further improved if the proposed scheme is applied to a better computing system or directly programmed on field programmable gate array (FPGA) boards.

TABLE 3. Comparison of computational time of 3840 × 3840 computergenerated holograms (CGH) according to magnification factor (computation time is obtained as the average of 100-time measurements).

Magnification Factor [M]	1	2	4	5	6	8	10
Image Resizing (ms)	N/A	43	34	30	35	33	32
Calculate ASM (ms)	3879	520	140	96	68	43	38
Lens Function Preparation (ms)	N/A	944	826	788	761	701	663
Arraying CGH and Multiply Lens Function (ms)	N/A	45	32	29	31	31	29
Total Computation Time (ms)	3879	1552	1032	943	895	808	762
Actual Calculation Time for Single Frame (ms)	3879	608	206	155	134	107	99

To compare the image quality between each magnification factor condition, which is shown numerically in Table 2, we demonstrate a hologram generation experiment as shown in Fig. 4. Here, we used the Peony-62a SLM kit (May Display, Suwon, Korea), which has a pixel resolution of 3840 × 2160 and pixel pitch of 3.6 μm. Therefore, we could directly use the CGH pattern calculated in the previous section after simply cropping 840 rows from the upper and lower sides. Figure 4(a) shows a photograph of the optical setup. Since the SLM kit basically contains green and blue laser sources, we filtered them by using a red filter, and a convex lens was used for magnifying the reconstructed hologram image. After that, a neutral density (ND) filter and iris were used for controlling the brightness of the hologram and blocking the non-diffracted beam from the SLM, respectively. The CGH patterns have an off-axis angle of 2° to avoid the non-diffracted beam signal, and the reconstructed hologram magnified by the lens was observed by a complementary metal-oxide semiconductor (CMOS) digital camera. Figure 4(b) shows a reconstructed hologram image from CGH generated by the conventional ASM method, while Figs. 4(c)–4(h) are reconstructed hologram images of CGH using the proposed method with magnification factors of M = 2, M = 4, M = 5, M = 6, M = 8, and M = 10, respectively. In this experiment, the size of the reconstructed hologram image was magnified three times by using two convex lenses for the convenience of photographing. According to the results shown in previous sections, the focal length of the first image (the image directly reconstructed by the SLM) is set to 17 cm. As expected, although the background noises slowly increase as the magnification increases, and image blurring starts to be observed at high magnification factor conditions such as M = 8 or M = 10, the quality of the reconstructed hologram still seems to be good compared to the original ASM method, so that we can find a reasonable condition of M based on the resolution of the original image.

Figure 4. Optical setup and experimental results: (a) optical setup of experiment, (b) reconstructed hologram image of computer-generated holograms (CGH) generated with a conventional method. Reconstructed hologram images of CGH generated by (c) ×2, (d) ×4, (e) ×5, (f) ×6, (g) ×8, and (h) ×10 magnification factor using the proposed method.

We propose a method to generate CGH with rapid calculation time by using a pre-calculated concave lens function dependent on focal length and reduced-resolution CGH pattern. Compared to the conventional ASM method, the proposed method was able to generate approximately 6 to 39 times faster depending on the magnification factor. Therefore, the CGH pattern was able to generate at a rate of 10 fps in ×10 magnification factors in relatively low-performance computing devices. Experiments with the SLM show that the quality of reconstructed hologram images does not fall significantly even in high magnification factor, except for slight blurring of the reconstructed image and increase in noise in the background; therefore, the proposed method is expected to solve space and cost problems in various holographic systems. The rapid calculation time proportional to 1/M2 of the proposed method with slight degradation of image quality can be used in various kinds of holographic applications such as HUD systems, real-time mobile hologram generation, and CGH data compression for hologram broadcasting.

Institute of Information & communications Technology Planning & Evaluation (IITP) funded by MSIT (No. 2019-0-00001, Development of Holo-TV Core Technologies for Hologram Media Services); Technology Innovation Program funded by the MOTIE, Korea (P20010672).