INTRODUCTION

Hyperspectral (HS) image processing is a branch of applied optics that studies raster images, each pixel of which is associated not with a separate value of light intensity, but with the full spectral decomposition of optical energy within a certain frequency range. The main idea of HS systems is the concept of a spectral signature, i.e., the assumption that the amount of electromagnetic radiation that is reflected or absorbed by a material varies with wavelength for any given material based on the inherent physical structure and chemical composition [1].

The term of hyperspectral image was first introduced in [2]. HS was originally developed for mining and geology, used for ore and oil prospecting, and was for a long time limited to laboratory use. Today, the situation is rapidly changing thanks to the advent of compact and affordable HS cameras based on sensors that are monolithically integrated into the imager.

HS technologies are beginning to be used in various fields, including control and inspection of agricultural crops. Miniaturization and low cost make them an ideal solution for medical devices. For example, HS devices are currently being developed for use in ophthalmology. In the future, HS technologies may well extend to household appliances and devices such as mobile phones, where the built-in hyperspectral sensor can be used to analyze food products in real time, allowing one to quickly determine the quality and freshness of products.

The results of HS imaging can be effectively used in any application to solve complex problems of classifying the types of the surface under study, determining its state and dynamics, identifying differences between similar classes of observed objects, etc.

The purpose of this work is to analyze existing approaches and develop a concept for constructing small-sized hyperspectrometers that are easy to use and allow for prompt recording of data for their further thematic processing.

HYPERSPECTRAL IMAGING

During hyperspectral recording, an image is a three-dimensional data array (data hypercube), two coordinates of which in \(X\), \(Y\) plane correspond to the spatial coordinates of the probed object, and the third coordinate \(\lambda\) correspond to the number of the spectral channel, i. e., a sweep for each pixel of the spatial image of the probed object is carried out along the wavelength \(\lambda\) (each point in the image corresponds to a full spectrum).

Analysis of modern developments of HS equipment shows the following trends.

—Significant increase in the production of HS spectrometers designed for ground-based systems and installed on quadrocopters.

—The desire to miniaturize HS spectrometers, reduce their weight and size characteristics, and attempts to completely abandon mechanical components in scanning and spectral decomposition units through the use of optoelectronic components in planar design.

HS IMAGING

In accordance with the definition of a hyperspectrometer, the main components of the device are as follows.

—A system for fragmenting the input image due to operational control of the input aperture implemented by mechanical scanning or using modern spatial light modulators based on micromirror arrays (MMAs).

—Spectral separation system based on a polychromator with a dispersing element that separates the input light flux into spectral components.

Figure 1 shows four modern modes of obtaining hyperspectral data. They are point scanning (whiskbroom), linear scanning (pushbroom), flat scanning (pushbroom), and single shot [3].

Fig. 1
figure 1

Hyperspectral modes of data acquisition. (A) Whiskbroom, (B) and (C) pushbroom, and (D) single shot.

Full size image

In the pushbroom mode (Fig. 1B), the object is scanned with a slit diaphragm instead of pixel scanning and the radiation is decomposed into a spectrum using dispersive elements. At each scanning step, one slice of the HS image is recorded in the \(X\), \(\lambda\) plane, so the time to form a full frame using such a camera is several seconds or even minutes. This mode is the most common and allows one to create compact devices with low weight, easier control, and a higher signal level. It is used, in particular, in space and aviation-based systems, where the surface under study is scanned due to the movement of the carrier vehicle.

In the flat scanning mode, shown in Fig. 1C, a channel-by-channel recording array is formed; is consists of several images taken sequentially, each of which contains one slice of the HS image in the \(X\), \(Y\) plane. Pushbroom mode should be used for a short enough time to avoid the risk of inconsistencies in the spectral band level (saturation or underexposure). Flat scanning is not suitable for moving environments and applications, in which the camera or surveillance objects move randomly in space.

The mode that receives all spatial and spectral data simultaneously (Fig. 1D) is known as a single shot. The technology of the mode is under development. The mode does not provide high spatial resolution and requires significant computing power.

SLIT TECHNOLOGIES

Slit light input systems are a mandatory feature of most HS scanning devices. The main purpose of the entrance slit is to clearly select a fragment of the image for feeding it to the spectrum separation system. The dimensions (width and height) of the entrance slit are a key factor that affects the throughput of the spectrometer. The spectral resolution of the device depends on the width of the image in the input hole if it exceeds the pixel width in the detector. The divergence angle of the light entering inside depends on the width of the slit. The slits can have different widths (from 5 to 800 \(\mu\)m or more); the slit height is 1–2 mm (standard). Basically, spectrometers use slits with widths of 10, 25, 50, 100, 200 \(\mu\)m, etc. In systems that use optical fibers to supply the light beam, the size of the fiber package must match the size of the entrance slit. This usually reduces light scattering and increases the throughput of the device [4, 5].

ALTERNATIVE TO SLIT SCANNING

One of the most promising ways to fragment the input image through operational control of the input aperture is the use of modern spatial light modulators (SLMs), which allow operational control of the input aperture of a dispersive spectrometer.

This approach provides the following possibilities:

—the ability to quickly program reconfigure the device to operate in different modes;

—simple implementation of the mode with electronically controlled scanning of the input slit formed by the SLM;

—high recording speed (frame-spectrum per milliseconds);

—software control of resolution over a wide range;

—the possibility of significantly accelerating the acquisition of hyperspectral information by working in an interactive mode with the selection of only informationally significant areas of the object for scanning.

The analysis shows that it is advisable to use as SLMs not liquid crystal (LC) transparency, but MMAs mass-produced by Texas Instruments (United States) for video projectors. A micromirror array is an ordered two-dimensional structure containing about \(10^{6}\) micromirrors with a size of \(10{-}15\) \(\mu\)m each, which can be independently switched from the ground state to two positions, the planes of which are rotated by an angle of \(\pm 10^{\circ}{-}12^{\circ}\) relative to the normal to the array plane.

Modern MMAs provide high spatial resolution (megapixels) and temporal resolution (operating frequency is up to 1800 Hz), high contrast, low light loss, high fill factor (\(>90\%\)), almost complete absence of spectral and polarization selectivity (unlike LC transparencies), and high reliability.

Currently, Texas Instruments offers eight high-resolution MMAs (with more than one million pixels) for hyperspectral scanning; they are DLP4500, DLP4500NIR, DLP6500FLQ, DLP6500FYE, DLP9000, DLP9000X, DLP9500, and DLP9500UV [5]. DLP4500NIR array is MMA module with spectral range of \(700{-}2500\) nm, operating frequency of 120 Hz, mirror array dimension of \(912\times 1140\), mirror size of 7.6 \(\mu\)m, and housing size of \(20.7\times 9.1\times 3.33\) mm.

Important advantages of the modulation carried out by the MMAs are the extremely high input speed, small pixel size, large number of pixels, and the natural absence of temporary fluctuations of the pixel signal during the frame time inherent in liquid crystal SLMs. A significant disadvantage of MMAs is the varying degrees of nonflatness of the working reflective field of the system, which is determined by the quality of manufacturing of the chip with micromirrors and the surface of the microcircuit housing window, leading to spatial phase distortions of the modulated signal. The frequency of switching the position of each mirror can reach several kilohertz. Currently, the rate of change of the entire modulator frame is limited by the capabilities of the control interface. It is actually possible to achieve input of images of megapixel dimensions with frame rates up to 1 MHz, and currently for the best serial devices the input speed is provided at the level of 20–30 kHz, which determines the speed of information input using MMAs at several tens of Gbit s\({}^{-1}\) [6, 7].

The use of MMAs for operational control of the input aperture makes it possible to fully provide all modes of data acquisition. Micromirror arrays are also successfully used for systems for generating dynamic scenes when testing thermal imaging devices in the far IR range [8–10].

SPECTRAL SEPARATION OF INPUT LIGHT

The key element of spectral devices that determines their characteristics is the dispersion device [5]. Currently, the main methods for constructing spectral images are wavelength scanning and spatial scanning [7].

Wavelength scanning methods implement a sequential analysis method, which can be performed using a circular variable filter, a filter bank, a liquid crystal tunable filter, or an acousto-optic tunable filter. In the spatial scanning, which is most widely used in the construction of hyperspectral equipment, a prism or grating is used as a dispersive element. Threaded gratings and holographic gratings are used. Threaded gratings are made on a surface with a mirror coating. Holographic gratings are created as a result of the interference of two UV laser beams on a photosensitive layer. They have stable spectral characteristics, but have lower diffraction efficiency. Threaded gratings scatter light quite strongly due to inaccurate manufacturing of the strokes, which leads to deterioration in optical performance. Holographic gratings can reduce the effect of light scattering and increase the output characteristics of the spectrometer. An important advantage of a holographic grating is the ability to create it on curved surfaces, which allows one to simultaneously use the grating as a focusing element. Grating spectrophotometers make it possible to measure spectra in the range from 180 nm to 12 \(\mu\)m with a resolution reaching 0.03 nm.

Their disadvantage is the presence of radiation of higher diffraction orders and the fact that scanning is carried out by rotating the gratings [11].

INTERFERENCE FILTERS ON THE MACH–ZEHNDER PLANAR ELECTROOPTICAL INTERFEROMETER

Devices for spectral separation of a polychromatic optical signal into individual spectral components are widely used in fiber-optic communication systems and are called demultiplexers. Modern demultiplexing technologies provide separation of up to 400 carriers with a step between them of 0.2–0.8 nm using a Mach–Zehnder planar electrooptical interferometer [12].

The fundamental difference between such an interferometer and a classical one is the electrooptical elements in the arms, the length of the optical path in which is determined by the change in the refractive index of the active medium due to the electric field (Pockels effect). The electro-optical effect is practically inertia-free (response speed is on the order of \(10^{-10}{-}10^{-11}\) s), due to which the upper limit of the frequency of oscillations of the electric field strength, in principle, can reach 100 GHz. Enormous progress in the development of fiber-optic communications allowed the developers of the element base to create one of the first photonic integrated circuits (a Mach–Zehnder planar interferometer) on a lithium niobate crystal.

The electrooptical effect in lithium niobate consists of a change in the refractive index of the crystal under the influence of an electric field. In the case when a light wave polarized along the \(Z\) axis propagates in a direction perpendicular to the \(Z\) axis, the refractive index depends linearly on the electric field strength.

Lithium niobate is transparent in the range from 0.4 to 5 \(\mu\)m, which makes it an ideal material for use in integrated optical devices [13].

DESIGN OF A MACH–ZEHNDER INTEGRATED ELECTROOPTICAL MODULATOR

The integrated electrooptical modulator is a Mach–Zehnder interferometer made on an \(X\) cut of a lithium niobate single crystal. An \(X\) cut is a thin plate cut from a single crystal in such a way that the crystallographic axis \(X\) is perpendicular, and the crystallographic axes \(Z\) and \(Y\) are parallel to the cut surface. The interferometer consists of two Y splitters connected by straight sections of channel waveguides (Fig. 2).

Fig. 2
figure 2

Block diagram of a Mach–Zehnder integrated electrooptical modulator.

Full size image

The waveguides are directed along the crystallographic axis \(Y\) and perpendicular to the axis \(Z\). Metal electrodes are deposited next to the waveguides.

When manufacturing a modulator, protective masks are applied to the surface of the wafer using photolithography, through which waveguides are formed and electrodes are deposited.

The refractive index of the channel light guide turns out to be greater than the refractive index in the bulk of the crystal. The difference in refractive indices between the waveguide and the substrate (waveguide contrast) is about 0.1. This method made it possible to create integrated interferometers with a half-wave voltage of 3.2 V.

Optical laser radiation \(P_{\text{in}}\) is fed to the input of the Mach–Zehnder interferometer via a Panda-type optical fiber that preserves the polarization of the radiation. In the Y1 splitter, the radiation is divided into two streams of equal power (\(50\pm 2\%\)) and enters two arms of the interferometer. Optical radiation in waveguides must be polarized in the direction of the \(Z\) axis of the crystal. The electric field strength vector between the electrodes is also directed along the \(Z\) axis. The \(X\) cut is the main cross section of the crystal for waves propagating along waveguides. A wave propagating along the waveguide in the direction \(Y\) and polarized along the \(Z\) axis is an extraordinary wave in the crystal.

When voltage \(U_{\text{in}}\) is applied to the electrodes, electric fields arise in the surface layer of the crystal through which the waveguides pass

$${}E_{z1}\approx E_{z2}\approx\frac{U_{\text{in}}}{\varepsilon_{e}d},$$

where \(d\) is the distance between the electrodes. The electric fields in the arms of the interferometer are directed in opposite directions (see Fig. 2), which causes equal and opposite sign changes in the refractive index in the waveguides due to the electrooptical effect.

When waves propagate along the arms of the interferometer, a phase difference arises between them, the magnitude of which depends on the optical lengths of the waveguides. The optical length is determined by both the physical length of the waveguide and its refractive index (in Fig. 3) [13, 14].

Fig. 3
figure 3

Cross section of the Mach–Zehnder modulator.

Full size image

Photline Technologies (France) produces integrated optical interferometers with an optical path length difference of up to 1000 \(\mu\)m. A number of models of modulators by iXBlue (France) are successfully used in spacecraft and radiophotonic systems for military purposes (COMJAM and AESA). The use of Mach–Zehnder planar interferometers for spectral separation of the input light flux in hyperspectral problems seems very promising, since they have been successfully tested for performing demultiplexing used in telecommunications technologies and many special applications [15].

SMALL-SIZED HYPERSPECTROMETER BASED ON MICROELECTROMECHANICAL SYSTEMS (MEMSs) AND PLANAR PHOTONIC CHIPS WITH FIBER-OPTIC COMMUNICATION BETWEEN OPTICAL ELEMENTS

Figure 4 shows one of the possible structural diagrams of a small-sized hyperspectrometer using a micromirror array and a Mach–Zehnder planar electro-optical interferometer. The input image is fed to a programmable megapixel MMA, which fragments it and transmits it through a matching device and a single-mode optical fiber to a Mach–Zehnder planar interferometer. Spectral separation is carried out due to a programmable phase shift caused by applying a constant voltage to the control electrodes of the interferometer. The selected spectral component is recorded by a matrix photodetector. The fiber simplifies the matching of optical elements; its diameter should be approximately two times the aperture of the photodetector.

Fig. 4
figure 4

Block diagram of a small-sized hyperspectrometer based on the Mach–Zehnder interferometer. (1) Input lens, (2) micromirror array, (3) coupling device, (4) Mach–Zehnder interferometer, (5) output lens, (6) photodetector, (7) computer, (8) power supply, (9) calibration unit, and (10) monitor.

Full size image

In this device, the Mach–Zehnder interferometer performs pixel-by-pixel sequential scanning of the input image with spectral decomposition by wavelength. Its advantages are very high wavelength resolution, ease of isolating spectral components by applying constant voltage to planar channels, and zero mechanical components.

TIME-LAPSE HYPERSPECTROMETER BASED ON SPECTRAL FILTERS APPLIED DIRECTLY TO THE SURFACE OF THE CMOS SENSOR

An alternative option is time-lapse hyperspectrometers (implementing single shot mode) with narrow spectral filters applied directly to the surface of the CMOS sensor (complementary metal oxide semiconductor) at the level of the semiconductor wafer [16–20]. The sensor of such devices is divided into groups of pixels, each of which has a specific spectral filter, and the resulting HS image is formed from the corresponding pixels of different groups simultaneously in a given field of view. For example, a two-megapixel CMOSIS sensor of CMV2000 model (Belgium) accommodates 32 tile sections with a resolution of 256 \(\times\) 256 pixels each. The result is a 32-channel 12-bit image with a resolution of 256 \(\times\) 256 pixels at a speed of about 30 fps in daylight and up to 340 fps (in 10-bit mode) in laboratory conditions with sensitivity in visible and near-IR ranges (600–1000 nm with a channel width of 12 nm).

Ximea GmbH (Germany), a developer of computer vision systems and industrial and scientific cameras, has integrated a two-megapixel CMOSIS sensor of CMV2000 model into its ultracompact and lightweight cameras of the xiQ and xiSpec series. The weight of such cameras is 27 g with sizes of \(26{.}4\times 26{.}4\times 21{.}6\) mm and power consumption of 1.8 W, which makes them the smallest industrial HSI USB3 cameras [21].

Such systems must be tailored to specific applications by determining the required number and width of spectral bands as well as the image resolution in each spectral range. The choice of spectral intervals, their number and location, as a rule, is made at the system design stage and does not change during operation [16–19]. However, there are attempts to create systems with reconfigurable characteristics.

The implementation of the selective spectral addressing mode is feasible, for example, using tunable acoustooptical filters [22, 23], in which it is possible to control the width of the spectral window and simultaneously obtain several spectral images at once [24, 25]. They are based on dynamic diffraction gratings excited by an acoustic ultrasonic wave and carry out spectral filtering of light beams without significant distortion of the images they carry. The main advantages of acoustooptical filters are software control with spectral tuning in microseconds, zero moving mechanical elements, small sizes, high quality of the resulting images, high spectral resolution (up to 0.1 nm), and the ability to create flexible hyperspectral analysis algorithms due to arbitrary spectral addressing within the operating wavelength range.

Control of the spatial resolution and quantization bit depth of individual pixels or their groups is possible on the basis, for example, of modern technologies of CMOS matrices for recording the light flux, which have flexible capabilities for changing the operating modes of the photodetector and allow adding signals from individual pixels and selecting from the total array working window of arbitrary location and size [26]. High-resolution cameras based on CMOS sensors allow one to switch the resolution downward by decimating elements or by cutting out the active window. In the active window selection mode, one can specify the position and size of this window anywhere in the matrix.

In this case, it is necessary to experimentally substantiate the feasibility of controlling information parameters (spatial resolution, spectral resolution, and signal brightness quantization bit depth) for the adaptive functioning of a promising hyperspectral system. Thus, the authors of [27–29] substantiated the effectiveness of spatial and spectral classification at different stages of processing HS-images, which consists in averaging the signals of individual groups of closely spaced pixels. The authors of [30-31] presented a methodology and an experimental substantiation of the possibility of reducing the spectral dimension of hyperspectral data when solving problems of classification of agricultural crops. It has been shown that selecting several spectral channels (10) and their width using a certain algorithm (combining those nearby from the selected channel) slightly reduces the quality of classification compared to classification over the entire sample of spectral channels (200). In considered systems, such a reduction in information content is inevitable and is due to their design features. However, it has certain limitations compared to digital processing of a complete set of HS data. In particular, spatial smoothing is carried out block by block, but not by a sliding window, as in spectral and spatial processing, and spectral channels must be selected from the available ones, nut not formed by their linear combination, as in the principal component analysis.

CONCLUSIONS

Thus, modern technologies make hyperspectrometers relatively accessible for many applications; they are quite compact, lightweight, and suitable for use as observation systems for various purposes.

Today, the most promising seems to be the creation of small-sized HS devices based on MEMSs and planar photonic microcircuits with fiber-optic communication between optical elements based on the use of commercially available key elements, the serial production of which is currently mastered by leading companies developing computer vision systems. Such systems pixel-by-pixel sequentially scann the input image with spectral decomposition along the wavelength using Mach–Zehnder interferometers with zero mechanical components and make it possible to create reliable, easily reconfigurable, and convenient in practice devices for the operational recording of hyperspectral images.

The time-lapse spectrometers with mosaic interference filters of different types applied directly to the surface of the CMOS sensor can be considered as an alternative devices. However, when creating them, it is necessary to experimentally substantiate the feasibility of controlling information parameters (spatial resolution, spectral resolution, and signal brightness quantization bit depth) to solve specific applied problems.