2.1.

Data Collection

Our experiments are performed on RGB RS images collected from eight reservoirs in Brazil using the Google Earth Pro^© software. GE images represent an integration of multiple satellite data sources, mainly DigitalGlobe’s QuickBird commercial satellite and EarthSat.²⁶ Aiming at improving the appearance of the images, the spectral information of images with more than three bands is reduced to RGB.²⁷ Furthermore, the appearance of GE images is improved using color balancing, warping, and mosaic processing.³ The GE platform presents important advantages, such as the fact of being an open database of RS images, of including historical images and of the flexibility in selecting images of different resolutions.

The eight studied reservoirs are Anta, Billings (the largest reservoir in São Paulo, Brazil), Dona Francisca, Guarapiranga, Jaguara, Luiz Barreto, Nova Avanhandav (Nova), and Salto Osório. Their geographic coordinates are listed in Table 1, and their locations are visualized in Fig. 2. For each reservoir, images over two different years are collected (Table 2). A total of 206 images, each with $2683\times 4800\text{pixels}$ are obtained, encompassing varying view altitudes and thus resulting in different resolutions (from $\sim 1\mathrm{up}$ to 2 m).

Table 1

Locations of the studied reservoirs.

Fig. 2

Visualization of the studied reservoirs locations.

Table 2

Acquisition years of each reservoir dataset. Some of the older year images of Luiz and Nova belong to 2004 and 2010, respectively.

2.2.

Data Preparation and Annotation

Data preparation involves two aspects: preprocessing for mosaic image formation and data annotation. The data preparation scheme is illustrated in Fig. 3 using Guarapiranga reservoir samples. The data preparation aims to prepare data for training the phase-1 and phase-2 segmentation models in Fig. 1.

Fig. 3

Proposed data preparation and annotation pipeline.

As shown in Fig. 3, the input images are initially mosaicked to eliminate overlapping areas in collected GE images. Subsequently, the images are annotated into two classes: reservoir and non-reservoir. Constructing the mosaic images is also essential for implementing the next steps.

Next, in order to simplify the contour around the reservoirs, a polygonal approximation is initially carried out.^28–30 This allows controlling the coarseness by the polygonal approximation parameter. Then a rectangular box connecting each pair of consecutive polygon corners is defined. These boxes are enlarged to cover a minimum distance from the border of the reservoir, which is set empirically. The RoIaR is defined as the union of these boxes (see Fig. 3) and used to mask the mosaic image.

The masked RoIaR image is annotated into man-made and non-man-made objects:

2.3.

Phase-1: Reservoir Map Generation

Reservoir segmentation: This step explores a deep neural network that segments input RGB patches into reservoir and non-reservoir. In this process, we trained and compared two encoder–decoder-based models, the U-Net, and SegNet architectures. Following our evaluation, the SegNet-based model emerged as the most effective choice in our experiments. Below, we briefly describe these two architectures.

The U-Net architecture introduced by Ref. 31 is based on a downsampling–upsampling procedure that concatenates feature maps between each encoder and corresponding decoder by skip connections (see Fig. 4). In each step in the encoder path, two $3\times 3$ convolutions followed by a ReLU and a $2\times 2$ max-pooling with stride two are repeated. Furthermore, the number of feature channels in each downsampling step is doubled. After each upsampling in the decoder path, a $2\times 2$ convolution that halves the number of feature channels is applied. These features are concatenated with the cropped feature of the corresponding encoder step, and then two $3\times 3$ convolution-ReLU blocks are implemented.

Fig. 4

U-Net architecture.

Due to the unpadded convolutions utilized in the U-Net, the output size of the model is smaller than the input. Therefore, we avoided unpadded convolutions to keep the size of each output equal to the corresponding input (named U-Net_p). On the other hand, a common strategy in DL research for training the CNNs properly and avoiding training from scratch is to use a pretrained CNN as the initializer or as the fixed feature extractor, called transfer learning. Therefore, we trained a U-Net model whose encoder path was replaced by VGG-16 (named U-Net_v) and initialized that with weights trained on the ImageNet dataset. However, the model overfitted the training set. The last trained U-Net-based model (named U-Net_s) has fewer features. In this model, there is only one convolution block in each layer that is also batch normalized.³²

The SegNet architecture was first introduced by Ref. 33. Similar to the U-Net, SegNet includes an encoder and a decoder part with the advantage that the need for learning to upsample is eliminated. Since each decoder uses pooling indices computed in the max-pooling step of the corresponding encoder. After each convolution layer in the encoder path, a ReLU non-linearity is used, whereas, in the decoder, no ReLU non-linearity is presented. Furthermore, the number of channels per layer is constant (see Fig. 5). In the employed architecture (called SegNet_d), despite the original form, the number of feature channels is doubled at each downsampling step. Batch normalization is applied after each convolution layer.

Fig. 5

SegNet architecture.

The collected images correspond to different reservoirs geographically spread in Brazil and have different visual properties. They may be obtained in different seasons, atmospheric conditions, geological conditions, and so forth. A possible approach to address such variability is to adopt domain adaptation techniques. Since this is out of the scope of this paper, we explored a data splitting approach to ensure variability in the training, validation, and test sets. Samples from every mosaic image are used in these sets in the following proportions: 60% for the training set, 20% for the validation set, and 20% for the test set.

Postprocessing: Feeding models by mosaic images instead of patches are impossible because of the available GPU memory limits. In many cases, patches do not contain important information about objects, such as their shapes, sizes, and locations in the images. However, this information is essential for detecting some water bodies from reservoirs. On the other hand, spectral similarities between objects of different classes also cause errors. Therefore, we proposed a postprocessing stage to fix these errors.

In this stage, the segmented patches are initially assembled to form the reservoir map. The morphological opening is applied to remove small false positive pixels and other non-interesting water objects that are segmented partially or completely as reservoir objects. Next, morphological closing is applied to remove small false negative objects inside the reservoir objects.

Applying morphological transformations with a large kernel causes changes in the shapes of objects predicted as the reservoir. In order to remove errors inside reservoirs (such as floating vegetation) and noisy objects (such as large water bodies around reservoirs), without removing reservoir objects that are separated because of constructed bridges, the following rules are proposed.

2.4.

Phase-2: RoIaR Man-Made Object Segmentation

RoIaR extraction: Once the reservoir is segmented, the next step is to detect and extract the RoIaR. Two possible approaches for RoIaR detection have been considered: polygonal approximation-based and mathematical morphology-based. The polygonal approximation approach has been described in Sec. 2.2, which is the one adopted for dataset annotation. Although this approach is useful for sparse data annotation (because we may control the polygonal approximation parameters), it produces patches of varying sizes that may not be suitable for analyzing man-made objects’ evolution, for instance.

Therefore, a mathematical morphological approach is also explored. Let $I$ denotes the segmented reservoir object and $s$ a structuring element. The dilated reservoir object is defined as $I_d=I\oplus s$, where $\oplus$ is the morphological dilation. The RoIaR $R$ is defined as $R=I_d-I$, where $-$ denotes set difference.

Following the data annotation procedure illustrated in Fig. 3, the detected RoIaR is applied as a mask to the mosaic image for RoIaR extraction. The extracted RoIaR is then segmented into man-made and non-man-made objects.

Man-made object segmentation: Two widely used network architectures for RS segmentation are the pyramid networks and encoder–decoder networks.³⁴ In phase-2, we assessed the PSPNet, FPN, and LinkNet networks, whose details are presented in the next paragraphs.

The PSPNet has been introduced by Ref. 11 and won the ImageNet Scene Parsing Challenge 2016. It is a pyramid pooling module that enables the network to capture the context of the whole image. In this module, the feature map is pooled at different sizes and passed through a convolution layer. Next, these features are upsampled and concatenated with the original feature map and passed through a convolution layer to produce the final prediction (see Fig. 6). We implemented PSPNet with different backbones in this study.

Fig. 6

An overview of PSPNet. The size of feature map channels is denoted below each box. The size of the last feature map in (a) is 1/8 of the input image size.

Figure 7 presents the FPN’s general schema, a network initially proposed by Ref. 35 for object detection. The construction of this architecture involves a bottom-up path, a top-down path, and lateral connections. The scaling step in the bottom-up path (and consequently in the top-down path) is two. Each lateral link combines feature maps from the bottom-up and top-down pathways with the same spatial size. Finally, the feature maps in the top-down stages are upsampled to be the same size as the input image. These feature maps are combined and used to produce the prediction map. The ResNet is used as the backbone, whereas in this study, other backbones have also been experimented.

Fig. 7

Overview of FPN.

The LinkNet architecture proposed by Ref. 36 is a fast semantic segmentation method that is constructed from an encoder and a decoder path (see Fig. 8). Each residual block in the encoder path consists of two consequent convolution blocks. The input of each residual block is bypassed to its output. The decoder blocks consist of three convolution layers, and the middle is a full convolution. The advantage of the proposed architecture is passing the input of each encoder block to the output of the corresponding decoder block.

Fig. 8

Overview of LinkNet architecture.

Splitting data into the training and test sets are reported to work well when the dataset size is modest. On the other hand, the training and test sets must represent possible distributions of the addressed problem. Therefore, 70% for the training set and 30% for the test set are selected randomly from each RoIaR.

Loss function: The focal loss proposed by Ref. 37 and the Dice loss (DL)³⁸ are utilized as the loss functions for training the networks. Focal loss down-weights easy examples and hence helps the model to learn complex examples better. It is reported by Ref. 39 that focal loss works best when the data is highly imbalanced. To see how it works, first, consider the binary cross entropy loss (CE):

Using this notation, we can rewrite Eq. (1) as $\mathrm{CE}(p_t)=-\log (p_t)$. To balance the importance of positive/negative examples, we can consider ${\alpha }_t$ as the weight for class 1 and $1-{\alpha }_t$ for class 0, then $\alpha$-balanced CE will be written as

Finally, to down-weight easy examples, they add factor ${(1-p_t)}^{\gamma }$ to CE where $\gamma$ > 0 is a tunable parameter. Based on the experiment, $\gamma =2$ works best and is used in this study too.

The DL is based on the dice coefficient (DC) [see Eq. (4)]. In the case of binary classification, $A$ is the set of all positive examples, and $B$ is the set of correct predicted positive examples:

Then DC can be expressed as the following form:

3.1.

Performance Evaluation Metrics

We adopted three common statistics, precision [Eq. (7)], recall [Eq. (8)], and $F_1$-score [Eq. (9)], as well as the confusion matrix of segmentation maps:

3.2.

Phase-1 Experimental Results

The trained architectures for this phase are modified versions of U-Net and SegNet. All trained models use the binary cross entropy as the loss function. The learning rate in the Adam optimizer (proposed by Ref. 42) is set to 0.001, which is reduced by a factor of 0.2 after every five epochs with no reduction in validation loss down to ${10}^{-7}$. Although the number of epochs is set to 100, training is stopped after 20 epochs with no reduction in the validation loss. Patches with $416\times 608\text{pixel}$ sizes are fed into the networks, and training, validation, and test sets contain 6017, 2009, and 1998 patches, respectively. Vertical and horizontal flips are two types of data augmentation that each one is applied randomly on 50% of training set patches. The $F_1$-score of trained models in segmenting the training and validation sets are presented in Table 3. As is illustrated in this table, the U-Net_v overfits the training set.

Table 3

Performance of trained architectures for phase-1 segmentation stage.

Table 4 presents the performance of models with healthy learning curves in the segmentation of the validation set, and it is worth noting that SegNet_d outperforms the U-Net-based models. The performance of SegNet_d in segmenting the test set is illustrated in Table 5. Some patches of studied reservoirs with different spectral properties besides their ground truths and SegNet_d, U-Net_s, and U-Net_p prediction outputs are shown in Fig. 9.

Table 4

Performance of models with healthy learning curves for phase-1 segmentation stage on the validation set.

Table 5

SegNet_d performance in segmenting the test set.

Fig. 9

Examples of the test set patches beside their corresponding ground truths and segmentation outputs.

In addition to errors that occur because of spectral similarities between reservoirs and some other objects (such as shadows), there are small water bodies in the images that are segmented as the reservoir by the models. This issue is unavoidable because of feeding patches to the models instead of the original images. Therefore, postprocessing the network outputs is an essential task. Figure 10 presents examples of non-interesting water bodies in the collected dataset and their segmentation results in the generated reservoir maps.

Fig. 10

Examples of non-reservoir water bodies in the collected dataset and corresponding segmentation results.

Morphological operations in postprocessing are highly effective in removing minor errors, as described above. The applied structuring element size for each reservoir is set to 100 divided by the spatial resolution. For instance, if the spatial resolution of a mosaic image is 1 m, the structuring element size is $100\times 100$.

As the reservoirs contain branches, applying morphological operations with large kernel sizes increases FP and FN objects. Accordingly, significant errors are removed by applying the two rules to objects in the produced segmentation maps. Postprocessing solely using rules is time-consuming because of the high number of FP and FN objects in prediction maps, whereas morphological operations speed up this process. Anta-2014 and Nova-2021 mosaic images, besides their ground truths, model outputs, and postprocessing outputs are illustrated in Fig. 11. The SegNet_d performance in segmenting these two reservoirs besides postprocessing performance are presented in Table 6. Applying the proposed postprocessing improves the accuracy of produced reservoir maps except for two of the 16 studied cases.

Fig. 11

Two examples of produced mosaic images, corresponding ground truths, prediction outputs, and postprocessing outputs. Anta-2014 with $\mathrm{11,687}\times \mathrm{14,430}\text{pixel}$ size, and Nova-2021 with $\mathrm{24,830}\times \mathrm{23,193}\text{pixel}$ size are depicted in the first and second columns, respectively.

Table 6

Prediction and refinement performance metrics for Anta-2014 an Nova-2021.

3.3.

Phase-2 Experimental Results

As discussed above, VGG-16, ResNet-50, and ResNet-101 are the most frequented backbones.⁷ In this study, these three backbones besides EfficientNet-B4 have been experimented. All backbones are initialized with weights trained on the ImageNet dataset. The Adam optimizer is used as the optimizer in all models. The initial learning rate is set to 0.0001 or 0.001, which is automatically reduced by a factor of 0.2 after every five epochs with no reduction in validation loss down to ${10}^{-7}$. The mini-batch size is set to two and power of two (up to the possible size based on the model’s size and available memory). The number of epochs in training all models is set to 80. The vertical and horizontal flips are two data augmentation methods that are implemented on different portions of images (up to 0.7). We added dropout regularization ($<0.3$) to the models with overfitting. Furthermore, experiments are on two training sets, a training set containing 70% of data or the oversampled training set. The oversampled images are images with at least 200 man-made objects pixels.

The evaluation metrics for the highest performance model constructed using each architecture are presented in Table 7. These models are all trained on the oversampled training set with a learning rate of 0.0001. Furthermore, the data augmentation rate in these models is set to 0.7 for each data augmentation method, and the dropout regularization is set to 0.0, 0.3, 0.3, and 0.0, respectively. Regarding the $F_1$-score, the best performance belongs to FPN; however, the differences are insignificant. The utilized backbones for each model in this table are ResNet50, VGG-16, VGG-16, and Efficientnet-B4, respectively. Except for the PSPNet that VGG-16 could improve the performance of the model significantly (2.34%), the performances of the rest models are slightly affected by changing their backbones ($<0.73\%$). In our experiments, oversampling images with more than 200 man-made object pixels improved the performances. Despite the expectation, increasing batch size did not increase the performance metrics in all cases. Adding the DL to the focal loss function significantly improved the models’ performances. Although increasing the data augmentation rate prevented overfitting in some cases, in other cases increasing dropout and data augmentation rates were both essential. Though the FPN outperforms the PSPNet, each epoch training time of PSPNet is less than one-third of the FPN. FPN performance in segmenting test set is presented in Table 8.

Table 7

Highest achieved performances using trained models for phase-2 segmentation on training and test sets.

Table 8

FPN performance in segmenting test set into the man-made and non-man-made objects pixels.

Moreover, the FPN performance in segmenting RoI of reservoirs located in the countryside and urban areas are computed separately and shown in Table 9. Some examples of patches besides their ground truths and segmentation outputs are illustrated in Fig. 12. This figure illustrates examples of different types of roads, rooftops, and urban and countryside constructions with different density levels.

Table 9

FPN performance in segmenting countryside and urban man-made objects. C and U are the abbreviations for countryside and urban.

Fig. 12

Examples of studied reservoirs RoI patches beside their corresponding ground truths and prediction outputs.

3.4.

Workflow Evaluation

We evaluated the proposed workflow using a dataset collected from the Barra Grande reservoir (Barra). Barra is located in Santa Catarina and Rio Grande do Sul states in Brazil. The collected images belong to 2021, and their spatial resolution is two meters. To evaluate the proposed workflow using the collected data, first, patches with $416\times 608\text{pixel}$ size are constructed from the mosaic RGB image of Barra. Next, patches are fed to the trained SegNet_d to be segmented into the reservoir and non-reservoir. The SegNet_d performance is evaluated by comparing model outputs with manually produced ground truths. In the next step, the SegNet_d outputs are assembled to be refined using the proposed postprocessing stage. The refined reservoir map is used to detach the RoI around Barra. The covered distance from the border of the reservoir is 200 meters. In Table 10, the performances of the phase-1 segmentation stage, besides the performance of proposed postprocessing, are reported. Table 11 shows the evaluation metrics for the phase-2 segmentation stage. Furthermore, some samples of phase-1 and phase-2 segmentation outputs are illustrated in Figs. 13 and 14, respectively.

Table 10

Performance of phase-1 segmentation and postprocessing stages in segmenting Barra dataset to reservoir and non-reservoir.

Table 11

Performance of phase-2 segmentation stage in segmenting Barra RoI to man-made and non-man-made.

Fig. 13

Two samples of Barra phase-1 patches, besides their corresponding ground truths and reservoir segmentation results.

Fig. 14

Two samples of Barra RoI patches, their corresponding ground truths, and man-made object segmentation results.

3.5.

Benchmark

In order to show the effectiveness of our proposed two-phase approach, we applied a single-phase network to segment images into reservoir, man-made, and non-man-made, as the baseline. In this model, the VGG-16 is used as the backbone, the learning rate is set to 0.0001, the number of epochs is set to 150, the early stopping is not applied, and the summation of Dice and focal losses is used as the loss function. The learning rate is reduced by a factor of 0.2 after every five epochs with no reduction in validation loss down to ${10}^{-7}$. Same as the phase-2 training phase, we constructed patches with $384\times 384\text{pixel}$ size and split them into two sets, training and test.

Since man-made objects inside RoIaR are annotated as man-made and outside as non-man-made (because they are not around the reservoir), the baseline performance is poor (see Table 12), as expected. This simple baseline approach illustrates the importance of our proposed two-phase approach.

Table 12

U-Net performance in segmenting the training and test sets into reservoir, man-made, and non.