Multi-Scale Spatio-Temporal Feature Extraction and Depth Estimation from Sequences by Ordinal Classification

Abstract

Depth estimation is a key problem in 3D computer vision and has a wide variety of applications. In this paper we explore whether deep learning network can predict depth map accurately by learning multi-scale spatio-temporal features from sequences and recasting the depth estimation from a regression task to an ordinal classification task. We design an encoder-decoder network with several multi-scale strategies to improve its performance and extract spatio-temporal features with ConvLSTM. The results of our experiments show that the proposed method has an improvement of almost 10% in error metrics and up to 2% in accuracy metrics. The results also tell us that extracting spatio-temporal features can dramatically improve the performance in depth estimation task. We consider to extend this work to a self-supervised manner to get rid of the dependence on large-scale labeled data.

1. Introduction

Depth estimation [1,2,3,4,5,6,7] is a longstanding and fundamental task in 3D computer vision and enables a wide variety of applications, e.g., autonomous driving [8,9,10], Augmented Reality (AR) and Virtual Reality (VR) [11,12], Simultaneous Localization And Mapping (SLAM) [13,14,15,16,17,18], 2D-3D video conversion [19] and 3D scene understanding [20,21]. Most methods to estimate depth fall into three categories: Monocular Depth Estimation, Stereo Depth Estimation and Depth from Motion (Sequence).

Monocular Depth Estimation [22,23,24,25,26,27,28] infers depth information from single RGB images and is demonstrated to be an ill-posed problem(In most cases, there are several possible outputs corresponding to a given input image and the problem can be seen as a task of selecting the most proper one from all the possible outputs [29]). Stereo Depth Estimation [30,31,32] needs specific devices to capture stereo images and can provide much more clues to estimate the depth than Monocular Depth Estimation. Depth from Motion or Sequence [33,34,35,36] tries to predict the depth of each pixel by taking successive frames into account, which is the most common situation in our life and many applications. We find that most recently proposed methods focus on Monocular Depth Estimation, as it is more difficult to solve academically. However, such methods ignore one of the most important features for determining depth in the human vision system, which is motion, and in most applications, the format of input is in sequence.

In this paper, we concentrate on depth estimation from monocular sequences by using a single moving camera. This choice is motivated because monocular systems have higher efficiency compared with other approaches. Another reason is that the processing of this three-dimensional spatio-temporal signal is also a key problem in Signal Processing and Machine Learning [37,38].

Before the age of Deep Learning, most methods [13,16,39] extracted and matched local features [40,41] from RGB images between neighbouring frames and got the pose of camera and depth information following the theory of Multiple view of Geometry [42]. Recently, Deep Learning [43] has been gradually applied into this field. Reference [44] regards the depth estimation as a pixel-wise regression problem and designs a multi-scale coarse-to-fine deep neural network to handle this problem. References [25,45] design a network to estimate the pose and depth from sequences in an unsupervised manner, but they just take single RGB images as input when predicting the depth, which is used as the intermediate results in their loss functions. Reference [46] predicts depth from video and extracts spatio-temporal features by ConvLSTM.

Most existing methods fail to extract the spatio-temporal features embedded in the sequences or do not make better use of it [46]. Moreover, it is observed that the uncertainty in depth prediction increases along with the underlying ground-truth depth, which indicates that it would be better to allow a relatively larger error when predicting a larger depth value to avoid over-strengthened influence of large depth values on the training process [47].

In this paper, we design a deep neural network to estimate the depth from sequences and our contributions are as follows:

• We design an encoder-decoder neural network with ConvLSTM to extract the spatio-temporal features from sequences.
• We combine several multi-scale strategies [48,49] to achieve an accurate high resolution estimation.
• Experimental results show that our network improves the performance of depth prediction.
• Experimental results show that spatio-temporal features play an important role in depth prediction.

This paper is organized as follows: In Section 2, we briefly review some related work. In Section 3, our proposed method is introduced in details. The experimental results can be found in Section 4. Finally, Section 5 concludes this paper.

2. Related Work

2.1. ConvLSTM

Long Short-Term Memory (LSTM) [50] is one of the most famous building-blocks of Recurrent Neural Network (RNN). It is proved to be stable and powerful in modeling long-range dependencies for sequences.

LSTM, as shown in Figure 1, has a memory cell $C_t$ which essentially acts as an accumulator of the state information. The cell is accessed, written and cleared by several self-parameterized controlling gates. Every time a new input comes, its information will be accumulated to the cell if the input gate it is activated. In addition, the past cell status $C_{t-1}$ could be “forgotten” in this process if the forget gate $f_t$ is on. Whether the latest cell output $C_t$ will be propagated to the final state $h_t$ is further controlled by the output gate $o_t$. The key equations of LSTM are as below:

$$\begin{array}{c}{f}_t=\sigma (W_f^x\ast x_t+W_f^h\ast h_{t-1})\hfill \\ i_t=\sigma (W_i^x\ast x_t+W_i^h\ast h_{t-1})\hfill \\ \widetilde{C_t}=tanh(W_{\tilde{C}}^x\ast x_t+W_{\tilde{C}}^h\ast h_{t-1})\hfill \\ o_t=\sigma (W_o^x\ast x_t+W_o^h\ast h_{t-1})\hfill \\ C_t=f_t\circ C_{t-1}+i_t\circ {\tilde{C}}_t\hfill \\ h_t=o_t\circ tanh\left(C_t\right)\hfill \end{array}$$

(1)

where ∗ denotes the matrix multiplication.

The major drawback of traditional LSTM in handling spatio-temporal data is its usage of full connections in input-to-state and state-to-state transitions in which no spatial information is encoded [51]. ConvLSTM makes all the inputs $x_1,\cdots ,x_t$, cell outputs $C_1,\cdots ,C_t$, hidden states $h_1,\cdots ,h_t$, and gates $i_t,f_t,o_t$ to be 3D tensors whose last two dimensions are spatial dimensions (rows and columns). So ConvLSTM can take RGB images or feature maps from convolution neural network as input. The key equations of ConvLSTM are the same in Equation (1), but the * denotes the convolution operator.

2.2. Ordinal Classification

When a variable is ordinal [52,53], its categories can be ranked from low to high, but the distances between adjacent categories are unknown. For example, if someone asks you about an idea, whether you strongly agree, agreed, have no opinion, disagree or strongly disagree with it, your opinion is ordinal.

Figure 2 shows a concrete example. We denote x as the features and $\beta$ as learnable weights, $ϵ$ is the random noise. The label y of this ordinal classification problem can be determined as follows.

$$\begin{array}{c}\hfill y=\left\{\begin{array}{cc}1\hfill & -\infty \le {\beta }^{T}x+ϵ<t_1\hfill \\ 2\hfill & t_1\le {\beta }^{T}x+ϵ<t_2\hfill \\ 3\hfill & t_2\le {\beta }^{T}x+ϵ<t_3\hfill \\ 4\hfill & t_3\le {\beta }^{T}x+ϵ<\infty \hfill \end{array}\right.\end{array}$$

We can find that( $t_0=-\infty$ and $t_4=\infty$)

$$\begin{array}{cc}\hfill P(y=i|x)& =P(t_{i-1}\le {\beta }^{T}x+ϵ<t_i)=P(t_{i-1}-{\beta }^{T}x\le ϵ<t_i-{\beta }^{T}x)\hfill \\ \hfill & =F(t_i-{\beta }^{T}x)-F(t_{i-1}-{\beta }^{T}x)\hfill \end{array}$$

where $F(·)$ is the cumulative density function of $ϵ$.

We can write the log-likelihood function:

$$\begin{array}{c}\hfill L\left(\beta \right)=\sum _j\sum _{k:y_k=j}log[F(t_j-{\beta }^T{x}_k)-F(t_{j-1}-{\beta }^T{x}_k)]\end{array}$$

This expression can be maximized with numerical methods to estimate $\beta$.

3. Our Method

3.1. Motivation

We describe the motivations of this paper from three aspects.

3.1.1. Depth from Sequences

We find that most applications [8,11,14] of depth estimation need to predict the depth of every frame from sequences. It is also known that successive frames in a sequence are highly related and embed numerous information of motion, which is believed to be important to predict depth [46]. So, it is a natural choice to train and test our model from sequences.

3.1.2. ConvLSTM

There are three main methods to extract spatio-temporal features, CNN-RNN [54], 3D CNN [55], ConvLSTM [51]. CNN-RNN first extracts spatial features by a CNN network and then sends the results to an RNN network for temporal features. It extracts two kinds of features separately and works very well in some applications such as image captioning [54]. 3D CNN extract spatio-temporal features simultaneously, but it is hard to train [55]. ConvLSTM introduces convolution to traditional LSTM and takes images as input. Both spatial and temporal features are extracted in this unit.

3.1.3. Ordinal Classification

As mentioned above, we are more confident about small depth, and should allow a relatively larger error when predicting a larger depth value. Most existing depth prediction deep learning networks output inverse depth or depth in log-space to solve this problem, and [47] shows that ordinal classification is another choice and achieves better performance. So we follow the idea in [47] and recast the depth estimation as an ordinal classification task.

3.2. Overview

The architecture of our proposed network can be found in Figure 3. It is an encoder-decoder [56] architecture. The encoder part consists of three ResNet [57] Bottleneck layers and ConvLSTM layers, it then extracts the spatio-temporal features of input images. The decoder part restores the original resolution of images by three Convolution layers and DeConvolution [58] layers. The Ordinal classification layers are attached to the DeConvolution layer to recover the depth map.

The structure of our network can be found in

Table 1. Structure of the proposed network.

Layers	Structure	Input Size	Output Size
Conv_0	Conv(3 × 3 × 32)	H × W × 3	H × W × 32
	Conv(1 × 1 × 16)	H × W × 32	H × W × 16
Conv1_1	Conv(3 × 3 × 16)	H × W × 16	H × W × 16
	Conv(1 × 1 × 32)	H × W × 16	H × W × 32
Conv1_2	ConvLSTM(3 × 3 × 64)	H × W × 32	H × W × 64
	MaxPooling	H × W × 64	H/2 × W/2 × 64
	Conv(1 × 1 × 32)	H/2 × W/2 × 64	H/2 × W/2 × 32
Conv2_1	Conv(3 × 3 × 32)	H/2 × W/2 × 32	H/2 × W/2 × 32
	Conv(1 × 1 × 64)	H/2 × W/2 × 32	H/2 × W/2 × 64
Conv2_2	ConvLSTM(3 × 3 × 128)	H/2 × W/2 × 64	H/2 × W/2 × 128
	MaxPooling	H/2 × W/2 × 128	H/4 × W/4 × 128
	Conv(1 × 1 × 64)	H/4 × W/4 × 128	H/4 × W/4 × 64
Conv3_1	Conv(3 × 3 × 64)	H/4 × W/4 × 64	H/4 × W/4 × 64
	Conv(1 × 1 × 128)	H/4 × W/4 × 64	H/4 × W/4 × 128
Conv3_2	ConvLSTM(3 × 3 × 256)	H/4 × W/4 × 128	H/4 × W/4 × 256
	MaxPooling	H/4 × W/4 × 256	H/8 × W/8 × 256
UpConv4_1	Conv(3 × 3 × 256)	H/8 × W/8 × 256	H/8 × W/8 × 256
UpConv4_2	DeConv(3 × 3 × 128.2)	H/8 × W/8 × 256	H/4 × W/4 × 128
UpConv5_1	Conv(3 × 3 × 128)	H/4 × W/4 × 128	H/4 × W/4 × 128
UpConv5_2	DeConv(3 × 3 × 64.2)	H/4 × W/4 × 128	H/2 × W/2 × 64
UpConv6_1	Conv(3 × 3 × 64)	H/2 × W/2 × 64	H/2 × W/2 × 64
UpConv6_2	DeConv(3 × 3 × 32.2)	H/2 × W/2 × 64	H × W × 32

. Conv(3 × 3 × 32) and ConvLSTM(3 × 3 × 32) means the kernel size is 3 × 3, the number of kernel is 32, the default value of stride is 1. DeConv(3 × 3 × 128.2) means the kernel size is 3 × 3, the number of kernel is 128, the stride is 2.

In the training stage, our network takes sequences(Every sequence is made up of three frames in our experiments) as input, ground-truth depth maps of corresponding frames as supervision. The encoder part tries to extract the embedded spatio-temporal features of input and obtains its feature map at 1/8 of original resolution (Conv3_2). Then the decoder attempts to recover the depth map from feature maps at different scale and compared the estimated depth map to the ground truth to obtain the error for back-propagation.

In the test stage, we just remove the Ordinal classification layers at ord1 and ord2 to estimate the depth map.

Details of our network are described in the following sections.

3.3. Multi-Scale Strategies

Pooling [43] is an important part of the deep convolution neural network. It reduces the complexity of the network and makes the model be invariant to some transformation. It is widely used in the task of classification [57,59], because classification needs to infer global information from the input, while depth estimation recovers local information from input. The repeated spatial pooling layers quickly reduce the spatial resolution of feature maps [47] (usually stride of 2), which is considered to have bad influence on performance of this task.

However, it is difficult to totally remove the pooling layers from the network. Hence, we adopt a multi-scale estimation to handle this problem.

3.3.1. Location of Pooling Layer

It is difficult to remove all pooling layers in network practically, or the number of parameters of the network will explode. So we put them just after the ConvLSTM layers, as shown in Table 1. Firstly, ResNet Bottleneck extracts spatial features at the same resolution with its input and sends them to the following ConvLSTM layer. Then, we perform a max pooling on the feature map from ConvLSTM to decrease the spatial resolution of feature maps. Although the spatial resolution is decreased, the important local information is encoded, extracted and stored in the ConvLSTM ahead.

3.3.2. Skip-Connection

In our network, there are three skip-connections [19] across the encoder and decoder to directly fuse the features at high resolution from the encoder to the decoder. This is one of the common strategies to improve the performance. Note that the width and height of input should be divisible by eight, or some error will occur in the decoder when concatenating them. For instance, if the size of input is 254 × 254 × 3, then the output size of Conv1_2, Conv2_2, Conv3_2 should be 127 × 127 × 64, 63 × 63 × 128, and 31 × 31 × 256. The output size of UpConv4_2 and UpConv5_1 is 62 × 62 × 128. Conv5_2 concatenates the output of Conv2_2 and UpConv5_1 and take them as input, so error occurs if their size is different.

3.3.3. Multi-Scale Estimation

In the training stage, our model outputs three depth maps at 1/4 (ord1), 1/2 (ord2) and original (ord3) resolution with an input frame. They are compared to the corresponding resized ground truth depth to calculate the error. This multi-scale estimation [44] forces the decoder of our model to recover the depth map progressively, and fine-tune the estimation from low resolution to high resolution.

3.4. Spacing-Increasing Discretization

Most existing methods regard the depth estimation as a pixel-wise regression problem. However, few methods output the depth directly, because it is well-known that the uncertainty in depth prediction increases along with the underlying ground-truth depth, which indicates that it would be better to allow a relatively larger error when predicting a larger depth value to avoid over-strengthened influence of large depth values on the training process [47]. A common solution is performing the regression in log space, but the results are still unsatisfactory [47].

Another idea is recasting the depth estimation as an ordinal classification task, and quantify the continuous depth value to several discrete label values. However, when the depth value becomes larger and larger, its confidence reduces dramatically, which means that the estimation error of larger depth values is generally larger. Hence, using the uniform discretization strategy would induce an over-strengthened loss for the large depth values. We adopt a Spacing-Increasing Discretization proposed in [47], which uniformly discretizes a given depth interval in log space to down-weight the training losses in regions with large depth values, so that our depth estimation network is capable to more accurately predict relatively small and medium depth and to rationally estimate large depth values.

Assuming that a depth interval $[a,b]$ needs to be discretized into K sub-intervals.

$$\begin{array}{cc}\hfill & a=t_0=e^{s_0},s_0=ln\left(a\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & b=t_K=e^{s_K},s_K=ln\left(b\right)\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & t_i=e^{s_i},i\in [0,K]\hfill \end{array}$$

(4)

The array ${\left\{s_i\right\}}_0^K$ is an arithmetic sequence, and

$$\begin{array}{cc}\hfill & s_i=s_0+\frac{i}{K}(s_K-s_0)=ln\left(a\right)+\frac{i}{K}(ln\left(b\right)-ln\left(b\right))\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & t_i=e^{s_i}=a{\left(\frac{b}{a}\right)}^{i/k}\hfill \end{array}$$

(6)

Figure 4 shows an example of Spacing-Increasing Discretization.

3.5. Ordinal Classification

After obtaining the discrete depth values, it is straightforward to turn the standard regression problem into an ordinal classification [47] problem.

Let ${\mathcal{X}}_i=\varphi \left(I\right)$ denote the output of DeConvolution layer at different scale ($i=1$ for UpConv4_2, $i=2$ for UpConv5_2, $i=3$ for UpConv6_2). ${\mathcal{Y}}_i=\Psi ({\mathcal{X}}_i,{\Theta }_i)$ denotes the output of ordinal classification layer given ${\mathcal{X}}_i$ with parameters ${\Theta }_i$ at scale i of size $W_i\times H_i\times 2K$. $W_i$ and $H_i$ are the width and height at scale i, and the value of depth are discretized into K sub-intervals. ${\mathcal{Y}}_i[w,h,2j+1]>{\mathcal{Y}}_i[w,h,2j]$ means the estimated ordinal labes ${\widehat{l}}_{(w,h)}>j$. ${\Theta }_i=\{{\theta }_0^i,{\theta }_1^i,\cdots ,{\theta }_{2k-1}^i\}$ are the parameters. The loss function at ith scale can be formulated as follows.

$$\begin{array}{cc}\hfill & {\mathcal{L}}_i=-\frac{1}{N}\sum _w\sum _h{\mathsf{\Lambda }}_i(w,h,{\mathcal{X}}_i,{\Theta }_i)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\mathsf{\Lambda }}_i(w,h,{\mathcal{X}}_i,{\Theta }_i)=\sum _{k=0}^{l_{(w,h)}-1}log\left({\mathcal{P}}_{(w,h)}^{i,k}\right)+\sum _{k=l_{(w,h)}}^{K-1}log(1-{\mathcal{P}}_{(w,h)}^{i,k})\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & {\mathcal{P}}_{(w,h)}^{i,k}=P({\widehat{l}}_{(w,h)}>k|{\mathcal{X}}_i,{\Theta }_i)\hfill \end{array}$$

(9)

where $l_{(w,h)}$ is the ground truth label of depth of image located at $(w,h)$, ${\widehat{l}}_{(w,h)}$ is the estimated label of depth if image located at $(w,h)$. Equation (7) is the log-likelihood function at $(w,h)$. ${\mathcal{P}}_{(w,h)}^{i,k}$ is calculated as follows

$$\begin{array}{c}\hfill {\mathcal{P}}_{(w,h)}^{i,k}=\frac{e^{y_{(w,h,2k+1)}^i}}{e^{y_{(w,h,2k)}^i}+e^{y_{(w,h,2k+1)}^i}}\end{array}$$

(10)

where $y_{(w,h,2k+1)}^i={\left({\theta }_{2k+1}^i\right)}^T{x}_{(w,h)}^i$, and $x_{(w,h)}^i\in {\mathcal{X}}_i$.

In the inference phase, after obtaining ordinal labels for each pixel of image, the predicted depth value can be calculated as follow(We ignore the superscript i of ${\mathcal{P}}_{(w,h)}^{i,k}$ for simplicity):

$$\begin{array}{cc}\hfill {\widehat{l}}_{(w,h)}=& \sum _{k=0}^K\mathbb{I}({\mathcal{P}}_{(w,h)}^k\ge 0.5)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\widehat{d}}_{(w,h)}=& \frac{t_{{\widehat{l}}_{(w,h)}}+t_{{\widehat{l}}_{(w,h)}+1}}{2}\hfill \end{array}$$

(12)

where $t_i$ is the discretized depth value, as shown in Figure 4, $\mathbb{I}(·)$ is the indicator function, which means $\mathbb{I}\left(\mathrm{true}\right)=1$ and $\mathbb{I}\left(\mathrm{false}\right)=0$.

3.6. Loss Function

The final loss function of our model is the sum of loss functions at different scale with weight parameters.

$$\begin{array}{c}\hfill \mathcal{L}={\lambda }_1{\mathcal{L}}_1+{\lambda }_2{\mathcal{L}}_2+{\lambda }_3{\mathcal{L}}_3\end{array}$$

(13)

Parameters ${\lambda }_1,{\lambda }_2,{\lambda }_3$ balance the weights among three scales and we set $\lambda =1/4,{\lambda }_2=1/2,$${\lambda }_3=1$ in the following experiments.

4. Experimental Results

Our model was trained and evaluated on the KITTI dataset [60]. The KITTI dataset consisted of video sequences of outdoor scenes along with their corresponding depth maps, procured using car-mounted cameras and Velodyne LiDAR sensors. We split the KITTI dataset into train and test following the description in [44], and we trained on 28 sequences and tested on the 697 images provided in [44]. The range of depth was set from 0 to 80 m, which meant $a=0,b=80$, and the depth was divided into $K=70$ sub-intervals. Throughout our experiments, the time-step of the ConvLSTM was set to 3. We evaluated our approach by using the standard metrics proposed by [44].

We compared the methods in two commonly used metrics: error metric and accurate metric. We denote y as the predicted depth and $y^{\ast }$ as ground truth depth, the key equations of these metrics are as follows:

$$\begin{array}{cc}\hfill \mathrm{Accuracy}\mathrm{Metric}:& \mathrm{percent}\mathrm{of}\mathrm{y}\mathrm{that}max(\frac{y}{y^{\ast }},\frac{y^{\ast }}{y})=\delta \hfill \\ \hfill \mathrm{Abs}\mathrm{Rel}:& \frac{1}{N}\sum |y-y^{\ast }|/y^{\ast }\hfill \\ \hfill \mathrm{Sq}\mathrm{Rel}:& \frac{1}{N}\sum {(y-y^{\ast })}^2/y^{\ast }\hfill \\ \hfill \mathrm{RMSE}:& \sqrt{\frac{1}{N}\sum {(y-y^{\ast })}^2}\hfill \\ \hfill \mathrm{RMSE}\mathrm{Log}:& \sqrt{\frac{1}{N}\sum {(log\left(y\right)-log{\left(y\right)}^{\ast })}^2}\hfill \end{array}$$

We conducted three experiments on our proposed method. Our${}_{vo}$ means the model took sequences as input and got results from the ordinal classification layer as shown in Figure 3. Our${}_{vc}$ replaced ordinal classification layers with convolution layers, namely Conv(3 × 3 × 1), and they were fine-tuned with training data. We trained this model to investigate the importance of ordinal classification layer. Our${}_{so}$ means the model took single RGB images as input and got results from the ordinal classification layer. Although the network was trained using sequences, the decoder part was designed to individually recover each state of the phase of the encoder. Doing so allowed us to use a single image as input, estimate its depth map, and compare its results to Our${}_{vo}$ to find out the how much performance improved by taking sequences as input.

Table 2. Results on KITTI.

Method	Error Metrics				Accuracy Metrics
	Abs Rel	Sq Rel	RMSE	RMSE Log	$\mathit{\delta }<1.25$	$\mathit{\delta }<1.{25}^2$	$\mathit{\delta }<1.{25}^3$
Eigen [44]	0.214	1.605	6.563	0.292	0.673	0.884	0.957
DepthNet [46]	0.137	1.019	5,187	0.218	0.809	0.928	0.971
Kuznietsov [22]	0.113	0.741	4.621	0.189	0.875	0.964	0.988
DORN [47]	0.072	0.307	2.727	0.120	0.932	0.984	0.994
Our${}_{so}$	0.129	0.865	4.663	0.194	0.881	0.959	0.972
Our${}_{vc}$	0.093	0.543	3.014	0.145	0.908	0.969	0.984
Our${}_{vo}$	0.062	0.264	2.313	0.101	0.953	0.991	0.995

shows the experimental results. We compared the proposed method with existing supervised method: DORN [47], DepthNet [46], Kuznietsov [22], and Eigen [44]. These four methods were trained with depth supervision, and Kuznietsov [22] had extra pose supervision.

In our experiments, we trained five different models for Our${}_{vo}$ and Our${}_{vc}$ respectively with the same training data and random initialization; the results of Our${}_{vo}$, Our${}_{vc}$ and Our${}_{so}$ are the average of 5 runs for each model.

The results show that Our${}_{vo}$ outperformed other methods. If compared with DORN, that had the best performances among four existing methods, Our${}_{vo}$ had a 13.88% improvement in Abs Rel (0.072->0.062), a 14% improvement in Sq Rel (0.307->0.264), a 15.18% improvement in RMSE (2.727->2.313), a 15.83% improvement in RMSE Log (0.307->0.264), a 2 percent point improvement when $\delta <1.25$ (93.2->95.3%), a 0.7 percent point improvement when $\delta <{1.25}^2$ (98.4->99.1%), a 0.1 percent point improvement when $\delta <{1.25}^3$ (99.4->99.5%). We can also get some conclusion by comparing the performance of several groups of methods.

Firstly, Our${}_{vc}$ had better results than DepthNet. Both of them took sequences as input, utilized ConvLSTM to extract spatio-temporal features, got output from the convolution layer; they also had similar network architecture. Nevertheless, our${}_{vc}$ chose ResNet Bottleneck to extract features from images and adopted a multi-scale strategy, by which we believe improvement was brought.

Secondly, the performance of Our${}_{vo}$ was better than DORN, while the performance of DORN was better than Our${}_{so}$. The main difference between DORN and Our${}_{so}$ was the architecture of the network. DORN designed a complex network with dense feature extractor, multi-scale feature learner, cross channel information learner and a full-image encoder, while Our${}_{so}$ lost its ability to infer through the time for taking single RGB images as input. However, Our${}_{vo}$ beat DORN with a much simpler network which indicates the importance of temporal feature in depth estimation.

Finally, Our${}_{vc}$ achieved better results than Our${}_{so}$ which shows that temporal feature may play a much more important role than ordinal classification layer.

Some visual result can be found in Figure 5. The results of DORN and Our${}_{so}$ was sharper and more jittery than Our${}_{vc}$ and Our${}_{vo}$, because Our${}_{vc}$ and Our${}_{vo}$ took sequences as input and had the ability to smooth the output in the time domain.

5. Conclusions

In this paper, we design a deep learning neural network to extract spatio-temporal features from sequences, motivated by [46], and predict the depth map from it by ordinal classification, inspired by [47]. The network encodes the input by ResNet Bottleneck and ConvLSTM, then decodes and recovers the resolution of input images by Convolution and DeConvolution with skip-connection from encoder. We train the network with a multi-scale loss function to improve the performance. The results of our experiments show that the proposed method has an improvement of almost 10% in error metrics and up to 2% in accuracy metrics when comparing with the recently proposed supervised depth estimation methods. These results also show us that extracting temporal features can significantly improve the performance in depth estimation task. In the future, we will follow this work and design a self-supervised network to get rid of the dependence on large-scale labeled data.