Huber Loss Meets Spatial Autoregressive Model: A Robust Variable Selection Method with Prior Information

Abstract

In recent times, the significance of variable selection has amplified because of the advent of high-dimensional data. The regularization method is a popular technique for variable selection and parameter estimation. However, spatial data is more intricate than ordinary data because of spatial correlation and non-stationarity. This article proposes a robust regularization regression estimator based on Huber loss and a generalized Lasso penalty to surmount these obstacles. Moreover, linear equality and inequality constraints are contemplated to boost the efficiency and accuracy of model estimation. To evaluate the suggested model’s performance, we formulate its Karush-Kuhn-Tucker (KKT) conditions, which are indicators used to assess the model’s characteristics and constraints, and establish a set of indicators, comprising the formula for the degrees of freedom. We employ these indicators to construct the AIC and BIC information criteria, which assist in choosing the optimal tuning parameters in numerical simulations. Using the classic Boston Housing dataset, we compare the suggested model’s performance with that of the model under squared loss in scenarios with and without anomalies. The outcomes demonstrate that the suggested model accomplishes robust variable selection. This investigation provides a novel approach for spatial data analysis with extensive applications in various fields, including economics, ecology, and medicine, and can facilitate the enhancement of the efficiency and accuracy of model estimation.

Appendix

Proof of Theorem 1 Let $P_{{\text {null}}}$ be the projection matrix of ${\text {null}}\left( G_{-A, B}\right)$, then $P_{{\text {null}}} G_{-A, B}^T=0$. Multiplying both sides of Eq. (2.21) by $P_{\text{ null }}$ yields:

$$\begin{aligned} \begin{aligned}&P_{\text{ null } } X_{-v}^*{ }^T X_{-v}^* \hat{\beta }^*=P_{\text{ null } } X_{-v}^*{ }^T y_{-v}-P_{\text{ null } } D_A^* \lambda s_A-P_{\text{ null } } G_{-A, B}^T \hat{\theta }_{-A, B}+P_{\text{ null } } X_v^* t s_v \\&=P_{\text{ null } } X_{-v}^*{ }^T y_{-v}-P_{\text{ null } } D_A^{* \top } \lambda s_A+P_{\text{ null } } X_v^{* \top } t s_\nu \\&\end{aligned} \end{aligned}$$

(A.1)

$\hat{\beta }^*$ can be decomposed into the sum of two parts as follows:

$$\begin{aligned} \begin{aligned} \hat{\beta }^*&=P_{\text{ null } } \hat{\beta }^*+P_{\text{ col } \left( G_{-A, B}^T \right) } \hat{\beta }^* \\&=P_{\text{ null } } \hat{\beta }^*+G_{-A, B}^T\left( G_{-A, B} G_{-A, B}^T\right) ^{+} G_{-A, B} \hat{\beta }^* \\&=P_{\text{ null } } \hat{\beta }^*-A g_{-A, B} \end{aligned} \end{aligned}$$

The last equality holds because $G_{-A, B} \hat{\beta }^*=g_{-A, B}$. Substituting the expression for $\hat{\beta }^*$ into (A.1) and simplifying, we obtain:

$$\begin{aligned} \begin{array}{r} P_{\text{ null } } X_{-v}^*{ }^T X_{-v}^* P_{\text{ null } } \hat{\beta }^*=P_{\text{ null } } X_{-v}^*{ }^T y_{-v}-P_{\text{ null } } D_A^*{ }^T \lambda s_A+P_{\text{ null } } X_v^{* \top } t s_v+P_{\text{ null } } X_{-v}^*{ }^T X_{-v}^* A g_{-A, B} \\ \quad =P_{\text{ null } } X_{-v}^*{ }^T\left[ y_{-v}-\left( X_{-v}^* P_{\text{ null } } X_{-v}^*{ }^T\right) + X_{-v}^* P_{\text{ null } }\left( D_A^* \lambda s_A-X_v^* t s_v\right) +X_{-v}^* A g_{-A, B}\right] \end{array} \end{aligned}$$

The last equality holds because from (A.1), we can deduce that $P_{\text{ null }} D_A^* \lambda s_A-P_{\text{ null }} X_v^{* T} t s_v \in {\text {col}}\left( P_{\text{ null }} X_{-v}^*{ }^{\top }\right)$. Further, we can derive:

$$\begin{aligned} P_{\text{ null } } D_A^{* T} \lambda s_A-P_{\text{ null } } X_v^{* T} t s_v=\left( P_{\text{ null } } X_{-v}^*{ }^T\right) \left( X_{-v}^* P_{\text{ null } } X_{-v}^*\right) +\left( X_{-v}^* P_{\text{ null } }\right) \left( P_{\text{ null } } D_A^{* T} \lambda s_A-P_{\text{ null } } X_v^{* T} t s_v\right) \end{aligned}$$

Therefore,

$$\begin{aligned} \begin{aligned}&X_{-v}^* P_{\text{ null } } \hat{\beta }^*=X_{-v}^* P_{\text{ null } }\left( P_{\text{ null } } X_{-v}^*{ }^T X_{-v}^* P_{\text{ null } }\right) + P_{\text{ null } } X_{-v}^*{ }^T \\&{\left[ y_{-v}-\left( X_{-v}^* P_{\text{ null } } X_{-v}^*{ }^T\right) + X_{-v}^* P_{\text{ null } }\left( D_{\mathcal {A}}^{* T} \lambda s_{\mathcal {A}}-X_v^{* T} t s_v\right) +X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}}\right] } \\&=P_{X_{-v}^* P_{\text{ null } }}\left[ y_{-v}-\left( X_{-v}^* P_{\text{ null } } X_{-v}^*{ }^T\right) + X_{-v}^* P_{\text{ null } }\left( D_{\mathcal {A}}^{* T} \lambda s_{\mathcal {A}}-X_v^{* T} t s_v\right) +X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}}\right] \end{aligned} \end{aligned}$$

Therefore, the fitted values $X_{-v}^* \hat{\beta }^*$ can be expressed as:

$$\begin{aligned} \begin{aligned} X_{-v}^* \hat{\beta }^*&=X_{-v}^* P_{\text{ null } } \hat{\beta }^*-X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}} \\&=P_{X_{-v}^* P_{\text{ null } }}\left[ y_{-v}-\left( X_{-v}^* P_{\text{ null } } X_{-v}^*{ }^T\right) + X_{-v}^* P_{\text{ null } }\left( D_{\mathcal {A}}^{* T} \lambda s_{\mathcal {A}}-X_v^{* T} t s_v\right) +X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}}\right] -X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}} \\&=P_{X_{-v}^* P_{\text{ null } }}\left[ y_{-v}-\left( X_{-v}^* P_{\text{ null } } X_{-v}^*{ }^T\right) + X_{-v}^* P_{\text{ null } }\left( D_{\mathcal {A}}^* \lambda s_{\mathcal {A}}-X_v^{* T} t s_v\right) \right] -\left( I-P_{X_{-v}^* P_{\text{ null } }}\right) X_{-v}^* A g_{-\mathcal {A}, \mathcal {B}} \end{aligned} \end{aligned}$$

Proof of Theorem 2 $\hat{\mu }(y)=\hat{y}$ is continuous and almost everywhere differentiable with respect to y. Therefore, we can use Stein’s lemma to calculate the degrees of freedom for the Space Huber Lcg-Lasso fitted values. Thus,

$$\begin{aligned} \textrm{df}(\hat{\mu })=\mathbb {E}\left[ \sum _{i\in V}^n \frac{\partial \hat{y}_i}{\partial y_i}\right] =\mathbb {E}\left[ \sum _{i\in V} \frac{\partial \hat{y}_i}{\partial y_i}+\sum _{i \in V^C} \frac{\partial \hat{y}_i}{\partial y_i}\right] \end{aligned}$$

As $\hat{\beta }^*$ depends only on $y_{-v}$, the derivatives of the fitted values $\hat{y}_i$ with respect to $y_i$ are 0 for $i \in \mathcal {V}$, i.e.,

$$\begin{aligned} \frac{\partial \hat{y}_i}{\partial y_i}=\frac{\partial x_i^T \hat{\beta }\left( y_{-v}\right) }{\partial y_i}=0 \text{, } \text{ for } i \in \mathcal {V} \end{aligned}$$

Thus, the expression for the degrees of freedom for the fitted values $\hat{\mu }$ becomes

$$\begin{aligned} \textrm{df}(\hat{\mu })=\mathbb {E}\left[ \sum _{i \in V^C} \frac{\partial \hat{y}_i}{\partial y_i}\right] =E\left[ \left( \nabla \cdot \hat{y}_{-v}\right) \left( y_{-v}\right) \right] \end{aligned}$$

Considering the expression of $\hat{y}_{-v}$ from Theorem 1, we have

$$\begin{aligned} \begin{aligned}&\left. \hat{y}_{-v}=P_{X_{-v}^* P_{\text{ null }}} y_{-v}-P_{X_{-v}^* P_{\text{ null }}}\left( X_{-v}^* P_{\text{ null } } X_{-v}^{* T}\right) + X_{-v}^* P_{\text{ null } }\left( D_{\mathcal {A}}^{* T} \lambda s_A-X_v^{* T} t s_v\right) \right] \\&-\left( I-P_{X_{-v}^* P_{\text{ null } }}\right) X_{-v}^* A g_{-A, B} \text{. } \end{aligned} \end{aligned}$$

The first term on the right-hand side depends directly on y, while the remaining parts depend only on the boundary sets $\mathcal {A}$, $\mathcal {B}$, $\mathcal {V}$, and the signs $s_{\mathcal {A}}$, $s_{\mathcal {V}}$. $\mathcal {A}$, $\mathcal {B}$, $\mathcal {V}$, $s_A$, and $s_V$ are locally constant in the neighborhood of y, implying their derivatives with respect to y are zero. Therefore,

$$\begin{aligned} \textrm{df}(\hat{\mu })=\mathbb {E}\left[ \left( \nabla \cdot \hat{y}_{-v}\right) \left( y_{-v}\right) \right] ={\text {tr}}\left( P_{X_{-v}^* P_{\text{ mul } }}\right) \end{aligned}$$

Since the trace of the projection matrix equals the dimension of the corresponding linear space, the theorem is established.