On the test of covariance between two high-dimensional random vectors

Abstract

We consider a problem of association test in high dimension. A new test statistic is proposed based on the covariance of random vectors and its asymptotic properties are derived under both the null hypothesis and the local alternatives. Furthermore power enhancement technique is utilized to boost the empirical power especially under sparse alternatives. We examine the finite-sample performances of the proposed test via Monte Carlo simulations, which show that the proposed test outperforms some existing procedures. An empirical analysis of a microarray data is demonstrated to detect the relationship between the genes.

Appendix

Proof of Theorem 1

Without loss of generality, we can assume that $\textrm{E}(\textbf{X})=0_{p\times 1}$ and $\textrm{E}(\textbf{Y})=0_{q\times 1}$ hereafter. Define that $\xi _c=\textrm{Var}\{h_c(\textbf{Z}_1,\ldots ,\textbf{Z}_{c})\}$, where $h_c(z_1,\ldots ,z_c)=\textrm{E}\{h(z_1,\ldots ,z_c,\textbf{Z}_{c+1},\ldots ,\textbf{Z}_{4})\}$ for $c=1,2,3,4$. Denote $G(\textbf{X},\textbf{X}')=\textbf{X}^\top \textbf{X}'$ and $H(\textbf{Y},\textbf{Y}')=\textbf{Y}^\top \textbf{Y}'$. Observe that $\textrm{E}\{L(\textbf{Z},\textbf{Z}')\}=\textrm{E}\{L(\textbf{Z},\textbf{Z}')|\textbf{Z}\}=\textrm{E}\{L(\textbf{Z},\textbf{Z}')|\textbf{Z}'\}=0$ under the null, and that $\textrm{E}\{G(\textbf{X},\textbf{X}')|\textbf{X}\}=\textrm{E}\{G(\textbf{X},\textbf{X}')|\textbf{X}'\}=\textrm{E}\{H(\textbf{Y},\textbf{Y}')|\textbf{Y}\}=\textrm{E}\{H(\textbf{Y},\textbf{Y}')|\textbf{Y}'\}=0$. Then we can obtain under the null hypothesis that

$$\begin{aligned} h_1(\textbf{Z}_1)= & {} 0,\\ h_2(\textbf{Z}_1,\textbf{Z}_2)= & {} \frac{1}{6}G(\textbf{X}_1,\textbf{X}_2)H(\textbf{Y}_1,\textbf{Y}_2),\\ h_3(\textbf{Z}_1,\textbf{Z}_2,\textbf{Z}_3)= & {} \frac{1}{12}[\{ 2G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_3)-G(\textbf{X}_2,\textbf{X}_3)\}H(\textbf{Y}_1,\textbf{Y}_2){}\\{} & {} +\{2G(\textbf{X}_1,\textbf{X}_3)-G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_2,\textbf{X}_3)\}H(\textbf{Y}_1,\textbf{Y}_3){}\\{} & {} +\{2G(\textbf{X}_2,\textbf{X}_3)-G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_3)\}H(\textbf{Y}_2,\textbf{Y}_3)],\\ h_4(\textbf{Z}_1,\textbf{Z}_2,\textbf{Z}_3,\textbf{Z}_4)= & {} \frac{1}{12}[ \{2G(\textbf{X}_1,\textbf{X}_2)+2G(\textbf{X}_3,\textbf{X}_4)-G(\textbf{X}_1,\textbf{X}_3)-G(\textbf{X}_1,\textbf{X}_4){}\\{} & {} {}-G(\textbf{X}_2,\textbf{X}_3)-G(\textbf{X}_2,\textbf{X}_4)\}\{H(\textbf{Y}_1,\textbf{Y}_2)+H(\textbf{Y}_3,\textbf{Y}_4)\}{}\\{} & {} {}+\{2G(\textbf{X}_1,\textbf{X}_3)+2G(\textbf{X}_2,\textbf{X}_4)-G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_4)\\{} & {} {}-G(\textbf{X}_2,\textbf{X}_3)-G(\textbf{X}_3,\textbf{X}_4)\}\{H(\textbf{Y}_1,\textbf{Y}_3)+H(\textbf{Y}_2,\textbf{Y}_4)\}{}\\{} & {} {}+\{2G(\textbf{X}_1,\textbf{X}_4)+2G(\textbf{X}_2,\textbf{X}_3) -G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_3){}\\{} & {} {}-G(\textbf{X}_2,\textbf{X}_4)-G(\textbf{X}_3,\textbf{X}_4)\}\{H(\textbf{Y}_1,\textbf{Y}_4)+H(\textbf{Y}_2,\textbf{Y}_3)\}]. \end{aligned}$$

Under Assumption (A1) we have

$$\begin{aligned} \xi _1= 0, \xi _2=\frac{\zeta ^2}{36}, \xi _3=o(n\zeta ^2), \xi _4=o(n^2\zeta ^2). \end{aligned}$$

Applying the Hoeffding’ decomposition in Serfling (1980), we get that

$$\begin{aligned} \frac{nT_{n,p,q}(\textbf{X},\textbf{Y})}{\sqrt{2\zeta ^2}}= \frac{\sum \limits _{1\le i< j\le n}L(\textbf{Z}_i,\textbf{Z}_j)}{\sqrt{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \zeta ^2}}+o_p(1). \end{aligned}$$

Write $M_j=\sum _{i=1}^{j-1}L(\textbf{Z}_i,\textbf{Z}_j)$, $S_u =\sum _{j=2}^u\sum _{i=1}^{j-1}L(\textbf{Z}_i,\textbf{Z}_j)=\sum _{j=2}^u M_j$ and the filtration $\mathscr {F}_u=\sigma \{\textbf{Z}_1, \ldots ,\textbf{Z}_u\}$. Since $\textrm{E}\{L(\textbf{Z},\textbf{Z}')\} =\textrm{E}\{L(\textbf{Z},\textbf{Z}')|\textbf{Z}\} =\textrm{E}[L(\textbf{Z},\textbf{Z}')|\textbf{Z}'\}=0$, then $\textrm{E}(S_u)=0$ and for $u<v$,

$$\begin{aligned} \textrm{E}(S_v|\mathscr {F}_u)= & {} S_u+\sum _{j=u+1}^v\sum _{i=1}^{u}\textrm{E}\{L(\textbf{Z}_i,\textbf{Z}_j)|\textbf{Z}_i\} +\sum _{j=u+2}^v\sum _{i=u+1}^{j-1}\textrm{E}\{L(\textbf{Z}_i,\textbf{Z}_j)\}\\= & {} S_u. \end{aligned}$$

This implies that $S_u$ is adaptive to $\mathscr {F}_u$ and thus it is a mean-zero martingale sequence.

To obtain the asymptotic normality of $\sum _{1\le i< j\le n}L(\textbf{Z}_i,\textbf{Z}_j))/\sqrt{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \zeta ^2}$, we only need to verify the two following conditions given by Corollary 3.1 in Hall and Heyde (1980).

Condition (1): the conditional variance

$$\begin{aligned} \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\overset{P}{\longrightarrow } 1, \end{aligned}$$

and Condition (2): the conditional Lindeberg condition, that is, for all $\varepsilon >0$,

$$\begin{aligned} \frac{1}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}\Big [M_j^2\mathbf{{I}}\{|M_j|>\varepsilon \sqrt{n(n-1)\zeta ^2} \}| \mathscr {F}_{j-1}\Big ]\overset{P}{\longrightarrow }\ 0. \end{aligned}$$

To prove condition (1), it is sufficient to prove that

$$\begin{aligned} \textrm{E}\left\{ \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\} =1, \end{aligned}$$

and

$$\begin{aligned} \textrm{Var}\left\{ \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\} \overset{}{\longrightarrow } 0. \end{aligned}$$

Notice again that $\textrm{E}\{L(\textbf{Z},\textbf{Z}')\}=\textrm{E}\{L(\textbf{Z},\textbf{Z}')|\textbf{Z}\} =\textrm{E}\{L(\textbf{Z},\textbf{Z}')|\textbf{Z}'\}=0$. We then show that

$$\begin{aligned}{} & {} \textrm{E}\left\{ \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\} \\{} & {} \quad =\frac{2}{n(n-1)\zeta ^2} \sum _{j=2}^n\textrm{E}\left\{ \sum _{i,i^{'}=1}^{j-1}L(\textbf{Z}_i,\textbf{Z}_j)L(\textbf{Z}_{i'},\textbf{Z}_j)\right\} \\{} & {} \quad = 1. \end{aligned}$$

Next we will prove that

$$\begin{aligned} \textrm{Var}\left\{ \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\} \overset{}{\longrightarrow } 0. \end{aligned}$$

Recall that $L_1(\textbf{Z},\textbf{Z}')=\textrm{E}\{L(\textbf{Z},\textbf{Z}'') L(\textbf{Z}',\textbf{Z}'')| (\textbf{Z},\textbf{Z}')\}$. It is easy to check that

$$\begin{aligned} \textrm{E}\{L_1(\textbf{Z},\textbf{Z}')\}=\textrm{E}\{L_1(\textbf{Z},\textbf{Z}')|\textbf{Z}\}=\textrm{E}\{L_1(\textbf{Z},\textbf{Z}')|\textbf{Z}'\}=0. \end{aligned}$$

Then we have that

$$\begin{aligned}{} & {} \textrm{Var}\left\{ \sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\} \\{} & {} \quad =\sum _{j=2}^n\textrm{Var}\Big \{\textrm{E}(M_j^2|\mathscr {F}_{j-1})\Big \} +2\sum _{ 2\le j<t\le n }\textrm{Cov}\Big \{\textrm{E}(M_j^2| \mathscr {F}_{j-1}),\textrm{E}(M_t^2| \mathscr {F}_{t-1})\Big \}\\{} & {} \quad =\frac{n(n-1)(2n-1)}{6}\textrm{Var}\{L_1(\textbf{Z},\textbf{Z})\}+\frac{n(n-1)^2(n-2)}{3}\textrm{E}\{L_1(\textbf{Z},\textbf{Z}')^2\}. \end{aligned}$$

When assumption (A1) is true, we thus obtain that

$$\begin{aligned} \textrm{Var}\left\{ \frac{2}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}(M_j^2|\mathscr {F}_{j-1})\right\}= & {} \frac{4}{n^2(n-1)^2\zeta ^4}\textrm{Var}\left\{ \sum _{j=2}^n\textrm{E}[M_j^2|\mathscr {F}_{j-1})\right\} \\\longrightarrow & {} 0, \end{aligned}$$

which ensures condition (1).

Next condition (2) will be verified. Direct calculation shows that

$$\begin{aligned} \sum _{j=2}^n\textrm{E}(M_j^4)= & {} \sum _{j=2}^n\sum _{i_1,i_2,i_3,i_4=1}^{j-1}\textrm{E}\{L(\textbf{Z}_{i_1},\textbf{Z}_j) L(\textbf{Z}_{i_2},\textbf{Z}_j)L(\textbf{Z}_{i_3},\textbf{Z}_j)L(\textbf{Z}_{i_4},\textbf{Z}_j)\}\\= & {} \frac{n(n-1)}{2}\textrm{E}\{L(\textbf{Z},\textbf{Z}')^4] +n(n-1)(n-2)\textrm{E}[L(\textbf{Z},\textbf{Z}')^2L(\textbf{Z},\textbf{Z}'')^2]. \end{aligned}$$

Again under assumption (A1), it is easy to show that

$$\begin{aligned} \frac{1}{n^4\zeta ^4}\sum _{j=2}^n\textrm{E}(M_j^4) \rightarrow 0, \end{aligned}$$

which implies that

$$\begin{aligned} \frac{1}{n^2(n-1)^2\zeta ^4}\sum _{j=2}^n\textrm{E}(M_j^4|\mathscr {F}_{j-1})\xrightarrow {P} 0. \end{aligned}$$

Notice that for all $\varepsilon >0$,

$$\begin{aligned}{} & {} \frac{1}{n(n-1)\zeta ^2}\sum _{j=2}^n\textrm{E}[M_j^2\mathbf{{I}}\{|M_j|>\varepsilon \sqrt{n(n-1)\zeta ^2} \}| \mathscr {F}_{j-1}]\\{} & {} \quad \le \frac{1}{\varepsilon ^2 n^2(n-1)^2\zeta ^4}\sum _{j=2}^n\textrm{E}(M_j^4| \mathscr {F}_{j-1}), \end{aligned}$$

which implies condition (2).

Therefore, we can show that

$$\begin{aligned} \frac{\sum \limits _{1\le i< j\le n}L(\textbf{Z}_i,\textbf{Z}_j)}{\sqrt{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \zeta ^2}} \overset{d}{\longrightarrow }\ {\mathcal {N}}(0,1). \end{aligned}$$

Using the Slutsky theorem we obtain that

$$\begin{aligned} \frac{nT_{n,p,q}((\textbf{X},\textbf{Y}))}{\sqrt{2\zeta ^2}} \overset{d}{\longrightarrow }\ {\mathcal {N}}(0,1). \end{aligned}$$

(41)

In the following, we will show the consistency of $\zeta ^2_n$. That is,

$$\begin{aligned} \frac{\zeta ^2_n}{\zeta ^2} \overset{P}{\longrightarrow }\ 1. \end{aligned}$$

(42)

To obtain this result, it is sufficient to verify $\textrm{E}(\zeta ^2_n/\zeta ^2)\rightarrow 1$ and $\textrm{Var}(\zeta ^2_n/\zeta ^2)\rightarrow 0$, respectively. Recall the definition of $A_{ij}$ and $B_{ij}$, we can rewrite them in the following form:

$$\begin{aligned} A_{ij}= & {} a_{ij}-\frac{1}{n-2}\sum _{l\ne i} a_{il}-\frac{1}{n-2}\sum _{k\ne j} a_{kj}+ \frac{1}{(n-1)(n-2)}\sum _{k\ne l} a_{kl}, \\= & {} \frac{n-3}{n-1}\left\{ G(\textbf{X}_i,\textbf{X}_j)-\frac{1}{n-2}\sum _{l\notin \{i,j\}}G(\textbf{X}_i,\textbf{X}_l) -\frac{1}{n-2}\sum _{k\notin \{i,j\}}G(\textbf{X}_k,\textbf{X}_j){}\right. \\{} & {} \left. {}+\frac{1}{(n-2)(n-3)}\sum _{\{k,l\}\bigcap \{i,j\}=\emptyset }G(\textbf{X}_k,\textbf{X}_l)\right\} \\:= & {} \frac{n-3}{n-1}(I_{n,1}+I_{n,2}+I_{n,3}+I_{n,4}), \\ B_{ij}= & {} b_{ij}-\frac{1}{n-2}\sum _{l\ne i} b_{il}-\frac{1}{n-2}\sum _{k\ne j}^n b_{kj}+ \frac{1}{(n-1)(n-2)}\sum _{k\ne l} b_{kl}, \\= & {} \frac{n-3}{n-1}\left\{ H(\textbf{Y}_i,\textbf{Y}_j)-\frac{1}{n-2}\sum _{l\notin \{i,j\}}H(\textbf{Y}_i,\textbf{Y}_l) -\frac{1}{n-2}\sum _{k\notin \{i,j\}}H(\textbf{Y}_k,\textbf{Y}_j){}\right. \\{} & {} \left. {}+\frac{1}{(n-2)(n-3)}\sum _{\{k,l\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_k,\textbf{Y}_l)\right\} \\:= & {} \frac{n-3}{n-1}(J_{n,1}+J_{n,2}+J_{n,3}+J_{n,4}). \end{aligned}$$

Notice that $\textrm{E}\{G(\textbf{X},\textbf{X}')|\textbf{X}\}=0$ and $\textrm{E}\{H(\textbf{Y},\textbf{Y}')|\textbf{Y}\}=0$, we have for $1\le i \ne j\le n$,

$$\begin{aligned}{} & {} \frac{(n-1)^4}{(n-3)^4}\textrm{E}(A_{ij}^2B_{ij}^2) \\{} & {} \quad = \textrm{E}\{(I_{n,1}+I_{n,2}+I_{n,3}+I_{n,4})^2 (J_{n,1}+J_{n,2}+J_{n,3}+J_{n,4})^2\} \\{} & {} \quad = \textrm{E}[\{I_{n,1}^2+2I_{n,1}(I_{n,2}+I_{n,3}+I_{n,4})+(I_{n,2}+I_{n,3}+I_{n,4})^2\} {}\\{} & {} \qquad \{J_{n,1}^2+2J_{n,1}(J_{n,2}+J_{n,3} +J_{n,4})+(J_{n,2}+J_{n,3}+J_{n,4})^2\}]\\{} & {} \quad =\textrm{E}(I_{n,1}^2J_{n,1}^2)+2\textrm{E}\{I_{n,1}^2J_{n,1}(J_{n,2}+J_{n,3}+J_{n,4})\} +2\textrm{E}\{J_{n,1}^2I_{n,1}(I_{n,2}+I_{n,3}+I_{n,4})\}{}\\{} & {} \qquad +\textrm{E}\{I_{n,1}^2(J_{n,2}+J_{n,3}+J_{n,4})^2\}+\textrm{E}\{J_{n,1}^2(I_{n,2}+I_{n,3}+I_{n,4})^2\} +\textrm{E}[\{2I_{n,1}(I_{n,2}{}\\{} & {} \qquad +I_{n,3}+I_{n,4})+(I_{n,2}+I_{n,3}+I_{n,4})^2\}\\{} & {} \qquad \{2J_{n,1}(J_{n,2}+J_{n,3}+J_{n,4})+(J_{n,2}+J_{n,3}+J_{n,4})^2\}], \end{aligned}$$

which implies that

$$\begin{aligned}{} & {} \left| \frac{(n-1)^4}{(n-3)^4}\textrm{E}(A_{ij}^2B_{ij}^2)-\textrm{E}(I_{n,1}^2J_{n,1}^2)\right| \\{} & {} \quad \le O\{\textrm{E}(I_{n,1}^2J_{n,2}^2)+\textrm{E}(I_{n,1}^2J_{n,3}^2)+\textrm{E}(I_{n,1}^2J_{n,4}^2){}\\{} & {} \qquad +\textrm{E}(I_{n,2}^2 J_{n,1}^2)+\textrm{E}(I_{n,2}^2J_{n,2}^2) +\textrm{E}(I_{n,2}^2J_{n,3}^2)+\textrm{E}(I_{n,2}^2J_{n,4}^2)\\{} & {} \qquad +\textrm{E}(I_{n,3}^2 J_{n,1}^2)+\textrm{E}(I_{n,3}^2 J_{n,2}^2)+\textrm{E}(I_{n,3}^2J_{n,3}^2)+\textrm{E}(I_{n,3}^2J_{n,4}^2){}\\{} & {} \qquad +\textrm{E}(I_{n,4}^2 J_{n,1}^2)+\textrm{E}(I_{n,4}^2 J_{n,2}^2)+\textrm{E}(I_{n,4}^2 J_{n,3}^2)+\textrm{E}(I_{n,4}^2J_{n,4}^2)\}. \end{aligned}$$

By the Cauchy–Schwarz inequality, we have under condition (A1) that

$$\begin{aligned} \textrm{E}\{G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}\le & {} \sqrt{\textrm{E}\{G(\textbf{X},\textbf{X}')^4H(\textbf{Y},\textbf{Y}'')^4\}} \\= & {} o(n\zeta ^2), \\ \textrm{E}\{G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}'')^2\}\le & {} \sqrt{\textrm{E}\{G(\textbf{X},\textbf{X}')^4\}\textrm{E}\{H(\textbf{Y},\textbf{Y}'')^4\}} \\= & {} o(n\zeta ^2). \\ \end{aligned}$$

Then it is trivial to check that

$$\begin{aligned} \textrm{E}(I_{n,1}^2J_{n,1}^2)= & {} \zeta ^2,\\ \textrm{E}(I_{n,1}^2J_{n,2}^2)= & {} \frac{1}{(n-2)^2}\textrm{E}\left[ G(\textbf{X}_i,\textbf{X}_j)^2\left\{ \sum _{l\notin \{i,j\}}H(\textbf{Y}_i,\textbf{Y}_l)^2{}\right. \right. \\{} & {} \left. \left. +\sum _{\{l,l'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_i,\textbf{Y}_l)H(\textbf{Y}_i,\textbf{Y}_{l'})\right\} \right] \\= & {} \frac{1}{(n-2)}\textrm{E}\{G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}\\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,1}^2J_{n,3}^2)= & {} \textrm{E}(I_{n,2}^2 J_{n,1}^2)=\textrm{E}(I_{n,3}^2 J_{n,1}^2)\\= & {} \frac{1}{(n-2)}\textrm{E}\{G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}\\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,1}^2J_{n,4}^2)= & {} \frac{1}{(n-2)^2(n-3)^2}\textrm{E}\left[ G(\textbf{X}_i,\textbf{X}_j)^2\left\{ \sum _{\{k,l\}\bigcap \{i,j\}=\emptyset } 2H(\textbf{Y}_k,\textbf{Y}_l)^2{}\right. \right. \\{} & {} +\sum _{\{k,l,l'\}\bigcap \{i,j\}=\emptyset }4H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k},\textbf{Y}_{l'}) {}\\{} & {} \left. \left. +\sum _{\{k,k',l,l'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k'},\textbf{Y}_{l'})\right\} \right] \\= & {} \frac{2}{(n-2)(n-3)}\textrm{E}\{G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}\\= & {} o(\zeta ^2), \end{aligned}$$

$$\begin{aligned} \textrm{E}(I_{n,4}^2 J_{n,1}^2)= & {} \frac{2}{(n-2)(n-3)}\textrm{E}\{G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}\\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,2}^2J_{n,2}^2)= & {} \frac{1}{(n-2)^4}\textrm{E}\left[ \left\{ \sum _{l\notin \{i,j\}} G(\textbf{X}_i,\textbf{X}_l)^2 +\sum _{\{l,l'\}\bigcap \{i,j\}=\emptyset }G(\textbf{X}_i,\textbf{X}_l)G(\textbf{X}_i,\textbf{X}_{l'})\right\} {}\right. \\{} & {} \left. \left\{ \sum _{l\notin \{i,j\}} H(\textbf{Y}_i,\textbf{Y}_l)^2 +\sum _{\{l,l'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_i,\textbf{Y}_l)H(\textbf{Y}_i,\textbf{Y}_{l'})\right\} \right] \\= & {} \frac{1}{(n-2)^3}\zeta ^2 +\frac{n-3}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}{}\\{} & {} + \frac{2(n-3)}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')H(\textbf{Y},\textbf{Y}')G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}'')\}\\\le & {} O\left\{ \frac{\zeta ^2+\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}}{n^2}\right\} \\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,3}^2J_{n,3}^2)= & {} \frac{1}{(n-2)^3}\zeta ^2 +\frac{n-3}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\},\\{} & {} + \frac{2(n-3)}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')H(\textbf{Y},\textbf{Y}')G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}'')\}\\\le & {} O\left\{ \frac{\zeta ^2+\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}}{n^2}\right\} \\= & {} o(\zeta ^2),\end{aligned}$$

$$\begin{aligned} \textrm{E}(I_{n,2}^2J_{n,3}^2)= & {} \frac{1}{(n-2)^4}\textrm{E}\left[ \left\{ \sum _{l\notin \{i,j\}} G(\textbf{X}_i,\textbf{X}_l)^2 +\sum _{\{l,l'\}\bigcap \{i,j\}=\emptyset }G(\textbf{X}_i,\textbf{X}_l)G(\textbf{X}_i,\textbf{X}_{l'})\right\} {}\right. \\{} & {} \left. \left\{ \sum _{k\notin \{i,j\}} H(\textbf{Y}_k,\textbf{Y}_j)^2 +\sum _{\{k,k'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_k,\textbf{Y}_j)H(\textbf{Y}_{k'},\textbf{Y}_j)\right\} \right] \\= & {} \frac{1}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\} +\frac{n-3}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}\\{} & {} + \frac{2(n-3)}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}''')G(\textbf{X}',\textbf{X}'')H(\textbf{Y}',\textbf{Y}''')\}\\\le & {} O\left\{ \frac{\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}+\textrm{E}\{ G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}}{n^2}\right\} \\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,3}^2J_{n,2}^2)= & {} \frac{1}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\} +\frac{n-3}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}\\{} & {} + \frac{2(n-3)}{(n-2)^3}\textrm{E}\{ G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}''')G(\textbf{X}',\textbf{X}'')H(\textbf{Y}',\textbf{Y}''')\}\\\le & {} O\left\{ \frac{\textrm{E}\{ G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2\}+\textrm{E}\{ G(\textbf{X},\textbf{X}')^2\}\textrm{E}\{H(\textbf{Y},\textbf{Y}')^2\}}{n^2}\right\} \\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,2}^2J_{n,4}^2)= & {} \frac{1}{(n-2)^4(n-3)^2}\textrm{E}\left[ \left\{ \sum _{l\notin \{i,j\}} G(\textbf{X}_i,\textbf{X}_l)^2 +\sum _{\{l,l'\}\bigcap \{i,j\}=\emptyset }G(\textbf{X}_i,\textbf{X}_l)G(\textbf{X}_i,\textbf{X}_{l'})\right\} {}\right. \\{} & {} \left\{ \sum _{\{k,l\}\bigcap \{i,j\}=\emptyset } 2H(\textbf{Y}_k,\textbf{Y}_l)^2 +\sum _{\{k,l,l'\}\bigcap \{i,j\}=\emptyset }4H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k},\textbf{Y}_{l'}){}\right. \\{} & {} \left. \left. +\sum _{\{k,k',l,l'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k'},\textbf{Y}_{l'})\right\} \right] \\= & {} \frac{1}{(n-2)^4(n-3)^2}\{4(n-2)(n-3)\textrm{E}( G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2){}\\ {}{} & {} \hspace{1.6cm}+2(n-2)(n-3)(n-4)\textrm{E}( G(\textbf{X},\textbf{X}')^2)\textrm{E}(H(\textbf{Y},\textbf{Y}')^2){}\\ {}{} & {} \hspace{1.6cm}+4(n-2)(n-3)\textrm{E}(G(\textbf{X},\textbf{X}')G(\textbf{X},\textbf{X}'')H(\textbf{Y}',\textbf{Y}'')^2)\\ {}{} & {} +8(n-2)(n-3)(n-4)\textrm{E}\{ G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}''')G(\textbf{X}',\textbf{X}'')H(\textbf{Y}',\textbf{Y}''')\}\\\le & {} O\left\{ \frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2)}{n^3}\} +O\{\frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2)\textrm{E}(H(\textbf{Y},\textbf{Y}')^2)}{n^3}\right\} \\= & {} o(\zeta ^2),\end{aligned}$$

$$\begin{aligned} \textrm{E}(I_{n,3}^2J_{n,4}^2)= & {} \textrm{E}(I_{n,4}^2 J_{n,2}^2)=\textrm{E}(I_{n,4}^2 J_{n,3}^2)\\\le & {} O\left\{ \frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2)}{n^3}\} +O\{\frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2)\textrm{E}(H(\textbf{Y},\textbf{Y}')^2)}{n^3}\right\} \\= & {} o(\zeta ^2),\\ \textrm{E}(I_{n,4}^2J_{n,4}^2)= & {} \frac{1}{(n-2)^4(n-3)^4}\textrm{E}\left[ \left\{ \sum _{\{k,l,l'\}\bigcap \{i,j\}=\emptyset }4G(\textbf{X}_k,\textbf{X}_l)G(\textbf{X}_{k},\textbf{X}_{l'}){}\right. \right. \\ {}{} & {} \left. +\sum _{\{k,l\}\bigcap \{i,j\}=\emptyset } 2G(\textbf{X}_k,\textbf{X}_l)^2+\sum _{\{k,k',l,l'\}\bigcap \{i,j\}=\emptyset }G(\textbf{X}_k,\textbf{X}_l)G(\textbf{X}_{k'},\textbf{X}_{l'})\right\} {}\\ {}{} & {} \left\{ \sum _{\{k,l\}\bigcap \{i,j\}=\emptyset } 2H(\textbf{Y}_k,\textbf{Y}_l)^2 +\sum _{\{k,l,l'\}\bigcap \{i,j\}=\emptyset }4H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k},\textbf{Y}_{l'}){}\right. \\ {}{} & {} +\left. \left. \left. \sum _{\{k,k',l,l'\}\bigcap \{i,j\}=\emptyset }H(\textbf{Y}_k,\textbf{Y}_l)H(\textbf{Y}_{k'},\textbf{Y}_{l'})\right\} \right. \right] \\= & {} \frac{1}{(n-2)^4(n-3)^4}[O(n^2\zeta ^2)+O\{n^3\textrm{E}( G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2)\}+O\{n^4\textrm{E}( {}\\ {}{} & {} G(\textbf{X},\textbf{X}')^2)\textrm{E}(H(\textbf{Y},\textbf{Y}')^2)\}+O\{n^3\textrm{E}(G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')H(\textbf{Y}',\textbf{Y}''))\}{}\\ {}{} & {} +O\{n^3\textrm{E}(H(\textbf{Y},\textbf{Y}')^2G(\textbf{X},\textbf{X}'')G(\textbf{X}',\textbf{X}''))\}\\ {}{} & {} +O\{n^3\textrm{E}( G(\textbf{X},\textbf{X}')G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}')H(\textbf{Y},\textbf{Y}''))\}{}\\ {}{} & {} + O\{n^3\textrm{E}( G(\textbf{X},\textbf{X}')G(\textbf{X},\textbf{X}'')H(\textbf{Y},\textbf{Y}')H(\textbf{Y}',\textbf{Y}''))\}\\ {}{} & {} + O\{n^4\textrm{E}( G(\textbf{X},\textbf{X}')G(\textbf{X},\textbf{X}'')H(\textbf{Y}''',\textbf{Y}')H(\textbf{Y}''',\textbf{Y}''))\}\\ {}{} & {} + O\{n^4\textrm{E}( G(\textbf{X},\textbf{X}')G(\textbf{X}'',\textbf{X}''')H(\textbf{Y},\textbf{Y}')H(\textbf{Y}'',\textbf{Y}'''))\}\\ {}{} & {} + O\{n^4\textrm{E}( G(\textbf{X},\textbf{X}')G(\textbf{X}'',\textbf{X}''')H(\textbf{Y},\textbf{Y}'')H(\textbf{Y}',\textbf{Y}'''))\}]\\\le & {} O\left\{ \frac{\zeta ^2}{n^4}\right\} +O\left\{ \frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2H(\textbf{Y},\textbf{Y}'')^2)}{n^4}\right\} +O\left\{ \frac{\textrm{E}( G(\textbf{X},\textbf{X}')^2)\textrm{E}(H(\textbf{Y},\textbf{Y}')^2)}{n^4}\right\} \\= & {} o(\zeta ^2). \end{aligned}$$

From the computations above we can conclude under assumption (A1) that

$$\begin{aligned} \textrm{E}\left( \frac{\zeta ^2_n}{\zeta ^2}\right) \longrightarrow 1. \end{aligned}$$

Using similar technique we also show under assumption (A1) that

$$\begin{aligned} \textrm{Var}\left( \frac{\zeta ^2_n}{\zeta ^2}\right) \longrightarrow 0. \end{aligned}$$

This proof is thus completed.

$\square$

Proof of Proposition 1

Note that $\textbf{Z}=(\textbf{X},\textbf{Y})^{^\top }$ is from multivariate normal distribution ${\mathcal {N}}(0_{(p+q)\times 1},\textbf{I}_{(p+q)})$. Then we can conclude that $\textbf{X}$ and $\textbf{Y}$ are independent, $\textbf{X}\sim {\mathcal {N}}(0_{p\times 1},\textbf{I}_{p})$ and $\textbf{Y}\sim {\mathcal {N}}(0_{q\times 1},\textbf{I}_{q})$. Therefore we only need to compute $\zeta ^2$, $\textrm{E}\{L_1(\textbf{Z},\textbf{Z}^{\prime })^2\}$ and $\textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}')^4\}$ since $\textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}')^4\}=\textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}'')^4\} =\textrm{E}\{(\textbf{X}^\top \textbf{X}')^4\}\textrm{E}\{(\textbf{Y}^\top \textbf{Y}'')^4\}.$ Direct calculations show that

$$\begin{aligned} \zeta ^2= & {} \textrm{E}\{\Vert \textbf{X}\Vert ^2\}\textrm{E}\{\Vert \textbf{Y}\Vert ^2\} = pq,\\ \textrm{E}\{L_1(\textbf{Z},\textbf{Z}')^2\}= & {} \textrm{E}\{(\textbf{X}^\top \textbf{X}')^2\}\textrm{E}\{(\textbf{Y}^\top \textbf{Y}')^2\} =pq,\\ \textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}')^4\}= & {} 9\textrm{E}\{\Vert \textbf{X}\Vert ^4\}\textrm{E}\{\Vert \textbf{Y}\Vert ^4\}=9pq(p+2)(q+2). \end{aligned}$$

$\square$

Proof of Theorem 2

Recall the expression of $h_4(\textbf{Z}_1,\textbf{Z}_2,\textbf{Z}_3,\textbf{Z}_4)$ in the proof of Theorem 1, and the fact that $\textrm{E}\{G(\textbf{X},\textbf{X}')|\textbf{X}\}=\textrm{E}\{G(\textbf{X},\textbf{X}')|\textbf{X}'\}=0$ and $\textrm{E}\{H(\textbf{Y},\textbf{Y}')|\textbf{Y}\}=\textrm{E}\{H(\textbf{Y},\textbf{Y}')|\textbf{Y}'\}=0$, we obtain the corresponding $h_1(\textbf{Z}_1),h_2(\textbf{Z}_1,\textbf{Z}_2)$ and $h_3(\textbf{Z}_1,\textbf{Z}_2,\textbf{Z}_3)$ under the local alternative as follows.

$$\begin{aligned} h_1(\textbf{Z}_1)= & {} \frac{1}{2}\{K(\textbf{X}_1,\textbf{Y}_1)+\textrm{tr}(\varvec{\Sigma }_{\textbf{X}\textbf{Y}}\varvec{\Sigma }_{\textbf{Y}\textbf{X}})\}, \\ h_2(\textbf{Z}_1,\textbf{Z}_2)= & {} \frac{1}{6}\{ G(\textbf{X}_1,\textbf{X}_2)H(\textbf{Y}_1,\textbf{Y}_2)+2K(\textbf{X}_1,\textbf{Y}_1)+2K(\textbf{X}_2,\textbf{Y}_2){}\\ {}{} & {} -K(\textbf{X}_2,\textbf{Y}_1)-K(\textbf{X}_1,\textbf{Y}_2)+\textrm{tr}(\varvec{\Sigma }_{\textbf{X}\textbf{Y}}\varvec{\Sigma }_{\textbf{Y}\textbf{X}})\},\\ h_3(\textbf{Z}_1,\textbf{Z}_2,\textbf{Z}_3)= & {} \frac{1}{12}[\{ 2G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_3)-G(\textbf{X}_2,\textbf{X}_3)\}H(\textbf{Y}_1,\textbf{Y}_2){}\\ {}{} & {} +\{2G(\textbf{X}_1,\textbf{X}_3)-G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_2,\textbf{X}_3)\}H(\textbf{Y}_1,\textbf{Y}_3){}\\ {}{} & {} +\{2G(\textbf{X}_2,\textbf{X}_3)-G(\textbf{X}_1,\textbf{X}_2)-G(\textbf{X}_1,\textbf{X}_3)\}H(\textbf{Y}_2,\textbf{Y}_3){}\\ {}{} & {} +2K(\textbf{X}_1,\textbf{Y}_1)-K(\textbf{X}_2,\textbf{Y}_1)-K(\textbf{X}_3,\textbf{Y}_1)+2K(\textbf{X}_2,\textbf{Y}_2){}\\ {}{} & {} -K(\textbf{X}_1,\textbf{Y}_2)-K(\textbf{X}_3,\textbf{Y}_2)+2K(\textbf{X}_3,\textbf{Y}_3)-K(\textbf{X}_1,\textbf{Y}_3){}\\ {}{} & {} -K(\textbf{X}_2,\textbf{Y}_3), \end{aligned}$$

where $K(\textbf{X}_i,\textbf{Y}_j)=\textbf{X}_i^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y}_j$. Again using the Hoeffding decomposition in Serfling (1980). we can show under assumptions (A1) and (A2) that

$$\begin{aligned} \frac{n(T_{n,p,q}(\textbf{X},\textbf{Y})-\textrm{tr}(\varvec{\Sigma }_{\textbf{X}\textbf{Y}}\varvec{\Sigma }_{\textbf{Y}\textbf{X}}))}{\sqrt{2\zeta ^2}}= \frac{\sum \limits _{1\le i< j\le n}\widetilde{L}(\textbf{Z}_i,\textbf{Z}_j)}{\sqrt{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \zeta ^2}}+o_p(1), \end{aligned}$$

where $\widetilde{L}(\textbf{Z}_i,\textbf{Z}_j) =L(\textbf{Z}_i,\textbf{Z}_j)-K(\textbf{X}_i,\textbf{Y}_i)-K(\textbf{X}_j,\textbf{Y}_j)+\textrm{tr}(\varvec{\Sigma }_{\textbf{X}\textbf{Y}}\varvec{\Sigma }_{\textbf{Y}\textbf{X}})$.

Note that $\textrm{E}(\widetilde{L}(\textbf{Z},\textbf{Z}'))=\textrm{E}(\widetilde{L}(\textbf{Z},\textbf{Z}')\mid \textbf{Z})=\textrm{E}(\widetilde{L}(\textbf{Z},\textbf{Z}')\mid \textbf{Z}')=0$. Using similar arguments by replacing $L(\textbf{Z}_i,\textbf{Z}_j)$ in the proof of Theorem 1 with $\widetilde{L}(\textbf{Z}_i,\textbf{Z}_j)$, we can show that

$$\begin{aligned} \frac{\sum \limits _{1\le i< j\le n}\widetilde{L}(\textbf{Z}_i,\textbf{Z}_j)}{\sqrt{\left( {\begin{array}{c}n\\ 2\end{array}}\right) \zeta ^2}}\overset{d}{\longrightarrow } {\mathcal {N}}(0,1), \end{aligned}$$

which yields that

$$\begin{aligned} \frac{n(T_{n,p,q}(\textbf{X},\textbf{Y})-\textrm{tr}(\varvec{\Sigma }_{\textbf{X}\textbf{Y}}\varvec{\Sigma }_{\textbf{Y}\textbf{X}}))}{\sqrt{2\zeta ^2}}\overset{d}{\longrightarrow }\ {\mathcal {N}}(0,1). \end{aligned}$$

Applying similar technique in proving the ratio-consistency in theorem 1, it is easy to show that

$$\begin{aligned} \frac{\zeta ^2_n}{\zeta ^2}\overset{P}{\longrightarrow }\ 1, \end{aligned}$$

provided that conditions (A1) and (A2) hold. We thus complete this proof.

$\square$

Proof of Proposition 2

For $\Vert \varvec{\beta }\Vert \ne 0$, we assume that U is a $p\times p$ orthogonal matrix whose first column is $\varvec{\beta }/\Vert \varvec{\beta }\Vert$. Then $\textbf{S}=(S_1,\ldots ,S_p)^\top =U^\top \textbf{X}\sim {\mathcal {N}}(0_{p\times 1},\textbf{I}_p)$. Moreover, $\textbf{S}'=(S_1',\ldots ,S_p')^\top =U^\top \textbf{X}'$ and $\textbf{S}''=(S_1'',\ldots ,S_p'')^\top =U^\top \textbf{X}''$ can be viewed as copies of $\textbf{S}$. Note that $\textrm{E}(S_1)=0$, $\textrm{E}(S_1^2)=1$ and $\textrm{E}(S_1^4)=3$. Elemental calculations show that

$$\begin{aligned}{} & {} \zeta ^2= \textrm{E}\{L(\textbf{Z},\textbf{Z}')^2\}\\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^2(\textbf{X}^\top \varvec{\beta }+\varepsilon )^2({\textbf{X}'}^\top \varvec{\beta }+\varepsilon ')^2\}\\{} & {} \quad =\textrm{E}\{(\textbf{X}^\top \textbf{X}')^2((\textbf{X}^\top \varvec{\beta })^2+1)(({\textbf{X}'}^\top \varvec{\beta })^2+1)\}\\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^2((\textbf{X}^\top \varvec{\beta })^2({\textbf{X}'}^\top \varvec{\beta })^2+(\textbf{X}^\top \varvec{\beta })^2+ ({\textbf{X}'}^\top \varvec{\beta })^2+1)\}\\{} & {} \quad = \textrm{E}\{(\textbf{S}^\top {\textbf{S}}')^2(\Vert \varvec{\beta }\Vert ^4S_1^2{S_1'}^2 +\Vert \varvec{\beta }\Vert ^2S_1^2+\Vert \varvec{\beta }\Vert ^2{S_1'}^2+1)\}\\{} & {} \quad =(p+8)\Vert \varvec{\beta }\Vert ^4+2(p+2)\Vert \varvec{\beta }\Vert ^2+p,\\{} & {} \textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}')^4\}\\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^4(\textbf{X}^\top \varvec{\beta }+\varepsilon )^4({\textbf{X}'}^\top \varvec{\beta }+\varepsilon ')^4\}\\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^4((\textbf{X}^\top \varvec{\beta })^4+6(\textbf{X}^\top \varvec{\beta })^2+3)(({\textbf{X}'}^\top \varvec{\beta })^4 +6({\textbf{X}'}^\top \varvec{\beta })^2+3)\}\\{} & {} \quad =\textrm{E}\{(\textbf{S}^\top {\textbf{S}}')^4 (\Vert \varvec{\beta }\Vert ^4S_1^4+6\Vert \varvec{\beta }\Vert ^2S_1^2+3) (\Vert \varvec{\beta }\Vert ^4{S'}_1^4+6\Vert \varvec{\beta }\Vert ^2{S'}_1^2+3)\}\\{} & {} \quad =O\{p^2(\Vert \varvec{\beta }\Vert ^2+1)^4\},\\{} & {} \textrm{E}\{(\textbf{X}^\top \textbf{X}'\textbf{Y}^\top \textbf{Y}'')^4\} \\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^4(\textbf{X}^\top \varvec{\beta }+\varepsilon )^4({\textbf{X}''}^\top \varvec{\beta }+\varepsilon '')^4\}\\{} & {} \quad = 9(\Vert \varvec{\beta }\Vert ^2+1)^2\textrm{E}\{\Vert \textbf{X}\Vert ^4((\textbf{X}^\top \varvec{\beta })^4+6(\textbf{X}^\top \varvec{\beta })^2+3)\}\\{} & {} \quad = 9(\Vert \varvec{\beta }\Vert ^2+1)^2\textrm{E}\{\Vert \textbf{S}\Vert ^4(\Vert \varvec{\beta }\Vert ^4S_1^4+6\Vert \varvec{\beta }\Vert ^2S_1^2+3)\}\\{} & {} \quad = O\{p^2(\Vert \varvec{\beta }\Vert ^2+1)^4\},\\{} & {} \textrm{E}\{(\textbf{X}^\top \textbf{X}')^4\}\textrm{E}\{(\textbf{Y}^\top \textbf{Y}')^4\} \\{} & {} \quad = \textrm{E}\{(\textbf{X}^\top \textbf{X}')^4\}\textrm{E}\{(\textbf{X}^\top \varvec{\beta }+\varepsilon )^4\}\textrm{E}\{({\textbf{X}'}^\top \varvec{\beta }+\varepsilon ')^4\}\\{} & {} \quad = 27(\Vert \varvec{\beta }\Vert ^2+1)^4\textrm{E}(\Vert \textbf{X}\Vert ^4)\\{} & {} \quad = 27p(p+2)(\Vert \varvec{\beta }\Vert ^2+1)^4. \end{aligned}$$

We then calculate $\textrm{E}\{(\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y})^2\}$ and $\textrm{E}\{K(\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y}')^2\}$. It is easy to show that $\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y}=\textbf{X}^\top \varvec{\beta }\textbf{Y}$ and $\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y}' =\textbf{X}^\top \varvec{\beta }\textbf{Y}'$. Then we can obtain that

$$\begin{aligned} \textrm{E}\{(\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y})^2\}= & {} \textrm{E}\{(\textbf{X}^\top \varvec{\beta })^2((\textbf{X}^\top \varvec{\beta })^2 +2\varepsilon \textbf{X}^\top \varvec{\beta }+\varepsilon ^2)\} \\= & {} \textrm{E}\{(\textbf{X}^\top \varvec{\beta })^4\}+\textrm{E}\{(\textbf{X}^\top \varvec{\beta })^2\} \\= & {} \Vert \varvec{\beta }\Vert ^4\textrm{E}(S_1^4)+\Vert \varvec{\beta }\Vert ^2\textrm{E}(S_1^2)\\= & {} 3\Vert \varvec{\beta }\Vert ^4+\Vert \varvec{\beta }\Vert ^2,\\ \textrm{E}\{(\textbf{X}^\top \varvec{\Sigma }_{\textbf{X}\textbf{Y}}\textbf{Y}')^2\}= & {} \textrm{E}\{(\textbf{X}^\top \varvec{\beta })^2\}\textrm{E}\{({\textbf{X}'}^\top \varvec{\beta })^2 +2\varepsilon '{\textbf{X}'}^\top \varvec{\beta }+{\varepsilon '}^2\} \\= & {} \textrm{E}\{(\textbf{X}^\top \varvec{\beta })^2\}\textrm{E}\{({\textbf{X}'}^\top \varvec{\beta })^2+1\} \\= & {} \Vert \varvec{\beta }\Vert ^2\textrm{E}(S_1^2)\textrm{E}(\Vert \varvec{\beta }\Vert ^2{S'}_1^2+1)\\= & {} \Vert \varvec{\beta }\Vert ^4+\Vert \varvec{\beta }\Vert ^2. \end{aligned}$$

Next we consider $\textrm{E}\{L_1(\textbf{Z},\textbf{Z}^{\prime })^2\}$.

Careful calculations show that

$$\begin{aligned}{} & {} \textrm{E}\{L_1(\textbf{Z},\textbf{Z}^{\prime })^2\}\\{} & {} \quad =\textrm{E}[\{\textrm{E}(L(\textbf{Z},\textbf{Z}'')L(\textbf{Z}',\textbf{Z}'')|(\textbf{Z},\textbf{Z}'))\}^2]\\{} & {} \quad = \textrm{E}[\{\textrm{E}(\textbf{Y}\textbf{Y}'{\textbf{X}''}^\top \textbf{X}{\textbf{X}'}^\top \textbf{X}''{\textbf{Y}''}^2|(\textbf{X},\textbf{Y},\textbf{X}',\textbf{Y}'))\}^2]\\{} & {} \quad = \textrm{E}[\{\textrm{E}(\textbf{Y}\textbf{Y}'{\textbf{X}''}^\top \textbf{X}{\textbf{X}'}^\top \textbf{X}''(({\textbf{X}''}^\top \varvec{\beta })^2+1)|(\textbf{X},\textbf{Y},\textbf{X}',\textbf{Y}'))\}^2]\\{} & {} \quad = \textrm{E}\left[ (\Vert \varvec{\beta }\Vert S_1+\varepsilon )^2(\Vert \varvec{\beta }\Vert S'_1+\varepsilon ')^2\right. \\{} & {} \left. \qquad \left\{ (3\Vert \varvec{\beta }\Vert ^2+1)S_1S'_1+(\Vert \varvec{\beta }\Vert ^2+1)\sum _{i=2}^pS_iS'_i\right\} ^2\right] \\{} & {} \quad = \textrm{E}\left[ (\Vert \varvec{\beta }\Vert ^4S_1^2{S'}_1^2+\Vert \varvec{\beta }\Vert ^2S_1^2+\Vert \varvec{\beta }\Vert ^2{S'}_1^2+1)\right. \\{} & {} \left. \qquad \left\{ (3\Vert \varvec{\beta }\Vert ^2+1)S_1S'_1+(\Vert \varvec{\beta }\Vert ^2+1)\sum _{i=2}^pS_iS'_i\right\} ^2\right] \\{} & {} \quad = (3\Vert \varvec{\beta }\Vert ^2+1)^4+(p-1)(\Vert \varvec{\beta }\Vert ^2+1)^4. \end{aligned}$$

Thus this proof is completed. $\square$

Proof of Theorem 3

(1) Let $\lambda _{n,p_n,q_n}=n(p_nq_n)^{-1/4}(\log n)^{-3/4}$. It suffices to show that, for any $\varepsilon >0$, almost surely $\lambda _{n,p_n,q_n}\max \limits _{1\le i \le p_n,1\le j \le q_n}|{\mathcal {R}}_n(X_i,Y_j)|<\varepsilon$ for all n sufficiently large, that is,

$$\begin{aligned} \Pr \Big (\lambda _{n,p_n,q_n}\max \limits _{1\le i \le p_n,1\le j \le q_n} |{\mathcal {R}}_n(X_i,Y_j)|>\varepsilon \text { for infinitely many} \hspace{2mm} n \Big )=0. \end{aligned}$$

Applying the Borel–Cantelli lemma, it suffices to show that

$$\begin{aligned} \sum _{k=0}^{\infty } \Pr \left( \max \limits _{2^k\le n\le 2^{k+1}} \left( \lambda _{n,p_n,q_n}\max \limits _{1\le i\le p_n,1\le j \le q_n}|{\mathcal {R}}_n(X_i,Y_j)|\right) >\varepsilon \right) <\infty . \end{aligned}$$

Notice that $T_{n,1,1}(X_i,X_i)$ and $T_{n,1,1}(Y_j,Y_j)$ are U-statistics of order 4. By the Theorem 5.4.C in Serfling (1980), it is easy to check that for each i, $1\le i \le p,1\le j\le q$, almost surely

$$\begin{aligned} \frac{T_{n,1,1}(X_i,X_i)}{\sigma ^2(X_i,X_i)} = 1+O\left( \frac{1}{\sqrt{n}}\right) , \frac{T_{n,1,1}(Y_j,Y_j)}{\sigma ^2(Y_j,Y_j)} = 1+O\left( \frac{1}{\sqrt{n}}\right) , \end{aligned}$$

provided the assumption (A3) holds. In addition, under $H_0,$ $\textrm{E}\{G(X_{i},X_{i}')H(Y_j,Y_j')\}=K(X_{i},Y_j) =0$. We can obtain from a standard result in section 32 of Loève (1978) and Markov’s inequality that

$$\begin{aligned} \begin{aligned}&\sum _{k=0}^{\infty }\Pr \left( \max \limits _{2^k\le n\le 2^{k+1}} (\lambda _{n,p_n,q_n}\max \limits _{1\le i\le p_n,1\le j\le q_n}|{\mathcal {R}}_n(X_i,Y_j)|)>\varepsilon \right) \\&\quad \le \sum _{k=0}^{\infty }\sum _{i=1}^{p_{2^{k+1}}}\sum _{j=1}^{q_{2^{k+1}}} \Pr \left( \max \limits _{2^k\le n\le 2^{k+1}} (\lambda _{n,p_n,q_n}|{\mathcal {R}}_n(X_i,Y_j)|)>\varepsilon \right) \\&\quad \le \sum _{k=0}^{\infty }\sum _{i=1}^{p_{2^{k+1}}}\sum _{j=1}^{q_{2^{k+1}}}\varepsilon ^{-4} \frac{2^{4(k+1)}}{(k+1)^{3}p_{2^{k+1}}q_{2^{k+1}}(\log 2)^{3}} \frac{\textrm{E}(|T_{2^k,1,1}(X_i,Y_j)|^4)}{\sigma ^4(X_i,X_i)\sigma ^4(Y_j,Y_j)}\\&\quad = \sum _{k=0}^{\infty } O((k+1)^{-3}), \end{aligned} \end{aligned}$$

where the last equality follows from Theorem 5.3.2 in Serfling (1980). Thus the result in (1) can be obtained.

(2) We need to show that almost surely

$$\begin{aligned} \Pr \left( \max \limits _{1\le i \le p,1\le j \le q} |{\mathcal {R}}_n(X_i,Y_j)| <\delta _{n,p,q} |S \ne \emptyset \right) \rightarrow 0, \hspace{5mm} n\rightarrow \infty . \end{aligned}$$

(43)

To obtain the result, it is sufficient to prove that

$$\begin{aligned} \Pr \left( \max \limits _{1\le i \le p,1\le j \le q} \frac{|T_{n,1,1}(X_i,Y_j)-\sigma ^2(X_i,Y_j)|}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}} >\delta _{n,p,q}|S \ne \emptyset \right) \rightarrow 0, \end{aligned}$$

since

$$\begin{aligned}{} & {} \Pr \left( \max \limits _{1\le i \le p,1\le j \le q} |{\mathcal {R}}_n(X_i,Y_j)| <\delta _{n,p,q}|S \ne \emptyset \right) \\{} & {} \quad \le \Pr \left( \max \limits _{1\le i \le p,1\le j \le q} \frac{|T_{n,1,1}(X_i,Y_j)-\sigma ^2(X_i,Y_j)|}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}} >\delta _{n,p,q}|S \ne \emptyset \right) . \end{aligned}$$

Write the projection of U-statistic $T_{n,1,1}(X_i,Y_j)$ as

$$\begin{aligned} \widehat{T}_{n,1,1}(X_i,Y_j) = \sum _{t=1}^n \textrm{E}\{T_{n,1,1}(X_i,Y_j)|(X_{ti},Y_{tj})\}-(n-1)\sigma ^2(X_i,Y_j). \end{aligned}$$

Thus we obtain that

$$\begin{aligned}{} & {} T_{n,1,1}(X_i,Y_j)-\sigma ^2(X_i,Y_j)\\{} & {} \quad =\frac{4}{n}\sum _{t=1}^n\left\{ K(X_{ti},Y_{tj})-\sigma ^2(X_i,Y_j)\right\} + (T_{n,1,1}(X_i,Y_j)-\widehat{T}_{n,1,1}(X_i,Y_j)), \end{aligned}$$

where $K(X_{ti},Y_{tj})=\sigma (X_i,Y_j)X_{ti}Y_{tj}$. Similar to prove (1), it is easy to check that almost surely

$$\begin{aligned} \max \limits _{1\le i \le p,1\le j \le q}\frac{|T_{n,1,1}(X_i,Y_j)-\widehat{T}_{n,1,1}(X_i,Y_j)|}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}} =o(\delta _{n,p,q}), \hspace{5mm} n \rightarrow \infty . \end{aligned}$$

Hence, we only need to show that

$$\begin{aligned} \Pr \left( \max \limits _{1\le i \le p,1\le j \le q} \left| \frac{4}{n}\sum _{t=1}^n\frac{K(X_{ti},Y_{tj})-\sigma ^2(X_i,Y_j)}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}}\right| >\delta _{n,p,q}|S \ne \emptyset \right) \rightarrow 0. \end{aligned}$$

Similarly, we have that

$$\begin{aligned}{} & {} \sum _{k=0}^{\infty }\Pr \left( \max \limits _{2^k\le n\le 2^{k+1}} (\lambda _{n,p_n,q_n}\max \limits _{1\le i \le p_n,1\le j \le q_n}\left| \frac{4}{n}\sum _{t=1}^n\frac{K(X_{ti},Y_{tj})-\sigma ^2(X_i,Y_j)}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}}\right| )>\varepsilon \right) \\{} & {} \quad \le \sum _{k=0}^{\infty } \sum _{i=1}^{p_{2^{k+1}}}\sum _{j=1}^{q_{2^{k+1}}}\varepsilon ^{-2}\frac{2^{2(k+1)}}{(k+1)^{3/2} (p_{2^{k+1}}q_{2^{k+1}})^{1/2} (\log 2)^{3/2}}\frac{\max \limits _{1\le i \le p_{2^{k+1}},1\le j \le q_{2^{k+1}}}\xi _{ij}}{2^{2k}\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}\\{} & {} \quad = \sum _{k=0}^{\infty } O((k+1)^{-3/2}), \end{aligned}$$

where the last equality is true when (A4) holds. Therefore, we have almost surely that

$$\begin{aligned} \max \limits _{1\le i \le p,1\le j\le q} \frac{|T_{n,1,1}(X_i,Y_j)-\sigma ^2(X_i,Y_j)|}{\sqrt{\sigma ^2(X_i,X_i)\sigma ^2(Y_j,Y_j)}} =o(\delta _{n,p,q}), \hspace{5mm} n \rightarrow \infty . \end{aligned}$$

This proof is thus completed.

$\square$