Structural Properties of the Wyner–Ziv Rate Distortion Function: Applications for Multivariate Gaussian Sources ^†

Abstract

The main focus of this paper is the derivation of the structural properties of the test channels of Wyner’s operational information rate distortion function (RDF), $\overline{R}\left({\Delta }_X\right)$, for arbitrary abstract sources and, subsequently, the derivation of additional properties for a tuple of multivariate correlated, jointly independent, and identically distributed Gaussian random variables, ${\{X_t,Y_t\}}_{t=1}^{\infty }$, $X_t:\Omega \to {\mathbb{R}}^{n_x}$, $Y_t:\Omega \to {\mathbb{R}}^{n_y}$, with average mean-square error at the decoder and the side information, ${\left\{Y_t\right\}}_{t=1}^{\infty }$, available only at the decoder. For the tuple of multivariate correlated Gaussian sources, we construct optimal test channel realizations which achieve the informational RDF, $\overline{R}\left({\Delta }_X\right)\stackrel{▵}{=}{inf}_{\mathcal{M}\left({\Delta }_X\right)}I(X;Z|Y)$, where $\mathcal{M}\left({\Delta }_X\right)$ is the set of auxiliary RVs Z such that ${\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X}$, $\widehat{X}=f(Y,Z)$, and $\mathbf{E}\left\{\right||X-\widehat{X}{\left|\right|}^2\}\le {\Delta }_X$. We show the following fundamental structural properties: (1) Optimal test channel realizations that achieve the RDF and satisfy conditional independence, ${\mathbf{P}}_{X|\widehat{X},Y,Z}={\mathbf{P}}_{X|\widehat{X},Y}={\mathbf{P}}_{X|\widehat{X}},\mathbf{E}\left\{X|\widehat{X},Y,Z\right\}=\mathbf{E}\left\{X|\widehat{X}\right\}=\widehat{X}.$ (2) Similarly, for the conditional RDF, $R_{X|Y}\left({\Delta }_X\right)$, when the side information is available to both the encoder and the decoder, we show the equality $\overline{R}\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$. (3) We derive the water-filling solution for $R_{X|Y}\left({\Delta }_X\right)$.

1. Introduction, Problem Statement, and Main Results

1.1. The Wyner and Ziv Lossy Compression Problem and Generalizations

Wyner and Ziv [1] derived an operational information definition for the lossy compression problem in Figure 1 with respect to a single-letter fidelity of reconstruction. The joint sequence of random variables (RVs) $\{(X_t,Y_t):t=1,2,\cdots \}$ takes values in sets of finite cardinality, $\{\mathcal{X},\mathcal{Y}\}$, and it is generated independently according to the joint probability distribution function ${\mathbf{P}}_{X,Y}$. Wyner [2] generalized [1] to RVs $\{(X_t,Y_t):t=1,2,\cdots \}$ that take values in abstract alphabet spaces $\{\mathcal{X},\mathcal{Y}\}$ and hence include continuous-valued RVs.

(A) Switch “A” Closed: When the side information $\{Y_t:t=1,2,\cdots \}$ is available non-causally at both the encoder and the decoder, Wyner [2] (see also Berger [3]) characterized the infimum of all achievable operational rates (denoted by ${\overline{R}}_1\left({\Delta }_X\right)$ in [2]), subject to a single-letter fidelity with average distortion less than or equal to ${\Delta }_X\in [0,\infty )$. The rate is given by the single-letter operational information theoretic conditional RDF:

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{{\mathcal{M}}_0\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y)\in [0,\infty ],{\Delta }_X\in [0,\infty )\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill =& \underset{{\mathbf{P}}_{\widehat{X}|X,Y}:\mathbf{E}\left\{d_X(X,\widehat{X})\right\}\le {\Delta }_X}{inf}I(X;\widehat{X}|Y)\hfill \end{array}$$

(2)

where ${\mathcal{M}}_0\left({\Delta }_X\right)$ is the set specified by

$$\begin{array}{cc}\hfill {\mathcal{M}}_0\left({\Delta }_X\right)\stackrel{▵}{=}& \{\widehat{X}:\Omega \to \widehat{\mathcal{X}}:{\mathbf{P}}_{X,Y,\widehat{X}}\mathrm{is}\mathrm{the}\mathrm{joint}\mathrm{measure}\mathrm{on}\mathcal{X}\times \mathcal{Y}\times \widehat{\mathcal{X}},\hfill \\ \hfill & \mathbf{E}\left\{d_X(X,\widehat{X})\right\}\le {\Delta }_X\},\hfill \end{array}$$

(3)

and $\widehat{X}$ is the reproduction of X. $I(X;\widehat{X}|Y)$ is the conditional mutual information between X and $\widehat{X}$ conditioned on Y, and $d_X(·,·)$ is the fidelity criterion between x and $\widehat{x}$. The infimum in (1) is over all elements of ${\mathcal{M}}_0\left({\Delta }_X\right)$ with induced joint distributions ${\mathbf{P}}_{X,Y,\widehat{X}}$ of the RVs $(X,Y,\widehat{X})$ such that the marginal distribution ${\mathbf{P}}_{X,Y}$ is the fixed joint distribution of the source $(X,Y)$. This problem is equivalent to (2) [4].

(B) Switch “A” Open: When the side information is available non-causally only at the decoder, Wyner [2] characterized the infimum of all achievable operational rates (denoted by $R^{\ast }\left({\Delta }_X\right)$ in [2]), subject to a single-letter fidelity with average distortion less than or equal to ${\Delta }_X$. The rate is given by the single-letter operational information theoretic RDF as a function of an auxiliary RV $Z:\Omega \to \mathcal{Z}$:

$$\begin{array}{cc}\hfill \overline{R}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{\mathcal{M}\left({\Delta }_X\right)}{inf}\left\{I(X;Z)-I(Y;Z)\right\}\in [0,\infty ],{\Delta }_X\in [0,\infty )\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill =& \underset{\mathcal{M}\left({\Delta }_X\right)}{inf}I(X;Z|Y)\hfill \end{array}$$

(5)

where $\mathcal{M}\left({\Delta }_X\right)$ is specified by the set of auxiliary RVs Z and defined as:

$$\begin{array}{cc}\hfill & \mathcal{M}\left({\Delta }_X\right)\stackrel{▵}{=}\{Z:\Omega \to \mathcal{Z}:{\mathbf{P}}_{X,Y,Z,\widehat{X}}\mathrm{is}\mathrm{the}\mathrm{joint}\mathrm{measure}\mathrm{on}\mathcal{X}\times \mathcal{Y}\times \mathcal{Z}\times \widehat{\mathcal{X}},\hfill \\ \hfill & {\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X},\exists \mathrm{meas}.\mathrm{fun}.f:\mathcal{Y}\times \mathcal{Z}\to \widehat{\mathcal{X}},\widehat{X}=f(Y,Z),\mathbf{E}\left\{d_X(X,\widehat{X})\right\}\le {\Delta }_X\}.\hfill \end{array}$$

(6)

Wyner’s realization of the joint measure ${\mathbf{P}}_{X,Y,Z,\widehat{X}}$ induced by the RVs $(X,Y,Z,\widehat{X})$ is illustrated in Figure 2, where Z is the output of the “test channel”, ${\mathbf{P}}_{Z|X}$. Clearly, $\overline{R}\left({\Delta }_X\right)$ involves two strategies, i.e., $f(·,·)$ and ${\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X}$. This makes it a much more complex problem compared to $R_{X|Y}\left({\Delta }_X\right)$ (which involves only ${\mathbf{P}}_{\widehat{X}|X,Y}$).

Throughout [2], the following assumption is imposed.

Assumption 1. 

$I(X;Y)<\infty$ (see [2]).

Wyner [2] considered scalar-valued jointly Gaussian RVs $(X,Y)$ with square-error distortion and constructed the optimal realizations $\widehat{X}$ and $(Z,\widehat{X})$ and the function $f(X,Z)$ from the sets ${\mathcal{M}}_0\left({\Delta }_X\right)$ and $\mathcal{M}\left({\Delta }_X\right)$, respectively. Also, it is shown that these realizations achieve the characterizations of the RDFs $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$, respectively, and that the two rates are equal, i.e., $\overline{R}\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$.

(C) Marginal RDF: If there is no side information, $\{Y_t:t=1,2,\dots \}$, or the side information is independent of the source, $\{X_t:t=1,2,\dots \}$, the RDFs $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ degenerate to the marginal RDF $R_X\left({\Delta }_X\right)$, defined by

$$\begin{array}{c}\hfill R_X\left({\Delta }_X\right)\stackrel{▵}{=}\underset{{\mathbf{P}}_{\widehat{X}|X}:\mathbf{E}\left\{d_X(X,\widehat{X})\right\}\le {\Delta }_X}{inf}I(X;\widehat{X})\in [0,\infty ],{\Delta }_X\in [0,\infty ).\end{array}$$

(7)

(D) Gray’s Lower Bounds: A lower bound on $R_{X|Y}\left({\Delta }_X\right)$ is given by Gray in [4] [Theorem 3.1]. This bound connects $R_{X|Y}\left({\Delta }_X\right)$ with the marginal RDF and the mutual information between X and Y as follows:

$$\begin{array}{c}\hfill R_{X|Y}\left({\Delta }_X\right)\ge R_X\left({\Delta }_X\right)-I(X;Y).\end{array}$$

(8)

Clearly, the lower bound is trivial for values of ${\Delta }_X\in [0,\infty )$ such that $R_X\left({\Delta }_X\right)-I(X;Y)<0$.

1.2. Main Contributions of the Paper

We first consider Wyner’s [2] RDFs $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ for arbitrary RVs $(X,Y)$ defined on abstract alphabet spaces, and we derive structural properties of the realizations that achieve the two optimal test channels. Subsequently, we generalize Wyner’s [2] results to multivariate-valued jointly Gaussian RVs $(X,Y)$. In other words, we construct the optimal multivariate-valued realizations $\widehat{X}$ and $(\widehat{X},Z)$ and the function $f(X,Z)$ which achieve the RDFs $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$, respectively. In the literature, it is often called achievability of the converse coding theorem. In addition, we use the realizations to prove the equality $\overline{R}\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$ and to derive the water-filling solution. Along the way, we verify that our results reproduce, for scalar-valued RVs $(X,Y)$, Wyner [2] RDFs and the optimal realizations. However, to our surprise, the existing results from the literature [[5], Theorem 4 and Abstract and [6], Theorem 3A], which deal with the more general multivariate-valued remote sensor problem (the RDF of the remote sensor problem is a generalization of Wyner’s RDF $\overline{R}\left({\Delta }_X\right)$, with the encoder observing a noisy version of the RVs generated by the source), do not degenerate to Wyner’s [2] RDFs, when specilized to scalar-valued RVs (we verify this in Remark 5 by also checking the correction suggested in https://tiangroup.engr.tamu.edu/publications/) (accessed on 3 January 2024). In Section 1.3, we give a detailed account of the main results of this paper. We should emphasize that preliminary results of this paper appeared in [7], mostly without the details of the proofs. This paper is extended [7] and contains complete proofs of the preliminary results of [7], which in some cases are lengthy (see, for example, Section 4, proofs of Theorems 3–5, Corollaries 1 and 2, etc.).

1.3. Problem Statement and Main Results

(a) We consider a tuple of jointly independent and identically distributed (i.i.d.) arbitrary RVs $(X^n,Y^n)=\{(X_t,Y_t):t=1,2,\dots ,n\}$ defined on abstract alphabet spaces, and we derive the following results.

(a.1) Lemma 1: Achievable lower bound on the conditional mutual information $I(X;\widehat{X}|Y)$, which strengthens Gray’s lower bound (8) [[4], Theorem 3.1].

(a.2) Theorem 2: Structural properties of the optimal reconstruction $\widehat{X}$, which achieves a lower bound on $R_{X|Y}\left({\Delta }_X\right)$ for mean-square error distortion. Theorem 2 strengthens the conditions for the equality to hold, $R_{X|Y}\left({\Delta }_X\right)=\overline{R}\left({\Delta }_X\right)$, given by Wyner [2] [Remarks, p. 65] (see Remark 1). However, for finite-alphabet-valued sources with Hamming distance distortion, it might be the case that $R_{X|Y}\left({\Delta }_X\right)<\overline{R}\left({\Delta }_X\right)$, as pointed out by Wyner and Ziv [1] [Section 3] for the doubly symmetric binary source.

(b) We consider a tuple of jointly i.i.d. multivariate Gaussian RVs $(X^n,Y^n)=\{(X_t,Y_t):t=1,2,\dots ,n\}$, with respect to the square-error fidelity, as defined below.

$$\begin{array}{cc}\hfill & X_t:\Omega \to {\mathbb{R}}^{n_x}=\mathcal{X},Y_t:\Omega \to {\mathbb{R}}^{n_y}=\mathcal{Y},t=1,2,\dots ,n,\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & X_t\in N(0,Q_X),Y_t\in N(0,Q_Y),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & Q_{(X_t,Y_t)}=\mathbb{E}\left\{\left[\begin{array}{c}{X}_t\\ Y_t\end{array}\right]{\left[\begin{array}{c}{X}_t\\ Y_t\end{array}\right]}^T\right\}=\left[\begin{array}{cc}{Q}_X& Q_{X,Y}\\ Q_{X,Y}^T& Q_Y\end{array}\right],\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & {\mathbf{P}}_{X_t,Y_t}={\mathbf{P}}_{X,Y}\mathrm{multivariate}\mathrm{Gaussian}\mathrm{distribution},\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & {\widehat{X}}_t:\Omega \to {\mathbb{R}}^{n_x}=\mathcal{X},t=1,2\dots ,n,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & D_X(x^n,{\widehat{x}}^n)=\frac{1}{n}\sum _{t=1}^n\left|\right|x_t-{\widehat{x}}_t{\left|\right|}_{{\mathbb{R}}^{n_x}}^2,\hfill \end{array}$$

(14)

where $n_x,n_y$ are arbitrary positive integers, $X\in N(0,Q_X)$ means X is a Gaussian RV, with zero mean and covariance matrix $Q_X$, and $\left|\right|·{\left|\right|}_{{\mathbb{R}}^{n_x}}^2$ is the Euclidean distance on ${\mathbb{R}}^{n_x}$. To give additional insight we often consider the following realization of side information (the condition $D{D}^T\succ 0$ ensures $I(X;Y)<\infty$, and hence, Assumption 1 is respected).

$$\begin{array}{cc}\hfill & Y_t=C{X}_t+D{V}_t,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & V_t\in N(0,Q_V),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & C\in {\mathbb{R}}^{n_y\times n_x},D\in {\mathbb{R}}^{n_y\times n_y},D{D}^T\succ 0,Q_V=I_{n_y},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & V^n\mathrm{independent}\mathrm{of}{X}^n,\hfill \end{array}$$

(18)

where $I_{n_y}$ denotes the $n_y\times n_y$ identity matrix. For the above specification of the source and distortion criterion, we derive the following results.

(b.1) Theorems 3 and 4: Structural properties of optimal realization of $\widehat{X}$, which achieves $R_{X|Y}\left({\Delta }_X\right)$, its closed form expression.

(b.2) Theorem 5: Structural properties of optimal realization of $\widehat{X}$ and $\widehat{X}=f(Y,Z)$, which achieve $\overline{R}\left({\Delta }_X\right)$ and the closed form expression of $\overline{R}\left({\Delta }_X\right)$.

(b.3) A proof that $\overline{R}\left({\Delta }_X\right)$ and $R_{X|Y}\left({\Delta }_X\right)$ coincide: Calculation of the distortion region such that Gray’s lower bound (8) holds with equality.

In Remark 4, we consider the tuple of scalar-valued, jointly Gaussian RVs $(X,Y)$ with square error distortion function and verify that our optimal realizations of $\widehat{X}$ and the closed form expressions for $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ are identical to Wyner’s [2] realizations and RDFs.

We should emphasize that our methodology is different from past studies in the sense that we focus on the structural properties of the realizations of the test channels, that achieve the characterizations of the two RDFs (i.e., verification of the converse coding theorem). Our derivations are generic and bring new insight into the construction of realizations that induce the optimal test channels of other distributed source coding problems (i.e., establishing the achievability of the converse coding theorem).

1.4. Additional Generalizations of the Wyner-Ziv [1] and Wyner [2] RDFs

Below, we discuss additional generalizations of Wyner and Ziv [1] and Wyner’s [2] RDFs.

(A) Draper and Wornell [8] Distributed Remote Source Coding Problem: Draper and Wornell [8] generalized the RDF $\overline{R}\left({\Delta }_X\right)$, when the source to be estimated at the decoder is $S:\Omega \to \mathcal{S}$, and it is not directly observed at the encoder. Rather, the encoder observes a RV $X:\Omega \to \mathcal{X}$ (which is correlated with S), while the decoder observes another RV, as side information, $Y:\Omega \to \mathcal{Y}$, which provides information on $(S,X)$. The aim is to reconstruct S at the decoder by $\widehat{S}:\Omega \to \widehat{\mathcal{S}}$, subject to an average distortion $\mathbf{E}\left\{d_S(S,\widehat{S})\right\}\le {\Delta }_S$, by a function $\widehat{S}=f(Y,Z)$. The RDF for this problem, called the distributed remote source coding problem, is defined by [8]

$$\begin{array}{cc}\hfill {\overline{R}}^{PO}\left({\Delta }_S\right)=& \underset{{\mathcal{M}}^{PO}\left({\Delta }_S\right)}{inf}I(X;Z|Y)\in [0,\infty ],\hfill \end{array}$$

(19)

where ${\mathcal{M}}^{PO}\left({\Delta }_S\right)$ is specified by the set of auxiliary RVs Z, and defined as:

$$\begin{array}{cc}\hfill {\mathcal{M}}^{PO}\left({\Delta }_S\right)\stackrel{▵}{=}& \{Z:\Omega \to \mathcal{Z}:{\mathbf{P}}_{S,X,Y,Z,\widehat{X}}\mathrm{is}\mathrm{the}\mathrm{joint}\mathrm{measure}\mathrm{on}\mathcal{S}\times \mathcal{X}\times \mathcal{Y}\times \mathcal{Z}\times \widehat{\mathcal{X}},\hfill \\ \hfill & {\mathbf{P}}_{Z|S,X,Y}={\mathbf{P}}_{Z|X},\exists \mathrm{measurable}\mathrm{function}{f}^{PO}:\mathcal{Y}\times \mathcal{Z}\to \widehat{\mathcal{S}},\hfill \\ \hfill & \widehat{S}=f^{PO}(Y,Z),\mathbf{E}\left\{d_S(S,\widehat{S})\right\}\le {\Delta }_S\}.\hfill \end{array}$$

(20)

Clearly, if $S=X-\mathrm{a}.\mathrm{s}$ (almost surely), then ${\overline{R}}^{PO}\left({\Delta }_S\right)$ degenerates (this implies the optimal test channel that achieves the characterization of the RDF ${\overline{R}}^{PO}\left({\Delta }_S\right)$ should degenerate to the optimal test channel that achieves the characterization of the RDF $\overline{R}\left({\Delta }_X\right)$) to $\overline{R}\left({\Delta }_X\right)$. For scalar-valued jointly Gaussian RVs $(S,X,Y,Z,\widehat{X})$ with square-error distortion, Draper and Wornell [8] [Equation (3) and Appendix A.1] derived the characterization of the RDF ${\overline{R}}^{PO}\left({\Delta }_S\right)$ and constructed the optimal realization $\widehat{S}=f^{PO}(Y,Z)$, which achieves this characterization.

In [5,6], the authors investigated the RDF ${\overline{R}}^{PO}\left({\Delta }_S\right)$ of [8] for the multivariate jointly Gaussian RVs $(S,X,Y,Z,\widehat{X})$, with square-error distortion, and derived a characterization for the RDF ${\overline{R}}^{PO}\left({\Delta }_S\right)$ in [[5], Theorem 4] and [[6], Theorem 3A] (see [[6], Equation (26)]). However, it will become apparent in Remark 5 that, when $S=X-$ almost surely (a.s.), and hence ${\overline{R}}^{PO}\left({\Delta }_S\right)=\overline{R}\left({\Delta }_X\right)$, the RDFs given in [[5], Theorem 4] and [[6], Theorem 3A], do not produce Wyner’s [2] value. We also show in Remark 5 that the same technical issues occur for the correction suggested in https://tiangroup.engr.tamu.edu/publications/ (accessed on 3 January 2024). Similarly, when $S=X-\mathrm{a}.\mathrm{s}.$ and $Y=X-\mathrm{a}.\mathrm{s}.$ [[5], Theorem 4] and [[6], Theorem 3A], do not produce the classical RDF $R_X\left({\Delta }_X\right)$ of the Gaussian source X.

(B) Additional Literature Review: The formulation of Figure 1 is generalized to other multiterminal or distributed lossy compression problems, such as relay networks, sensor networks, etc., under various code formulations and assumptions. Oohama [9] analyzed lossy compression problems for a tuple of scalar correlated Gaussian memoryless sources with square error distortion criterion. Also, he determined the rate-distortion region, in the special case when one source provides partial side information to the other source. Furthermore, Oohama in [10] analyzed separate lossy compression problems for $L+1$ scalar correlated Gaussian memoryless sources, when L of the sources provide partial side information at the decoder for the reconstruction of the remaining source and gave a partial answer to the rate distortion region. Additionally, ref. [10] proved that the problem of [10] includes, as a special case, the additive white Gaussian CEO problem analyzed by Viswanathan and Berger [11]. Extensions of [10] are derived by Ekrem and Ulukus [12] and Wang and Chen [13], where an outer bound on the rate region is derived for the vector Gaussian multiterminal source. Additional works are [14,15,16] and the references therein.

The vast literature on multiterminal or distributed lossy compression of jointly Gaussian sources with square-error distortion (including the references mentioned above), is often confined to scalar-valued correlated RVs. Moreover, as easily verified, not much emphasis is given in the literature on the structural properties of the realizations of RVs that induce the optimal test channels that achieve the characterizations of the RDFs.

The rest of the paper is organized as follows. In Section 2, we review Wyner’s [2] operational definition of lossy compression. We also state a fundamental theorem on mean-square estimation that we use throughout the paper regarding the analysis of (b). The main Theorems are presented in Section 3; some of the proofs, including the structural properties, are given in Section 4. Connections between our results and the past literature are provided in Section 5. A simulation to show the gap between the two rates is given in the same section.

2. Preliminaries

In this section, we review the Wyner [2] source coding problems with fidelity in Figure 1. We begin with the notation, which follows closely [2].

2.1. Notation

Let $\mathbb{Z}\stackrel{▵}{=}\{\dots ,-1,0,1,\dots \}$ the set of all integers, $\mathbb{N}\stackrel{▵}{=}\{0,1,2,\dots ,\}$ the set of natural integers, ${\mathbb{Z}}_{+}\stackrel{▵}{=}\{1,2,\dots ,\}$. For $n\in {\mathbb{Z}}_{+}$, denote the following finite subset of the above defined set, ${\mathbb{Z}}_n\stackrel{▵}{=}\{1,2,\dots ,n\}$. Denote the real numbers by $\mathbb{R}$ and the set of positive and of strictly positive real numbers, by ${\mathbb{R}}_{+}=[0,\infty )$ and ${\mathbb{R}}_{++}=(0,\infty )$, respectively.

For any matrix $A\in {\mathbb{R}}^{p\times m},(p,m)\in {\mathbb{Z}}_{+}\times {\mathbb{Z}}_{+}$, we denote its kernel by $ker\left(A\right)$ its transpose by $A^T$, and for $m=p$, we denote its trace by $\mathrm{trace}\left(A\right)$, and by $\mathrm{diag}\left\{A\right\}$, the matrix with diagonal entries $A_{ii},i\in {\mathbb{Z}}_p$, and zero elsewhere. The determinant of a square matrix A is denoted by $det\left(A\right)$. The identity matrix with dimensions $p\times p$ is designated as $I_p$. Denote an arbitrary set or space by $\mathcal{U}$ and the product space formed by n copies of it by ${\mathcal{U}}^n\stackrel{▵}{=}{\times }_{t=1}^n\mathcal{U}$. $u^n\in {\mathcal{U}}^n$ denotes the set of $n-$tuples $u^n\stackrel{▵}{=}(u_1,u_2,\dots ,u_n)$, where $u_k\in \mathcal{U},k\in {\mathbb{Z}}_k$ are its coordinates. Denote a probability space by $(\Omega ,\mathcal{F},\mathbb{P})$. For a sub-sigma-field $\mathcal{G}\subseteq \mathcal{F}$, and $A\in \mathcal{F}$, denote by $\mathbb{P}\left(A\right|\mathcal{G})$ the conditional probability of A given $\mathcal{G}$; i.e., $\mathbb{P}\left(A\right|\mathcal{G})=\mathbb{P}(A\left|\mathcal{G}\right)\left(\omega \right),\omega \in \Omega$ is a measurable function on $\Omega$.

On the above probability space, consider two-real valued random variables (RV) $X:\Omega \to \mathcal{X},Y:\Omega \to \mathcal{X}$, where $(\mathcal{X},\mathcal{B}(\mathcal{X}\left)\right),(\mathcal{Y},\mathcal{B}(\mathcal{Y}\left)\right)$ are arbitrary measurable spaces. The measure (or joint distribution if $\mathcal{X},\mathcal{Y}$ are Euclidean spaces) induced by $(X,Y)$ on $\mathcal{X}\times \mathcal{Y}$ is denoted by ${\mathbf{P}}_{X,Y}$ or $\mathbf{P}(dx,dy)$ and their marginals on $\mathcal{X}$ and $\mathcal{Y}$ by ${\mathbf{P}}_X$ and ${\mathbf{P}}_Y$, respectively. The conditional measure of RV X conditioned on Y is denoted by ${\mathbf{P}}_{X|Y}$ or $\mathbf{P}\left(dx\right|y)$, when $Y=y$ is fixed. On the above probability space, consider three-real values RVs $X:\Omega \to \mathcal{X},Y:\Omega \to \mathcal{X}$, $Z:\Omega \to \mathcal{Z}$. We say that RVs $(Y,Z)$ are conditional independent given RV X if ${\mathbf{P}}_{Y,Z|X}={\mathbf{P}}_{Y|X}{\mathbf{P}}_{Z|X}-\mathrm{a}.\mathrm{s}.$ (almost surely) or equivalently ${\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X}-\mathrm{a}.\mathrm{s}$; the specification a.s is often omitted. We often denote the above conditional independence by the Markov chain (MC) $Y\leftrightarrow X\leftrightarrow Z$.

Finally, for RVs $X,Y$, etc., $H\left(X\right)$ denotes differential entropy of X, $H\left(X\right|Y)$ conditional differential entropy of X given Y, and $I(X;Y)$ the mutual information between X and Y, as defined in standard books on information theory [17,18]. We use $log(·)$ to denote the natural logarithm. The notation $X\in N(0,Q_X)$ means X is a Gaussian distributed RV with zero mean and covariance $Q_X⪰0$, where $Q_X⪰0$ (resp. $Q_X\succ 0$) means $Q_X$ is positive semidefinite (respectively, positive definite). We denote the covariance of X and Y by

$$\begin{array}{c}\hfill Q_{X,Y}\stackrel{▵}{=}\mathrm{cov}\left(X,Y\right).\end{array}$$

(21)

We denote the covariance of X conditioned on Y by

$$\begin{array}{cc}\hfill Q_{X|Y}& \stackrel{▵}{=}\mathrm{cov}(X,X|Y)\hfill \\ \hfill & =\mathbf{E}\left\{\left(X-\mathbf{E}\left(X|Y\right)\right){\left(X-\mathbf{E}\left(X|Y\right)\right)}^T\right\}\mathrm{if}(X,Y)\mathrm{is}\mathrm{jointly}\mathrm{Gaussian},\hfill \end{array}$$

(22)

where the second equality is due to a property of jointly Gaussian RVs.

2.2. Mean-Square Estimation of Conditionally Gaussian RVs

Below, we state a well-known property of conditionally Gaussian RVs from [19], which we use in our derivations.

Proposition 1. 

Conditionally Gaussian RVs [19]. Consider a pair of multivariate RVs $X={(X_1,\dots ,X_{n_x})}^T:\Omega \to {\mathbb{R}}^{n_x}$ and $Y={(Y_1,\dots ,Y_{n_y})}^T:\Omega \to {\mathbb{R}}^{n_y}$, $(n_x,n_y)\in {\mathbb{Z}}_{+}\times {\mathbb{Z}}_{+}$, defined on some probability distribution $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$. Let $\mathcal{G}\subseteq \mathcal{F}$ be a sub$-\sigma -$algebra. Assume the conditional distribution of $(X,Y)$ conditioned on $\mathcal{G}$, i.e., $\mathbf{P}(dx,dy|\mathcal{G})$ is $\mathbb{P}-\mathrm{a}.\mathrm{s}.$ (almost surely) Gaussian, with conditional means

$$\begin{array}{c}\hfill {\mu }_{X|\mathcal{G}}\stackrel{▵}{=}\mathbf{E}\left\{X|\mathcal{G}\right\},{\mu }_{Y|\mathcal{G}}\stackrel{▵}{=}\mathbf{E}\left\{Y|\mathcal{G}\right\},\end{array}$$

(23)

and conditional covariances

$$\begin{array}{c}\hfill Q_{X|\mathcal{G}}\stackrel{▵}{=}\mathrm{cov}\left(X,X|\mathcal{G}\right),Q_{Y|\mathcal{G}}\stackrel{▵}{=}\mathrm{cov}\left(Y,Y|\mathcal{G}\right),\end{array}$$

(24)

$$\begin{array}{c}\hfill Q_{X,Y|\mathcal{G}}\stackrel{▵}{=}\mathrm{cov}\left(X,Y|\mathcal{G}\right).\end{array}$$

(25)

Then, the vectors of conditional expectations ${\mu }_{X|Y,\mathcal{G}}\stackrel{▵}{=}\mathbf{E}\left\{X|Y,\mathcal{G}\right\}$ and matrices of conditional covariances $Q_{X|Y,\mathcal{G}}\stackrel{▵}{=}\mathrm{cov}\left(X,X|Y,\mathcal{G}\right)$ are given, $\mathbb{P}-\mathrm{a}.\mathrm{s}.$, by the following expressions (If $Q_{Y|\mathcal{G}}\succ 0$ then the inverse exists and the pseudoinverse is $Q_{Y|\mathcal{G}}^{†}=Q_{Y|\mathcal{G}}^{-1}$):

$$\begin{array}{cc}\hfill & {\mu }_{X|Y,\mathcal{G}}={\mu }_{X|\mathcal{G}}+Q_{X,Y|\mathcal{G}}{Q}_{Y|\mathcal{G}}^{†}\left(Y-{\mu }_{Y|\mathcal{G}}\right),\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & Q_{X|Y,\mathcal{G}}\stackrel{▵}{=}{Q}_{X|\mathcal{G}}-Q_{X,Y|\mathcal{G}}{Q}_{Y|\mathcal{G}}^{†}{Q}_{X,Y|\mathcal{G}}^T.\hfill \end{array}$$

(27)

If $\mathcal{G}$ is the trivial information, i.e., $\mathcal{G}=\{\Omega ,\varnothing \}$, then $\mathcal{G}$ is removed from the above expressions.

Note that, if $\mathcal{G}=\{\Omega ,\varnothing \}$, then (26) and (27) reduce to the well-known conditional mean and conditional covariance of X conditioned on Y.

For Gaussian RVs, we make use of the following properties.

Proposition 2. 

Let $X:\Omega \to {\mathbb{R}}^n,n\in {\mathbb{Z}}_{+}$, $X\in N(0,Q_X),Q_X⪰0$, $S\in {\mathbb{R}}^{n_1\times n},n_1\in {\mathbb{Z}}_{+}$, and denote by ${\mathcal{F}}^X$ and ${\mathcal{F}}^{SX}$ the $\sigma -$algebra generated by the RVs X and $SX$, respectively. The following hold.

(a) ${\mathcal{F}}^{SX}\subseteq {\mathcal{F}}^X$.

(b) ${\mathcal{F}}^{SX}={\mathcal{F}}^X$ if and only if $ker\left(Q_X\right)=ker\left(S{Q}_X\right)$.

Proof. 

This is well-known in measure theory, see [20]. □

Proposition 3. 

Let $X:\Omega \to {\mathbb{R}}^n,n\in {\mathbb{Z}}_{+}$, $X\in N(0,Q_X),Q_X⪰0$, $\mathrm{rank}\left(Q_X\right)=n_1,n_1\in {\mathbb{Z}}_{+},n_1<n$. Then, there exists a linear transformation $S\in {\mathbb{R}}^{n_1\times n}$ such that, if $X_1:\Omega \to {\mathbb{R}}^{n_1}$, $X_1=SX$, then $X_1\in N(0,Q_{X_1}),Q_{X_1}\succ 0$, ${\mathcal{F}}^X={\mathcal{F}}^{X_1}$.

Proof. 

This is well-known in probability theory, see [20]. □

2.3. Wyner’s Coding Theorems with Side Information at the Decoder

For the sake of completeness, we introduce certain results from Wyner’s work in [2], which we use in this paper. On a probability space $(\Omega ,\mathcal{F},\mathbb{P})$, consider a tuple of jointly i.i.d. RVs $(X^n,Y^n)=\{(Y_t,Y_t):t\in {\mathbb{Z}}_n\}$,

$$\begin{array}{c}\hfill X_t:\Omega \to \mathcal{Y},Y_t:\Omega \to \mathcal{Y},t\in {\mathbb{Z}}_n,\end{array}$$

(28)

with induced distribution ${\mathbf{P}}_{X_t,Y_t}={\mathbf{P}}_{X,Y},\forall t$. Consider also the measurable function $d_X:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty )$, for a measurable space $\widehat{\mathcal{X}}$. Let

$$\begin{array}{c}\hfill {\mathcal{I}}_M\stackrel{▵}{=}\left\{0,1,\dots ,M-1\right\},M\in {\mathbb{Z}}_M,\end{array}$$

(29)

be a finite set.

A code $(n,M,D_X)$, when switch “A” is open (see Figure 1), is defined by two measurable functions, the encoder $F_E$ and the decoder $F_D$, with average distortion, as follows.

$$\begin{array}{cc}\hfill & F_E:{\mathcal{X}}^n⟶{\mathcal{I}}_M,F_D:{\mathcal{I}}_M\times {\mathcal{Y}}^n⟶{\widehat{\mathcal{X}}}^n,\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill & \frac{1}{n}\mathbf{E}\left\{\sum _{t=1}^n{d}_X(X_t,{\widehat{X}}_t)\right\}=D_X,\hfill \end{array}$$

(31)

where ${\widehat{X}}^n$ is again a sequence of RVs, ${\widehat{X}}^n=F_D(F_E\left(X^n\right),Y^n)\in {\widehat{\mathcal{X}}}^n$. A non-negative rate distortion pair $(R,{\Delta }_X)$ is said to be achievable if for every $ϵ>0$, and n sufficiently large, there exists a code $(n,M,D_X)$ such that

$$M\le 2^{n(R+ϵ)},D_X\le {\Delta }_X+ϵ.$$

(32)

Let $\mathcal{R}$ denote the set of all achievable pairs $(R,{\Delta }_X)$, and define, for ${\Delta }_X\ge 0$, the infimum of all achievable rates by

$$R^{\ast }\left({\Delta }_X\right)=\underset{(R,{\Delta }_X)\in \mathcal{R}}{inf}R.$$

(33)

If for some ${\Delta }_X$ there is no $R<\infty$ such that $(R,{\Delta }_X)\in \mathcal{R}$, then set $R^{\ast }\left({\Delta }_X\right)=\infty .$ For arbitrary abstract spaces Wyner [2] characterized the infimum of all achievable rates $R^{\ast }\left({\Delta }_X\right)$ by the single-letter RDF, $\overline{R}\left({\Delta }_X\right)$ given by (5) and (6), in terms of an auxiliary RV $Z:\Omega \to \mathcal{Z}$. Wyner’s realization of the joint measure ${\mathbf{P}}_{X,Y,Z,\widehat{X}}$ induced by the RVs $(X,Y,Z,\widehat{X})$ is illustrated in Figure 2, where Z is the output of the “test channel”, ${\mathbf{P}}_{Z|X}$. Wyner proved the following coding theorems.

Theorem 1. 

Wyner [[2], Theorems, pp. 64–65]. Suppose Assumption 1 holds.

(a) Converse Theorem. For any ${\Delta }_X\ge 0$, $R^{\ast }\left({\Delta }_X\right)\ge \overline{R}\left({\Delta }_X\right)$.

(b) Direct Theorem. If the conditions stated in ([2], pages 64-65, (i), (ii)) hold, then $R^{\ast }\left({\Delta }_X\right)\le \overline{R}\left({\Delta }_X\right)$, $0\le {\Delta }_X<\infty$.

In Figure 1, when switch A is closed and the tuple of jointly independent and identically distributed RVs $(X^n,Y^n)$ is defined as in Section 2.3, Wyner [2] generalized Berger’s [3] characterization of all achievable pairs $(R,{\Delta }_X)$, from finite alphabet spaces to abstract alphabet spaces.

A code $(n,M,D_X)$, when switch “A” is closed, (see Figure 1), is defined as in Section 2.3, with the encoder $F_E$, replaced by

$$\begin{array}{c}\hfill F_E:{\mathcal{X}}^n\times {\mathcal{Y}}^n⟶{\mathcal{I}}_M.\end{array}$$

(34)

Let ${\mathcal{R}}_1$ denote the set of all achievable pairs $(R,{\Delta }_X)$, again as defined in Section 2.3. For ${\Delta }_X\ge 0$, define the infimum of all achievable rates by

$${\overline{R}}_1\left({\Delta }_X\right)=\underset{(R,{\Delta }_X)\in {\mathcal{R}}_1}{inf}R.$$

(35)

Wyner [2] characterized the infimum of all achievable rates ${\overline{R}}_1\left({\Delta }_X\right)$ by the single-letter RDF $R_{X|Y}\left({\Delta }_X\right)$ given by (1) and (3). The coding Theorems are given by Theorem 1 with $R^{\ast }\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ replaced by ${\overline{R}}_1\left({\Delta }_X\right)$ and $R_{X|Y}\left({\Delta }_X\right)$, respectively. That is, ${\overline{R}}_1\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$ (using Wyner’s notation [[2], Appendix A.1]) These coding theorems generalized earlier work of Berger [3] for finite alphabet spaces. Wyner also derived a fundamental lower bound on $R^{\ast }\left({\Delta }_X\right)$ in terms of ${\overline{R}}_1\left({\Delta }_X\right)$, as stated in the next remark.

Remark 1. 

Wyner [[2], Remarks, p. 65]

(A) For $Z\in \mathcal{M}\left({\Delta }_X\right)$, $\widehat{X}=f(Y,Z)$, and thus ${\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X}$. Then, by a property of conditional mutual information and the data processing inequality:

$$\begin{array}{c}\hfill I(X;Z|Y)=I(X;Z,f(Y,Z)|Y)\ge I(X;\widehat{X}|Y)\ge R_{X|Y}\left({\Delta }_X\right),\end{array}$$

(36)

where the last equality is defined since $\widehat{X}\in {\mathcal{M}}_0\left({\Delta }_X\right)$ (see [[2], Remarks, p. 65]. Moreover, minimizing (36) over $Z\in \mathcal{M}\left({\Delta }_X\right)$ gives

$$\begin{array}{c}\hfill R^{\ast }\left({\Delta }_X\right)\ge R_{X|Y}\left({\Delta }_X\right).\end{array}$$

(37)

(B) Inequality (37) holds with equality, i.e., $R^{\ast }\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$ if $\widehat{X}\in {\mathcal{M}}_0\left({\Delta }_X\right)$, which achieves $I(X;\widehat{X}|Y)=R_{X|Y}\left({\Delta }_X\right)$ can be generated as in Figure 2 with $I(X;Z|Y)=I(X;\widehat{X}|Y)$. This occurs if and only if $I(X;Z|\widehat{X},Y)=0$, and follows from the identity and lower bound

$$\begin{array}{cc}\hfill I(X;Z|Y)=& I(X;Z,\widehat{X}|Y)=I(X;Z|Y,\widehat{X})+I(X;\widehat{X}|Y)\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill \ge & I(X;\widehat{X}|Y),\hfill \end{array}$$

(39)

where the inequality holds with equality if and only if $I(X;Z|\widehat{X},Y)=0$.

3. Main Theorems and Discussion

In this section, we state the main results of this paper. These are the achievable lower bounds of Lemma 1 and Theorem 2, which hold for RVs defined on general abstract alphabet spaces, and Theorems 4 and 5, which hold for multivariate Gaussian RVs.

3.1. Side Information at Encoder and Decoder for an Arbitrary Source

We start with the following achievable lower bound on the conditional mutual information $I(X;\widehat{X}|Y)$, which appears in the definition of $R_{X|Y}\left({\Delta }_X\right)$ of (1); this strengthens Gray’s lower bound (8) [[4], Theorem 3.1].

Lemma 1. 

Achievable lower bound on conditional mutual information. Let $(X,Y,\widehat{X})$ be a triple of arbitrary RVs taking values in the abstract spaces $\mathcal{X}\times \mathcal{Y}\times \widehat{\mathcal{X}}$, with distribution ${\mathbf{P}}_{X,Y,\widehat{X}}$ and joint marginal the fixed distribution ${\mathbf{P}}_{X,Y}$ of $(X,Y)$. Then, the following hold.

(a) The inequality holds:

$$\begin{array}{c}\hfill I(X;\widehat{X}|Y)\ge I(X;\widehat{X})-I(X;Y).\end{array}$$

(40)

Moreover, the equality holds

$$\begin{array}{c}\hfill I(X;\widehat{X}|Y)=I(X;\widehat{X})-I(X;Y)\in [0,\infty ),\end{array}$$

(41)

if and only if

$$\begin{array}{c}\hfill {\mathbf{P}}_{X|\widehat{X},Y}={\mathbf{P}}_{X|\widehat{X}}-a.s.orequivalentlyY\leftrightarrow \widehat{X}\leftrightarrow XisaMC.\end{array}$$

(42)

(b) If $Y\leftrightarrow \widehat{X}\leftrightarrow X$ is a Markov chain then the equality holds

$$\begin{array}{c}\hfill R_{X|Y}\left({\Delta }_X\right)=R_X\left({\Delta }_X\right)-I(X;Y),{\Delta }_X\in {\mathcal{D}}_C\left(X\right|Y),\end{array}$$

(43)

i.e., for all ${\Delta }_X$ that belong to strictly positive set ${\mathcal{D}}_C\left(X\right|Y)\subseteq [0,\infty )$.

Proof. 

See Appendix A.1. □

The next theorem which holds for arbitrary RVs is further used to derive the characterization of $R_{X|Y}\left({\Delta }_X\right)$ for multivariate Gaussian RVs.

Theorem 2. 

Achievable lower bound on conditional mutual information and mean-square error estimation

(a) Let $(X,Y,\widehat{X})$ be a triple of arbitrary RVs on the abstract spaces $\mathcal{X}\times \mathcal{Y}\times \widehat{\mathcal{X}}$, with distribution ${\mathbf{P}}_{X,Y,\widehat{X}}$ and joint marginal the fixed distribution ${\mathbf{P}}_{X,Y}$ of $(X,Y)$.

Define the conditional mean of X conditioned on $(\widehat{X},Y)$ by

$$\begin{array}{c}\hfill {\overline{X}}^{cm}\stackrel{▵}{=}\mathbf{E}\left(X|Y,\widehat{X}\right)=e(Y,\widehat{X}),\end{array}$$

(44)

for some measurable function $f:\mathcal{Y}\times \widehat{\mathcal{X}}\to \mathcal{X}$.

(1) The inequality holds:

$$I(X;\widehat{X}|Y)\ge I(X;{\overline{X}}^{cm}|Y).$$

(45)

(2) The equality holds, $I(X;\widehat{X}|Y)=I(X;{\overline{X}}^{cm}|Y)$ if anyone of the conditions (i) or (ii) holds.

$$\begin{array}{cc}\hfill & \left(i\right){\overline{X}}^{cm}=\widehat{X}-a.s\hfill \\ \hfill & \left(ii\right)Forafixedy\in \mathcal{Y}thefunctione(y,·):\widehat{\mathcal{X}}\to \mathcal{X},e(y,\widehat{x})={\overline{x}}^{cm}uniquelydefines\widehat{x}\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill & i.e.,e(y,·)isaninjectivefunctiononthesupportof\widehat{x}.\hfill \end{array}$$

(47)

(b) In part (a) let $(X,Y,\widehat{X})$ be a triple of arbitrary RVs on $\mathcal{X}\times \mathcal{Y}\times \widehat{\mathcal{X}}={\mathbb{R}}^{n_x}\times {\mathbb{R}}^{n_y}\times {\mathbb{R}}^{n_x}$, $(n_x,n_y)\in {\mathbb{Z}}_{+}\times {\mathbb{Z}}_{+}$.

For all measurable functions $(y,\widehat{x})⟼g(y,\widehat{x})\in {\mathbb{R}}^{n_x}$, the mean-square error satisfies

$$\begin{array}{c}\hfill \mathbf{E}\left\{\left|\right|X-g(Y,\widehat{X}){\left|\right|}_{{\mathbb{R}}^{n_x}}^2\right\}\ge \mathbf{E}\left\{\left|\right|X-\mathbf{E}\left(X|Y,\widehat{X}\right){\left|\right|}_{{\mathbb{R}}^{n_x}}^2\right\},\forall g(·).\end{array}$$

(48)

Proof. 

See Appendix A.2. □

3.2. Side Information at Encoder and Decoder for Multivariate Gaussian Source

The characterizations of the RDFs $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ for a multivariate Gaussian source are encapsulated in Theorems 3–5; these are proved in Section 4. These theorems include the structural properties of optimal test channels or realizations of $(\widehat{X},Z)$, which induce joint distributions. Furthermore, they achieve the RDFs; the closed form expressions of the RDFs are based on a water-filling. The realization of the optimal test channel of $R_{X|Y}\left({\Delta }_X\right)$ is shown in Figure 3.

The following theorem gives a parametric realization of optimal test channel that achieves the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$.

Theorem 3. 

Characterization of $R_{X|Y}\left({\Delta }_X\right)$ by test channel realization. Consider the RDF $R_{X|Y}\left({\Delta }_X\right)$ defined by (1), for the multivariate Gaussian source with mean-square error distortion defined by (9)–(18). The following hold.

(a) The optimal realization $\widehat{X}$ that achieves $R_{X|Y}\left({\Delta }_X\right)$ is parametrized by the matrices $(H,Q_W)$ and represented by

$$\begin{array}{cc}\hfill \widehat{X}& =H\left(X-Q_{X,Y}{Q}_Y^{-1}Y\right)+Q_{X,Y}{Q}_Y^{-1}Y+W\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill & =H\left(X-Q_{X,Y}{Q}_Y^{-1}Y\right)+Q_{X,Y}{Q}_Y^{-1}Y+H\Psi ,if{H}^{-1}exists,\hfill \end{array}$$

(50)

where

$$\begin{array}{cc}\hfill & H{Q}_{X|Y}=Q_{X|Y}{H}^T\stackrel{▵}{=}{Q}_{X|Y}-{\Sigma }_{\Delta }⪰0,\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill & Widependentof(X,Y),Q_W\in N(0,Q_W),\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill & Q_W\stackrel{▵}{=}H{Q}_{X|Y}-H{Q}_{X|Y}{H}^T=H{\Sigma }_{\Delta }={\Sigma }_{\Delta }-{\Sigma }_{\Delta }{Q}_{X|Y}^{-1}{\Sigma }_{\Delta }={\Sigma }_{\Delta }H⪰0,\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill & W=H\Psi ,\Psi \in N(0,Q_{\Psi }),Q_{\Psi }\stackrel{▵}{=}{\Sigma }_{\Delta }{H}^{-1}=H^{-1}{\Sigma }_{\Delta },if{H}^{-1}exists,\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill & {\Sigma }_{\Delta }\stackrel{▵}{=}\mathbf{E}\left\{\left(X-\widehat{X}\right){\left(X-\widehat{X}\right)}^T\right\},\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill & Q_{\widehat{X}|Y}=Q_{X|Y}-{\Sigma }_{\Delta }⪰0,\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill & Q_{X|Y}=Q_X-Q_{X,Y}{Q}_Y^{-1}{Q}_{X,Y}^T\succ 0,Q_{X,Y}=Q_X{C}^T,Q_Y=C{Q}_X{C}^T+D{D}^T.\hfill \end{array}$$

(57)

Moreover, the optimal parametric realization of $\widehat{X}$ satisfies the following structural properties.

$$\begin{array}{cc}\hfill & \left(i\right){\mathbf{P}}_{X|\widehat{X},Y}={\mathbf{P}}_{X|\widehat{X}},if{Q}_X\succ {\Sigma }_{\Delta },\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill & \left(ii\right)\mathbf{E}\left\{X|Y\right\}=\mathbf{E}\left\{\widehat{X}|Y\right\},if{Q}_X⪰{\Sigma }_{\Delta },\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & \left(iii\right)\mathrm{cov}(X,\widehat{X}|Y)=\mathrm{cov}(\widehat{X},\widehat{X}|Y),if{Q}_{X|Y}\succ {\Sigma }_{\Delta },\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill & \left(iv\right)\mathbf{E}\left\{X|\widehat{X},Y\right\}=\mathbf{E}\left\{X|\widehat{X}\right\}=\widehat{X},if{Q}_{X|Y}\succ {\Sigma }_{\Delta }.\hfill \end{array}$$

(61)

(b) The RDF $R_{X|Y}\left({\Delta }_X\right)$ is given by

$$R_{X|Y}\left({\Delta }_X\right)=\underset{{\Sigma }_{\Delta }⪰0,Q_{X|Y}-{\Sigma }_{\Delta }⪰0,\mathrm{trace}\left({\Sigma }_{\Delta }\right)\le {\Delta }_X}{inf}\frac{1}{2}logmax\left\{1,det\left(Q_{X|Y}{\Sigma }_{\Delta }^{-1}\right)\right\}.$$

(62)

Proof. 

The proof is given in Section 4. □

The next theorem gives additional structural properties of the optimal test channel realization of Theorem 3 and uses these properties to characterize RDF $R_{X|Y}\left({\Delta }_X\right)$ via a water-filling solution.

Theorem 4. 

Characterization of $R_{X|Y}\left({\Delta }_X\right)$ via water-filling solution. Consider the RDF $R_{X|Y}\left({\Delta }_X\right)$ defined by (1), for the multivariate Gaussian source with mean-square error distortion defined by (9)–(18), and its characterization in Theorem 3. The following hold.

(a) The matrices of the parametric realization of $\widehat{X}$,

$$\begin{array}{cc}\hfill \{{\Sigma }_{\Delta },& Q_{X|Y},H,Q_W\}havespectral\hfill \\ \hfill & decompositionswithrespecttothesameunitarymatrixU{U}^T=I_{n_x},U^{T}U=I_{n_x},\hfill \end{array}$$

(63)

where the realization coefficients are

$$\begin{array}{cc}\hfill & Q_W=H{\Sigma }_{\Delta }=U\mathrm{diag}({\sigma }_{W_1}^2,\cdots ,{\sigma }_{W_{n_x}}^2)U^T,{\Sigma }_{\Delta }=U\mathrm{diag}({\delta }_1,\cdots ,{\delta }_{n_x})U^T,\hfill \end{array}$$

(64)

$$\begin{array}{cc}\hfill & H=I_{n_x}-Q_{X|Y}^{-1}{\Sigma }_{\Delta }=U\mathrm{diag}(h_1,\cdots ,h_{n_x})U^T,Q_{X|Y}=U\mathrm{diag}({\lambda }_1,\cdots ,{\lambda }_{n_x})U^T,\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & {\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_{n_x}>0,{\delta }_1\ge {\delta }_2\ge \dots \ge {\delta }_{n_x}>0,\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill & {\sigma }_{W_1}^2\ge {\sigma }_{W_2}^2\ge \dots \ge {\sigma }_{W_{n_x}}\ge 0,h_1\ge h_2\ge \cdots \ge h_{n_x}\ge 0,{\sigma }_{W_i}^2=h_i{\delta }_i,h_i\stackrel{▵}{=}1-\frac{{\delta }_i}{{\lambda }_i},\hfill \end{array}$$

(67)

and the eigenvalues ${\sigma }_{W_i}^2$ and $h_i$ are given by

$$\begin{array}{cc}\hfill & {\sigma }_{W_i}^2=\frac{min({\lambda }_i,{\delta }_i)({\lambda }_i-min({\lambda }_i,{\delta }_i))}{{\lambda }_i},h_i=\frac{{\lambda }_i-min({\lambda }_i,{\delta }_i)}{{\lambda }_i},\sum _{i=1}^{n_x}min({\lambda }_i,{\delta }_i)={\Delta }_X.\hfill \end{array}$$

(68)

Moreover, if ${\sigma }_{W_i}^2=0$, then $h_i=0$, and vice versa.

(b) The RDF $R_{X|Y}\left({\Delta }_X\right)$ is given by the water-filling solution:

$$R_{X|Y}\left({\Delta }_X\right)=\frac{1}{2}logmax\left\{1,det\left(Q_{X|Y}{\Sigma }_{\Delta }^{-1}\right)\right\}=\frac{1}{2}\sum _{i=1}^{n_x}log\frac{{\lambda }_i}{{\delta }_i},$$

(69)

where

$$\mathbf{E}\left\{\left|\right|X-\widehat{X}{\left|\right|}_{{\mathbb{R}}^{n_x}}^2\right\}=\mathrm{trace}\left({\Sigma }_{\Delta }\right)=\sum _{i=1}^{n_x}{\delta }_i={\Delta }_X,{\delta }_i=\left\{\begin{array}{ccc}\mu ,\hfill & if\hfill & \mu <{\lambda }_i\hfill \\ {\lambda }_i,\hfill & if\hfill & \mu \ge {\lambda }_i\hfill \end{array}\right.$$

(70)

and $\mu \in (0,\infty )$ is a Lagrange multiplier (obtained from the Kuch–Tucker conditions).

(c) Figure 3 depicts the parallel channel scheme that realizes the optimal $\widehat{X}$ of parts (a), (b), which achieves $R_{X|Y}\left({\Delta }_X\right)$.

(d) If X and Y are independent or Y is replaced by a RV that generates the trivial information, i.e., the $\sigma -$algebra of Y is $\sigma \left\{Y\right\}=\{\Omega ,\varnothing \}$ (or $C=0$ in (15)), then (a)–(c) hold with $Q_{X|Y}=Q_X,Q_{X,Y}=0$, and $R_{X|Y}\left({\Delta }_X\right)=R_X\left({\Delta }_X\right)$, i.e., reduces to the marginal RDF of X.

Proof. 

The proof is given in Section 4. □

The proof of Theorem 4 (see Section 4) is based on the identification of structural properties of the test channel distribution. Some of the implications are briefly described below.

Conclusion 1: The construction and the structural properties of the optimal test channel ${\mathbf{P}}_{X|\widehat{X},Y}$ that achieves the water-filling characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ of Theorems 3 and 4 are not documented elsewhere in the literature.

(i) Structural properties (58) and (61) strengthen Gray’s inequality [[4], Theorem 3.1], (see proof of (8)) to the equality. That is, structural property (58) implies that Gray’s [[4], Theorem 3.1] lower bound (8) holds with equality for a strictly positive surface (See Gray [4] for definition) ${\Delta }_X\in {\mathcal{D}}_C\left(X\right|Y)\subseteq [0,\infty )$, i.e.,

$$\begin{array}{c}\hfill R_{X|Y}\left({\Delta }_X\right)=R_X\left({\Delta }_X\right)-I(X;Y),{\Delta }_X\in {\mathcal{D}}_C\left(X\right|Y)\stackrel{▵}{=}\left\{{\Delta }_X\in [0,\infty ):{\Delta }_X\le n_x{\lambda }_{n_x}\right\}.\end{array}$$

(71)

The set ${\mathcal{D}}_C\left(X\right|Y)$ excludes values of ${\Delta }_X\in [0,\infty )$ for which water-filling is active in (69) and (70).

By the realization of the optimal reproduction $\widehat{X}$, it follows that the subtraction of equal quantities $\mathbf{E}\left\{X|Y\right\}$ at the encoder and decoder does not affect the information measure, noting that $\mathbf{E}\left\{X|Y\right\}=\mathbf{E}\left\{\widehat{X}|Y\right\}$.

Theorem 4 points (a) and (b) are obtained with the aid of Theorem 3 and Hadamard’s inequality, which shows $Q_{X|Y}$ and ${\Sigma }_{\Delta }$ have the same eigenvectors.

(ii) Structural properties of realizations of Theorems 3 and 4: The matrices $\{{\Sigma }_{\Delta },Q_{X|Y},H,Q_W\}$ are nonnegative symmetric and have a spectral decomposition with respect to the same unitary matrix $U{U}^T=I_{n_x}$ [21]. This implies that the test channel is equivalently represented by parallel additive Gaussian noise channels (subject to pre-processing and post-processing at the encoder and decoder).

(iii) In Remark 4, we show that the realization of optimal $\widehat{X}$ in Figure 3, which achieves the RDF of Theorem 4, degenerates to Wyner’s [2] optimal realization, which attains the RDF $R_{X|Y}\left({\Delta }_X\right)$, for the tuple of scalar-valued, jointly Gaussian RVs $(X,Y)$ with square error distortion function.

3.3. Side Information Only at Decoder for Multivariate Gaussian Source

Theorem 5 gives the optimal test channel that achieves the characterization of the RDF $\overline{R}\left({\Delta }_X\right)$ and further states that there is no loss of compression rate if side information is only available at the decoder. That is, although in general, $\overline{R}\left({\Delta }_X\right)\ge R_{X|Y}\left({\Delta }_X\right)$, an optimal reproduction $\widehat{X}=f(Y,Z)$ of X, where $f(·,·)$ is linear, is constructed such that the inequality holds with equality.

Theorem 5. 

Characterization and water-filling solution of $\overline{R}\left({\Delta }_X\right)$. Consider the RDF $\overline{R}\left({\Delta }_X\right)$ defined by (5) for the multivariate Gaussian source with mean-square error distortion, defined by (9)–(18). Then, the following hold.

(a) The characterization of the RDF, $\overline{R}\left({\Delta }_X\right)$ satisfies

$$\begin{array}{c}\hfill \overline{R}\left({\Delta }_X\right)\ge R_{X|Y}\left({\Delta }_X\right),\end{array}$$

(72)

where $R_{X|Y}\left({\Delta }_X\right)$ is given in Theorem 4b.

(b) The optimal realization $\widehat{X}=f(Y,Z)$, which achieves the lower bound in (72), i.e., $\overline{R}\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)$, is represented by

$$\begin{array}{cc}\hfill \widehat{X}=& f(Y,Z)\hfill \end{array}$$

(73)

$$\begin{array}{cc}\hfill =& \left(I-H\right)Q_{X,Y}{Q}_Y^{-1}Y+Z,\hfill \end{array}$$

(74)

$$\begin{array}{cc}\hfill Z=& HX+W,\hfill \end{array}$$

(75)

$$\begin{array}{cc}\hfill (H,& Q_W)givenby\left(51\right)–\left(57\right),and\left(63\right)holds.\hfill \end{array}$$

(76)

Moreover, the following structural properties hold:

(1) The optimal test channel satisfies

$$\begin{array}{cc}\hfill & \left(i\right){\mathbf{P}}_{X|\widehat{X},Y,Z}={\mathbf{P}}_{X|\widehat{X},Y}\stackrel{\left(\alpha \right)}{=}{\mathbf{P}}_{X|\widehat{X}},where\left(\alpha \right)holdsif{Q}_X\succ {\Sigma }_{\Delta },\hfill \end{array}$$

(77)

$$\begin{array}{cc}\hfill & \left(ii\right)\mathbf{E}\left\{X|\widehat{X},Y,Z\right\}=\mathbf{E}\left\{X|\widehat{X},Y\right\}\stackrel{\left(\beta \right)}{=}\mathbf{E}\left\{X|\widehat{X}\right\}\stackrel{\left(\gamma \right)}{=}\widehat{X},where\left(\beta \right),\left(\gamma \right)holdif{Q}_{X|Y}\succ {\Sigma }_{\Delta },\hfill \end{array}$$

(78)

$$\begin{array}{cc}\hfill & \left(iii\right){\mathbf{P}}_{Z|X,Y}={\mathbf{P}}_{Z|X}.\hfill \end{array}$$

(79)

(2) Structural property (2) of Theorem 4a holds.

Proof. 

It is given in Section 4. □

The proof of Theorem 5 is based on the derivation of the structural properties and Theorem 4. Some implications are discussed below.

Conclusion 2: The optimal reproduction $\widehat{X}=f(X,Z)$ or test channel distribution ${\mathbf{P}}_{X|\widehat{X},Y,Z}$, which achieves $\overline{R}\left({\Delta }_X\right)$ of Theorem 5, are not reported in the literature.

(i) From the structural property (1) of Theorem 5, i.e., (77), it follows that the lower bound $\overline{R}\left({\Delta }_X\right)\ge R_{X|Y}\left({\Delta }_X\right)$ is achieved by the realization $\widehat{X}=f(Y,Z)$ of Theorem 5b; i.e., for a given $Y=y$, then $\widehat{X}$ uniquely defines Z.

(ii) If X is independent of Y or Y generates trivial information, then the RDFs $\overline{R}\left({\Delta }_X\right)={\overline{R}}_{X|Y}\left({\Delta }_X\right)$ degenerate to the classical RDF of the source X, i.e., $R_X\left({\Delta }_X\right)$, as expected. This is easily verified from (73) and (76), i.e., $Q_{X,Y}=0$, which implies $\widehat{X}=Z$.

For scalar-valued RVs, $X:\Omega \to \mathbb{R},Y:\Omega \to \mathbb{R},X\in N(0,{\sigma }_X^2)$, and X independent of Y, then the optimal realization reduces to

$$\begin{array}{cc}\hfill & \widehat{X}=Z=\left(1-\frac{{\Delta }_X}{{\sigma }_X^2}\right)X+\sqrt{\left(1-\frac{{\Delta }_X}{{\sigma }_X^2}\right){\Delta }_X}\overline{W},\overline{W}\in N(0,1),{\sigma }_X^2\le {\Delta }_X,\hfill \end{array}$$

(80)

$$\begin{array}{cc}\hfill & Q_{\widehat{X}}=Q_Z={\sigma }_{\widehat{X}}^2={\sigma }_X^2-{\Delta }_X\ge 0,\hfill \end{array}$$

(81)

as expected.

(iii) In Remark 4, we show that the realization of optimal $\widehat{X}=f(Y,Z)$, which achieves the RDF $\overline{R}\left({\Delta }_X\right)$ of Theorem 5, degenerates to Wyner’s [2] realization that attains the RDF $\overline{R}\left({\Delta }_X\right)$, of the tuple of scalar-valued, jointly Gaussian RVs $(X,Y)$, with the square error distortion function.

4. Proofs of Theorems 3–5

In this section, we derive the statements of Theorems 3–5 by making use of Theorem 2 (which holds for general abstract alphabet spaces) by restricting attention to multivariate jointly Gaussian $(X,Y)$.

4.1. Side Information at Encoder and Decoder

For jointly Gaussian RVs $(X,Y,\widehat{X})$, in the next theorem we identify simple sufficient conditions for the lower bound of Theorem 2 to be achievable.

Theorem 6. 

Sufficient conditions for the lower bounds of Theorem 2 to be achievable. Consider the statement of Theorem 2 for a triple of jointly Gaussian RVs $(X,Y,\widehat{X})$ on ${\mathbb{R}}^{n_x}\times {\mathbb{R}}^{n_y}\times {\mathbb{R}}^{n_x}$, $(n_x,n_y)\in {\mathbb{Z}}_{+}\times {\mathbb{Z}}_{+}$, i.e., ${\mathbf{P}}_{X,Y,\widehat{X}}={\mathbf{P}}_{X,Y,\widehat{X}}^G$ and joint marginal the fixed Gaussian distribution ${\mathbf{P}}_{X,Y}={\mathbf{P}}_{X,Y}^G$ of $(X,Y)$

Then,

$$\begin{array}{cc}\hfill & {\overline{X}}^{cm}\stackrel{▵}{=}\mathbf{E}\left(X|Y,\widehat{X}\right)=e^G(Y,\widehat{X}),\hfill \end{array}$$

(82)

$$\begin{array}{cc}\hfill & e^G(Y,\widehat{X})=\mathbf{E}\left(X|Y\right)+\mathrm{cov}(X,\widehat{X}|Y){\left\{\mathrm{cov}(\widehat{X},\widehat{X}|Y)\right\}}^{†}\left(\widehat{X}-\mathbf{E}\left(\widehat{X}|Y\right)\right).\hfill \end{array}$$

(83)

Moreover, the following hold.

Case (i). $\mathrm{cov}(\widehat{X},\widehat{X}|Y)\succ 0$, that is, $\mathrm{rank}\left(Q_{\widehat{X}|Y}\right)=n_x$. Condition (84) is sufficient for $I(X;\widehat{X}|Y)=I(X;{\overline{X}}^{cm}|Y)$.

$$\begin{array}{c}\hfill {\overline{X}}^{cm}\stackrel{▵}{=}\mathbf{E}\left(X|Y,\widehat{X}\right)=e^G(Y,\widehat{X})=\widehat{X}-a.s.\end{array}$$

(84)

In addition, Conditions 1 and 2 below are sufficient for (84) to hold.

$$\begin{array}{cc}\hfill & Condition1.\mathbf{E}\left(X|Y\right)=\mathbf{E}\left(\widehat{X}|Y\right)\hfill \end{array}$$

(85)

$$\begin{array}{cc}\hfill & Condition2.\mathrm{cov}(X,\widehat{X}|Y)\mathrm{cov}{(\widehat{X},\widehat{X}|Y)}^{-1}=I_{n_x}\hfill \end{array}$$

(86)

Case (ii). $\mathrm{cov}(\widehat{X},\widehat{X}|Y)⪰0$ but not $\mathrm{cov}(\widehat{X},\widehat{X}|Y)\succ 0$; that is, $\mathrm{rank}\left(Q_{\widehat{X}|Y}\right)=n_1<n_x$. Condition (87) is sufficient for $I(X;\widehat{X}|Y)=I(X;{\overline{X}}^{cm}|Y)$.

$$\begin{array}{c}\hfill e^G(·,·)definedby\left(83\right)satisfies\left(47\right).\end{array}$$

(87)

In addition, a sufficient condition for (87) to hold is, for a fixed $Y=y\in \mathcal{Y}$, the $\sigma -$algebras satisfy ${\mathcal{F}}^{\widehat{X}}={\mathcal{F}}^{e^G(y,\widehat{X})}$.

Proof. 

Note that identity (83) follows from Proposition 1, (26), by letting $Y=\widehat{X}$ and $\mathcal{G}$ be the information generated by Y. Consider Case (i); If (84) holds then $I(X;\widehat{X}|{\overline{X}}^{cm},Y)=0$. By (83), Conditions 1 and 2 are sufficient for (84) to hold. Consider Case (ii). Sufficient condition (87) follows from Theorem 2, and implies $I(X;\widehat{X}|{\overline{X}}^{cm},Y)=0$. The statement below (87) follows from Proposition 2. □

Now, we turn our attention to the optimization problem $R_{X|Y}\left({\Delta }_X\right)$ defined by (1) for the multivariate Gaussian source with mean-square error distortion defined by (9)–(18). In the next lemma, we derive a preliminary parametrization of the optimal reproduction distribution ${\mathbf{P}}_{\widehat{X}|X,Y}$ of the RDF $R_{X|Y}\left({\Delta }_X\right)$.

Lemma 2. 

Preliminary parametrization of optimal reproduction distribution of $R_{X|Y}\left({\Delta }_X\right)$. Consider the RDF $R_{X|Y}\left({\Delta }_X\right)$ defined by (1) for the multivariate Gaussian source, i.e., ${\mathbf{P}}_{X,Y}={\mathbf{P}}_{X,Y}^G$, with mean-square error distortion defined by (9)–(18).

(a) For every joint distribution ${\mathbf{P}}_{X,Y,\widehat{X}}$ there exists a jointly Gaussian distribution denoted by ${\mathbf{P}}_{X,Y,\widehat{X}}^G$, with marginal the fixed distribution ${\mathbf{P}}_{X,Y}^G$, which minimizes $I(X;\widehat{X}|Y)$ and satisfies the average distortion constraint, i.e., with $d_X(x,\widehat{x})=\left|\right|x-\widehat{x}{\left|\right|}_{{\mathbb{R}}^{n_x}}^2$.

(b) The conditional reproduction distribution of the RDF $R_{X|Y}\left({\Delta }_X\right)$ is ${\mathbf{P}}_{\widehat{X}|X,Y}={\mathbf{P}}_{\widehat{X}|X,Y}^G$ and induced by the parametric realization of $\widehat{X}$ (in terms of $H,G,Q_W$),

$$\begin{array}{cc}\hfill & \widehat{X}=HX+GY+W,\hfill \end{array}$$

(88)

$$\begin{array}{cc}\hfill & H\in {\mathbb{R}}^{n_x\times n_x},G\in {\mathbb{R}}^{n_x\times n_y},\hfill \end{array}$$

(89)

$$\begin{array}{cc}\hfill & W\in N(0,Q_W),Q_W⪰0,\hfill \end{array}$$

(90)

$$\begin{array}{cc}\hfill & Windependentof(X,Y),\hfill \end{array}$$

(91)

and $\widehat{X}$ is a Gaussian RV.

(c) $R_{X|Y}\left({\Delta }_X\right)$ is characterized by the optimization problem.

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{{\mathcal{M}}_0^G\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y),{\Delta }_X\in [0,\infty ),\hfill \end{array}$$

(92)

where ${\mathcal{M}}_0^G\left({\Delta }_X\right)$ is specified by the set

$$\begin{array}{cc}\hfill {\mathcal{M}}_0^G\left({\Delta }_X\right)\stackrel{▵}{=}& \left\{\widehat{X}:\Omega \to \widehat{\mathcal{X}}:\left(88\right)-\left(91\right)hold,and\mathbf{E}\left\{\left|\right|X-\widehat{X}{\left|\right|}_{{\mathbb{R}}^{n_x}}^2\right\}\le {\Delta }_X\right\}.\hfill \end{array}$$

(93)

(d) If there exists $(H,G,Q_W)$ such that (84) or (87) hold, then a further lower bound on $R_{X|Y}\left({\Delta }_X\right)$ is achieved in the subset ${\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)\subseteq {\mathcal{M}}_0^G\left({\Delta }_X\right)$ defined by

$$\begin{array}{cc}\hfill {\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)\stackrel{▵}{=}& \{\widehat{X}:\Omega \to \widehat{\mathcal{X}}:\left(88\right)–\left(91\right)hold,\left(84\right)or\left(87\right)hold,\hfill \\ \hfill & \mathbf{E}\left\{\left|\right|X-\widehat{X}{\left|\right|}_{{\mathbb{R}}^{n_x}}^2\right\}\le {\Delta }_X\},\hfill \end{array}$$

(94)

and the corresponding characterization of the RDF is

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{{\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y),{\Delta }_X\in [0,\infty ).\hfill \end{array}$$

(95)

Proof. 

(a) This is omitted since it is similar to the classical unconditional RDF $R_X\left({\Delta }_X\right)$ of a Gaussian message $X\in N(0,Q_X)$. (b) By (a), the conditional distribution ${\mathbf{P}}_{\widehat{X}|X,Y}^G$ is such that, its conditional mean is linear in $(X,Y)$, its conditional covariance is nonrandom, i.e., constant, and for fixed $(X,Y)=(x,y)$, ${\mathbf{P}}_{\widehat{X}|X,Y}^G$ is Gaussian. Such a distribution is induced by the parametric realization (88)–(91). (c) Follows from parts (a) and (b). (d) Follows from Theorem 6 and (48) due to the achievability of the lower bounds. □

In the next theorem, we identify the optimal triple $(H,G,Q_W)$ such that (84) or (87) hold (i.e., establish its existence), characterize the RDF by $R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}{inf}_{{\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)}I(X;\widehat{X}|Y)$, and construct a realization $\widehat{X}$ that achieves it.

Theorem 7. 

Characterization of RDF $R_{X|Y}\left({\Delta }_X\right)$. Consider the RDF $R_{X|Y}\left({\Delta }_X\right)$, defined by (1), for the multivariate Gaussian source with mean-square error distortion, defined by (9)–(18). The characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ is

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{\mathcal{Q}\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y)\hfill \end{array}$$

(96)

$$\begin{array}{cc}\hfill =& \underset{{\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y)\hfill \end{array}$$

(97)

$$\begin{array}{cc}\hfill =& \underset{\mathcal{Q}\left({\Delta }_X\right)}{inf}\frac{1}{2}log\left\{det\left(Q_{X|Y}{\Sigma }_{\Delta }^{-1}\right)\right\},\hfill \end{array}$$

(98)

where

$$\begin{array}{cc}\hfill \mathcal{Q}\left({\Delta }_X\right)\stackrel{▵}{=}& \left\{{\Sigma }_{\Delta }⪰0:Q_{X|Y}-{\Sigma }_{\Delta }⪰0,\mathrm{trace}\left({\Sigma }_{\Delta }\right)\le {\Delta }_X\right\},\hfill \end{array}$$

(99)

$$\begin{array}{cc}\hfill {\Sigma }_{\Delta }\stackrel{▵}{=}& \mathbf{E}\left\{\left(X-\widehat{X}\right){\left(X-\widehat{X}\right)}^T\right\},\hfill \end{array}$$

(100)

$$\begin{array}{cc}\hfill Q_{X|Y}=& Q_X-Q_{X,Y}{Q}_Y^{-1}{Q}_{X,Y}^T,\hfill \end{array}$$

(101)

$$\begin{array}{cc}\hfill Q_{X,Y}=& Q_X{C}^T,Q_Y=C{Q}_X{C}^T+D{D}^T.\hfill \end{array}$$

(102)

The realization of the optimal reproduction $\widehat{X}\in {\mathcal{M}}_0^{G,o}\left({\Delta }_X\right)$, which achieves $R_{X|Y}\left({\Delta }_X\right)$, is given in Theorem 3a, also satisfies the properties stated under Theorem 3a. (i)–(iv).

Proof. 

See Appendix A.3. □

Remark 2. 

Structural properties of the optimal realization of Theorem 4a. For the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ of Theorem 7, which is achieved by $\widehat{X}$ defined in Theorem 3a in terms of the matrices $\left\{{\Sigma }_{\Delta },Q_{X|Y},H,Q_W\right\}$, we show in Corollary 2, the statements of Theorem 4a, i.e.,

$$\begin{array}{cc}\hfill \left(i\right)& H=H^T⪰0,\hfill \end{array}$$

(103)

$$\begin{array}{cc}\hfill \left(ii\right)& \left\{{\Sigma }_{\Delta },{\Sigma }_{X|Y},H,Q_W\right\}havespectralrepres.withrespecttothesameunitarymatrixU{U}^T=I_{n_x}.\hfill \end{array}$$

(104)

To prove the structural property of Remark 2, we use the next corollary, which is a degenerate case of [[22], Lemma 2] (i.e., the structural properties of test channel of Gorbunov and Pinsker [23] nonanticipatory RDF of Markov sources).

Corollary 1. 

Structural properties of realization of optimal $\widehat{X}$ of Theorem 4a. Consider the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ of Theorem 7. Suppose $Q_{X|Y}\succ 0$ and ${\Sigma }_{\Delta }⪰0$ commute, that is,

$$\begin{array}{c}\hfill Q_{X|Y}{\Sigma }_{\Delta }={\Sigma }_{\Delta }{Q}_{X|Y}.\end{array}$$

(105)

Then,

$$\begin{array}{cc}\hfill \left(1\right)& H=I_{n_x}-{\Sigma }_{\Delta }{Q}_{X|Y}^{-1}=H^T,Q_W={\Sigma }_{\Delta }{H}^T={\Sigma }_{\Delta }H=H{\Sigma }_{\Delta }=Q_W^T⪰0\hfill \\ \hfill \left(2\right)& \left\{{\Sigma }_{\Delta },Q_{X|Y},H,Q_W\right\}havespectral\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill & decompositionswithrespecttothesameunitarymatrixU{U}^T=I_{n_x},U^{T}U=I_{n_x}.\hfill \end{array}$$

(107)

that is, the following hold.

$$\begin{array}{cc}\hfill & Q_{X|Y}=U\mathrm{diag}\{{\lambda }_1,\cdots ,{\lambda }_{n_x}\}{U}^T,{\lambda }_1\ge {\lambda }_2\ge \dots \ge {\lambda }_{n_x}>0,\hfill \end{array}$$

(108)

$$\begin{array}{cc}\hfill & {\Sigma }_{\Delta }=U\mathrm{diag}\{{\delta }_1,\cdots ,{\delta }_{n_x}\}{U}^T,{\delta }_1\ge {\delta }_2\ge \dots \ge {\delta }_{n_x}\ge 0,\hfill \end{array}$$

(109)

$$\begin{array}{cc}\hfill & H=U\mathrm{diag}\{1-\frac{{\delta }_1}{{\lambda }_1},\cdots ,1-\frac{{\delta }_{n_x}}{{\lambda }_{n_x}}\}{U}^T,\hfill \end{array}$$

(110)

$$\begin{array}{cc}\hfill & Q_W=U\mathrm{diag}\{(1-\frac{{\delta }_1}{{\lambda }_1}){\delta }_1,\cdots ,(1-\frac{{\delta }_{n_x}}{{\lambda }_{n_x}}){\delta }_{n_x}\}{U}^T,\mathrm{and}(1-\frac{{\delta }_k}{{\lambda }_k}){\delta }_k\ge 0.\hfill \end{array}$$

(111)

Proof. 

See Appendix A.4. □

In the next corollary, we re-express the realization of $\widehat{X}$ of Theorem 4a, which characterizes the RDF of Theorem 7 using a translation of X and $\widehat{X}$ by subtracting their conditional means with respect to Y, making use of property $\mathbf{E}\left\{X|Y\right\}=\mathbf{E}\left\{\widehat{X}|Y\right\}$ of (78). This is the the realization shown in Figure 3.

Corollary 2. 

Equivalent characterization of $R_{X|Y}\left({\Delta }_X\right)$. Consider the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ of Theorem 7 and the realization of $\widehat{X}$ of Theorem 3a and Theorem 4a. Define the translated RVs

$$\mathbf{X}\stackrel{▵}{=}X-\mathbf{E}\left\{X|Y\right\}=X-Q_{X,Y}{Q}_Y^{-1}Y,\widehat{\mathbf{X}}\stackrel{▵}{=}\widehat{X}-\mathbf{E}\left\{X|Y\right\}=\widehat{X}-Q_{X,Y}{Q}_Y^{-1}Y.$$

(112)

Let

$$\begin{array}{cc}\hfill & Q_{X|Y}=U\mathrm{diag}\{{\lambda }_1,\cdots ,{\lambda }_{n_x}\}{U}^T,U{U}^T=I_{n_x},U^{T}U=I_{n_x},{\lambda }_1\ge {\lambda }_2\ge \dots \ge {\lambda }_{n_x},\hfill \end{array}$$

(113)

$$\begin{array}{cc}\hfill & \overline{\mathbf{X}}\stackrel{▵}{=}{U}^T\mathbf{X},\widehat{\overline{\mathbf{X}}}\stackrel{▵}{=}{U}^T\widehat{\mathbf{X}}.\hfill \end{array}$$

(114)

Then,

$$\begin{array}{cc}\hfill & \widehat{\mathbf{X}}=H\mathbf{X}+W,\hfill \end{array}$$

(115)

$$\begin{array}{cc}\hfill & I(X;\widehat{X}|Y)=I(\mathbf{X};\widehat{\mathbf{X}})=I(U^T\mathbf{X};U^T\widehat{\mathbf{X}}),\hfill \end{array}$$

(116)

$$\begin{array}{cc}\hfill & \mathbf{E}\parallel X-\widehat{X}{\parallel }_{{\mathbb{R}}^{n_x}}^2=\mathbf{E}\parallel \mathbf{X}-\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2=\mathbf{E}\parallel U^T\mathbf{X}-U^T\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2=\mathrm{trace}\left({\Sigma }_{\Delta }\right),\hfill \end{array}$$

(117)

where $(H,Q_W)$ are given in Theorem 3a.

Further, the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ (98) satisfies the following equalities and inequality:

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)\stackrel{▵}{=}& \underset{\mathcal{Q}\left({\Delta }_X\right)}{inf}I(X;\widehat{X}|Y)=\underset{\mathcal{Q}\left({\Delta }_X\right)}{inf}\frac{1}{2}logmax\left\{1,det\left(Q_{X|Y}{\Sigma }_{\Delta }^{-1}\right)\right\}\hfill \end{array}$$

(118)

$$\begin{array}{cc}\hfill =& \underset{\mathbf{E}\parallel \mathbf{X}-\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2\le {\Delta }_X}{inf}I(\mathbf{X};\widehat{\mathbf{X}})\hfill \end{array}$$

(119)

$$\begin{array}{cc}\hfill =& \underset{\mathbf{E}\parallel U^T\mathbf{X}-U^T\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2\le {\Delta }_X}{inf}I(U^T\mathbf{X};U^T\widehat{\mathbf{X}})\hfill \end{array}$$

(120)

$$\begin{array}{cc}\hfill \ge & \underset{\mathbf{E}\parallel U^T\mathbf{X}-U^T\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2\le {\Delta }_X}{inf}\sum _{t=1}^{n_x}I({\overline{\mathbf{X}}}_t;{\widehat{\overline{\mathbf{X}}}}_t)\hfill \end{array}$$

(121)

Moreover, the inequality (121) is achieved if $Q_{X|Y}⪰0$ and ${\Sigma }_{\Delta }⪰0$ commute; that is, if (105) holds, then

$$R_{X|Y}\left({\Delta }_X\right)=\underset{{\sum }_{i=1}^{n_x}{\delta }_i\le {\Delta }_X}{inf}\frac{1}{2}\sum _{i=1}^{n_x}logmax\left\{1,\frac{{\lambda }_i}{{\delta }_i}\right\}$$

(122)

where

$$\begin{array}{c}\hfill \mathrm{diag}\{\mathbf{E}\left(U^T\mathbf{X}-U^T\widehat{\mathbf{X}}\right){\left(U^T\mathbf{X}-U^T\widehat{\mathbf{X}}\right)}^T\}=\mathrm{diag}\{{\delta }_1,{\delta }_2,\dots ,{\delta }_{n_x}\}.\end{array}$$

(123)

Proof. 

By Theorem 3a,

$$\begin{array}{cc}\hfill \widehat{X}=& HX+\left(I-H\right)Q_{X,Y}{Q}_Y^{-1}Y+W\hfill \end{array}$$

(124)

$$\begin{array}{cc}\hfill =& H\left(X-Q_{X,Y}{Q}_Y^{-1}Y\right)+Q_{X,Y}{Q}_Y^{-1}Y+W\hfill \end{array}$$

(125)

$$\begin{array}{cc}\hfill ⟹& \widehat{X}-Q_{X,Y}{Q}_Y^{-1}Y=H\left(X-Q_{X,Y}{Q}_Y^{-1}Y\right)+W\hfill \end{array}$$

(126)

$$\begin{array}{cc}\hfill ⟹& \widehat{\mathbf{X}}=H\mathbf{X}+W.\hfill \end{array}$$

(127)

The last equation establishes (115). By properties of conditional mutual information and the properties of optimal realization $\widehat{X}$, the following equalities hold.

$$\begin{array}{cc}\hfill I(X;\widehat{X}|Y)=& I(X-Q_{X,Y}{Q}_Y^{-1}Y;\widehat{X}-Q_{X,Y}{Q}_Y^{-1}Y|Y)\hfill \end{array}$$

(128)

$$\begin{array}{cc}\hfill =& I(\mathbf{X};\widehat{\mathbf{X}}|Y),\mathrm{by}\left(112\right)\hfill \end{array}$$

(129)

$$\begin{array}{cc}\hfill =& H\left(\widehat{\mathbf{X}}\right|Y)-H\left(\widehat{\mathbf{X}}\right|Y,\mathbf{X})\hfill \end{array}$$

(130)

$$\begin{array}{cc}\hfill =& H\left(\widehat{\mathbf{X}}\right)-H\left(\widehat{\mathbf{X}}\right|Y,\mathbf{X}),\mathrm{by}\mathrm{indep}.\mathrm{of}(\mathbf{X},W)\mathrm{and}Y\hfill \end{array}$$

(131)

$$\begin{array}{cc}\hfill =& H\left(\widehat{\mathbf{X}}\right)-H\left(\widehat{\mathbf{X}}\right|\mathbf{X}),\mathrm{by}\mathrm{indep}.\mathrm{of}W\mathrm{and}Y\mathrm{for}\mathrm{fixed}X\hfill \end{array}$$

(132)

$$\begin{array}{cc}\hfill =& I(\mathbf{X};\widehat{\mathbf{X}})\hfill \end{array}$$

(133)

$$\begin{array}{cc}\hfill =& I(U^T\mathbf{X};U^T\widehat{\mathbf{X}})\hfill \end{array}$$

(134)

$$\begin{array}{cc}\hfill =& I({\overline{\mathbf{X}}}_1,{\overline{\mathbf{X}}}_2,\dots ,{\overline{\mathbf{X}}}_{n_x};{\widehat{\overline{\mathbf{X}}}}_1,{\widehat{\overline{\mathbf{X}}}}_2,\dots ,{\widehat{\overline{\mathbf{X}}}}_{n_x})\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill \ge & \sum _{t=1}^{n_x}I({\overline{\mathbf{X}}}_t;{\widehat{\overline{\mathbf{X}}}}_t),\mathrm{by}\mathrm{mutual}\mathrm{independence}\mathrm{of}{\overline{\mathbf{X}}}_t,t=1,2,\dots ,n_x.\hfill \end{array}$$

(136)

Moreover, inequality (136) holds with equality if $({\overline{\mathbf{X}}}_t;{\widehat{\overline{\mathbf{X}}}}_t),t=1,2,\dots ,n_x$ are jointly independent. The average distortion function is then given by

$$\begin{array}{cc}\hfill \mathbf{E}\parallel X-\widehat{X}{\parallel }_{{\mathbb{R}}^{n_x}}^2& =\mathbf{E}\parallel X-\widehat{X}-Q_{X,Y}{Q}_Y^{-1}Y+Q_{X,Y}{Q}_Y^{-1}Y{\parallel }_{{\mathbb{R}}^{n_x}}^2\hfill \end{array}$$

(137)

$$\begin{array}{cc}\hfill & =\mathbf{E}\parallel \mathbf{X}-\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2,by\left(112\right)\hfill \end{array}$$

(138)

$$\begin{array}{cc}\hfill & =\mathbf{E}\parallel U^T\mathbf{X}-U^T\widehat{\mathbf{X}}{\parallel }_{{\mathbb{R}}^{n_x}}^2=\mathrm{trace}\left({\Sigma }_{\Delta }\right),byU{U}^T=I_{n_x}.\hfill \end{array}$$

(139)

By Corollary 1, if (105) holds, that is, $Q_{X|Y}\succ 0$ and ${\Sigma }_{\Delta }⪰0$ satisfy $Q_{X|Y}{\Sigma }_{\Delta }={\Sigma }_{\Delta }{Q}_{X|Y}$ (i.e., commute), then (106)–(108) hold, and by (122) we obtain

$$\begin{array}{cc}\hfill \widehat{\overline{\mathbf{X}}}\stackrel{▵}{=}& U^T\widehat{\mathbf{X}}=U^{T}H\mathbf{X}+U^{T}W=U^T\widehat{\mathbf{X}}=U^{T}HU{U}^T\mathbf{X}+U^{T}W\hfill \end{array}$$

(140)

$$\begin{array}{cc}\hfill =& U^{T}HU\overline{\mathbf{X}}+U^{T}W,U^{T}HU\mathrm{is}\mathrm{diagonal}\mathrm{and}{U}^{T}W\mathrm{has}\mathrm{indep}.\mathrm{components}.\hfill \end{array}$$

(141)

Hence, if (105) holds, then the lower bound in (136) holds with equality because $({\overline{\mathbf{X}}}_t;{\widehat{\overline{\mathbf{X}}}}_t),t\in {\mathbb{Z}}_{n_x}$ are jointly independent. Moreover, if (105) holds, then from, say, (118), the expressions (122) and (123) are obtained. The above equations establish all claims. □

Proposition 4. 

Theorem 4 is correct.

Proof. 

By invoking Corollary 2, Theorem 7 and the convexity of $R_{X|Y}\left({\Delta }_X\right)$ given by (122), then we arrive at the statements of Theorem 4, which completely characterize the RDF $R_{X|Y}\left({\Delta }_X\right)$ and construct a realization of the optimal $\widehat{X}$ that achieves it. □

Next, we discuss the degenerate case, when the statements of Theorems 3, 4 and 7 reduce to the RDF $R_X\left({\Delta }_X\right)$ of a Gaussian RV X with square-error distortion function. We illustrate that the identified structural property of the realization matrices $\left\{{\Sigma }_{\Delta },Q_{X|Y},H,Q_W\right\}$ leads to to the well-known water-filling solution.

Remark 3. 

Degenerate case of Theorem 7 and realization $\widehat{X}$ of Theorem 4a. Consider the characterization of the RDF $R_{X|Y}\left({\Delta }_X\right)$ of Theorem 7, the realization of $\widehat{X}$ Theorem 3a, Theorem 3, and assume X and Y are independent or Y generates the trivial information; i.e., the $\sigma -$algebra of Y is $\sigma \left\{Y\right\}=\{\Omega ,\varnothing \}$ or $C=0$ in (15)–(18).

(a) By the definitions of $Q_{X,Y},Q_{X|Y}$ then

$$\begin{array}{c}\hfill Q_{X,Y}=0,Q_{X|Y}=Q_X.\end{array}$$

(142)

Substituting (142) into the expressions of Theorem 7, the RDF $R_{X|Y}\left({\Delta }_X\right)$ reduces to

$$\begin{array}{cc}\hfill R_{X|Y}\left({\Delta }_X\right)=& R_X\left({\Delta }_X\right)\stackrel{▵}{=}\underset{\mathcal{Q}\left({\Delta }_X\right)}{inf}I(X;\widehat{X})\hfill \end{array}$$

(143)

$$\begin{array}{cc}\hfill =& \underset{{\mathcal{Q}}^m\left({\Delta }_X\right)}{inf}\frac{1}{2}log\left\{det\left(Q_X{\Sigma }_{\Delta }^{-1}\right)\right\},\hfill \end{array}$$

(144)

where

$$\begin{array}{cc}\hfill {\mathcal{Q}}^m\left({\Delta }_X\right)\stackrel{▵}{=}& \left\{{\Sigma }_{\Delta }⪰0:Q_X⪰{\Sigma }_{\Delta },\mathrm{trace}\left({\Sigma }_{\Delta }\right)\le {\Delta }_X\right\},\hfill \end{array}$$

(145)

and the optimal reproduction $\widehat{X}$ reduces to

$$\begin{array}{cc}\hfill \widehat{X}=& \left(I_{n_x}-{\Sigma }_{\Delta }{Q}_X^{-1}\right)X+W,Q_X⪰{\Sigma }_{\Delta },\hfill \end{array}$$

(146)

$$\begin{array}{cc}\hfill Q_W=& \left(I_{n_x}-{\Sigma }_{\Delta }{Q}_X^{-1}\right){\Sigma }_{\Delta }⪰0.\hfill \end{array}$$

(147)

Thus, $R_X\left({\Delta }_X\right)$ is the well-known RDF of a multivariate memoryless Gaussian RV X with square-error distortion.

(b) For the RDF $R_X\left({\Delta }_X\right)$ of part (a), it is known [24] that ${\Sigma }_{\Delta }$ and $Q_X$ have a spectral decomposition with respect to the same unitary matrix, that is,

$$\begin{array}{cc}\hfill & Q_X=U{\Lambda }_X{U}^T,{\Sigma }_{\Delta }=U\Delta U^T,U{U}^T=I,\hfill \end{array}$$

(148)

$$\begin{array}{cc}\hfill & {\Lambda }_X=\mathrm{diag}\{{\lambda }_{X,1},\cdots ,{\lambda }_{X,n_x}\},\Delta =\mathrm{diag}\{{\delta }_1,\cdots ,{\delta }_{n_x}\},\hfill \end{array}$$

(149)

where the entries of $({\Lambda }_X,\Delta )$ are in decreasing order.

Define

$$\begin{array}{c}\hfill {\mathsf{X}}^p\stackrel{▵}{=}{U}^{T}X,{\widehat{\mathsf{X}}}^p\stackrel{▵}{=}{U}^T\widehat{X},{\mathsf{W}}^p\stackrel{▵}{=}{U}^{T}W.\end{array}$$

(150)

Then, a parallel channel realization of the optimal reproduction ${\widehat{\mathsf{X}}}^p$ is obtained as follows:

$$\begin{array}{cc}\hfill & {\widehat{\mathsf{X}}}^p=\mathsf{H}{\mathsf{X}}^p+{\mathsf{W}}^p,\hfill \end{array}$$

(151)

$$\begin{array}{cc}\hfill & \mathsf{H}=I_{n_x}-\Delta {\Lambda }_X^{-1}=\mathrm{diag}\{1-\frac{{\delta }_1}{{\lambda }_{X,1}},\cdots ,1-\frac{{\delta }_{n_x}}{{\lambda }_{X,n_x}}\},\hfill \end{array}$$

(152)

$$\begin{array}{cc}\hfill & Q_{{\mathsf{W}}^p}=\mathsf{H}\Delta =\mathrm{diag}\{\left(1-\frac{{\delta }_1}{{\lambda }_{X,1}}\right){\delta }_1,\cdots ,\left(1-\frac{{\delta }_{n_x}}{{\lambda }_{X,n_x}}\right){\delta }_{n_x}\}.\hfill \end{array}$$

(153)

The RDF $R_X\left({\Delta }_X\right)$ is then computed from the reverse water-filling equations as follows.

$$R_X\left({\Delta }_X\right)=\frac{1}{2}\sum _{i=1}^{n_x}log\frac{{\lambda }_{X,i}}{{\delta }_i},$$

(154)

where

$$\sum _{i=1}^{n_x}{\delta }_i={\Delta }_X,{\delta }_i=\left\{\begin{array}{ccc}\mu ,\hfill & if\hfill & \mu <{\lambda }_{X,i}\hfill \\ {\lambda }_{X,i},\hfill & if\hfill & \mu \ge {\lambda }_{X,i}\hfill \end{array}\right.$$

(155)

and $\mu \in [0,\infty )$ is a Lagrange multiplier (obtained from the Kuch–Tucker conditions).

4.2. Side Information Only at Decoder

In general, when the side information is available only at the decoder, the achievable operational rate $R^{\ast }\left({\Delta }_X\right)$ is greater than the achievable operational rate ${\overline{R}}_1\left({\Delta }_X\right)$ when the side information is available to the encoder and the decoder [2]. By Remark 1, $\overline{R}\left({\Delta }_X\right)\ge R_{X|Y}\left({\Delta }_X\right)$, and equality holds if $I(X;Z|\widehat{X},Y)=0$.

In view of the characterization of $R_{X|Y}\left({\Delta }_X\right)$ and the realization of the optimal reproduction $\widehat{X}$ of Theorem 3, which is presented in Figure 3, we observe that we can re-write (49) as follows.

$$\begin{array}{cc}\hfill \widehat{X}=& HX+\left(I_{n_x}-H\right)Q_{X,Y}{Q}_Y^{-1}Y+W,\hfill \end{array}$$

(156)

$$\begin{array}{cc}\hfill =& \left(I_{n_x}-H\right)Q_{X,Y}{Q}_Y^{-1}Y+Z\hfill \end{array}$$

(157)

$$\begin{array}{cc}\hfill =& f(Y,Z)\hfill \end{array}$$

(158)

$$\begin{array}{cc}\hfill Z=& HX+W,\hfill \end{array}$$

(159)

$$\begin{array}{cc}\hfill H=& I_{n_x}-{\Sigma }_{\Delta }{Q}_{X|Y}^{-1},Q_W=H{\Sigma }_{\Delta },\mathrm{defined}\mathrm{by}\left(51\right)–\left(63\right),\hfill \end{array}$$

(160)

$$\begin{array}{cc}\hfill {\mathbf{P}}_{Z|X,Y}=& {\mathbf{P}}_{Z|X},(\widehat{X},Y)\mathrm{uniquely}\mathrm{define}Z,\mathrm{which}\mathrm{implies}I(X;Z|\widehat{X},Y)=0.\hfill \end{array}$$

(161)

Proposition 5. 

Theorem 5 is correct.

Proof. 

From the above realization of $\widehat{X}=f(Y,Z)$, we have the following. (a) By Wyner, see Remark 1, then the inequalities (36) and (37) hold, and equalities hold if $I(X;Z|\widehat{X},Y)=0$. That is, for any $\widehat{X}=f(Y,Z)$, and by the properties of conditional mutual information, then

$$\begin{array}{cc}\hfill I(X;Z|Y)& \stackrel{\left(\alpha \right)}{=}I(X;Z,\widehat{X}|Y)\hfill \end{array}$$

(162)

$$\begin{array}{cc}\hfill & \stackrel{\left(\beta \right)}{=}I(X;Z|\widehat{X},Y)+I(X;\widehat{X}|Y)\hfill \end{array}$$

(163)

$$\begin{array}{cc}\hfill & \stackrel{\left(\gamma \right)}{\ge }I(X;\widehat{X}|Y),\hfill \end{array}$$

(164)

where $\left(\alpha \right)$ is due to $\widehat{X}=f(Y,Z)$, $\left(\beta \right)$ is due to the chain rule of mutual information, and $\left(\gamma \right)$ is due to $I(X;Z|\widehat{X},Y)\ge 0$. Hence, (72) is obtained (as in Wyner [2] for a tuple of scalar jointly Gaussian RVs). (b) Equality holds in (164) if there exists an $\widehat{X}=f(Y,Z)$ such that $I(X;Z|\widehat{X},Y)=0$, and the average distortion is satisfied. Taking $\widehat{X}=f(Y,Z)=(I_{n_x}-H)Q_{X,Y}{Q}_Y^{-1}Y+Z$, where $Z=g(X,W)$ is specified by (156)–(160), then $I(X;Z|\widehat{X},Y)=0$ and the average distortion is satisfied. Since the realization (156)–(160) is identical to the realization (73)–(76), then part (b) is also shown. (c) This follows directly from the optimal realization. □

5. Connection with Other Works and Simulations

In this section, we illustrate that for the special case of scalar-valued jointly Gaussian RVs $(X,Y)$, our results reproduce Wyner’s [2] results. In addition, we show that the characterizations of the RDFs of the more general problems considered in [5,6] (i.e., where a noisy version of source is available at the encoder) do not reproduce Wyner’s [2] results. Finally, we present simulations.

5.1. Connection with Other Works

Remark 4. 

The degenerate case to Wyner’s [2] optimal test channel realizations. Now, we verify that for the tuple of scalar-valued, jointly Gaussian RVs $(X,Y)$, with square error distortion function specified below, our optimal realizations of $\widehat{X}$ and closed form expressions for $R_{X|Y}\left({\Delta }_X\right)$ and $\overline{R}\left({\Delta }_X\right)$ are identical to Wyner’s [2] realizations and RDFs (see Figure 4). Let us define:

$$\begin{array}{cc}\hfill & X:\Omega \to \mathcal{X}\stackrel{▵}{=}\mathbb{R},Y:\Omega \to \mathcal{Y}\stackrel{▵}{=}\mathbb{R},\widehat{X}:\Omega \to \widehat{\mathcal{X}}\stackrel{▵}{=}\mathbb{R},\hfill \end{array}$$

(165)

$$\begin{array}{cc}\hfill & d_X(x,\widehat{x})={\left(x-\widehat{x}\right)}^2,\hfill \end{array}$$

(166)

$$\begin{array}{cc}\hfill & X\in N(0,{\sigma }_X^2),{\sigma }_X^2>0,Y=\alpha \left(X+U\right),\hfill \end{array}$$

(167)

$$\begin{array}{cc}\hfill & U\in N(0,{\sigma }_U^2),{\sigma }_U^2>0,\alpha >0.\hfill \end{array}$$

(168)

(a) RDF $R_{X|Y}\left({\Delta }_X\right)$: By Theorem 4a applied to (165)–(168), we obtain

$$\begin{array}{cc}\hfill & Q_X={\sigma }_X^2,Q_{X,Y}=\alpha {\sigma }_X^2,Q_Y={\sigma }_Y^2={\alpha }^2{\sigma }_X^2+{\alpha }^2{\sigma }_U^2,Q_{X|Y}=c{\sigma }_U^2,c\stackrel{▵}{=}\frac{{\sigma }_X^2}{{\sigma }_X^2+{\sigma }_U^2},\hfill \end{array}$$

(169)

$$\begin{array}{cc}\hfill & H=1-{\Delta }_X{Q}_{X|Y}^{-1}=\frac{c{\sigma }_U^2-d}{c{\sigma }_U^2}\equiv a,Q_{X,Y}{Q}_Y^{-1}=\frac{c}{\alpha },H{Q}_{X,Y}{Q}_Y^{-1}=\frac{ac}{\alpha },\hfill \end{array}$$

(170)

$$\begin{array}{cc}\hfill & W=H\Psi =a\Psi ,Q_{\Psi }=H^{-1}{\Delta }_X=\frac{{\Delta }_X}{a}=\frac{c{\sigma }_U^2{\Delta }_X}{c{\sigma }_U^2-{\Delta }_X},c{\sigma }_U^2-{\Delta }_X>0.\hfill \end{array}$$

(171)

Moreover, by Theorem 4b the optimal reproduction $\widehat{X}\in {\mathcal{M}}_0\left(d\right)$ and $R_{X|Y}\left(d\right)$ are,

$$\begin{array}{cc}\hfill & \widehat{X}=a(X-\frac{c}{\alpha }Y)+\frac{c}{\alpha }Y+a\Psi ,c{\sigma }_U^2-{\Delta }_X>0\hfill \end{array}$$

(172)

$$\begin{array}{cc}\hfill & R_{X|Y}\left({\Delta }_X\right)=\left\{\begin{array}{cc}\frac{1}{2}log\frac{c{\sigma }_U^2}{{\Delta }_X},\hfill & 0<{\Delta }_X<c{\sigma }_U^2\hfill \\ 0,\hfill & {\Delta }_X\ge c{\sigma }_U^2.\hfill \end{array}\right.\hfill \end{array}$$

(173)

This shows our realization of Figure 3 degenerates to Wyner’s [2] realization of Figure 4a.

(b) RDF $\overline{R}\left({\Delta }_X\right)$: By Theorem 5b applied to (165)–(168), and using the calculations (169)–(172), we obtain

$$\begin{array}{cc}\hfill & \widehat{X}=f(Y,Z)=\frac{c}{\alpha }(1-a)Y+Zby\left(172\right),\left(175\right),\hfill \end{array}$$

(174)

$$\begin{array}{cc}\hfill & Z=a\left(X+\Psi \right),(a,\Psi )definedin\left(170\right),\left(171\right)\hfill \end{array}$$

(175)

$$\begin{array}{cc}\hfill & \overline{R}\left({\Delta }_X\right)=R_{X|Y}\left({\Delta }_X\right)=\left(173\right)byevaluatingI(X;Z)-I(Y;Z),using\left(4\right)and\left(175\right).\hfill \end{array}$$

(176)

This shows our value of $\overline{R}\left({\Delta }_X\right)$ and optimal realization $\widehat{X}=f(Y,Z)$ reproduce Wyner’s optimal realization and the value of $\overline{R}\left({\Delta }_X\right)$ given in [2] (i.e., Figure 4b).

In the following Remark, we show that, when $S=X$-a.s., the realization of the auxiliary RV Z, which is used in the proofs in [5,6] to show the converse coding theorem does not coincide with Wyner’s realization [2]. Also, their realizations do not reproduce Wyner’s RDF (this observation is verified for modified realization given in the correction note without proof in https://tiangroup.engr.tamu.edu/publications/ (accessed on 3 January 2024)). The deficiency of the realizations in [5,6] to show the converse was first pointed out in [7], using an alternative proof.

Remark 5. 

Optimal test channel realization of [5,6]

(a) The derivation of [[5], Theorem 4], uses the following representation of RVs (see [[5], Equation (4)] adopted to our notation using (19)):

$$\begin{array}{c}\hfill X=\left(K_{xs}{K}_{sy}+K_{xy}\right)Y+K_{xs}{N}_1+N_2,S=K_{sy}Y+N_1,\end{array}$$

where $N_1$ and $N_2$ are independent Gaussian RVs with zero mean, $N_1$ is independent Y and $N_2$ is independent of $(S,Y)$.

To reduce [5,6] to the Wyner and Ziv RDF, we set $X=S-\mathrm{a}.\mathrm{s}.$, which then implies, $K_{xs}=I,N_2=0-\mathrm{a}.\mathrm{s}$ and $K_{xy}=0$. According to the derivation of the converse [[5], Theorem 4] (see [[5], 3 lines above Equation (32)] using our notation), the optimal realization of the auxiliary RV $Z_{\mathrm{T}}$ used to achieve the RDF is

$$\begin{array}{c}\hfill Z_{\mathrm{T}}=U^{T}X+N_3,\end{array}$$

(177)

where $Q_{X|Y}=U\mathrm{diag}({\lambda }_1,\cdots ,{\lambda }_n)U^T$ and U is a unitary matrix, $N_3\in N(0,Q_{N_3})$ such that $Q_{N_3}$ is a diagonal covariance matrix, with elements given by (for the value of ${\sigma }_{3,i}^2$, we considered the one given in the correction note in https://tiangroup.engr.tamu.edu/publications/ (accessed on 3 January 2024) (although no derivation is given), where it is stated that ${\sigma }_{3,i}^2$ that appeared in the derivation [[5], proof of theorem 4] should be multiplied by ${\lambda }_i$),

$$\begin{array}{c}\hfill {\sigma }_{3,i}^2=\frac{min({\lambda }_i,{\delta }_i)}{{\lambda }_i-min({\lambda }_i,{\delta }_i)}{\lambda }_i,\sum _{i=1}^{n}min({\lambda }_i,{\delta }_i)={\Delta }_X.\end{array}$$

(178)

(b) It is easy to verify that the above realization of $Z_{\mathrm{T}}$ that uses the correction of footnote 6 is precisely the realization given in [[6], Theorem 3A].

(c) Special Case: For scalar-valued RVs the auxiliary RV $Z_{\mathrm{T}}$ reduces to

$$\begin{array}{c}\hfill Z_{\mathrm{T}}=X+N_3,N_3\in N\left(0,\frac{{\Delta }_X{Q}_{X|Y}}{Q_{X|Y}-{\Delta }_X}\right),Q_{X|Y}>{\Delta }_X\end{array}$$

(179)

Now, we examine whether the realization (179) corresponds to Wyner’s realization and induces Wyner’s RDF. Recall that the Wyner’s [2] RDF, denoted by $R_{X;Z|Y}\left({\Delta }_X\right)$ and corresponding to auxiliary RV Z, is

$$\begin{array}{cc}\hfill & Z=HX+W,H=\frac{Q_{X|Y}-{\Delta }_X}{Q_{X|Y}},W\in N(0,H{\Delta }_X),\hfill \end{array}$$

(180)

$$\begin{array}{cc}\hfill & R_{X;Z|Y}\left({\Delta }_X\right)=I(X;Z|Y)=\frac{1}{2}log\left(\frac{Q_{X|Y}}{{\Delta }_X}\right),{\Delta }_X\le Q_{X|Y}.\hfill \end{array}$$

(181)

Clearly, the two realizations (179) and (180) are different. Let ${\widehat{R}}_{X;Z_{\mathrm{T}}|Y}\left({\Delta }_X\right)$ denote the RDF corresponding to the realization $Z_{\mathrm{T}}$. Then ${\widehat{R}}_{X;Z_{\mathrm{T}}|Y}\left({\Delta }_X\right)$ can be computed using $I(X;Z_{\mathrm{T}}|Y)=I(X;Z_{\mathrm{T}})-I(Y;Z_{\mathrm{T}})=-H\left(Z_{\mathrm{T}}\right|X)+H\left(Z_{\mathrm{T}}\right|Y)$ where $H(·|·)$ denotes the conditional differential entropy. Then, by using

$$\begin{array}{cc}\hfill & Q_{Z_{\mathrm{T}}|X}=Q_{N_3}=\frac{{\Delta }_X{Q}_{X|Y}}{Q_{X|Y}-{\Delta }_X},\hfill \end{array}$$

(182)

$$\begin{array}{cc}\hfill & Q_{Z_{\mathrm{T}}|Y}=Q_{N_3}+Q_{X|Y}.\hfill \end{array}$$

(183)

it is straightforward to show that

$$\begin{array}{cc}\hfill {\widehat{R}}_{X;Z_{\mathrm{T}}|Y}\left({\Delta }_X\right)=& -H\left(Z_{\mathrm{T}}\right|X)+H\left(Z_{\mathrm{T}}\right|Y)\hfill \end{array}$$

(184)

$$\begin{array}{cc}\hfill =& -\frac{1}{2}log\left(2\pi e\frac{{\Delta }_X{Q}_{X|Y}}{Q_{X|Y}-{\Delta }_X}\right)+\frac{1}{2}log\left(2\pi e\frac{Q_{X|Y}^2}{Q_{X|Y}-{\Delta }_X}\right),{\Delta }_X<Q_{X|Y}\hfill \end{array}$$

(185)

However, we note that (i) unlike Wyner’s RDF given in (181), which gives $R_{X;Z|Y}\left({\Delta }_X\right)=0$ at ${\Delta }_X=Q_{X|Y}$, the corresponding ${\widehat{R}}_{X;Z_{\mathrm{T}}|Y}\left({\Delta }_X\right)=-\infty +\infty$ at ${\Delta }_X=Q_{X|Y}$, and (ii) Wyner’s test channel realization is $Z=HX+W,$$H=\frac{Q_{X|Y}-{\Delta }_X}{Q_{X|Y}}$ and $W\in N(0,H{\Delta }_X)$, which is different from the test channel realization in (179). In particular, if $Q_{X|Y}={\Delta }_X$, then $H=0\Rightarrow W\in N(0,0)$ and $Z=0-\mathrm{a}.\mathrm{s}$. On the other hand, for the test channel in (179), if $Q_{X|Y}={\Delta }_X$, then $N_3\in N\left(0,+\infty \right)$, and thus the variance of $Z_{\mathrm{T}}$ in (179) is not zero.

Further, in Proposition 6, we prove that for the multi-dimensional source, the test channel realization in (179) does not achieve the RDF when water-filling is active, i.e., when at least one component of the source is not reproduced.

(d) Special Case Classical RDF: The classical RDF is obtained as a special case if we assume X and Y are independent or Y generates the trivial information $\{\Omega ,\varnothing \}$; i.e., Y is nonrandom. Clearly, in this case, the RDF ${\widehat{R}}_{S;Z_{\mathrm{T}}|Y}\left({\Delta }_X\right)$ should degenerate to the classical RDF of the source X, i.e., $R_X\left({\Delta }_X\right)$, and it should be that $\widehat{X}=Z_{\mathrm{T}}$. However, for this case, (179) gives $Q_{Z_{\mathrm{T}}}=Q_X+\frac{{\Delta }_X{Q}_X}{Q_X-{\Delta }_X}=\frac{Q_X^2}{Q_X-{\Delta }_X}$, which is fundamentally different from Wyner’s degenerate, and correct values $Q_{\widehat{X}}=Q_Z=max\{0,Q_X-{\Delta }_X\}$.

Proposition 6. 

When $S=X$-a.s., Wyner’s [2] auxiliary RV Z and the auxiliary RV $Z_{\mathrm{T}}$ given in (177) i.e., the degenerate case of [5,6] (with the correction of footnote 6), are not related by an invertible function. As a result, the computed RDF based on the two realizations are different.

Proof. 

Recall that, if the two auxiliary RVs $Z_{\mathrm{T}}$ and Z are not related by an invertible function, i.e., $Z=f\left(Z_{\mathrm{T}}\right)$, where $f(·)$ is invertible and both f and its inverse are measurable, then $I(X;Z_{\mathrm{T}})-I(Y;Z_{\mathrm{T}})\ne I(X;Z)-I(Y;Z)$. It was shown earlier in this paper (and also in [7]) that for the multivariate Wyner’s RDF, the auxiliary RV takes the form

$$\begin{array}{c}\hfill Z=HX+W,W\in N(0,Q_W),\end{array}$$

(186)

where $Q_W=H{\Sigma }_{\Delta }=U\mathrm{diag}({\sigma }_{w,1}^2,\cdots ,{\sigma }_{w,n}^2)U^T$, ${\Sigma }_{\Delta }=U\mathrm{diag}({\delta }_1,\cdots ,{\delta }_n)U^T$, $H=I-Q_{X|Y}^{-1}{\Sigma }_{\Delta }=U\mathrm{diag}(h_1,\cdots ,h_n)U^T$ and $Q_{X|Y}=U\mathrm{diag}({\lambda }_1,\cdots ,{\lambda }_n)U^T$, where U is a unitary matrix. The eigenvalues ${\sigma }_{w,i}^2$ and $h_i$ are given by

$$\begin{array}{cc}\hfill & {\sigma }_{w,i}^2=\frac{min({\lambda }_i,{\delta }_i)({\lambda }_i-min({\lambda }_i,{\delta }_i))}{{\lambda }_i},\hfill \end{array}$$

(187)

$$\begin{array}{cc}\hfill & h_i=\frac{{\lambda }_i-min({\lambda }_i,{\delta }_i)}{{\lambda }_i},\hfill \end{array}$$

(188)

where ${\sum }_{i=1}^{n}min({\lambda }_i,{\delta }_i)={\Delta }_X$. Hence, Equations (186), (187), and (188), imply that if ${\sigma }_{w,i}^2=0$ then $h_i=0$, and vice versa. Such zero values correspond to compression with water-filling. On the other hand, from (177) and (178), if water-filling is active, then ${\sigma }_{3,i}^2=\frac{{\lambda }_i^2}{{\lambda }_i-{\lambda }_i}$. Moreover, by comparing Equations (187) with (178) and (188) with (177), it is straightforward to show that $f(·)=HU$. If $HU$ is not an invertible matrix for all values of the distortion ${\Delta }_X$, then $I(X;Z_{\mathrm{T}})-I(Y;Z_{\mathrm{T}})\ne I(X;Z)-I(Y;Z)$.

By (188) it is easy to show that if $min({\lambda }_i,{\delta }_i)={\lambda }_i$, $HU$ is not invertible. This implies $I(X;Z_{\mathrm{T}})-I(Y;Z_{\mathrm{T}})\ne I(X;Z)-I(Y;Z)$. □

5.2. Simulations

In this section, we provide an example to show the gap between the classical rate distortion $R_X\left({\Delta }_X\right)$ defined in (154), the conditional distortion function $R_{X|Y}\left({\Delta }_X\right)$ (69), and to verify the validity of Gray’s lower bound (8). Note that in Theorem 5 it is shown that $R_{X|Y}\left({\Delta }_X\right)=\overline{R}\left({\Delta }_X\right)$, and hence the plot for $\overline{R}\left({\Delta }_X\right)$ is omitted. For the evaluation, we pick a joint covariance matrix (11) given by

$$\begin{array}{cc}\hfill & Q_{(X,Y)}=\left[\begin{array}{cccc}2.5000& 1.1250& 0.4750& 0.6125\\ 1.1250& 0.8125& 0.2750& 0.3063\\ 0.4750& 0.2750& 0.1525& 0.1625\\ 0.6125& 0.3063& 0.1625& 0.2031\end{array}\right],X:\Omega \to {\mathbb{R}}^2,Y:\Omega \to {\mathbb{R}}^2.\hfill \end{array}$$

In order to compute the rates, we first have to find $Q_X,Q_Y,Q_{XY}$ and $Q_{X|Y}$. From the definition of $Q_{(X,Y)}$ given in (11), it is easy to see that the covariance of X, Y, and the joint covariance of X and Y are equal to

$$\begin{array}{c}\hfill Q_X=\left[\begin{array}{cc}2.5000& 1.1250\\ 1.1250& 0.8125\end{array}\right],Q_Y=\left[\begin{array}{cc}0.1525& 0.1625\\ 0.1625& 0.2031\end{array}\right],Q_{XY}=\left[\begin{array}{cc}0.4750& 0.6125\\ 0.2750& 0.3063\end{array}\right].\end{array}$$

Then, the conditional covariance $Q_{X|Y}$, which appears in $R_{X|Y}\left({\Delta }_X\right)$, can be computed from (27). Using Singular Value Decomposition (SVD), we can calculate the eigenvalues of $Q_{X|Y}$. For this case, the eigenvalues of the conditional covariance are $\{0.7538,0.2\}$. Similarly, the eigenvalues of $Q_X$ can be determined. Finally, the eigenvalues of $Q_X$ and $Q_{X|Y}$ are passed to the water-filling to compute the $R_X\left({\Delta }_X\right)$ and $R_{X|Y}\left({\Delta }_X\right)$, respectively.

The classical rate distortion, the conditional RDF, and the Gray’s lower bound for the joint covariance above are illustrated in Figure 5. It is clear that $R_{X|Y}\left({\Delta }_X\right)$ is smaller, and as the distortion ${\Delta }_X$ increases, the gap between the classical and conditional RDF becomes larger. Gray’s lower bound is achievable for some positive distortion values, as provided in (71), i.e., for ${\Delta }_X\in \{{\Delta }_X\in [0,\infty ):{\Delta }_X\le n_x{\lambda }_{n_x}\}$. Recall that the set of eigenvalues of $Q_{X|Y}$ is $\{0.7538,0.2\}$, and the lower bound is achievable for ${\Delta }_X\le 2·0.2=0.4$; i.e., for these values $R_{X|Y}\left({\Delta }_X\right)=R_X\left({\Delta }_X\right)-I(X;Y)$.

6. Conclusions

We derived nontrivial structural properties of the optimal test channel realizations that achieve the optimal test channel distributions of the characterizations of RDFs for a tuple of multivariate jointly independent and identically distributed Gaussian random variables with mean-square error fidelity for two cases. Initially, the side information was available at the encoder and decoder, and then it was only available at the decoder. Using the realizations of the optimal test channels, we showed that when the side information is known to the encoder and the decoder, it does not achieve a better compression compared to when side information is only known to the decoder.