Best-effort adaptation

Abstract

We study a problem of best-effort adaptation motivated by several applications and considerations, which consists of determining an accurate predictor for a target domain, for which a moderate amount of labeled samples are available, while leveraging information from another domain for which substantially more labeled samples are at one’s disposal. We present a new and general discrepancy-based theoretical analysis of sample reweighting methods, including bounds holding uniformly over the weights. We show how these bounds can guide the design of learning algorithms that we discuss in detail. We further show that our learning guarantees and algorithms provide improved solutions for standard domain adaptation problems, for which few labeled data or none are available from the target domain. We finally report the results of a series of experiments demonstrating the effectiveness of our best-effort adaptation and domain adaptation algorithms, as well as comparisons with several baselines. We also discuss how our analysis can benefit the design of principled solutions for fine-tuning.

Appendices

Appendix A: Best-effort adaptation

1.1 A.1 Theorems and proofs

Below we will work with a notion of discrepancy extended to finite signed measures, as defined in ().

Theorem 1

Fix a vector $\textsf {q}$ in $[0, 1]^{[m + n]}$. Then, for any $\delta > 0$, with probability at least $1 - \delta$ over the draw of an i.i.d. sample S of size m from ${\mathscr {Q}}$ and an i.i.d. sample $S'$ of size n from ${\mathscr {P}}$, the following holds for all $h \in {\mathscr {H}}$:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) \le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}}\big ) + 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) + \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}}. \end{aligned}$$

Proof

Let $S = ((x_1, y_1), \ldots , (x_m, y_m))$ be a sample of size m drawn i.i.d. from ${\mathscr {Q}}$ and similarly $S' = ((x_{m + 1}, y_{m + 1}), \ldots , (x_{m + n}, y_{m + n}))$ a sample of size n drawn i.i.d. from ${\mathscr {P}}$. Let T denote the sample formed by S and $S'$, $T = (S, S')$. For any such sample T, define $\Phi (T)$ as follows:

$$\begin{aligned} \Phi (T) = \sup _{h \in {\mathscr {H}}} \{ \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h) - \mathcal {L}_T(\textsf {q}, h)\}, \end{aligned}$$

with $\mathcal {L}_T(\textsf {q}, h) = \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i)$. Changing point $x_i$ to some other point $x'_i$ affects $\Phi (T)$ by at most $\textsf {q}_i$. Thus, by McDiarmid’s inequality, for any $\delta > 0$, with probability at least $1 - \delta$, the following holds for all $h \in {\mathscr {H}}$:

$$\begin{aligned} \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h) \le \mathcal {L}_T(\textsf {q}, h) + \mathbb E[\Phi (T)] + \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}}. \end{aligned}$$

(A.1)

Now, let $T' = ((x'_1, y'_1), \ldots , (x'_m, y'_m), (x'_{m + 1}, y'_{m + 1}), \ldots , (x'_{m + n}, y'_{m + n})))$ be a sample drawn according to the same distribution as T, then we can write:

$$\begin{aligned} \underset{T'}{\mathbb E}[\mathcal {L}_{T'}(q, h)]&= \sum _{i=1}^m q_i \mathbb E[\ell (h(x'_i), y'_i)] + \sum _{i = m + 1}^{m+n} q_i \mathbb E[\ell (h(x'_i), y'_i)] \nonumber \\&{(\text {linearity of expectation and weights } \textsf {q}_i \text { independent of } T)} \nonumber \\&= \sum _{i = 1}^m q_i \mathcal {L}({\mathscr {Q}}, h) + \sum _{i=m+1}^{m+n} q_i \mathcal {L}({\mathscr {P}}, h) {(\text {i.i.d. sample})} \nonumber \\&= \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}})\mathcal {L}({\mathscr {P}}, h). \end{aligned}$$

(A.2)

In light of that equality, we can analyze the expectation term as follows:

$$\begin{aligned} \mathbb E[\Phi (T)]&= \underset{T}{\mathbb E}\left[ {\sup _{h \in {\mathscr {H}}} \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h) - \mathcal {L}_T(\textsf {q}, h)}\right] \\&= \underset{T}{\mathbb E}\left[ {\sup _{h \in {\mathscr {H}}} \underset{T'}{\mathbb E}\ \left[ {\mathcal {L}_{T'}(\textsf {q}, h)}\right] - \mathcal {L}_T(\textsf {q}, h)}\right] \\&= \underset{T}{\mathbb E}\left[ {\sup _{h \in {\mathscr {H}}} \underset{T'}{\mathbb E}\ \left[ {\mathcal {L}_{T'}(\textsf {q}, h) - \mathcal {L}_T(\textsf {q}, h)}\right] }\right] \quad \quad \quad \quad \quad \quad {(\mathcal {L}_{T'} (\textsf {q}, h) \text { independent of } T')}\\&\le \underset{T, T'}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \mathcal {L}_{T'}(\textsf {q}, h) - \mathcal {L}_T(\textsf {q}, h)}\right] \quad \quad \quad \quad \quad \quad \quad \ \ {(\text {sub-additivity of supremum})}\\&= \underset{T, T'}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x'_i), y'_i) - \textsf {q}_i \ell (h(x_i), y_i)}\right] \\&= \underset{T, T', {\varvec{\sigma }}}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \sum _{i = 1}^{m + n} \sigma _i \big ({\textsf {q}_i \ell (h(x'_i), y'_i) - \textsf {q}_i \ell (h(x_i), y_i)}\big )}\right] \\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \ \ {(\text {introducing Rademacher variables } \sigma _i)}\\&\le \underset{T', {\varvec{\sigma }}}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \sum _{i = 1}^{m + n} \sigma _i \textsf {q}_i \ell (h(x'_i), y'_i)}\right] + \underset{T, {\varvec{\sigma }}}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \sum _{i = 1}^{m + n} -\sigma _i \textsf {q}_i \ell (h(x_i), y_i)}\right] \\&\qquad \qquad \qquad \qquad \qquad \qquad \quad \ {(\text {sub-addivity of supremum and linearity of expectation})}\\&= 2 \underset{T, {\varvec{\sigma }}}{\mathbb E}\ \left[ {\sup _{h \in {\mathscr {H}}} \sum _{i = 1}^{m + n} \sigma _i \textsf {q}_i \ell (h(x_i), y_i)}\right] \quad \ \ {(-\sigma _i \text { and } \sigma _i \text { follow the same distribution}}\\&= 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}). \end{aligned}$$

Finally, using the upper bound

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) - \left[ {\overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h)}\right]&= \left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] \mathcal {L}({\mathscr {P}}, h) - \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h)\\&\le \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}}\big ), \end{aligned}$$

inequality (A.1), and the upper bound on $\mathbb E[\Phi (T)]$, we obtain:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h)&\le \left[ {\overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) + (\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h)}\right] + \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}}\big )\\&\le \mathcal {L}_T(\textsf {q}, h) + \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}}\big ) + 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) + \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}}, \end{aligned}$$

which completes the proof. $\square$

Next, we show that the learning bound just proven is tight in terms of the weighted-discrepancy term.

Theorem 2

Fix a distribution $\textsf {q}$ in the simplex $\Delta _{m + n}$. Then, for any $\epsilon > 0$, there exists $h \in {\mathscr {H}}$ such that, for any $\delta > 0$, the following lower bound holds with probability at least $1 - \delta$ over the draw of an i.i.d. sample S of size m from ${\mathscr {Q}}$ and an i.i.d. sample $S'$ of size n from ${\mathscr {P}}$:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) \ge \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) - 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) - \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}} -\epsilon . \end{aligned}$$

In particular, for $\big \Vert {\textsf {q}}\big \Vert _2,\mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) \in O\big ({\frac{1}{\sqrt{m + n}}}\big )$, we have:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) \ge \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \Omega \big ({\frac{1}{\sqrt{m + n}}}\big ). \end{aligned}$$

Proof

Let $\mathcal {L}(\textsf {q}, h)$ denote $\sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i)$. By definition of discrepancy as a supremum, for any $\epsilon > 0$, there exists $h \in {\mathscr {H}}$ such that $\mathcal {L}({\mathscr {P}}, h) - \mathcal {L}({\mathscr {Q}}, h) \ge \text {dis}({\mathscr {P}}, {\mathscr {Q}}) - \epsilon$. For that h, we have

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) - \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) - \mathcal {L}(\textsf {q}, h)&\ge \mathcal {L}({\mathscr {P}}, h) - \overline{\textsf {q}}\big ({\mathcal {L}({\mathscr {P}}, h) - \mathcal {L}({\mathscr {Q}}, h)}\big ) - \mathcal {L}(\textsf {q}, h) - \epsilon \\&= (1 - \overline{\textsf {q}}) \mathcal {L}({\mathscr {P}}, h) + \overline{\textsf {q}}\mathcal {L}({\mathscr {Q}}, h) - \mathcal {L}(\textsf {q}, h) - \epsilon \\&= \mathbb E[\mathcal {L}(\textsf {q}, h)] - \mathcal {L}(\textsf {q}, h) - \epsilon . \end{aligned}$$

By McDiarmid’s inequality, with probability at least $1 - \delta$, we have $\mathbb E[\mathcal {L}(\textsf {q}, h)] - \mathcal {L}(\textsf {q}, h) \ge - 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) - \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}}$. Thus, we have:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h) - \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) - \mathcal {L}(\textsf {q}, h) \ge - 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) - \Vert \textsf {q}\Vert _2 \sqrt{\frac{\log \frac{1}{\delta }}{2}} - \epsilon . \end{aligned}$$

The last inequality follows directly by using the assumptions and Lemma 10. $\square$

Theorem 3

For any $\delta > 0$, with probability at least $1 - \delta$ over the draw of an i.i.d. sample S of size m from ${\mathscr {Q}}$ and an i.i.d. $S'$ of size n from ${\mathscr {P}}$, the following holds for all $h \in {\mathscr {H}}$ and $\textsf {q}\in \{\textsf {q}:\Vert \textsf {q}- \textsf {p}^0 \Vert _1 < 1\}$:

$$\begin{aligned}&\mathcal {L}({\mathscr {P}}, h) \le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}}\big ) + \text {dis}(\textsf {p}^0, \textsf {q})\\&+ 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) + 7 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1 + \left[ {\big \Vert {\textsf {q}}\big \Vert _2 + 2 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1}\right] \left[ { \sqrt{\log \log _2 \tfrac{2}{1 - \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1}} + \sqrt{\tfrac{\log \frac{2}{\delta }}{2}} }\right] . \end{aligned}$$

Proof

Consider two sequences $(\epsilon _k)_{k \ge 0}$ and $(\textsf {q}^k)_{k \ge 0}$. By Theorem 1, for any fixed $k \ge 0$, we have:

$$\begin{aligned} \mathbb {P}\Bigg [ \mathcal {L}({\mathscr {P}}, h) > \sum _{i = 1}^{m + n} \textsf {q}^k_i \ell (h(x_i), y_i) + \text {dis}\left( {\left[ {(1 - \big \Vert {\textsf {q}^k}\big \Vert _1) + \overline{\textsf {q}}^k}\right] {\mathscr {P}}, \overline{\textsf {q}}^k {\mathscr {Q}}}\right) \\ + 2 \mathfrak {R}_{\textsf {q}^k}(\ell \circ {\mathscr {H}}) + \frac{\Vert \textsf {q}^k \Vert _2}{\sqrt{2}} \epsilon _k \Bigg ] \le e^{-\epsilon _k^2}. \end{aligned}$$

Choose $\epsilon _k = \epsilon + \sqrt{2 \log (k + 1)}$. Then, by the union bound, we can write:

$$\begin{aligned} \mathbb {P}\Bigg [&\exists k \ge 1 :\mathcal {L}({\mathscr {P}}, h) > \sum _{i = 1}^{m + n} \textsf {q}^k_i \ell (h(x_i), y_i) + \text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}^k}\big \Vert _1) + \overline{\textsf {q}}^k}\right] {\mathscr {P}}, \overline{\textsf {q}}^k {\mathscr {Q}}}\big ) \\&+ 2 \mathfrak {R}_{\textsf {q}^k}(\ell \circ {\mathscr {H}}) + \frac{\Vert \textsf {q}^k \Vert _2}{\sqrt{2}} \epsilon _k \Bigg ] \nonumber \\&\le \sum _{k = 0}^{+\infty } e^{-\epsilon _k^2} \le \sum _{k = 0}^{+\infty } e^{-\epsilon ^2 - \log ((k + 1)^2)} = e^{-\epsilon ^2} \sum _{k = 1}^{+\infty } \frac{1}{k^2} = \frac{\pi ^2}{6} e^{-\epsilon ^2} \le 2 e^{-\epsilon ^2}.\nonumber \end{aligned}$$

(A.3)

We can choose $\textsf {q}^k$ such that $\Vert \textsf {q}^k - \textsf {p}^0 \Vert _1 = 1 - \frac{1}{2^k}$. Then, for any $\textsf {q}\in \{\textsf {q}:\Vert \textsf {q}- \textsf {p}^0 \Vert _1 < 1\}$, there exists $k \ge 0$ such that $\Vert \textsf {q}^k - \textsf {p}^0 \Vert _1 \le \Vert \textsf {q}- \textsf {p}^0 \Vert _1 < \Vert \textsf {q}^{k + 1} - \textsf {p}^0 \Vert _1$ and thus such that

$$\begin{aligned} \sqrt{2 \log (k + 1)} = \sqrt{2 \log \log _2 \frac{1}{1 - \big \Vert {\textsf {q}^{k + 1} - \textsf {p}^0}\big \Vert _1}}&= \sqrt{2 \log \log _2 \frac{2}{1 - \big \Vert {\textsf {q}^k - \textsf {p}^0}\big \Vert _1}}\\&\le \sqrt{2 \log \log _2 \frac{2}{1 - \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1}}. \end{aligned}$$

Furthermore, for that k, the following inequalities hold:

$$\begin{aligned} \sum _{i = 1}^{m + n} \textsf {q}^k_i \ell (h(x_i), y_i)&\le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \text {dis}(\textsf {q}^k, \textsf {q})\\&\le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \text {dis}(\textsf {q}^k, \textsf {p}^0) + \text {dis}(\textsf {p}^0, \textsf {q})\\&\le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \big \Vert {\textsf {q}^k - \textsf {p}^0}\big \Vert _1 + \text {dis}(\textsf {p}^0, \textsf {q})\\&\le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1 + \text {dis}(\textsf {p}^0, \textsf {q}),\\ \text {dis}\left( {\left[ {(1 - \big \Vert {\textsf {q}^k}\big \Vert _1) + \overline{\textsf {q}}^k}\right] {\mathscr {P}}, \overline{\textsf {q}}^k {\mathscr {Q}}}\right)&\le \text {dis}\left( \left[ {\left( 1 - \big \Vert {\textsf {q}}\big \Vert _1\right) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}\right) \\&\qquad + \big \Vert { \left[ {(\big \Vert {\textsf {q}}\big \Vert _1 - \overline{\textsf {q}}) - (\big \Vert {\textsf {q}^k}\big \Vert _1 - \overline{\textsf {q}}^k)}\right] {\mathscr {P}}\!+\! \left[ {\overline{\textsf {q}}- \overline{\textsf {q}}^k}\right] {\mathscr {Q}}}\big \Vert _1\\&\le \text {dis}\left( \left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}\right) + \big \Vert {\textsf {q}^k - \textsf {q}}\big \Vert _1\\&\le \text {dis}\left( \left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}\right) + 2 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1,\\ \mathfrak {R}_{\textsf {q}^k}(\ell \circ {\mathscr {H}})&\le \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) + \big \Vert {\textsf {q}^k - \textsf {q}}\big \Vert _1 \le \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) \!+\! 2 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1,\\ \text {and} \qquad \big \Vert {\textsf {q}^k}\big \Vert _2&\le \big \Vert {\textsf {q}}\big \Vert _2 + \big \Vert {\textsf {q}^k - \textsf {q}}\big \Vert _2\\&\le \big \Vert {\textsf {q}}\big \Vert _2 + \big \Vert {\textsf {q}^k - \textsf {q}}\big \Vert _1 \le \big \Vert {\textsf {q}}\big \Vert _2 + 2 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1. \end{aligned}$$

Plugging in these inequalities in (A.3) concludes the proof. $\square$

Corollary 4

For any $\delta > 0$, with probability at least $1 - \delta$ over the draw of an i.i.d. sample S of size m from ${\mathscr {Q}}$ and an i.i.d. sample $S'$ of size n from ${\mathscr {P}}$, the following holds for all $h \in {\mathscr {H}}$ and $\textsf {q}\in \{\textsf {q}:\Vert \textsf {q}- \textsf {p}^0 \Vert _1 < 1\}$:

$$\begin{aligned} \mathcal {L}({\mathscr {P}}, h)&\le \sum _{i = 1}^{m + n} \textsf {q}_i \ell (h(x_i), y_i) + \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \text {dis}(\textsf {p}^0, \textsf {q}) + 2 \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}})\\&\quad + 8 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1 + \left[ {\big \Vert {\textsf {q}}\big \Vert _2 + 2 \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1}\right] \left[ { \sqrt{\log \log _2 \tfrac{2}{1 - \big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1}} + \sqrt{\frac{\log \tfrac{2}{\delta }}{2}} }\right] . \end{aligned}$$

Proof

Note that the discrepancy term of the bound of Theorem 3 can be further upper bounded as follows:

$$\begin{aligned}&\text {dis}\big ({\left[ {(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\right] {\mathscr {P}}, \overline{\textsf {q}}{\mathscr {Q}}} \big )\\&= \underset{h \in {\mathscr {H}}}{\sup }\ \left\{ \big [{(1 - \big \Vert {\textsf {q}}\big \Vert _1) + \overline{\textsf {q}}}\big ] \underset{(x, y) \sim {\mathscr {P}}}{\mathbb E}\ [\ell (h(x), y)] - \overline{\textsf {q}}\underset{(x, y) \sim {\mathscr {Q}}}{\mathbb E}\ [\ell (h(x), y)]\right\} \\&\le \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \left| {1 - \big \Vert {\textsf {q}}\big \Vert _1}\right| \underset{h \in {\mathscr {H}}}{\sup }\ \underset{(x, y) \sim {\mathscr {P}}}{\mathbb E}\ [\ell (h(x), y)]\\&\le \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \left| {1 - \big \Vert {\textsf {q}}\big \Vert _1}\right| \\&= \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \left| {\big \Vert {\textsf {p}^0}\big \Vert _1 - \big \Vert {\textsf {q}}\big \Vert _1} \right| \\&\le \overline{\textsf {q}}\text {dis}({\mathscr {P}}, {\mathscr {Q}}) + \big \Vert {\textsf {p}^0 - \textsf {q}}\big \Vert _1. \end{aligned}$$

Plugging this in the right-hand side in the bound of Theorem 3 completes the proof. $\square$

Lemma 10

Fix a distribution $\textsf {q}$ over $[m + n]$. Then, the following holds for the $\textsf {q}$-weighted Rademacher complexity:

$$\begin{aligned} \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) \le \Vert \textsf {q}\Vert _\infty (m + n) \, \mathfrak {R}_{m + n}(\ell \circ {\mathscr {H}}). \end{aligned}$$

Proof

Since for any $i \in [m + n]$, the function $\varphi _i :x \mapsto \textsf {q}_i x$ is $\textsf {q}_i$-Lipschitz and thus $\big \Vert {\textsf {q}}\big \Vert _\infty$-Lipschitz, the result is an application of the result of Meir and Zhang [88, Theorem 7]. $\square$

Note that the bound of the lemma is tight: equality holds when $\textsf {q}$ is chosen to be the uniform distribution. By McDiarmid’s inequality, the $\textsf {q}$-weighted Rademacher complexity can be estimated from the empirical quantity

$$\begin{aligned} \widehat{\mathfrak {R}}_{\textsf {q}, S, S'}(\ell \circ {\mathscr {H}}) = \underset{\varvec{\sigma }}{\mathbb E}\ \left[ {\underset{h \in {\mathscr {H}}}{\sup }\sum _{i = 1}^{m + n} \sigma _i \textsf {q}_i \ell (h(x_i), y_i)}\right] , \end{aligned}$$

modulo a term in $O(\Vert \textsf {q}\Vert _2)$.

1.2 A.2 Convex optimization solution

In the case of the squared loss with the hypothesis set of linear functions or kernel-based functions, the optimization algorithm for best can be formulated as a convex optimization problem.

We can proceed as follows when $\ell$ is the squared loss. We introduce new variables ${\textsf {u}}_i = 1/\textsf {q}_i$, ${\textsf {v}}_i = 1/{\textsf {p}^0_i}$ and define the convex set ${\mathscr {U}}= \{ {\textsf {u}}:{\textsf {u}}_i \ge 1 \}$. Using the following four expressions:

$$\begin{aligned} \textsf {q}_i (h(x_i) - y_i)^2 = \frac{(h(x_i) - y_i)^2}{{\textsf {u}}_i},&\big \Vert {\textsf {q}}\big \Vert _2^2 = \sum _i \frac{1}{{\textsf {u}}_i^2},\\ \big \Vert {\textsf {q}}\big \Vert _\infty \big \Vert {h}\big \Vert ^2 = \max _i \frac{\big \Vert {h}\big \Vert ^2}{{\textsf {u}}_i} = \frac{\big \Vert {h}\big \Vert ^2}{{\textsf {u}}_{\min }},&\big \Vert {\textsf {q}- \textsf {p}^0}\big \Vert _1 \le \sum _i \left| {{\textsf {v}}_i - {\textsf {u}}_i}\right| = \big \Vert {{\textsf {u}}- {\textsf {v}}}\big \Vert _1, \end{aligned}$$

leads to the following convex optimization problem with

new hyperparameters $\gamma _\infty , \gamma _1, \gamma _2$:

$$\begin{aligned} \min _{h \in {\mathscr {H}}, {\textsf {u}}\in {\mathscr {U}}}&\sum _{i = 1}^{m + n} \frac{(h(x_i) - y_i)^2 + d_i}{{\textsf {u}}_i} + \text {dis}\big ({\left( { \tfrac{1}{{\textsf {u}}_i}}\right) _i, \left( {\tfrac{1}{{\textsf {v}}_i}}\right) _i}\big ) \\&+ \gamma _\infty \frac{\big \Vert {h}\big \Vert ^2}{{\textsf {u}}_{\min }} + \gamma _1 \big \Vert {{\textsf {u}}- {\textsf {v}}}\big \Vert _1 + \gamma _2 \sum _{i = 1}^{m + n} \frac{1}{{\textsf {u}}_i^2}. \end{aligned}$$

Note that the first term is jointly convex as a sum of quadratic-over-linear or matrix fractional functions [14]. When ${\mathscr {H}}$ is a subset of the reproducing kernel Hilbert space associated to a positive definite kernel K, for a fixed ${\textsf {u}}$, the problem coincides with a standard kernel ridge regression problem. Thus, we can rewrite it in terms of dual variables $\varvec{\alpha }$, the kernel matrix K, $Y = (y_1, \ldots , y_{m + n})^\top$ and $U = (u_1, \ldots , u_{m + n})^\top$ as follows:

$$\begin{aligned} \min _{{\textsf {u}}\in {\mathscr {U}}} \max _{\varvec{\alpha }}&- \varvec{\alpha }^\top \big ({K + \frac{\gamma _\infty }{{\textsf {u}}_{\min }} U}\big ) \varvec{\alpha }+ 2 \varvec{\alpha }^\top Y + \sum _{i = 1}^{m + n} \frac{d_i}{{\textsf {u}}_i}\\&\text {dis}\big ({\left( { \tfrac{1}{{\textsf {u}}_i}}\right) _i, \left( {\tfrac{1}{{\textsf {v}}_i}}\right) _i}\big ) + \gamma _1 \big \Vert {{\textsf {u}}- {\textsf {v}}}\big \Vert _1 + \gamma _2 \sum _{i = 1}^{m + n} \frac{1}{{\textsf {u}}_i^2}. \end{aligned}$$

Solving for $\varvec{\alpha }$ yields the following convex optimization problem:

$$\begin{aligned} \min _{{\textsf {u}}\in {\mathscr {U}}} Y^\top \big ({K + \frac{\gamma _\infty }{{\textsf {u}}_{\min }} U}\big )^{-1} Y + \sum _{i = 1}^{m + n} \frac{d_i}{{\textsf {u}}_i} + \gamma _1 \big \Vert {{\textsf {u}}- {\textsf {v}}}\big \Vert _1 + \gamma _2 \sum _{i = 1}^{m + n} \frac{1}{{\textsf {u}}_i^2}. \end{aligned}$$

Standard descent methods such as SGD can be used to solve this problem. Note that the above can be further simplified using the upper bound $1/{{\textsf {u}}_{\min }} \le \sum _{i = 1}^{m + n} 1/{u_i}$.

1.3 A.3 Discrepancy estimation

First, note that if the ${\mathscr {P}}$-drawn labeled sample at our disposal is sufficiently large, we can reserve a sub-sample of size $n_1$ to train a relatively accurate model $h_{\mathscr {P}}$. Thus, we can subsequently reduce ${\mathscr {H}}$ to a ball ${\textsf {B}}(h_{\mathscr {P}}, r)$ of radius $r \sim \frac{1}{\sqrt{n_1}}$. This helps us work with a finer local labeled discrepancy since the maximum in the definition is then taken over a smaller set.

We do not have access to the discrepancy value $\text {dis}({\mathscr {P}}, {\mathscr {Q}})$, which defines $d_i$s. Instead, we can use the labeled samples from ${\mathscr {Q}}$ and ${\mathscr {P}}$ to estimate it. Our estimate $\widehat{d}$ of the discrepancy is given by

$$\begin{aligned} \widehat{d} = \max _{h \in {\mathscr {H}}} \left\{ \frac{1}{n} \sum _{i = m + 1}^{m + n} \ell (h(x_i), y_i) - \frac{1}{m} \sum _{i = 1}^m \ell (h(x_i), y_i) \right\} . \end{aligned}$$

(A.4)

Thus, for a convex loss $\ell$, the optimization problems for computing $\widehat{d}$ can be naturally cast as DC-programming problem, which can be tackled using the DCA algorithm [123] and related methods already discussed for sbest. For the squared loss, the DCA algorithms is guaranteed to converge to a global optimum [123].

Fig. 2

Alternate minimization procedure for best effort adaptation

By McDiarmid’s inequality, with high probability, $\left| {\text {dis}({\mathscr {P}}, {\mathscr {Q}}) - \widehat{d}}\right|$ can be bounded by $O(\sqrt{\frac{m + n}{mn}})$. More refined bounds such as relative deviation bounds or Bernstein-type bounds provide more favorable guarantee when the discrepancy is relatively small. When ${\mathscr {H}}$ is chosen to be a small ball ${\textsf {B}}(h_{\mathscr {P}}, r)$, our estimate of the discrepancy is further refined.

The optimization problem (A.4) can be equivalently solved via the following minimization:

$$\begin{aligned} \widehat{d}&= \min _{h \in {\mathscr {H}}} \left\{ \frac{1}{m} \sum _{i = 1}^m \ell (h(x_i), y_i) -\frac{1}{n} \sum _{i = m + 1}^{m + n} \ell (h(x_i), y_i)\right\} . \end{aligned}$$

The DCA solution for this problem then consists of solving a sequence of T convex optimization problems where $h_1 \in {\mathscr {H}}$ is chosen arbitrarity and where $h_{t + 1}$, $t \in [T]$ is obtained as follows

$$\begin{aligned} h_{t + 1}&\in \underset{h \in {\mathscr {H}}}{\arg \min }\ \left\{ \frac{1}{m} \sum _{i = 1}^m \ell (h(x_i), y_i) -\frac{1}{n} \sum _{i = m + 1}^{m + n} \nabla \ell (h_t(x_i), y_i) \cdot (h - h_t)\right\} . \end{aligned}$$

The second term of the objective is obtained by a linearization of the loss.

1.4 A.4 Pseudocode of alternate minimization procedure

1.5 A.5 $\alpha$-reweighting method

Let $d\! = \!\text {dis}({\mathscr {P}}, {\mathscr {Q}})$, $\widehat{d}$ and $\widehat{d} = \text {dis}(\widehat{\mathscr {Q}}, \widehat{\mathscr {P}})$. Consider the following simple, and in general suboptimal, choice of $\textsf {q}$ as a distribution defined by:

$$\begin{aligned} \overline{\textsf {q}}&= \frac{\alpha m}{m + n} \qquad \textsf {q}_i = {\left\{ \begin{array}{ll} \frac{\overline{\textsf {q}}}{m} = \frac{\alpha }{m + n} &{} \text {if } i \in [m];\\[.25cm] \frac{1 - \overline{\textsf {q}}}{n} = \frac{m (1 - \alpha ) + n}{(m + n) n} &{} \text {otherwise}, \end{array}\right. } \end{aligned}$$

where $\alpha = \Psi (1 - d)$ for some non-decreasing function $\Psi$ with $\Psi (0) = 0$ and $\Psi (1) = 1$. We will compare the right-hand side of the bound of Theorem 1, which we denote by B, with its right-hand side $B_0$ for $\textsf {q}$ chosen to be uniform over $S'$ corresponding to supervised learning on just $S'$:

$$\begin{aligned} B_0 = \mathcal {L}(\widehat{\mathscr {P}}, h) + 2 \mathfrak {R}_n(\ell \circ {\mathscr {H}}) + \sqrt{\frac{\log \frac{1}{\delta }}{2n}}. \end{aligned}$$

We now show that under some assumptions, we have $B - B_0 \le 0$. Thus, even for this sub-optimal choice of $\overline{\textsf {q}}$, under those assumptions, the guarantee of the theorem is then strictly more favorable than the one for training on $S'$ only, uniformly over $h \in {\mathscr {H}}$.

By definition of $\widehat{d}$, we can write:

$$\begin{aligned} \mathcal {L}(\textsf {q}, h)&= \overline{\textsf {q}}\mathcal {L}(\widehat{\mathscr {Q}}, h) + (1 - \overline{\textsf {q}}) \mathcal {L}(\widehat{\mathscr {P}}, h) \le \overline{\textsf {q}}\widehat{d} + \mathcal {L}(\widehat{\mathscr {P}}, h). \end{aligned}$$

By definition of the $\textsf {q}$-Rademacher complexity and the sub-additivity of the supremum, the following inequality holds:

$$\begin{aligned} \mathfrak {R}_{\textsf {q}}(\ell \circ {\mathscr {H}}) \le \overline{\textsf {q}}\mathfrak {R}_{m}(\ell \circ {\mathscr {H}}) + (1 - \overline{\textsf {q}}) \mathfrak {R}_{n}(\ell \circ {\mathscr {H}}). \end{aligned}$$

By definition of $\textsf {q}$, we can write:

$$\begin{aligned} \big \Vert {\textsf {q}}\big \Vert ^2_2 n = n \left[ {m \big ({\frac{\overline{\textsf {q}}}{m}}\big )^2 + n \big ({\frac{1 - \overline{\textsf {q}}}{n}}\big )^2}\right]&= \frac{n}{m} \overline{\textsf {q}}^2 + (1 - \overline{\textsf {q}})^2 \\&= 1 - 2 \overline{\textsf {q}}+ \frac{m + n}{m} \overline{\textsf {q}}^2\\&= 1 - (2 - \alpha ) \overline{\textsf {q}}\le 1 - \overline{\textsf {q}}. \end{aligned}$$

Thus, using the inequality $\sqrt{1 - x} \le 1 - \frac{x}{2}$, $x \le 1$, we have:

$$\begin{aligned} B - B_0&\le 2 \overline{\textsf {q}}\left[ {\mathfrak {R}_{m}(\ell \circ {\mathscr {H}}) - \mathfrak {R}_{n}(\ell \circ {\mathscr {H}}) }\right] + \overline{\textsf {q}}(d + \widehat{d}) + \left[ {\sqrt{1 - \overline{\textsf {q}}} - 1}\right] \left[ {\tfrac{\log \frac{1}{\delta }}{2n}}\right] ^{\frac{1}{2}}\\&\le 2 \overline{\textsf {q}}\left[ {\mathfrak {R}_{m}(\ell \circ {\mathscr {H}}) - \mathfrak {R}_{n}(\ell \circ {\mathscr {H}}) }\right] + \overline{\textsf {q}}(d + \widehat{d}) - \overline{\textsf {q}}\left[ {\tfrac{\log \frac{1}{\delta }}{8n}}\right] ^{\frac{1}{2}}. \end{aligned}$$

Suppose we are in the regime of relatively small discrepancies and that, given n, both the discrepancy and the empirical discrepancies are upper bounded as follows: $\max \{d, \overline{d}\} < \sqrt{\frac{\log 1/{\delta }}{32n}}$. Assume also that for $m \gg n$ (which is the setting we are interested in), we have $\mathfrak {R}_{m}(\ell \circ {\mathscr {H}}) - \mathfrak {R}_{n}(\ell \circ {\mathscr {H}}) \le 0$. Then, the first term is non-positive and, regardless of the choice of $\alpha < 1$, we have $B - B_0 \le 0$. Thus, even for this sub-optimal choice of $\overline{\textsf {q}}$, under some assumptions, the guarantee of the theorem is then strictly more favorable than the one for training on $S'$ only, uniformly over $h \in {\mathscr {H}}$.

Note that the assumption about the difference of Rademacher complexities is natural. For example, for a kernel-based hypothesis set ${\mathscr {H}}$ with a normalized kernel such as the Gaussian kernel and the norm of the weight vectors in the reproducing kernel Hilbert space (RKHS) bounded by $\Lambda$, it is known that the following inequalities hold: $\frac{1}{\sqrt{2}} \frac{ \Lambda }{\sqrt{m}} \le \mathfrak {R}_m({\mathscr {H}}) \le \frac{ \Lambda }{\sqrt{m}}$ [90]. Thus, for $m > 2n$, we have $\mathfrak {R}_m({\mathscr {H}}) - \mathfrak {R}_n({\mathscr {H}}) \le \frac{ \Lambda }{\sqrt{m}} - \frac{ \Lambda }{\sqrt{2n}} < 0$.

Appendix B: Domain adaptation

1.1 B.1 Proof of Lemma 8

Lemma 8

Let $\ell$ be the squared loss. Then, for any hypothesis $h_0$ in ${\mathscr {H}}$, the following upper bound holds for the labeled discrepancy:

$$ \text {dis}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) \le \overline{\text {dis}}_{{\mathscr {H}}\times \{ h_0 \} }(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) + 2 \delta _{{\mathscr {H}}, h_0}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}). $$

Proof

For any $h_0$, using the definition of the squared loss, the following inequalities hold:

$$\begin{aligned} \text {dis}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})&= \sup _{h \in {\mathscr {H}}} \, \left| { \mathbb E_{(x, y) \sim \widehat{\mathscr {P}}}[\ell (h(x), y)] - \mathbb E_{(x, y) \sim \widehat{\mathscr {Q}}}[\ell (h(x), y)] }\right| \\&\le \sup _{h \in {\mathscr {H}}} \, \left| { \mathbb E_{(x, y) \sim \widehat{\mathscr {P}}}[\ell (h(x), h_0(x))] - \mathbb E_{(x, y) \sim \widehat{\mathscr {Q}}}[\ell (h(x), h_0(x))]}\right| \\&+ \sup _{h \in {\mathscr {H}}} \, \bigg | \mathbb E_{(x, y) \sim \widehat{\mathscr {P}}}[\ell (h(x), y)] - \mathbb E_{(x, y) \sim \widehat{\mathscr {P}}}[\ell (h(x), h_0(x))] \\&+ \mathbb E_{(x, y) \sim \widehat{\mathscr {Q}}}[\ell (h(x), h_0(x))] - \mathbb E_{(x, y) \sim \widehat{\mathscr {Q}}}[\ell (h(x), y)] \bigg |\\&= \overline{\text {dis}}_{{\mathscr {H}}\times \{ h_0 \} }(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})\\&\quad + 2 \sup _{h \in {\mathscr {H}}} \, \left| { \mathbb E_{(x, y) \sim \widehat{\mathscr {P}}} \left[ {h(x) \big ({y - h_0(x)}\big )}\right] - \mathbb E_{(x, y) \sim \widehat{\mathscr {Q}}} \left[ {h(x) \big ({y - h_0(x)}\big )}\right] }\right| \\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad {(\text {def. of squared loss})}\\&= \overline{\text {dis}}_{{\mathscr {H}}\times \{ h_0 \} }(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) + 2 \delta _{{\mathscr {H}}, h_0}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}).\qquad \qquad \qquad \qquad \ \ {(\text {def. of local discrepancy})} \end{aligned}$$

This completes the proof. $\square$

1.2 B.2 Proof of Lemma 9

Lemma 9

Let $\ell$ be a loss function that is $\mu$-Lipschitz with respect to its second argument. Then, for any hypothesis $h_0$ in ${\mathscr {H}}$, the following upper bound holds for the labeled discrepancy:

$$\begin{aligned} \text {dis}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) \le \overline{\text {dis}}_{{\mathscr {H}}\times \{ h_0 \} }(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) + \mu \, \eta _{{\mathscr {H}}, h_0}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}). \end{aligned}$$

Proof

When the loss function $\ell$ is $\mu$-Lipschitz with respect to its second argument, we can use the following upper bound:

$$\begin{aligned} \text {dis}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})&= \sup _{h \in {\mathscr {H}}} \, \left| { \underset{(x, y) \sim \widehat{\mathscr {P}}}{\mathbb E}[\ell (h(x), y)] - \underset{(x, y) \sim \widehat{\mathscr {Q}}}{\mathbb E}[\ell (h(x), y)] }\right| \\&\le \sup _{h \in {\mathscr {H}}} \, \left| { \underset{(x, y) \sim \widehat{\mathscr {P}}}{\mathbb E}[\ell (h(x), h_0(x))] - \underset{(x, y) \sim \widehat{\mathscr {Q}}}{\mathbb E}[\ell (h(x), h_0(x))]}\right| \\&\quad + \sup _{h \in {\mathscr {H}}} \, \bigg | \underset{(x, y) \sim \widehat{\mathscr {P}}}{\mathbb E}[\ell (h(x), y)] - \underset{(x, y) \sim \widehat{\mathscr {P}}}{\mathbb E}[\ell (h(x), h_0(x))] \\&\quad + \underset{(x, y) \sim \widehat{\mathscr {Q}}}{\mathbb E}[\ell (h(x), h_0(x))] - \underset{(x, y) \sim \widehat{\mathscr {Q}}}{\mathbb E}[\ell (h(x), y)] \bigg |\\&\le \overline{\text {dis}}_{{\mathscr {H}}\times \{ h_0 \} }(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}}) + \mu \underset{(x, y) \sim \widehat{\mathscr {P}}}{\mathbb E}\left[ {\left| {y - h_o(x)}\right| }\right] + \mu \underset{(x, y) \sim \widehat{\mathscr {Q}}}{\mathbb E}\left[ {\left| {y - h_o(x)}\right| }\right] . \\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \ \quad {(\ell \text { assumed } \mu -\text {Lipschitz})} \end{aligned}$$

This completes the proof. $\square$

1.3 B.3 Sub-Gradients and estimation of unlabeled discrepancy terms

Here, we first describe how to compute the sub-gradients of the unlabeled weighted discrepancy term $\text {dis}(\textsf {q}', \textsf {p})$ that appears in the optimization problem for domain adaptation (), and similarly $\overline{\text {dis}}(\textsf {p}^0, (\textsf {q}, \textsf {q}'))$, in the case of the squared loss with linear functions. Next, we show how the same analysis can be used to compute the empirical discrepancy term $\overline{\text {dis}}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})$, which provides an accurate estimate of $\overline{d} = \overline{\text {dis}}({\mathscr {P}}, {\mathscr {Q}})$.

1.3.1 B.3.1 Sub-Gradients of unlabeled weighted discrepancy terms

Let $\ell$ be the squared loss and let ${\mathscr {H}}$ be the family of linear functions defined by ${\mathscr {H}}= \{x \mapsto {\textbf{w}}\cdot \varvec{\Phi }(x) :\big \Vert {{\textbf{w}}}\big \Vert _2 \le \Lambda \}$, where $\varvec{\Phi }$ is a feature mapping from ${\mathscr {X}}$ to $\mathbb {R}^k$.

We can analyze the unlabeled discrepancy term $\overline{\text {dis}}(\textsf {q}', \textsf {p})$ using an analysis similar to that of [21]. By definition of the unlabeled discrepancy, we can write:

$$\begin{aligned} \overline{\text {dis}}(\textsf {q}', \textsf {p})&= \sup _{h, h' \in {\mathscr {H}}} \left\{ \sum _{i = 1}^{n} \textsf {q}'_i \ell (h(x_{m + i}), h'(x_{m + i})) - \sum _{i = 1}^{m} \textsf {p}_i \ell (h(x_{i}), h'(x_{i}))\right\} \\&= \sup _{\big \Vert {{\textbf{w}}}\big \Vert _2 , \big \Vert {{\textbf{w}}'}\big \Vert _2 \le \Lambda } \left\{ \sum _{i = 1}^{n} \textsf {q}'_i \left[ {({\textbf{w}}- {\textbf{w}}') \cdot \varvec{\Phi }(x_{m + i})}\right] ^2 - \sum _{i = 1}^{m} \textsf {p}_i \left[ {({\textbf{w}}- {\textbf{w}}') \cdot \varvec{\Phi }(x_i)}\right] ^2\right\} \\&= \sup _{\big \Vert {{\textbf{u}}}\big \Vert _2 \le 2\Lambda } \left\{ \sum _{i = 1}^{n} \textsf {q}'_i \left[ {{\textbf{u}}\cdot \varvec{\Phi }(x_{m + i})}\right] ^2 - \sum _{i = 1}^{m} \textsf {p}_i \left[ {{\textbf{u}}\cdot \varvec{\Phi }(x_i)}\right] ^2\right\} \\ \end{aligned}$$

$$\begin{aligned}&= \sup _{\big \Vert {{\textbf{u}}}\big \Vert _2 \le 2\Lambda } \left\{ \sum _{i = 1}^{n} \textsf {q}'_i {\textbf{u}}^\top \varvec{\Phi }(x_{m + i}) \varvec{\Phi }(x_{m + i})^\top {\textbf{u}}- \sum _{i = 1}^{m} \textsf {p}_i {\textbf{u}}^\top \varvec{\Phi }(x_i) \varvec{\Phi }(x_i)^\top {\textbf{u}}\right\} \\&= \sup _{\big \Vert {{\textbf{u}}}\big \Vert _2 \le 2\Lambda } \left\{ {\textbf{u}}^\top \left[ {\sum _{i = 1}^{n} \textsf {q}'_i \varvec{\Phi }(x_{m + i}) \varvec{\Phi }(x_{m + i})^\top - \sum _{i = 1}^{m} \textsf {p}_i \varvec{\Phi }(x_i) \varvec{\Phi }(x_i)^\top }\right] {\textbf{u}}\right\} \\&= 4\Lambda ^2 \sup _{\big \Vert {{\textbf{u}}}\big \Vert _2 \le 1} {\textbf{u}}^\top {\textbf{M}}(\textsf {q}', \textsf {p}) {\textbf{u}}\\&= 4\Lambda ^2 \max \left\{ 0, \sup _{\big \Vert {{\textbf{u}}}\big \Vert _2 = 1} {\textbf{u}}^\top {\textbf{M}}(\textsf {q}', \textsf {p}) {\textbf{u}}\right\} \\&= 4\Lambda ^2 \max \left\{ 0, \lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )\right\} , \end{aligned}$$

where ${\textbf{M}}(\textsf {q}', \textsf {p}) = \sum _{i = 1}^{n} \textsf {q}'_i \varvec{\Phi }(x_{m + i}) \varvec{\Phi }(x_{m + i})^\top - \sum _{i = 1}^{m} \textsf {p}_i \varvec{\Phi }(x_i) \varvec{\Phi }(x_i)^\top$ and where $\lambda _{\max }$ $\big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )$ denotes the maximum eigenvalue of the symmetric matrix ${\textbf{M}}(\textsf {q}', \textsf {p})$. Thus, the unlabeled discrepancy $\overline{\text {dis}}(\textsf {q}', \textsf {p})$ can be obtained from the maximum eigenvalue of a symmetric matrix that is an affine function of $\textsf {q}'$ and $\textsf {p}$. Since $\lambda _{\max }$ is a convex function and since composition with an affine function preserves convexity, $\lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )$ is a convex function of $\textsf {q}'$ and $\textsf {p}$. Since the maximum of two convex function is convex, $\max \left\{ 0, \lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )\right\}$ is also convex.

Rewriting $\lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )$ as $\max _{\big \Vert {{\textbf{u}}}\big \Vert _2 = 1} {\textbf{u}}^\top {\textbf{M}}(\textsf {q}', \textsf {p}) {\textbf{u}}$ helps derive the sub-gradient of $\lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )$ using the sub-gradient calculation of the maximum of a set of functions:

$$\begin{aligned} \nabla _{(\textsf {q}', \textsf {p})} \lambda _{\max } \big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big ) = \begin{bmatrix} {\textbf{u}}^\top \varvec{\Phi }(x_{m + 1}) \varvec{\Phi }(x_{m + 1})^\top {\textbf{u}}\\ \vdots \\ {\textbf{u}}^\top \varvec{\Phi }(x_{m + n}) \varvec{\Phi }(x_{m + n})^\top {\textbf{u}}\\ -{\textbf{u}}^\top \varvec{\Phi }(x_1) \varvec{\Phi }(x_1)^\top {\textbf{u}}\\ \vdots \\ -{\textbf{u}}^\top \varvec{\Phi }(x_m) \varvec{\Phi }(x_m)^\top {\textbf{u}}\end{bmatrix} = \begin{bmatrix} \big ({\varvec{\Phi }(x_{m + 1}) \cdot {\textbf{u}}}\big )^2\\ \vdots \\ \big ({\varvec{\Phi }(x_{m + n}) \cdot {\textbf{u}}}\big )^2\\ -\big ({\varvec{\Phi }(x_1) \cdot {\textbf{u}}}\big )^2\\ \vdots \\ - \big ({\varvec{\Phi }(x_m) \cdot {\textbf{u}}}\big )^2 \end{bmatrix}, \end{aligned}$$

where ${\textbf{u}}$ is the eigenvector corresponding to the maximum eigenvalue of ${\textbf{M}}(\textsf {q}', \textsf {p})$. Alternatively, we can approximate the maximum eigenvalue via the softmax expression

$$\begin{aligned} f(\textsf {q}', \textsf {p}) = \frac{1}{\mu } \log \left[ {\sum _{j = 1}^k e^{\mu \lambda _j\big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )}}\right] = \frac{1}{\mu } \log \left[ {{{\,\textrm{Tr}\,}}\big ({ e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}}\big ) }\right] , \end{aligned}$$

where $e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}$ denotes the matrix exponential of $\mu {\textbf{M}}(\textsf {q}', \textsf {p})$ and $\lambda _j\big ({{\textbf{M}}(\textsf {q}', \textsf {p})}\big )$ the jth eigenvalue of ${\textbf{M}}(\textsf {q}', \textsf {p})$. The matrix exponential can be computed in $O(k^3)$ time by computing the singular value decomposition (SVD) of the matrix. We have:

$$\begin{aligned} \lambda _{\max } ({\textbf{M}}(\textsf {q}', \textsf {p})) \le f(\textsf {q}', \textsf {p}) \le \lambda _{\max } ({\textbf{M}}(\textsf {q}', \textsf {p})) + \frac{\log k}{\mu }. \end{aligned}$$

Thus, for $\mu = \frac{\log k}{\epsilon }$, $f(\textsf {q}', \textsf {p})$ provides a uniform $\epsilon$-approximation of $\lambda _{\max } ({\textbf{M}}(\textsf {q}', \textsf {p}))$. The gradient of $f(\textsf {q}', \textsf {p})$ is given for all $j \in [n]$ and $i \in [m]$ by

$$\begin{aligned}&\nabla _{\textsf {q}'_j} f(\textsf {q}', \textsf {p}) = \frac{\left\langle {e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}, \varvec{\Phi }(x_{m + j})\varvec{\Phi }(x_{m + j})^\top }\right\rangle }{{{\,\textrm{Tr}\,}}\big ({ e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}}\big ) } = \frac{\varvec{\Phi }(x_{m + j})^\top e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})} \varvec{\Phi }(x_{m + j})}{{{\,\textrm{Tr}\,}}\big ({ e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}}\big ) }\\&\nabla _{\textsf {p}_i} f(\textsf {q}', \textsf {p}) = - \frac{\left\langle {e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}, \varvec{\Phi }(x_{i})\varvec{\Phi }(x_{i})^\top }\right\rangle }{{{\,\textrm{Tr}\,}}\big ({ e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}}\big ) } = \frac{\varvec{\Phi }(x_{i})^\top e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})} \varvec{\Phi }(x_{i})}{{{\,\textrm{Tr}\,}}\big ({ e^{\mu {\textbf{M}}(\textsf {q}', \textsf {p})}}\big ) }. \end{aligned}$$

The sub-gradient of the unlabeled discrepancy term $\overline{\text {dis}}(\textsf {p}^0, (\textsf {q}, \textsf {q}'))$ or a smooth approximation can be derived in a similar, using the same analysis as above.

1.3.2 B.3.2 Estimation of unlabeled discrepancy terms

The unlabeled discrepancy $\overline{d} = \overline{\text {dis}}({\mathscr {P}}, {\mathscr {Q}})$ can be accurately estimated from its empirical version $\overline{\text {dis}}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})$ [84]. In view of the analysis of the previous section, we have

$$\begin{aligned} \overline{\text {dis}}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})&= 4 \Lambda ^2 \lambda _{\max } \big ({{\textbf{M}}(\widehat{\mathscr {P}}, \widehat{\mathscr {Q}})}\big )\\&= 4 \Lambda ^2 \lambda _{\max } \left( { \frac{1}{n} \sum _{i = 1}^{n} \varvec{\Phi }(x_{m + i}) \varvec{\Phi }(x_{m + i})^\top - \frac{1}{m} \sum _{i = 1}^{m} \varvec{\Phi }(x_i) \varvec{\Phi }(x_i)^\top }\right) . \end{aligned}$$

Thus, this last expression can be used in place of $\overline{d}$ in the optimization problem for domain adaptation.

Appendix C: Further details about experimental settings

In this section we provide further details on our experimental setup starting with best effort adaptation.

1.1 C.1 Best-effort adaptation

Recall that in this setting we have labeled data from both source and target, however the amount of labeled data from the source is much larger. We start by describing the baselines that we compare our algorithms with. For the best-effort adaptation problem two natural baselines are to learn a hypothesis solely on the target ${\mathscr {P}}$, or train solely on the source ${\mathscr {Q}}$. A third baseline that we consider is the $\alpha$-reweighted q as discussed in Section 3.2. Note, $\alpha =1$ corresponds to training on all the available data with a uniform weighting.

1.1.1 C.2 Simulated data

We first consider a simulated scenario where n samples from the target distribution ${\mathscr {P}}$ are generated by first drawing the feature vector x i.i.d. from a normal distribution with zero mean and spherical covariance matrix, i.e, $N(0, I_{d \times d})$. Given x, a binary label $y \in \{ -1, +1 \}$ is generated as $\text {sgn}(w_p \cdot x)$ for a randomly chosen unit vector $w_p \in \mathbb {R}^{d}$. For a fixed $\eta \in (0.5, 1)$, $m=1\mathord {,}000$ i.i.d. samples from the source distribution ${\mathscr {Q}}$ are generated by first drawing $(1-\eta )m$ examples from $N(0, I_{d \times d})$ and labeled according to $\text {sgn}(w_q \cdot x)$ where $\Vert w_p - w_q\Vert \le \epsilon$, for a small value of $\epsilon$. Notice that when $\epsilon$ is small, the $(1-\eta )m$ samples are highly relevant for learning the target ${\mathscr {P}}$. The remaining $\eta m$ examples from ${\mathscr {Q}}$ are all set to a fixed vector u and are labeled as $+1$. These examples represent the noise in ${\mathscr {Q}}$ and as $\eta$ increases the presence of such examples makes $\text {dis}({\mathscr {P}}, {\mathscr {Q}})$ larger. In our experiments we set $d = 20, \epsilon =0.01$, and vary $\eta \in \{0.05, 0.1, 0.15, 0.2\}$.

On the above adaptation problem we evaluate the performance of the previously discussed baselines with our proposed $\text {sbest}$ algorithm implemented via the alternate minimization, sbest-AM, and the DC-programming algorithms, sbest-DC, where the loss function considered is the logistic loss and the hypothesis set is the set of linear models with zero bias. For each value of $\eta$, the results are averaged over 50 independent runs using the data generation process described above.

Figure 3 shows the performance of the different algorithms for various values of the noise level $\eta$ and as the number of examples n from the target increases. As can be seen from the figure, both $\alpha$-reweighting and the baseline that trains solely on ${\mathscr {Q}}$ degrade significantly in performance as $\eta$ increases. This is due to the fact the $\alpha$-reweighting procedure cannot distinguish between non-noisy and noisy data points within the m samples generated from ${\mathscr {Q}}$.

In Fig. 4(Left) we plot the best $\alpha$ chosen by the $\alpha$-reweighting procedure as a function of n. For reference we also plot the amount of mass on the non-noisy points from ${\mathscr {Q}}$, i.e., $(1-\eta ) \cdot m/(m+n)$. As can be seen from the figure, as n increases the amount of mass selected over the source ${\mathscr {Q}}$ decreases. Furthermore, as expected this decrease is sharper as the amount of noise level increases. In particular, $\alpha$-reweighting is not able to effectively use the non-noisy samples from ${\mathscr {Q}}$.

On the other hand, both sbest-AM and sbest-DC are able to counter the effect of the noise by generating $\textsf {q}$-weightings that are predominantly supported on the non-noisy samples. In Fig. 4(Right) we plot the amount of probability mass that the alternate minimization and the DC-programming implementations of $\text {sbest}$ assign to the noisy data points.

As can be seen from the figure, the total probability mass decreases with n and is also decreasing with the noise levels. These results also demonstrate that our algorithms that compute a good q-weighting can do effective outlier detection since they lead to solutions that assign much smaller mass to the noisy points.

Fig. 3

Comparison of $\text {sbest}$ against the baselines on simulated data in the classification setting. As the noise rate and therefore the discrepancy between ${\mathscr {P}}$ and ${\mathscr {Q}}$ increases the performance of the baselines degrades. In contrast, both the alternate minimization and the DC-programming algorithms effectively find a good $\textsf {q}$-weighting and can adapt to the target

Fig. 4

(Left) Best $\alpha$ chosen by $\alpha$-reweighting as a function of n. (Right) Total probability mass assigned by $\text {sbest}$ to the noisy points

1.1.2 C.1.2 Real-world data: classification and regression

Classification Next we evaluate our proposed algorithms and baselines for three real-world datasets obtained from the UCI machine learning repository [31]. We first describe the datasets and our choices of the source and target domains in each case. The first dataset we consider is the Adult-Income dataset. This is a classification task where the goal is to predict whether the income of a given individual is greater than or equal to $\$50$K. The dataset has 32, 561 examples. We form the source domain ${\mathscr {Q}}$ by taking examples where the attribute gender equals ‘Male’ and the target domain ${\mathscr {P}}$ corresponds to examples where the gender is ‘Female’. This leads to 21, 790 examples from ${\mathscr {Q}}$ and 10, 771 examples from ${\mathscr {P}}$.

The second dataset we consider is the South-German-Credit dataset. This dataset consists of $1\mathord {,}000$ examples and the goal is to predict whether a given individual has good credit or bad credit. We form the source domain ${\mathscr {Q}}$ by condition on the residence attribute and taking all examples where the attribute value is in $\{3, 4\}$ (indicating that the individual has lived at the current residence for 3 or more than 4 years.) The target domain is formed by taking examples where the residence attribute value is in $\{1,2\}$. This split leads to 620 examples from ${\mathscr {Q}}$ and 380 training examples from ${\mathscr {P}}$.

The third dataset we consider is the Speaker-Accent-Recognition dataset. In this dataset the goal is to predict the accent of a speaker given the speech signal. We consider the source ${\mathscr {Q}}$ to be examples where the accent is ’US’ or ’UK’ and the target to be examples where the accent is in {’ES’, ’FR’, ’GE’, ’IT’}. This split leads to 150 training examples from ${\mathscr {Q}}$ and 120 training examples from ${\mathscr {P}}$.

In each case we randomly split the examples from ${\mathscr {P}}$ into a training set of $70\%$ examples and a test set of $20\%$ examples. The remaining $10\%$ of the data is used for cross validation. We provide results averaged over 10 such random splits. For the six tasks from the Newsgroups dataset we follow the same methodology as in [127] to create the tasks.

In each of the above three cases we consider training a logistic regression classifier and compare the performance of $\text {sbest}$ with the baselines that we previously discussed. The results are shown in Table 1 in the main paper.

Regression Next we consider the following regression datasets from the UCI repository.

The wind dataset [51] where the task is to predict wind speed from the given features. The source consists of data from months January to November and the target is the data from December. This leads to a total of 5, 500 examples from ${\mathscr {Q}}$, 350 examples from ${\mathscr {P}}$ used for training and validation and 200 examples from ${\mathscr {P}}$ for testing. We create 10 random splits by dividing the 300 examples from ${\mathscr {P}}$ into a train set of size 150 and a validation set of size 200.

The airline dataset is derived from [63]. We create the task of predicting the amount of time the flight is delayed from various features such as the arrival time, distance, whether or not the flight was diverted, and the day of the week. We take a subset of the data for the Chicago O’Haire International Airport (ORD) in 2008. The source and target consists of datat from different hours of the day. This leads to 16, 000 examples from ${\mathscr {Q}}$ and 500 examples from ${\mathscr {P}}$ (used as 200 for training and 300 for validation) and 300 examples for testing.

The gas dataset [31, 111, 126] where the task is to predict the concentration level from various sensor measurements. The dataset consists of pre-determined batches and we take the first six to be the source and the last batch of size 360, 000 as the target (600 for training and 1000 for validation and 1000 for testing).

The news dataset [31, 34] where the goal is to predict the popularity of an article. Our source data consists of articles from Monday to Saturday and the target consists of articles from Sunday. This leads to 32500 examples from the source and 2737 examples for the target (737 for training, 1000 for validation and 1000 for testing).

The traffic dataset from the Minnesota Department of Transportation [31, 75] where the goal is to predict the traffic volume. We create source and target by splitting based on the time of the day. This leads to 2200 examples from the source and 1000 examples from the test set (200 for training, 400 for validation and 400 for testing).

In each of the datasets above we create 10 random splits based on the shuffling of the training and validation set and report mean and average values over the splits. We compare as baselines the KMM [61] algorithm and the DM algorithm [21]. Since both the algorithms were originally designed for the setting when the target has no labels we modify them in the following way. We run KMM (DM) on the source vs. target data to get a weight distribution $\textsf {q}$ over the source data. Finally, we perform weighted loss minimization by using the weights in $\textsf {q}$ for the source and uniform 1/n weights on the target of size n. The results are shown in Table 4. As can be seen sbest consistently outperforms the baselines.

Table 4 MSE of the sbest algorithm against baselines

1.2 C.2 Fine-tuning tasks

In this section we demonstrate the effectiveness of our proposed algorithms for the purpose of fine-tuning pre-trained representations. In the standard pre-training/fine-tuning paradigm [108] a model is first pre-trained on a generalist dataset (which is identified as coming from distribution ${\mathscr {Q}}$). Once a good representation is learned, the model is then fine-tuned on a task specific dataset (generated from target ${\mathscr {P}}$). Two of the predominantly used fine-tuning approaches in the literature are last layer fine-tuning [67, 116] and full model fine-tuning [60]. In the former approach the representations obtained from the last layer of the pre-trained model are used to train a simple model (often a linear hypothesis) on the data coming from ${\mathscr {P}}$. In our experiments we fix the choice of the simple model to be a multi-class logistic regression model. In the latter approach, the model when train on ${\mathscr {P}}$, is initialized from the pre-trained model and all the parameters of the model are fine-tuned (via gradient descent) on the target distribution ${\mathscr {P}}$. In this section we explore the additional advantages of combining data from both ${\mathscr {P}}$ and ${\mathscr {Q}}$ during the fine-tuning stage via our proposed algorithms. There has been recent interest in carefully combining various tasks/data for the purpose of fine-tuning and avoid the phenomenon of “negative transfer” [2]. Our proposed theoretical results present a principled approach towards this.

To evaluate the effectiveness of our theory for this purpose, we consider the CIFAR-10 vision dataset [72]. The dataset consists of 50000 training and 10000 testing examples belonging to 10 classes. We form a pre-training task on data from ${\mathscr {Q}}$, by combining all the data belonging to classes: {’airplane’, ’automobile’, ’bird’, ’cat’, ’deer’, ’dog’}. The fine-tuning task consists of data belonging to classes: {’frog’, ’horse’, ’ship’, ’truck’}. We consider both the approaches of last layer fine-tuning and full-model fine-tuning and compare the standard approach of fine-tuning only using data from ${\mathscr {P}}$ with our proposed algorithms. We use $60\%$ of the data from the source for pre-training, and the remaining $40\%$ is used in fine-tuning.

We split the fine-tuning data from ${\mathscr {P}}$ randomly into a $70\%$ training set to be used in fine-tuning, $10\%$ for cross validation and and the remaining $20\%$ to be used as a test set. The results are reported over 5 such random splits. We perform pre-training on a standard ResNet-18 architecture [52] by optimizing the cross-entropy loss via the Adam optimizer. As can be seen in Table 2 both gapBoost and $\text {sbest}$ that combine data from ${\mathscr {P}}$ and ${\mathscr {Q}}$ lead to a classifier with better performance for the downstream task, however, sbest clearly outperforms gapBoost.

The second dataset we consider is the Civil Comments dataset [97]. This dataset consists of text comments in online forums and the goal is to predict whether a given comment is toxic or not. Each data point is also labeled with identity terms that describes which subgroup the text in the comment is related to. We create a subsample of the dataset where the target consists of examples from the data points where the identity terms is “asian” and the source is the remaining set of points. This leads to 394, 000 points from the source and 20, 000 points from the target. We create 5 random splits of the data by randomly partitioning the target data into 10, 000 examples for finetuning, 2000 for validation and 8000 for testing. We perform pre-training on a BERT-small model [29] starting from the default checkpoint as obtained from the standard tensorflow implementation of the model.

1.3 C.3 Domain adaptation

In this section we evaluate the effectiveness of our proposed $\text {best-da}$ objective for adaptation in settings where the target has very little to no labeled data. In order to do this we consider multi-domain sentiment analysis dataset of [12] that has been used in prior works on domain adaptation. The dataset consists of text reviews associated with a star rating from 1 to 5 for various different categories such as books, dvd, etc. We specifically consider four categories namely books, dvd, electronics, and kitchen. Inspired form the methodology adapted in prior works [21, 89], for each category, we form a regression task by converting the review text to a 128 dimensional vector and fitting a linear regression model to predict the rating. In order to get the features we first combine all the data from the four tasks and convert the raw text to a TF-IDF representation using scikit-learn’s feature extraction library [98]. Following this, we compute the top 5000 most important features by using scikit-learn’s feature selection library, that in turn uses a chi-squared test to perform feature selection. Finally, we project the obtained onto a 128 dimensional space via performing principal component analysis.

After feature extraction, for each task we fit a ridge regression model in the 128 dimensional space to predict the ratings. The predictions of the model are then defined as the ground truth regression labels. Following the above pre-processing we form 12 adaptation problems for each pair of distinct tasks: (TaskA, TaskB) where TaskA, TaskB are in {books, dvd, electronics, kitchen}. In each case we form the source domain (${\mathscr {Q}}$) by taking 500 labeled samples from TaskA and 200 labeled examples from TaskB. The target (${\mathscr {P}}$) is formed by taking 300 unlabeled examples from TaskB. To our knowledge, there exists no principled method for cross-validation in fully unsupervised domain adaptation. Thus, in our adaptation experiments, we used a small labeled validation set of size 50 to determine the parameters for all the algorithms. This is consistent with experimental results reported in prior work (e.g., [21]).

We compare our $\text {best-da}$ algorithm with the discrepancy minimization (DM) algorithm of [21], and the (GDM) algorithm, [22], which is a state of the art adaptation algorithm for regression problems. We also compare with the popular Kernel Mean Matching (KMM) algorithm, [61], for domain adaptation. the results averaged over 10 independent source and target splits, where we normalize the mean squared error (MSE) of $\text {best-da}$ to be 1.0 and present the relative MSE achieved by the other methods. The results show that in most adaptation problems, $\text {best-da}$ outperforms (boldface) or ties with (italics) existing methods.

1.3.1 C.3.1 Domain adaptation – covariate-shift

Here we perform experiments for domain adaptation only under covariate shift and compare the performance of our proposed best-da objective with previous state of the art algorithms. We again consider the multi-domain sentiment analysis dataset [12] from the previous section and in particular focus on the books category. We use the same feature representation as before and define the ground truth as $y = w^* \cdot x + \sigma ^2$ where $w^*$ is obtained by fitting a ridge regression classifier. We let the target be the uniform distribution over the entire dataset. We define the source as follows: for a fixed value of $\epsilon$, we pick a random hyperplane w and consider a mixture distribution with mixture weight 0.99 on the set $w \cdot x \ge \epsilon$ and the mixture weight of 0.01 on the set $w \cdot x < \epsilon$. The performance of best-da as compared to DM and KMM is shown in Table 5. As can be seen our proposed algorithm either matches or outperforms current algorithms.

Table 5 MSE achieved by $\text {best-da}$ as compared to DM and KMM on the covariate shift task for various values of $\epsilon$

Hyperparameters for the algorithms

For our proposed sbest and sbest-da algorithms the hyperparameters $\lambda _\infty , \lambda _1, \lambda _2$ were chosen via cross validation in the range $\{1e-3, 1e-2, 1e-1\} \cup \{0,1,2,\dots , 10\} \cup \{0, 1000, 2000, 10000, 50000, 100000\}$. The h optimization step of alternate minimization was performed using sklearn’s linear regression/logistic regression methods [98]. During full layer fine-tuning on ResNet/BERT models we use the Adam optimizer for the h optimization step with a learning rate of $1e-3$ used for the CIFAR-10 dataset and a learning rate of $1e-5$ for the BERT-small models.

For the q optimization we used projected gradient descent and the step size was chosen via cross validation in the range $\{1e-3, 1e-2, 1e-1\}$.

We re-implemented the gapBoost algorithm [127] in Python. Following the prescription by the authors of gapBoost we set the parameter $\gamma = 1/n$ where n is the size of the target. We tune parameters $\rho _S, \rho _T$ in the range $\{0.1, 0.2, \ldots ,1\}$ and the number of rounds of boosting in the range $\{5,10,15,20\}$. We also re-implemented baselines DM [21] and the GDM algorithm [22]. These DM algorithm was implemented via gradient descent and the second stage of the GDM algorithm was implemented via alternate minimization. The learning rates in each case searched in the range $\{1e-3, 1e-2, 1e-1\}$ and the regularization parameters were searched in the range $\{1e-3, 1e-2, 1e-1, 0, 10, 100\}$. The radius parameter for GDM was searched in the range [0.01, 1] in steps of 0.01. In line with our proposed algorithms, all baselines were implemented without incorporating a bias term.

To our knowledge, there exists no principled method for cross-validation in fully unsupervised domain adaptation. Thus, in our unsupervised adaptation experiments, we used a small labeled validation set of size 50 to determine the parameters for all the algorithms. This is consistent with experimental results reported in prior work [21, 22].