Smooth determinantal varieties and critical loci in multiview geometry

Abstract

Linear projections from $\mathbb {P}^k$ to $\mathbb {P}^h$ appear in computer vision as models of images of dynamic or segmented scenes. Given multiple projections of the same scene, the identification of sufficiently many correspondences between the images allows, in principle, to reconstruct the position of the projected objects. A critical locus for the reconstruction problem is a variety in $\mathbb {P}^k$ containing the set of points for which the reconstruction fails. Critical loci turn out to be determinantal varieties. In this paper we determine and classify all the smooth critical loci, showing that they are classical projective varieties.

1 Introduction

In this paper we classify the smooth determinantal varieties arising in the multiview geometry and computer vision settings as critical loci for reconstruction problems. Since critical loci and determinantal varieties belong to different research fields, it is mandatory to explain the relation between them.

Photos of static three-dimensional scenes taken from pinhole cameras are usually modelled by linear projections from ${{\mathbb {P}}}^{3}$ to ${{\mathbb {P}}}^{2}$ . Similarly, in computer vision, linear projections from ${{\mathbb {P}}}^{k}$ to ${{\mathbb {P}}}^{h},$ are used to describe videos or images of particular dynamic and segmented scenes ( [18, 19, 32]). For this reason a camera can be identified with a linear projection $\pi : \mathbb {P}^k \dashrightarrow \mathbb {P}^h$.

The reconstruction problem is the following: given a set of points in $\mathbb {P}^k$ with unknown coordinates, called scene, and n images of it in n target spaces ${{\mathbb {P}}}^{h_i}$, $i=1,...,n$, taken from unknown cameras, the goal is to recover the positions of cameras and scene points in the ambient space ${{\mathbb {P}}}^{k}$ .

Sufficiently many images and sufficiently many corresponding points in the given images should in principle allow for a successful projective reconstruction, where corresponding points in the targets are images of the same point in the scene. Nonetheless, there exist sets of points in the ambient space ${{\mathbb {P}}}^{k},$ for which the projective reconstruction fails. These configurations of points are called critical, which means that there exist other non projectively equivalent sets of points and cameras that give the same images in the target spaces.

Critical loci turn out to be algebraic varieties and have been studied by many authors, indeed there is a wide literature on the subject. With analysis ad hoc, in the classical case of projections from $\mathbb {P}^3$ to $\mathbb {P}^2$ [11, 16, 17, 24,25,24, 31], the critical loci can be twisted cubic curves [11], or quadric surfaces [23, 24]. In the case of projections onto $\mathbb {P}^2$ from higher dimensional spaces, [4, 5, 9], critical loci have been proven to be minimal degree varieties [9] for one projection, or, in a more general setting and under suitable genericity assumptions, either hypersurfaces, if the ambient space is odd dimensional, or determinantal varieties of codimension two if the ambient space is even dimensional [5].

Later, in [2, 7,8,8] the study of the ideal of critical loci has been formalized making use of the so-called Grassmann tensor introduced in [18]. A seminal case of this approach has been considered in [2], where the authors computed the equations of the critical locus for three projections from $\mathbb {P}^2$ to $\mathbb {P}^1$. In [6, 8] the case of three projections from $\mathbb {P}^4$ to $\mathbb {P}^2$ is studied in detail. When the projections are general enough, critical loci are shown to be Bordiga surfaces, and conversely, every Bordiga surface is shown to be critical for suitable choices of three projections. When the genericity assumptions are not fulfilled, critical loci are shown to be not irreducible, with components of different dimensions. Finally, in [7], critical loci which are hypersurfaces in the ambient space are investigated.

On the other hand, the classification of embedded smooth projective varieties is a classical problem in algebraic geometry. For low degree or dimension and codimension, the classical approach to a classification problem consists in applying suitable techniques to get a finite list of possible cases, followed by the construction of explicit examples in order to establish the effectiveness of the list. Determinantal varieties are quite classical varieties, whose study takes advantage of homological techniques. The seminal result in the subject is Hilbert-Burch Theorem, but it is worth mentioning the structure theorem of codimension 3 Gorenstein ideals by D. Buchsbaum and D. Eisenbud, or Buchsbaum-Rim and Eagon-Northcott complexes. See [1, 10, 12, 25, 26] for results on determinantal varieties, and for further references on the subject.

Under this view point, since critical loci are in the class of determinantal varieties, in this paper we approach in full generality the problem of determining which smooth determinantal varieties appear as critical loci and of classifying them. More precisely, we determine under what assumptions the critical locus for a reconstruction problem for n projections from $\mathbb {P}^k$ to $\mathbb {P}^{h_i}$ for $i=1, \dots, n$ is a smooth variety, and provide a complete and effective classification of smooth varieties that can be critical.

The classification results are summarized as follows, where n is the number of projections.

• $n=2$

  All smooth critical loci are minimal degree varieties. Conversely, with the only exception of Veronese surface in $\mathbb {P}^5$, every minimal degree variety embedded in $\mathbb {P}^k$, with codimension c, $c \le k \le 2c+1$, is the critical locus for suitable pairs of projections.

• $n=3$

  X in $\mathbb {P}^k$ is a smooth critical locus if and only if X is

  • a cubic plane curve, in the case of three projections from $\mathbb {P}^2$ to $\mathbb {P}^1$, $\mathbb {P}^1$, and $\mathbb {P}^1$;

  • a cubic surface in $\mathbb {P}^3$, in the case of three projections from $\mathbb {P}^3$ to $\mathbb {P}^1$, $\mathbb {P}^1$, and $\mathbb {P}^2$;

  • a Bordiga surface in $\mathbb {P}^4$ in the case of three projections from $\mathbb {P}^4$ to $\mathbb {P}^2$, $\mathbb {P}^2$, and $\mathbb {P}^2$.

• $n=4$

  Smooth critical loci are quartic determinantal surfaces in $\mathbb {P}^3$, in the case of $4-$uples of projections from $\mathbb {P}^3$ to $\mathbb {P}^1$, $\mathbb {P}^1$, $\mathbb {P}^1$, and $\mathbb {P}^1$, containing four pairwise skew lines. Conversely, given four pairwise skew lines, it is possible to construct a smooth determinantal surface of degree 4 through them that is the critical locus for a suitable reconstruction problem as above.

The plan of the paper is as follows. In Sect. 2, we introduce the setting of multiple view geometry and we recall the construction of the Grassmann tensor. In Sect. 3, we introduce critical loci and determine the generators of their ideals, showing in particular that critical loci are determinantal varieties. In Sect. 4, we give some numerical bounds for critical loci to be smooth, and in particular we show that a critical locus can be smooth only if the number n of projections is at most 4. The remaining Sects. 5, 6, 7 are devoted to the study of critical loci in the cases $n=2, n=3, n=4$, respectively.

2 On multiview geometry and Grassmann tensors

In this section we fix notation and terminology and give a short overview of classical facts in Computer Vision related to the problem of projective reconstruction of scenes and cameras from multiple views.

In this context, a camera P is a linear projection from ${{\mathbb {P}}}^{k}$ onto ${{\mathbb {P}}}^{h},$ from a linear subspace C of dimension $k-h-1$, called center of projection. The target space ${{\mathbb {P}}}^{h}$ is called view. A scene is a set of points ${\mathbf {X}}_i\in {{\mathbb {P}}}^{k}, i=1, \dots , N$.

Using homogeneous coordinates in ${{\mathbb {P}}}^{k}$ and ${{\mathbb {P}}}^{h},$ we identify P with a $(h+1) \times (k+1)$ matrix of maximal rank, defined up to a multiplicative constant. Hence C comes out to be the right annihilator of P.

Let us consider a set of n cameras $P_j:{{\mathbb {P}}}^{k}\setminus C_{j} \rightarrow {{\mathbb {P}}}^{h_j}$ projecting the same scene in $\mathbb {P}^k$ and the corresponding set of images in the different target spaces. In this setting, proper linear subspaces $L_i \subseteq \mathbb {P}^{h_i}, i=1, \dots, n$, are said to be corresponding if there exists at least a point ${\mathbf {X}} \in {{\mathbb {P}}}^{k}$ such that $P_i({\mathbf {X}})\in L_i$ for all $i=1, \dots, n$.

In the context of multiple view geometry, the problem of projective reconstruction of a scene, given multiple images of it, is the following: given sufficiently many scene points in $\mathbb {P}^k$ and identified a suitable number of corresponding subspaces on each image, one wants to get the projection matrices (up to projective transformations), i.e. the cameras, and the coordinates in $\mathbb {P}^k$ of the scene points.

2.1 The Grassmann tensors

Hartley and Schaffalitzky, [18], have constructed a set of multiview tensors, called Grassmann tensors, encoding the relations between sets of corresponding subspaces. We recall here the basic elements of their construction.

We consider n projections $P_j:{{\mathbb {P}}}^{k}\setminus {C_j} \rightarrow \mathbb {P}^{h_j},$ $j = 1, \dots , n,$ with centers $C_{1}, \dots , C_{n}$.

First generality assumption: we assume that the intersection $C_1 \cap \dots \cap C_n$ is empty.

Let $L_j \subseteq \mathbb {P}^{h_j}$ be a general linear subspaces of codimension $\alpha _j$, $j=1, \dots , n$. We say that $(L_1, \dots , L_n)$ is a n–tuple of corresponding subspaces if and only if $\overline{(P_1)^{-1}(L_1)} \cap \dots \cap \overline{(P_n)^{-1}(L_n)}$ is not empty, where ${\overline{Y}}$ is the Zariski closure of Y. We allow $\alpha _j$ to be equal to 0 for some j. If this happens, the associated view does not impose any constrain to the reconstruction problem, and so the effect of setting $\alpha _j = 0$ is to decrease the number of views.

We remark that the Computer Vision community uses a slightly different definition of corresponding spaces: the spaces are said to be corresponding if $(P_1)^{-1}(L_1) \cap \dots \cap (P_n)^{-1}(L_n)$ is not empty. The difference is that the centers $C_j$ are not considered when the inverse images are intersected in this setting, while we prefer to include them, so to get projective varieties, and not only open subsets of them.

From the Grassmann formula, if $\sum _j \alpha _j = k+1,$ the existence of points in the previous intersection gives a constrain which allows us to construct the Grassmann tensor. Hartley and Schaffalitzky call the n–tuple $(\alpha _1, \dots , \alpha _n)$ a profile for the reconstruction problem. We remark that we allow $\alpha _j = 0$, too, while, in [18], $\alpha _j \ge 1$ for every $j=1, \dots , n$.

Let $L_1, \dots , L_n$ be n general linear subspaces as above and let $S_j$ be the maximal rank matrix of type $(h_j+1) \times (h_j - \alpha _j +1)$ whose columns are a basis for $L_j$. By definition, if the $L_j$’s are corresponding subspaces, there exists a point ${\mathbf {Y}} \in {{\mathbb {P}}}^{k}$ such that $P_j({\mathbf {Y}})\in L_j$ for every j. In other words there exist n vectors $\mathbf {v_j} \in \mathbb {C}^{h_j -\alpha _j +1}$ $j = 1,\dots ,n$ such that:

$$\begin{aligned} \left( \begin{array}{cccccc} P_1 &{} S_1 &{} 0 &{} \dots &{} 0 &{} 0 \\ P_2 &{} 0 &{} S_2 &{} \dots &{} 0 &{} 0 \\ \vdots &{} &{} &{} &{} &{} \vdots \\ P_n &{} 0 &{} 0 &{} \dots &{} 0 &{} S_n \\ \end{array} \right) \left( \begin{array}{c} {\mathbf {Y}} \\ \mathbf {v_1} \\ \mathbf {v_2} \\ \vdots \\ \mathbf {v_n} \\ \end{array} \right) =\left( \begin{array}{c} 0 \\ 0 \\ \vdots \\ 0 \\ \end{array} \right) . \end{aligned}$$

(1)

The coefficient matrix $T^{P_1, \dots , P_n}_{S_1 , \dots , S_n}$ is square of order $n + \sum h_j = k + 1 + \sum (h_j - \alpha _j + 1)$, where the left side is the number of rows, the right side is the number of columns, and they coincide due to our assumptions on the profile. The existence of a non–trivial solution $({\mathbf {Y}}, \mathbf {v_1},\dots ,\mathbf {v_n})$ of system (1) implies that the determinant of $T^{P_1, \dots , P_n}_{S_1 , \dots , S_n}$ is zero.

The determinant ${\mathcal {T}}^{P_1, \dots , P_n}(L_1, \dots , L_n) = \det (T^{P_1, \dots , P_n}_{S_1, \dots , S_n})$ can be thought of as a n–linear form (tensor) in the Plücker coordinates of the spaces $L_j$’s, in the corresponding Grassmann variety. This tensor is called Grassmann tensor. From the above discussion, it follows that this tensor vanishes if and only if the linear spaces $L_1, \dots , L_n$ are corresponding. In [18], the authors show that the Grassmann tensor allows the reconstruction of the projection matrices, up to the only case when all target spaces are $\mathbb {P}^1$. For this reason, the Computer Vision community does not consider the case above in reconstruction problems.

3 Critical loci and their ideals

Roughly speaking, one guesses that the reconstruction problem can be successfully solved if sufficiently many views and sufficiently many sets of corresponding points in the given views are known. This is generally true, but even in the classical set–up of two projections from ${{\mathbb {P}}}^{3}$ to ${{\mathbb {P}}}^{2}$ one can have non projectively equivalent pairs of scenes and cameras that produce the same images in the view planes, thus preventing reconstruction. Such configurations and the loci they describe are referred to as critical. In [5], critical loci for projective reconstruction of camera centers and scene points from multiple views for projections from ${{\mathbb {P}}}^{k}$ to ${{\mathbb {P}}}^{2}$ have been introduced and studied.

Now we recall the basic definition.

Definition 3.1

Given n projections $Q_i: \mathbb {P}^k \dashrightarrow \mathbb {P}^{h_i}$, a set of points $\{ {\mathbf {X}}_1, \dots , {\mathbf {X}}_N\}$ in ${{\mathbb {P}}}^{k}$ is said to be a critical configuration for projective reconstruction for $Q_1, \dots , Q_n$ if there exists another set of n projections $P_i : \mathbb {P}^k \dashrightarrow \mathbb {P}^{h_i}$ and another set $\{{\mathbf {Y}}_1, \dots , {\mathbf {Y}}_N \} \subset {{\mathbb {P}}}^{k}$, non-projectively equivalent to $\{ {\mathbf {X}}_1, \dots , {\mathbf {X}}_N \},$ such that, for all $i = 1, \dots , n$ and $j = 1, \dots , N$, we have $P_i({\mathbf {Y}}_j) = Q_i({\mathbf {X}}_j)$, up to homography in the targets. The two sets $\{{\mathbf {X}}_j\}$ and $\{{\mathbf {Y}}_j\}$ are called conjugate critical configurations, with associated conjugate projections $\{Q_i\}$ and $\{P_i\}$.

In Proposition 3.1, we prove that points in critical configurations fill an algebraic variety, called critical locus ${\mathcal {X}}$, whose ideal can be obtained by making use of the Grassmann tensor introduced above.

Indeed, the Grassmann tensor ${\mathcal {T}}^{P_1,\dots ,P_n}(L_1, \dots , L_n)$ encodes the algebraic relations between corresponding subspaces in the different views of the projections $P_1,\dots ,P_n$. Hence by definition of critical set, if $\{{\mathbf {X}}_j,{\mathbf {Y}}_j\}$ are conjugate critical configurations, then, for each j, the projections $Q_1({\mathbf {X}}_j), \dots , Q_n({\mathbf {X}}_j)$ are corresponding points not only for the projections $Q_1, \dots ,Q_n,$ but for the projections $P_1, \dots ,P_n$, too.

Following the construction above, we first choose a point ${\mathbf {X}}$ in the critical locus and a profile $(\alpha _1, \dots , \alpha _n)$, where $\alpha _i$ is the codimension of $L_i$ in $\mathbb {P}^{h_i}$, for $i=1, \dots , n$. If $Q_i({\mathbf {X}}) \in L_i$, for every $i=1, \dots , n$, then ${\mathcal {T}}^{P_1,\dots ,P_n}(L_1, \dots , L_n) = 0$. The previous condition is fulfilled if $L_i$ is spanned by $Q_i({\mathbf {X}})$ and any other $h_i-\alpha _i$ independent points in $\mathbb {P}^{h_i}$. So, we can suppose

$$\begin{aligned} S_i = \left( \begin{array}{cccc} Q_i({\mathbf {X}})&{\mathbf {x}}_{i1}&\dots&{\mathbf {x}}_{i,h_i-\alpha _i} \end{array} \right) = \left( \begin{array}{cc} Q_i({\mathbf {X}})&S'_i \end{array} \right) \end{aligned}$$

of maximal rank $h_i-\alpha _i+1$, that is to say, $S'_i$ is a general $(h_i+1) \times (h_i-\alpha _i)$ matrix of rank $h_i-\alpha _i$. Due to this choice, the matrix $T^{P_1, \dots , P_n}_{S_1, \dots , S_n}$ becomes

$$\begin{aligned} T^{P_1, \dots , P_n}_{S_1, \dots , S_n} = \left( \begin{array}{cccccccccc} P_1 &{} Q_1({\mathbf {X}}) &{} S'_1 &{} 0 &{} 0 &{} 0 &{} \dots &{} 0 &{} 0 &{} 0 \\ P_2 &{} 0 &{} 0 &{} Q_2({\mathbf {X}}) &{} S'_2 &{} 0 &{} \dots &{} 0 &{} 0 &{} 0 \\ \vdots &{} &{} &{} &{} &{} &{} &{} &{} &{} \vdots \\ P_n &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} \dots &{} 0 &{} Q_n({\mathbf {X}}) &{} S'_n \end{array} \right) . \end{aligned}$$

The determinant $\det (T^{P_1,\dots , P_n}_{S_1, \dots , S_n})$ is a sum of products of maximal minors of $S'_1, \dots , S'_n$, and maximal minors of the matrix

$$\begin{aligned} M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n} = \left( \begin{array}{ccccccc} P_1 &{} Q_1({\mathbf {X}}) &{} 0 &{} 0 &{} \dots &{} 0 &{} 0 \\ P_2 &{} 0 &{} Q_2({\mathbf {X}}) &{} 0 &{} \dots &{} 0 &{} 0 \\ \vdots &{} &{} &{} &{} &{} &{} \vdots \\ P_n &{} 0 &{} 0 &{} 0 &{} \dots &{} 0 &{} Q_n({\mathbf {X}}) \end{array} \right) . \end{aligned}$$

(2)

Such a matrix is a $(n+ \displaystyle \sum _{i=1}^n h_i) \times (n+k+1)$ matrix, the last n columns of which are of linear forms, while the first $k+1$ columns are of constants.

More explicitly, if we consider $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ as a block matrix, the coefficients are the minors obtained by delating $h_i-\alpha _i$ rows from the i–th block

$$\begin{aligned} \left( \begin{array}{ccccccc} P_i&0&\dots&0&Q_i({\mathbf {X}})&0&\dots \end{array} \right) , \end{aligned}$$

for every block.

If we allow the profile to change, because ${\mathbf {X}}$ is in the critical locus independently from the profile, we get all the possible maximal minors of $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$. The discussion above is part of the proof of the following result.

Proposition 3.1

The ideal $I({\mathcal {X}})$ of the critical locus ${\mathcal {X}}$ is generated by the maximal minors of $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$, and so ${\mathcal {X}}$ is a determinantal variety. Moreover, ${\mathcal {X}}$ contains the centers of the projections $Q_j$’s.

Proof

We only have to prove that the center $C'_j$ of $Q_j$ is contained in ${\mathcal {X}}$ for every $j=1, \dots , n$. $C'_j$ is the zero locus of $Q_j({\mathbf {X}})$, and so $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ drops rank at every point in $C'_j$, for each j. $\square$

We remark that, if $\alpha _i \ge 1$, then some maximal minors of $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ do not appear in $\det (T^{P_1,\dots , P_n}_{S_1, \dots , S_n})$ for whatever profile, and so they should not be among the generators of $I({\mathcal {X}})$. This fact supports our choice to allow $\alpha _i = 0$.

From the first generality assumption, it follows both that the first $k+1$ columns of $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ are linearly independent, and that the linear forms in the last n columns of the above matrix span a linear space of dimension $k+1$ in $R_1 = \left( R = K[x_0,\dots , x_k]\right) _1$, where $\mathbb {P}^k = \text{ Proj }(R)$. In fact, no point is common to either the centers of the $P_i$’s or of the $Q_j$’s.

As in [8], we write the matrix $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ as the following block matrix

$$\begin{aligned} M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n} = \left( \begin{array}{cc} A &{} B \\ C &{} D \end{array} \right) \end{aligned}$$

where A is a $(n -k-1+ \displaystyle \sum _{i=1}^n h_i) \times (k+1)$ matrix, B is a $(n -k-1+ \displaystyle \sum _{i=1}^n h_i) \times n$ matrix, C is an order $(k+1)$ square matrix, and, finally, D is a $(k+1) \times n$ matrix.

Due to the first generality assumption, $\left( \begin{array}{c} A \\ C \end{array} \right)$ has rank $k+1$. Hence, it contains an invertible square submatrix of order $k+1$. Up to row exchanges, we can assume that the last $k+1$ rows give us the required invertible matrix, so we will assume C to be invertible.

By performing elementary operations on columns and rows, we can reduce $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ to the following easier form

$$\begin{aligned} \left( \begin{array}{cc} 0 &{} N_{{\mathcal {X}}} \\ I_{k+1} &{} 0 \end{array} \right) \end{aligned}$$

where $N_{{\mathcal {X}}} = B - AC^{-1}D$ is a $(n -k-1+ \displaystyle \sum _{i=1}^n h_i) \times n$ matrix of linear forms. Furthermore, the maximal minors of $M^{P_1, \dots , P_n}_{Q_1, \dots , Q_n}$ span the same ideal as the maximal minors of $N_{{\mathcal {X}}}$. Hence, we have the following result.

Corollary 3.1

$I({\mathcal {X}})$ is generated by the maximal minors of $N_{{\mathcal {X}}} = B - A C^{-1} D$, with the same notations as above.

Since the critical locus ${\mathcal {X}}$ is a determinantal variety whose ideal is generated by the maximal minors of a matrix of linear forms, the expected dimension of ${\mathcal {X}}$ is

$$\begin{aligned} ed_{{\mathcal {X}}} = k - \left( 1 + (n -k-1+ \sum _{i=1}^n h_i) - n\right) = 2k - \sum _{i=1}^n h_i. \end{aligned}$$

(3)

From Porteous’s formula ( [1], formula 4.2, p. 86),we get, if $\dim ({\mathcal {X}}) = ed_{{\mathcal {X}}}$,

$$\begin{aligned} \deg ({\mathcal {X}}) = \left( {\begin{array}{c}n-k-1+\sum _{i=1}^n h_i\\ n-1\end{array}}\right) . \end{aligned}$$

(4)

Second generality assumption: we assume projections $P_1, \dots , P_n$, and $Q_1, \dots , Q_n$ are general enough to guarantee that the critical locus ${\mathcal {X}}$ has the expected dimension $2k - \sum _{i=1}^n h_i$.

In Computer Vision, real experiments are performed on noisy measurements, and computer simulations are performed on randomly generated data. Both practises have the effect of producing projections that are as general as possible. The associated critical locus has then the smallest dimension allowed by the number and dimension of the views. In particular, the dimension of the critical locus is the expected one when the views satisfy the bounds we discuss in next section. Nevertheless, it is worthwhile to analyse the cases in which this assumption is not fulfilled, both from a Computer Vision point of view and from the geometric one. In fact, from the Computer Vision perspective, when the projections are close to cases in which the critical locus has dimension bigger than expected, that is to say the second generality assumption is not fulfilled, the reconstruction procedure is numerically unstable, in general, and so one tries to avoid such cases. On the other hand, from a geometric point of view, such cases are quite interesting and a full classification seems to be not available, yet. For example, in [17] the authors study in details the case of three projections from $\mathbb {P}^3$ to $\mathbb {P}^2$, while in [6], the case of three projections from $\mathbb {P}^4$ to $\mathbb {P}^2$ is considered. In both papers, the classification of the possible cases is achieved by ad hoc arguments and is quite involved.

From now on, every time we assert we are in the general case, we assume that both the generality assumptions hold.

4 Numerical bounds

In this section, we deduce both a lower bound for $\sum h_i$, and an upper bound on the number n of views to get smooth critical loci.

Lemma 4.1

In the same notations as above, we have

$$\begin{aligned} k+1 \le \sum _{i=1}^n h_i. \end{aligned}$$

(5)

Proof

As $0 \le \alpha _i \le h_i$ for every $i=1, \dots , n$, and $\sum _{i=1}^n \alpha _i = k+1$, then we have $k + 1 \le \displaystyle \sum _{i=1}^n h_i$. $\square$

We are interested in studying the case ${\mathcal {X}}$ is irreducible and non–singular. To begin, we relate the projection centers to singular critical loci.

Lemma 4.2

If two centers of the projections $Q_1, \dots , Q_n$ intersect, the critical locus is singular.

Proof

If the centers of $Q_1$ and $Q_2$, for example, have a common point, two columns of matrix (2) vanish, and so its rank is at most $k+n-1$. From generalities on determinantal varieties, it follows that the critical locus is singular. $\square$

Now, we can compute an upper bound on the number n of views to get an associated smooth critical locus.

Theorem 4.1

Let ${\mathcal {X}}$ be the codimension $c \ge 2$ critical locus for a couple of $n \ge 4$ projections $P_1, \dots , P_n$ and $Q_1, \dots , Q_n$ from $\mathbb {P}^k$ to $\mathbb {P}^{h_i}$, $i=1, \dots , n$. Then, either ${\mathcal {X}}$ is not irreducible, or is singular.

Proof

The center $C_i$ of $Q_i$ has dimension $k - h_i -1$. Since $C_i \subseteq {\mathcal {X}}$, we have that $\dim C_i \le \dim {\mathcal {X}}$, and so $k - h_i - 1 \le k - c$, or equivalently,

$$\begin{aligned} h_i \ge c-1. \end{aligned}$$

Let us assume that $c-1 \le h_1 \le \dots \le h_n$.

If $h_1 = c-1$, then the center $C_1$ of $Q_1$ has codimension c and is contained in ${\mathcal {X}}$. Then, ${\mathcal {X}}$ is not irreducible.

Assume now that $h_1 \ge c$. Since $\dim {\mathcal {X}} = 2k - \sum _{i=1}^n h_i = k-c$, we get

$$\begin{aligned} k = \sum _{i=1}^n h_i -c. \end{aligned}$$

On the other hand, the center $C_ j$ of projection $Q_j$ has dimension

$$\begin{aligned} \dim C_j = k - h_j -1 = \sum _{i \not = j} h_i - c -1. \end{aligned}$$

It is easy to check that $(n-2)c \ge c+2$ for every $n \ge 4$ and $c \ge 2$. On the other hand, since $h_i \ge c$, we have $\sum _{i=1}^{n-2} h_i \ge c+2$. This last inequality implies that $\dim (C_n) + \dim (C_{n-1}) - k \ge 0$, and so $C_n$ and $C_{n-1}$ meet. This implies that ${\mathcal {X}}$ is singular, as stated. $\square$

The same proof allows us to state also the following result.

Theorem 4.2

Let ${\mathcal {X}}$ be the codimension 1 critical locus for a couple of $n \ge 5$ projections $P_1, \dots , P_n$ and $Q_1, \dots , Q_n$ from $\mathbb {P}^k$ to $\mathbb {P}^{h_i}$, $i=1, \dots , n$. Then, either ${\mathcal {X}}$ is not irreducible, or is singular.

Hence, we have to study the cases $n=2$ and $n=3$, for every codimension c and $n=4$ for $c=1$, only.

5 The $n = 2$ view case

In this section, we want to prove that, under some mild assumptions, the critical locus for $n = 2$ is a smooth and irreducible variety of minimal degree, and, conversely, that every smooth irreducible variety of minimal degree, but the Veronese surface, is critical for a suitable couple of projections. In this way, we classify all smooth critical loci in Computer Vision for 2 views. E.g., when the codimension is 1, the critical locus for two projections from $\mathbb {P}^3$ to $\mathbb {P}^2$ is a quadric surface, and this is well–known in the Computer Vision community; when $c=2$, the critical locus for two projections from $\mathbb {P}^4$ to $\mathbb {P}^3$ is a rational normal scroll; when $c=3$ the critical locus for two projections from $\mathbb {P}^5$ to $\mathbb {P}^4$ is either $\mathbb {P}({\mathcal {O}}_{\mathbb {P}^1}(2) \oplus {\mathcal {O}}_{\mathbb {P}^1}(2))$, or $\mathbb {P}({\mathcal {O}}_{\mathbb {P}^1}(1) \oplus {\mathcal {O}}_{\mathbb {P}^1}(3))$.

When $n=2$, we have $h_1 + h_2 = k + c$. Moreover, $k > h_2 \ge h_1 \ge c+1$. We remark that the last inequality on the right is a consequence of the previous equality.

Now, we prove that the critical locus is a variety of minimal degree.

Proposition 5.1

In the general case, the codimension c critical loci for two views are minimal degree varieties.

Proof

Corollary 3.1 implies that the ideal $I({\mathcal {X}})$ is generated by the maximal minors of $N_{{\mathcal {X}}}$ whose type is $(c+1) \times 2$, and so it is generated by quadrics. Furthermore, from Eq. (4), we get that $\deg ({\mathcal {X}}) = 1+c$. This description proves that ${\mathcal {X}} \subseteq \mathbb {P}^k$ is a minimal degree variety (see [13]). $\square$

The generality assumption in Proposition 5.1 implies that the minors of $N_{{\mathcal {X}}}$ define a variety of the expected codimension c.

From the classification of minimal degree varieties in [13], we get that ${\mathcal {X}}$ is singular as soon as $k \ge 2(c+1)$. Hence, smooth irreducible varieties of minimal degree that can be critical loci are embedded in $\mathbb {P}^k$ for $c+2 \le k \le 2c+1$.

Now, we consider the converse of Proposition 5.1.

Proposition 5.2

With the only exception of Veronese surfaces in $\mathbb {P}^5$, every codimension c minimal degree variety embedded in $\mathbb {P}^k$ with $c+2 \le k \le 2c+1$, is the critical locus for a suitable pair of projections.

Proof

Let us consider matrix M in the case of 2 projections, and the matrix $N_{{\mathcal {X}}}$ we obtain from it, as discussed in Sect. 2. In the 2 view case, we have

$$\begin{aligned} M = \left( \begin{array}{ccc} P_1 &{} Q_1(X) &{} 0 \\ P_2 &{} 0 &{} Q_2(X) \end{array} \right) = \left( \begin{array}{cc} A &{} B \\ C &{} D \end{array} \right) , \end{aligned}$$

where $P_1$ ($P _2$, respectively) is a $(h_1+1) \times (k+1)$ ($(h_2+1) \times (k+1)$, respectively) full rank matrix, and, up to transposition, $Q_1(X) = (Q_{11}(X), \dots , Q_{1,h_1+1}(X))$ and $Q_2(X) = (Q_{21}(X), \dots , Q_{2,h_2+1}(X))$ and the linear forms in each one of them are linearly independent. Moreover, A is of type $(c+1) \times (k+1)$, C is of type $(k+1) \times (k+1)$, and we assume it is invertible, B is of type $(c+1) \times 2$, and finally, D is of type $(k+1) \times 2$. The assumption on the rank of C is always fulfilled up to collecting rows of M in a different way. Notice that we can take all rows from $P_2$ and make suitable exchanges among the rows of $P_1$ so to get that the last $k+1$ rows of $\left( \begin{array}{c} A \\ C \end{array} \right)$ are linearly independent.

We recall that $h_i \ge c+1$ for $i=1,2$.

When computing $N_{{\mathcal {X}}}$, we get

$$\begin{aligned} N_{{\mathcal {X}}} = \left( \left( \begin{array}{c} Q_{11}(X) \\ \vdots \\ Q_{1,c+1}(X) \end{array} \right) \right. \left. - E \left( \begin{array}{c} Q_{1,c+2}(X) \\ \vdots \\ Q_{1,h_1+1}(X) \end{array} \right) \right| \left. - F \left( \begin{array}{c} Q_{21}(X) \\ \vdots \\ Q_{2,h_2+1}(X) \end{array} \right) \right) \end{aligned}$$

where $A C^{-1} = (E \vert F)$. It is evident that the first column of $N_{{\mathcal {X}}}$ depends on the first view, and the second from the second view.

Let us consider now a codimension c variety V of minimal degree, and let N be the $(c+1) \times 2$ matrix of linear forms associated to V (see [13]). To fix notation, let $n_{ij}$ be the elements of N. By comparing N and $N_{{\mathcal {X}}}$, we can choose E and $Q_{1,c+2}(X), \dots , Q_{1,h_1+1}(X)$ as general as possible, and we get

$$\begin{aligned} \left( \begin{array}{c} Q_{11}(X) \\ \vdots \\ Q_{1,c+1}(X) \end{array} \right) = \left( \begin{array}{c} n_{11} \\ \vdots \\ n_{c+1,1} \end{array} \right) + E \left( \begin{array}{c} Q_{1,c+2}(X) \\ \vdots \\ Q_{1,h_1+1}(X) \end{array} \right) . \end{aligned}$$

We choose F as $(-I \vert F')$, $F'$ being general, and $Q_{2,c+2}(X), \dots , Q_{2,h_2+1}(X)$ arbitrary. Similarly to the previous case, we get

$$\begin{aligned} \left( \begin{array}{c} Q_{21}(X) \\ \vdots \\ Q_{2,c+1}(X) \end{array} \right) = \left( \begin{array}{c} n_{12} \\ \vdots \\ n_{c+1,2} \end{array} \right) + F' \left( \begin{array}{c} Q_{2,c+2}(X) \\ \vdots \\ Q_{2,h_2+1}(X) \end{array} \right) . \end{aligned}$$

Once we choose a general invertible matrix C of order $k+1$, we compute $A = (E \vert F) C$ and so we get the matrix M as required. We remark that the assumptions on the ranks of $P_1, P_2, Q_1(X)$ and $Q_2(X)$ are satisfied by the generality of the choices in the construction. $\square$

6 The $n = 3$ view case

In this section, we classify all smooth varieties that can be obtained as critical loci for couples of three projections.

Before approaching the problem, we briefly recall the list of codimension c smooth determinantal varieties associated to matrices of type $(c+2) \times 3$. In the case under consideration, since there are three views, we have $h_1 + h_2 + h_3 = k + c$. From the proof of Theorem 4.1, we know that, if a view verifies $h_i \le c-1$, then either the critical locus does not have codimension c, or is not irreducible. Then, we can assume $h_i \ge c$ for $i=1,2,3$, from which we get that $k \ge 2c$. On the other hand, a determinantal variety of codimension c as the ones we consider, is singular when embedded in $\mathbb {P}^k$ with $k \ge 2c+2$. Hence, smooth determinantal varieties can be critical loci for two triples of projection only if embedded in a projective space $\mathbb {P}^k$ with $k = 2c$ or $k = 2c+1$. The degree of such varieties is the expected one, namely $\deg ({\mathcal {X}}) = \left( {\begin{array}{c}c+2\\ 2\end{array}}\right)$, as it follows from Eq. (4). Thanks to classification results on smooth varieties with small invariants, in the case we are dealing with, the list of smooth varieties is complete for $c \le 3$. This is not a limitation for us because, as we will see in Theorem 6.1, smooth critical loci appear only when $c \le 2$.

For sake of completeness, we list the smooth determinantal varieties of degree $\left( {\begin{array}{c}c+2\\ 2\end{array}}\right)$ for $c \le 3$:

$c=1$:

plane cubic curves: they can be critical for a couple of three projections from $\mathbb {P}^2$ to $\mathbb {P}^1$. Even if this case is not of interest for the Computer Vision community, we include it for completeness from a geometric perspective;

$c=1$:

cubic surfaces: they can be critical for a couple of three projections from $\mathbb {P}^3$ to $\mathbb {P}^1, \mathbb {P}^1, \mathbb {P}^2$;

$c=2$:

Bordiga surfaces: they can be critical for a couple of three projections from $\mathbb {P}^4$ to $\mathbb {P}^2$;

$c=2$:

Bordiga scrolls: they can be critical for a couple of three projections from $\mathbb {P}^5$ to $\mathbb {P}^2, \mathbb {P}^2, \mathbb {P}^3$;

$c=3$:

3-fold scrolls on $\mathbb {P}^2$: they can be critical for a couple of three projections from $\mathbb {P}^6$ to $\mathbb {P}^3$;

$c=3$:

4-fold scrolls on $\mathbb {P}^2$: they can be critical for a couple of three projections from $\mathbb {P}^7$ to $\mathbb {P}^3, \mathbb {P}^3, \mathbb {P}^4$.

The codimension 1 cases are part of classical results on the classification of smooth hypersurfaces, the codimension 2 ones are in [20], while the codimension 3 ones are in [14]. We briefly describe the codimension 2 and 3 varieties.

The Bordiga surface is the embedding in $\mathbb {P}^4$ of the blow–up of $\mathbb {P}^2$ at 10 general points via the linear system of plane quartics through the points. Let $Z \subseteq \mathbb {P}^2$ be a set of 10 general points, and $B \subseteq \mathbb {P}^4$ the associated Bordiga surface. The ideal sheaves ${\mathcal {I}}_Z$ and ${\mathcal {I}}_B$ are described from the following exact sequences:

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^4(-5) {\mathop {\longrightarrow }\limits ^{N_Z}} {\mathcal {O}}_{\mathbb {P}^2}^5(-4) \rightarrow {\mathcal {I}}_Z \rightarrow 0 \quad \text{ and } \quad 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^4}^3(-4) {\mathop {\longrightarrow }\limits ^{N_B}} {\mathcal {O}}_{\mathbb {P}^4}^4(-3) \rightarrow {\mathcal {I}}_B \rightarrow 0. \end{aligned}$$

The matrices $N_Z$ and $N_B$ are not independent since it holds

$$\begin{aligned} (x_0, \dots , x_4) N_Z = (z_0, z_1, z_2) N_B^T \end{aligned}$$

(6)

where $x_0, \dots , x_4$ are coordinates in $\mathbb {P}^4$ and $z_0, z_1, z_2$ are coordinates in $\mathbb {P}^2$.

The Bordiga scroll X is $\mathbb {P}({\mathcal {E}})$ embedded in $\mathbb {P}^5$ via the tautological bundle $\xi$, where ${\mathcal {E}}$ is any rank 2 vector bundle defined by the extension

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2} \rightarrow {\mathcal {E}} \rightarrow {\mathcal {I}}_Z(4) \rightarrow 0 \end{aligned}$$

where $Z \subseteq \mathbb {P}^2$ is a set of 10 general points, as for the Bordiga surface (see [27]). By comparing the resolution of ${\mathcal {I}}_Z$ and the defining extension for ${\mathcal {E}}$, we get the following exact sequence

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^4(-1) {\mathop {\longrightarrow }\limits ^{N_{{\mathcal {E}}}}} {\mathcal {O}}_{\mathbb {P}^2}^6 \rightarrow {\mathcal {E}} \rightarrow 0. \end{aligned}$$

The minimal free resolution of X is

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^5}^3(-4) {\mathop {\longrightarrow }\limits ^{N_X}} {\mathcal {O}}_{\mathbb {P}^5}^4(-3) \rightarrow {\mathcal {I}}_X \rightarrow 0, \end{aligned}$$

and, as for the Bordiga surface, the matrices $N_X$ and $N_{{\mathcal {E}}}$ are related in the equation

$$\begin{aligned} (x_0, \dots , x_5) N_{{\mathcal {E}}} = (z_0, z_1, z_2) N_X^T. \end{aligned}$$

In the codimension 3 case, the resolutions of the two scrolls are obtained by means of the Eagon–Northcott complex, and it follows that they both have sectional genus 6. In such a case, we can construct them both similarly to the case of the Bordiga scroll. We remark that, in principle, the 3–fold scroll could be constructed as blow–up of a scroll at four double points, but it is not known whether it exists. The starting point is now a set Z of 15 general points in $\mathbb {P}^2$, and a rank 2 vector bundle ${\mathcal {E}}$ defined by the extension

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2} \rightarrow {\mathcal {E}} \rightarrow {\mathcal {I}}_Z(5) \rightarrow 0. \end{aligned}$$

Since the minimal free resolution of ${\mathcal {I}}_Z$ is

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^5(-6) \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^6(-5) \rightarrow {\mathcal {I}}_Z \rightarrow 0, \end{aligned}$$

we get the following presentation of ${\mathcal {E}}$:

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^5(-1) {\mathop {\longrightarrow }\limits ^{N_{{\mathcal {E}}}}} {\mathcal {O}}_{\mathbb {P}^2}^7 \rightarrow {\mathcal {E}} \rightarrow 0. \end{aligned}$$

Let $N_X$ be the $5 \times 3$ matrix that satisfy the equation

$$\begin{aligned} (x_0, \dots , x_6) N_{{\mathcal {E}}} = (z_0, z_1, z_2) N_X^T. \end{aligned}$$

Then, the defining ideal I(X) of X is generated by the $3 \times 3$ minors of $N_X$.

Since the construction of the 4–fold scroll is analogous to the one of the 3–fold scroll, we only stress the differences. This time, the vector bundle to consider is the rank 3 one defined by the extension

$$\begin{aligned} 0 \rightarrow {\mathcal {O}}_{\mathbb {P}^2}^2 \rightarrow {\mathcal {E}} \rightarrow {\mathcal {I}}_Z(5) \rightarrow 0, \end{aligned}$$

where Z is a set of 15 general points as in the previous case. Then, the matrix $N_X$ is obtained as in the previous case, but $N_{{\mathcal {E}}}$ is now a matrix with type $8 \times 5$.

Now, we address the problem of getting the above smooth varieties as critical loci for the reconstruction problem. As we have seen above, we consider a codimension c, determinantal variety $X \subseteq \mathbb {P}^k$, with $k = 2c$ or $k = 2c+1$, whose defining ideal is generated by the $3 \times 3$ minors of a $(c+2) \times 3$ matrix N of linear forms. The result is contained in the following classification Theorem.

Theorem 6.1

Let $X \subseteq \mathbb {P}^k$ be a smooth codimension c variety.

X is the critical locus for a suitable couple of three projections from $\mathbb {P}^k$ if and only if either $X \subseteq \mathbb {P}^2$ is a cubic curve, or $X \subseteq \mathbb {P}^3$ is a cubic surface, or, finally, $X \subseteq \mathbb {P}^4$ is a Bordiga surface. In particular, $c \le 2$.

Proof

From the previous discussion, it follows that $h_1 = h_2 = c$, and $h_3 = c + \varepsilon$ for $k = 2c + \varepsilon$, $\varepsilon = 0$ or 1. Let us consider general projections $P_i, Q_i : \mathbb {P}^k \rightarrow \mathbb {P}^{h_i}$ for $i=1,2,3$.

At first, we compute the matrix $N_{{\mathcal {X}}}$ as in Corollary 3.1, and so we get

$$\begin{aligned} N_{{\mathcal {X}}} = \left( \begin{array}{ccc} Q_{11}(X) &{} 0 &{} 0 \\ \vdots \\ Q_{1,c+1}(X) &{} 0 &{} 0 \\ 0 &{} Q_{21}(X) &{} 0 \end{array} \right) - A C^{-1} \left( \begin{array}{ccc} 0 &{} Q_{22}(X) &{} 0 \\ \vdots \\ 0 &{} Q_{2,c+1}(X) &{} 0 \\ 0 &{} 0 &{} Q_{31}(X) \\ \vdots \\ 0 &{} 0 &{} Q_{3,h_3+1}(X) \end{array} \right) . \end{aligned}$$

To write $N_{{\mathcal {X}}}$ as above, we need that the centers of $P_2$ and $P_3$ do not meet. If this is not the case, we select rows from $P_1, P_2$ and $P_3$ to get an invertible submatrix C. In such a case, $N_{{\mathcal {X}}}$ is computed by using different submatrices of $M_{{\mathcal {X}}}$, but the result holds as well.

If we multiply the matrices above, and perform elementary operations on the rows, we get the matrix $N = (n_{ij})$ such that: (i) its j-th column depend on $Q_j$ only, for $j=1,2,3$; (ii) $n_{c+2,1} = n_{c+1,2} = 0$. In the case $h_3 = c$, it is possible to perform elementary operations of the rows of N so that (iii) $n_{c,3} = 0$.

It follows that the critical locus is actually a codimension c scheme whose defining ideal is generated by the $3 \times 3$ minors of a matrix with type $(c+2) \times 3$ of linear forms.

It is known that the defining equation of every smooth cubic surface $X \subseteq \mathbb {P}^3$ can be written as $L_1L_2L_3 + M_1M_2M_3 = 0$ for suitable linear forms $L_i, M_i$, $i=1,2,3$. An equation of this kind for the cubic surface is called Cayley–Salmon. We remark that, for a given surface, there are 120 different Cayley–Salmon equations that define it (see [15]). The lines defined by $L_i = M_j = 0$ are contained in the cubic surface X. The Cayley–Salmon equation is the locus where

$$\begin{aligned} N_X = \left( \begin{array}{ccc} L_1 &{} M_2 &{} 0 \\ M_1 &{} 0 &{} L_3 \\ 0 &{} L_2 &{} M_3 \end{array} \right) \end{aligned}$$

drops rank. If we add scalar multiples of the first two columns to the third one, we get a matrix $N_X$ that verifies constrains (i), (ii) above, and the linear forms on the third column are linearly independent. If we compare matrices $N_{{\mathcal {X}}}$ and $N_X$ under the simplifying assumption that

$$\begin{aligned} (E \vert F) = -\left( \begin{array}{cccc} 1 &{} 1 &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 \\ -e_{31} &{} 0 &{} 0 &{} 1 \end{array} \right) , \end{aligned}$$

we get $Q_{11} = L_1, Q_{12} = M_1, Q_{21} = e_{31} M_2 + L_2, Q_{22} = M_2$, and $Q_{3j} = n_{j3}$ for $j=1,2,3$, and so we obtain the projections $Q_1, Q_2, Q_3$. If we choose a general invertible matrix C of order 4, we get $A = (E \vert F) C$, and so we obtain also the projections $P_1, P_2, P_3$.

We have then proven that every smooth cubic surface is the critical locus for a suitable couple of three projections from $\mathbb {P}^3$ to $\mathbb {P}^1, \mathbb {P}^1, \mathbb {P}^2$, as claimed.

A plane cubic curve is obtained as a section of a smooth cubic surface with a general plane. Hence, the argument above shows also that every plane cubic curve is critical for a suitable couple of three projections from $\mathbb {P}^2$ to $\mathbb {P}^1$, as claimed. We remark that, for plane cubic curves, one has to start from the Cayley–Salmon equation and does not have to perform further elementary operations on the columns of the matrix.

The Bordiga surface has been considered from the point of view of critical loci in [8]. In that paper, the authors proved that the critical locus for a couple of three projections from $\mathbb {P}^4$ to $\mathbb {P}^2$ is in the irreducible component of the Hilbert scheme containing the Bordiga surface as general element ( [8], Proposition 5.1), and conversely, that every Bordiga surface X is actually critical for suitable couples of three projections ( [8], Theorem 5.1). The key point of the proof of Theorem 5.1 is that, if the unit points in $\mathbb {P}^2$ are in Z, then the matrix $N_X$ fulfils the constraints (i), (ii), (iii) above, since it is related to $N_Z$ in Eq. (6).

To complete the proof, we’ll prove that the critical locus ${\mathcal {X}}$ is never smooth in the remaining cases.

To this end, we consider a general critical locus ${\mathcal {X}}$, its associated matrix $N_{{\mathcal {X}}}$ as obtained at the beginning of the proof, and we take the codimension $c+3$ linear space $L \subseteq {\mathcal {X}}$ defined by $n_{11} = \dots = n_{c+1,1} = n_{c+2,2} = n_{c+2,3} = 0$. As ${\mathcal {X}}$ is at least 3–dimensional in the cases we are considering, L is not empty. To prove that the points in L are singular for ${\mathcal {X}}$, we evaluate the Jacobian matrix at them. Without loss of generality, we can make a change of coordinates, so that $n_{i,1} = x_i$ for $i=1, \dots , c+1$, $n_{c+2,2} = x_{c+2}$ and $n_{c+2,3} = x_{c+3}$. Furthermore, we assume that, at a point in L, the rank of the $(c+1) \times 2$ matrix obtained by removing the first column and the last row in $N_{{\mathcal {X}}}$ is 2. In the case this does not hold, the point is singular by general properties of determinantal varieties. To simplify notation, we denote $(i_1i_2)$ the determinant of the minor of the above matrix obtained by taking rows $i_1$ and $i_2$. Let $f_{ijh}$ be the determinant of the submatrix of $N_{{\mathcal {X}}}$ obtained by taking rows i, j, h with $1 \le i< j < h \le c+2$. The derivative of $f_{ijh}$ with respect to any variable is the sum of 3 determinants, two columns of which are from $N_{{\mathcal {X}}}$ and the third column is the derivative of the corresponding column in $N_{{\mathcal {X}}}$. We have to evaluate the derivatives at $P \in L$. If $h = c+2$, then the gradient of $f_{ij,c+2}$ at P is the null matrix, as it is easy to check. If $h \le c+1$, we get

$$\begin{aligned} \nabla f_{ijh}(P) = (jh) \vec {e}_i - (ih) \vec {e}_j + (ij) \vec {e}_h \end{aligned}$$

where $\vec {e}_k$ is the k–th element of the canonical basis. Without loss of generality, we can assume $(12) \not = 0$, equivalent to the rank two assumption. The matrices $\nabla f_{123}(P), \dots ,$ $\nabla f_{12,c+1}(P)$ are linearly independent, since $(12) I_{c-1}$ is a submatrix of the Jacobian matrix corresponding to the above generators. Let us consider now $f_{ijh}$ with $2< j < h \le c+1$ and $i=1$ or $i=2$. We have

$$\begin{aligned} (12) \nabla f_{ijh}(P) - (ij) \nabla f_{12h}(P) + (ih) \nabla f_{12j}(P) = [(12)(jh) - (1j)(2h) + (1h)(2j)] \vec {e}_i = 0 \end{aligned}$$

because the equation in square brackets is a Plücker relation that holds for rank 2 matrices of type $(c+1) \times 2$ for every $c \ge 3$. Finally, we consider $f_{ijh}$ with $2< i< j < h \le c+1$. We have

$$\begin{aligned} (12) \nabla f_{ijh}(P) - (ij) \nabla f_{12h} + (ih) \nabla f_{12j}(P) - (jh)\nabla f_{12i}(p)=\\ =[(2i)(jh) - (2j)(ih) + (2h)(ij)] \vec {e}_1 + [(1i)(jh) - (1j)(ih)+(1h)(ij)]\vec{e}_2= 0 \end{aligned}$$

because we get Plücker relations once more. Hence, $\nabla f_{ijh}(P)$ is in the span of $\nabla f_{123}(P),$ $\dots , \nabla f_{12,c+1}(P)$ for every $1 \le i< j < h \le c+2$, and so the Jacobian matrix has rank $c-1$ at most at every $P \in L$. This proves that every $P \in L$ is singular for the critical locus ${\mathcal {X}}$, and so the proof is complete. $\square$

7 The $n = 4$ view case

According to Theorem 4.2, when we have 4 views, the codimension of the critical locus is 1, otherwise the critical locus is either not irreducible or singular. In such a case, $h_1 + h_2 + h_3 + h_4 = k+1$. On the other hand, a degree 4, determinantal hypersurface is singular if embedded in $\mathbb {P}^k$ with $k \ge 4$. Hence, the only possible case is $k=3$, and $h_i = 1$ for every $i=1, \dots , 4$. As previously said, this case is not of interest for the Computer Vision community, and we insert it for sake of completeness from a geometrical point of view.

The study of quartic determinantal surfaces in $\mathbb {P}^3$ is a classical topic, and we briefly recall the main results (see [21] for more results on the subject).

Quartic surfaces in $\mathbb {P}^3$ are parameterized by points in $\mathbb {P}^{34}$. It is known that the general quartic surface in $\mathbb {P}^3$ is not determinantal, and that the locus of determinantal ones is a divisor in $\mathbb {P}^{34}$. Determinantal quartic surfaces are characterized as the ones that contain a non–hyperelliptic curve C of degree 6 and genus 3 (see, e.g., [3]). Such a curve is also called Schur’s sextic.

Moreover, a general quartic surface does not contain any line. It is known that not ruled quartic surfaces can contain any number of lines in the range 1 to 52, or 54, 56, 60, 64 lines. In [30,31,30], the author studied quartic determinantal surfaces containing one or two lines. In particular, if N is an order 4 matrix of linear forms in $\mathbb {P}^3$ whose determinant is the defining equation of the quartic surface S, and $\ell \subset S$ is a line, then, up to elementary operations on rows and columns of N, the linear forms defining $\ell$ are either in a row or column of N, or in a $3 \times 2$ or $2 \times 3$ submatrix of N.

Now we discuss the connections between quartic determinantal surfaces containing lines and the reconstruction problem in Computer Vision.

Proposition 7.1

Let $P_i, Q_i: \mathbb {P}^3 \rightarrow \mathbb {P}^1, i = 1, \dots , 4$, be two 4–tuples of projections. Then, in the general case, the associated critical locus is a smooth quartic determinantal surface.

Proof

The matrix M associated to the two 4–tuples of projections is described in Eq. (2), and is a square matrix of order 8. From M, we get matrix $N_{{\mathcal {X}}} = B - A C^{-1} D$ of order 4 whose determinant defines the critical locus ${\mathcal {X}}$. In the considered case, matrices B, D are

$$\begin{aligned} B = \left( \begin{array}{cccc} Q_{11}(X) &{} 0 &{} 0 &{} 0 \\ Q_{12}(X) &{} 0 &{} 0 &{} 0 \\ 0 &{} Q_{21}(X) &{} 0 &{} 0 \\ 0 &{} Q_{22}(X) &{} 0 &{} 0 \end{array} \right) , \qquad D = \left( \begin{array}{cccc} 0 &{} 0 &{} Q_{31}(X) &{} 0 \\ 0 &{} 0 &{} Q_{32}(X) &{} 0 \\ 0 &{} 0 &{} 0 &{} Q_{41}(X) \\ 0 &{} 0 &{} 0 &{} Q_{42}(X) \end{array} \right) . \end{aligned}$$

Also in this case, we can repeat analogous arguments as in Theorem 6.1 on the construction of $N_{{\mathcal {X}}}$.

Then, the first two columns of $N_{{\mathcal {X}}}$ are the first two columns of B, while the last two columns of $N_{{\mathcal {X}}}$ depend of the two non–zero columns of D. Then, the critical locus is a quartic determinantal surface. When computing ${\mathcal {X}}$ in a random case, we get a smooth surface, and so the general critical locus is smooth. $\square$

Remark 7.1

In the notation of [28], the four lines, centers of projections $Q_1, \dots , Q_4$, are of type $4''$.

Now we highlight a geometrical property of such critical loci.

Proposition 7.2

In the same hypotheses as above, the critical locus contains twisted cubic curves meeting three of the four lines at two points.

Proof

Let $N_{{\mathcal {X}}}$ be the matrix constructed in the proof of Proposition 7.1, and let $N'$ its sumatrix consisting of the first three columns. The maximal minors of $N'$ define a Schur curve containing the centers of projections $Q_1, Q_2, Q_3$ as components. To fix notation, we set

$$\begin{aligned} N' = \left( \begin{array}{ccc} Q_{11}(X) &{} 0 &{} n'_{13} \\ Q_{12}(X) &{} 0 &{} n'_{23} \\ 0 &{} Q_{21}(X) &{} n'_{33} \\ 0 &{} Q_{22}(X) &{} n'_{43} \end{array} \right) \end{aligned}$$

where the center of $Q_1$ is the line $Q_{11}(X) = Q_{12}(X) = 0$, the center of $Q_2$ is the line $Q_{21}(X) = Q_{22}(X) = 0$ and the center of $Q_3$ is $n'_{13} = n'_{23} = 0$, also defined by $n'_{33} = n'_{43} = 0$. The two couples of generators of the third line are related by the equation

$$\begin{aligned} \left( \begin{array}{c} n'_{33} \\ n'_{43} \end{array} \right) = \left( \begin{array}{cc} a_{11} &{} a_{12} \\ a_{21} &{} a_{22} \end{array} \right) \left( \begin{array}{c} n'_{13} \\ n'_{23} \end{array} \right) \end{aligned}$$

where $A = (a_{ij})$ is invertible. The residual curve is the twisted cubic curve defined by the $2 \times 2$ minors of

$$\begin{aligned} \left( \begin{array}{c} Q_{11}(X) \\ Q_{12}(X) \end{array} \right| \left. \text{ Adj }(A) \left( \begin{array}{c} Q_{21}(X) \\ Q_{22}(X) \end{array} \right) \right| \left. \begin{array}{c} n'_{13} \\ n'_{23} \end{array} \right) , \end{aligned}$$

as it can be checked. Each center meets the twisted cubic curve above at two points, because the generators of each line vanish two quadrics of the three that generate the twisted cubic. $\square$

Now, we consider a partial converse of the above Proposition.

Proposition 7.3

Let $\ell _1, \dots , \ell _4 \subset \mathbb {P}^3$ be 4 lines, pairwise skew. Then there is a quartic determinantal surface S containing the 4 lines that is critical for two 4–tuples of projections from $\mathbb {P}^3$ to $\mathbb {P}^1$. The given lines are centers for the four projections $Q_1, \dots , Q_4$.

Proof

Let $\ell _i$ be the line defined by $I(\ell _i) = \langle q_{i1}, q_{i2} \rangle$, where $q_{i1}, q_{i2}$ are linearly independent linear forms in $\mathbb {C}[x_0, \dots , x_3]$. Let us consider the matrix

$$\begin{aligned} N = \left( \begin{array}{cccc} q_{11} &{} 0 &{} e_{11} q_{31} + e_{12} q_{32} &{} e_{13} q_{41} + e_{14} q_{42} \\ q_{12} &{} 0 &{} e_{21} q_{31} + e_{22} q_{32} &{} e_{23} q_{41} + e_{24} q_{42} \\ 0 &{} q_{21} &{} e_{31} q_{31} + e_{32} q_{32} &{} e_{33} q_{41} + e_{34} q_{42} \\ 0 &{} q_{22} &{} e_{41} q_{31} + e_{42} q_{32} &{} e_{43} q_{41} + e_{44} q_{42} \end{array} \right) , \end{aligned}$$

and let S be the surface defined by $\det (N) = 0$. We assume $E = (e_{ij})$ to be invertible. Hence, we can reconstruct M from N by choosing a general invertible matrix C and by setting $A = -E C$. Hence, S is critical for suitable projections as claimed. $\square$

To complete the section, we make some final remarks on quartic surfaces that contain four skew lines that explain why it is not possible to give a stronger converse of Proposition 7.1.

Remark 7.2

Given 4 pairwise skew general lines in $\mathbb {P}^3$, the linear subspace V of $\mathbb {C}^{35}$ containing quartic surfaces through the lines has dimension 15. Here, four lines are general if they are not contained in the same quadric. Since the conditions of being determinantal and of containing lines are independent, we expect that the quartic determinantal surfaces containing the four lines are a locus of dimension 14 in V.

From a parameter count, quartic determinantal surfaces that are critical loci depend on the elements of E. Since we can get the same quartic surface for different choices of matrix E (see the proof of Proposition 7.3), we have that the parameters are actually less than 16. Algebraically, from every rows of E, different from the first one, an element can be disregarded. So, the locus in V of critical surfaces has dimension 13. From a geometrical point of view, a quartic determinantal surface containing 4 lines as above is critical if, and only if, it contains a twisted cubic curve meeting three of the four lines at 2 points, and not meeting the last line, for every choice of three among the four lines. Since a twisted cubic curve is uniquely determined by 6 points, we get such a curve by choosing two points of the first three lines. If we choose a second twisted cubic curve meeting all lines but the third one, we need 6 more points, two for each of the three selected lines. Once those two twisted cubics are selected, the quartic surface through the four lines is given, and it is possible to get its defining equation as determinant of a matrix as in the proof of Proposition 7.3. The parameters from which this construction depends are 12 (the points on the lines), and one more because we can multiply the matrix by a scalar so that the determinant defines the same surface. Hence, we get once more a locus of dimension 13.

In conclusion, we expect the critical quartic surfaces to fill a codimension 2 subset in V.