Hardness estimates of the code equivalence problem in the rank metric

Abstract

In this paper, we analyze the hardness of the Matrix Code Equivalence (MCE) problem for matrix codes endowed with the rank metric, and provide the first algorithms for solving it. We do this by making a connection to another well-known equivalence problem from multivariate cryptography—the Isomorphism of Polynomials (IP). Under mild assumptions, we give tight reductions from MCE to the homogenous version of the Quadratic Maps Linear Equivalence (QMLE) problem, and vice versa. Furthermore, we present reductions to and from similar problems in the sum-rank metric, showing that MCE is at the core of code equivalence problems. On the practical side, using birthday techniques known for IP, we present two algorithms: a probabilistic algorithm for MCE running in time $q^{\frac{2}{3}(n+m)}$ up to a polynomial factor, and a deterministic algorithm for MCE with roots, running in time $q^{\min \{m,n,k\}}$ up to a polynomial factor. Lastly, to confirm these findings, we solve randomly-generated instances of MCE using these two algorithms.

1 Introduction

Given two mathematical objects of the same type, an equivalence problem asks the question whether there exists an equivalence map between these objects—and how to find it—that preserves some important property of the objects. These kind of problems come in different flavors depending on the objects—groups, graphs, curves, codes, quadratic forms, etc.—and quite often the interesting maps are isomorphisms or isometries. Interestingly, equivalence problems are one of the core hard problems underlying the security of many public-key cryptosystems, especially post-quantum ones. Many multivariate and code-based systems employ an equivalence transformation as a hiding technique, and thus intrinsically rely on the assumption that a particular equivalence problem is intractable, for example [10, 16, 20, 21, 36, 40, 43]. In addition, quite remarkably, a hard equivalence problem gives rise to a Sigma protocol and, through the Fiat–Shamir transform, a provably secure digital signature scheme [27]. This idea has been revisited many times, being the basis of several signature schemes [4, 11, 18, 19, 30, 43]. Two such schemes actually appeared during the writing of this manuscript [23, 48] as a result of NIST’s announcement for an additional fourth round on signatures in the post quantum standardization process [38]. Understanding the hardness of these equivalence problems is an essential task in choosing appropriate parameters that attain a certain security level of these cryptographic schemes.

One of these problems is the Code Equivalence problem, which given two codes (with the Hamming metric), asks for an isometry (equivalence transformation that preserves the metric) that maps one code to the other. It was first studied by Leon [35] who proposed an algorithm that takes advantage of the Hamming weight being invariant under monomial permutations. It was improved very recently by Beullens [9] using collision-based techniques. Sendrier [46] proposed another type of algorithm, the Support Splitting Algorithm (SSA), that is exponential in the dimension of the hull (the intersection of a code and its dual). Interestingly, in low characteristic, random codes have very small hull, rendering the problem easy.

In this work, we focus on the code equivalence problem, but for matrix codes (an ${\mathbb {F}}_{q}$-linear subspace of the space of $m\times n$ matrices over ${\mathbb {F}}_{q}$) endowed with the rank metric—Matrix Code Equivalence (MCE). Evaluating the hardness of this problem is only natural—rank-based cryptography has become serious competition for its Hamming-based counterpart, showing superiority in key sizes for the same security level [2, 3, 7, 37]. MCE, and variations of it, has been introduced by Berger in [8], but it was only recently that the first concrete statements about its hardness were shown in two concurrent independent works publicly available as preprints [17, 32].¹ Couvreur et al. [17] showed that MCE is at least as hard as the (Monomial) Code Equivalence problem in the Hamming metric, while for only right equivalence, or when the codes are ${\mathbb {F}}_{q^m}$-linear, the problem becomes easy. Grochow and Qiao [32] show the same reduction from (Monomial) Code Equivalence to MCE but using a completely different technique of linear algebra coloring gadgets which makes the reduction looser than the one in [17].

1.1 Our contributions

In this paper, we investigate the theoretical and practical hardness of the Matrix Code Equivalence (MCE) problem. Our contributions can be summarized as follows:

First, we link in a straightforward manner the MCE problem to hard problems on systems of polynomials by showing that MCE is polynomial-time equivalent to the Bilinear Maps Linear Equivalence (BMLE) problem. We then extend this result by proving that MCE is polynomial-time equivalent to the Quadratic Maps Linear Equivalence (QMLE) problem, under a mild assumption of trivial automorphism groups of the codes in question. While our technique fails to give a proof without this assumption, we consider it to be reasonable for randomly generated codes and for cryptographic purposes. As the QMLE problem is considered to be the hardest equivalence problem for systems of multivariate polynomials, it is essential to understand under which conditions MCE and QMLE reduce to one another. Note that previous work² requires much stronger assumptions for related results [6, 29, 32], such as algebraically closed fields or existence of square or third roots. Our reduction to QMLE is tight and gives a tight upper bound on the hardness of MCE. Furthermore, it is very simple, thus establishing connection between code equivalence problems and polynomial equivalence problems that is usable in practice. This is the basis of our contributions on the practical hardness of MCE.

Fig. 1

Reductions around Matrix Code Equivalence. Dashed arrows are contributions from this work, dotted arrows are trivial reductions. “${\textsf {A}} \longrightarrow {\textsf {B}}$” means that “Problem ${\textsf {A}}$ reduces to Problem ${\textsf {B}}$ in polynomial time”. Results with * assume trivial automorphism groups

Second, using similar techniques, and under the same assumptions, we show that MCE is polynomial-time equivalent to other code equivalence problems, such as Matrix Sum-Rank Code Equivalence Problem, and at least as hard as the Vector Sum-Rank Code Equivalence Problem. All these connections and our results are visualized in Fig. 1.

On the practical side, we provide the first two non-trivial algorithms for solving MCE using the connection to QMLE. The first algorithm is a generalization of a known birthday-based algorithm for QMLE [14, 15] for systems of polynomials with the same number of variables as equations. We show that this algorithm extends to different invariance properties and code dimensions, which helps us prove complexity of $q^{\frac{2}{3}(n+m)}$ up to a polynomial factor for MCE for $m\times n$ matrix codes. The algorithm is probabilistic with success probability that can be made arbitrarily close to 1, and can be used for code dimensions up to $2(m+n)$. For larger dimensions, the complexity becomes $q^{(n+m)}$ up to a polynomial factor, but the algorithm is deterministic. The birthday-based algorithm for QMLE [14] assumed existence of a polynomial-time solver for the inhomogeneous variant of QMLE to achieve these complexities. Interestingly, due to the specific instances of the inhomogeneous QMLE arising from the collision search, the problem seems to be much harder than for random instances—a fact previously overlooked in [14]. In contrast, [15] uses a non-polynomial estimate for this solver. We analyse the most recent results regarding such solvers, and show that for parameter sets of cryptographical interest the above complexities hold, even if such solvers do not achieve polynomial time.

Our second algorithm uses the bilinear structure of the polynomials arising from MCE. Because matrix codes show symmetry between the parameters, as given in Lemma 26, the complexity of solving MCE using this result and Algorithm 2 becomes $q^{\min \{m,n,k\}}$ up to a polynomial factor. The algorithm is deterministic and does not require a polynomial-time solver for the inhomogeneous QMLE instance, but the weaker assumption that the solver has a complexity of ${\mathcal {O}}(q^{\min \{m,n,k\}})$ at most. This general result, valid for any m,n, and k, is summarized in our main result Theorem 41.

Lastly, to verify the results and performance of these algorithms in practice, we have implemented both and solved randomly generated instances of MCE for different parameter sets. The results of these experiments show that our assumptions are reasonable and the above complexities hold. Our implementations are open source and available at: https://github.com/mtrimoska/matrix-code-equivalence

2 Preliminaries

Let ${\mathbb {F}}_{q}$ be the finite field of q elements. ${\text {GL}}_n(q)$ and ${\text {AGL}}_n(q)$ denote respectively the general linear group and the general affine group of degree n over ${\mathbb {F}}_{q}$.

We use bold letters to denote vectors ${{\textbf {a}}},{{\textbf {c}}},{{\textbf {x}}},\dots$, and matrices ${{\textbf {A}}}, {{\textbf {B}}}, \dots$. The entries of a vector ${{\textbf {a}}}$ are denoted by $a_i$, and we write ${{\textbf {a}}}=(a_1,\dots ,a_n)$ for a (row) vector of dimension n over some field and ${{\textbf {a}}}^{\top }=(a_1,\ldots ,a_n)^{\top }$ for the respective column vector. Similarly, the entries of a matrix ${{\textbf {A}}}$ are denoted by $A_{ij}$. A matrix ${{\textbf {A}}}$ is called symmetric if ${{\textbf {A}}}^\top ={{\textbf {A}}}$ and skew-symmetric if ${{\textbf {A}}}^\top =-{{\textbf {A}}}$. The space of matrices over ${\mathbb {F}}_{q}$ of size $m \times n$ is denoted ${\mathcal {M}}_{m,n}(q)$. The set of k-subsets of ${\mathcal {M}}_{m,n}(q)$ is denoted by ${\mathcal {M}}^{[k]}_{m,n}(q)$.

Random sampling from a set S is denoted by $a \xleftarrow {{ \$ }} S$. We use the notation $\tilde{{\mathcal {O}}}(f(n))$ to denote ${\mathcal {O}}( f(n)\log (f(n)) )$ whenever we want to omit polynomial factors from the complexity expression. We use the notation $f = \Theta (g)$ whenever f is bounded from below and above by g asymptotically.

For a computational problem ${\textsf {P}}$, if we want to emphasize a list of parameters p defining the size of the inputs and the input set S, we will use the notation ${\textsf {P}}(p,S)$. If these are not relevant, clear from context, or the set S is the entire universe U, we will use only ${\textsf {P}}(p)$ or ${\textsf {P}}$.

Our results in Sect. 3 use the following standard notion of Turing reduction.

Definition 1

Given two computational problems ${\textsf {P}}({{p}},S)$ and ${\textsf {P}}'({{p}}',S')$, with inputs coming from sets S and $S'$ respectively, we say that ${\textsf {P}}({{p}},S)$ reduces to ${\textsf {P}}'({{p}}',S')$ if there exists a probabilistic polynomial-time oracle machine ${\mathcal {B}}$ such that for every oracle ${\mathcal {A}}$ that solves ${\textsf {P}}'({{p}}',S')$ on all inputs from $S'$, ${\mathcal {B}}^{{\mathcal {A}}}$ (${\mathcal {B}}$ given access to ${\mathcal {A}}$) solves ${\textsf {P}}({{p}},S)$ on all inputs from S.

Note that our reductions are meaningful only as worst-case to worst-case, and therefore in the definition we include the statement that the oracles solve the problems on all inputs. On the other hand, we do not always require the oracle ${\mathcal {A}}$ to be able to solve ${\textsf {P}}'$ on the entire universe $U'$ of inputs in order for ${\mathcal {B}}^{{\mathcal {A}}}$ to be able to solve ${\textsf {P}}$ on the entire universe U of inputs. When this is the case, it will be emphasized through the definition of the input sets S and $S'$. These restrictions, however, can not be used to show a stronger statement such as worst-case to average-case reduction.

2.1 The Matrix Code Equivalence problem

This section introduces basic notions on matrix codes and their equivalences. A more thorough introduction on matrix codes can be found in [31]. The usual choice for measuring distance between matrices over a finite field is the so called rank metric, defined as follows.

Definition 2

Let ${{\,\textrm{Rank}\,}}({{\textbf {M}}})$ denote the rank of a matrix ${{\textbf {M}}} \in {\mathcal {M}}_{m,n}(q)$. The rank distance between two $m \times n$ matrices ${{\textbf {A}}}$ and ${{\textbf {B}}}$ over ${\mathbb {F}}_{q}$ is defined as

$$\begin{aligned} d({{\textbf {A}}}, {{\textbf {B}}}) = {{\,\textrm{Rank}\,}}({{\textbf {A}}}- {{\textbf {B}}}). \end{aligned}$$

An isometry is a map $\mu : {\mathcal {M}}_{m,n}(q) \rightarrow {\mathcal {M}}_{m,n}(q)$ that preserves the rank, i.e. ${{\,\textrm{Rank}\,}}(\mu ({{\textbf {M}}})) = {{\,\textrm{Rank}\,}}({{\textbf {M}}})$ for all ${{\textbf {M}}} \in {\mathcal {M}}_{m,n}(q)$.

By symmetry, without loss of generality, in the rest of the text we assume $n\geqslant m$.

Definition 3

A matrix code is a subspace ${\mathcal {C}}$ of $m \times n$ matrices over ${\mathbb {F}}_{q}$ endowed with the rank metric. Let k denote the dimension of ${\mathcal {C}}$ as a subspace of ${\mathbb {F}}_{q}^{m\times n}$ and its basis by $\langle {{\textbf {C}}}_{{1}}, \ldots , {{\textbf {C}}}_{{k}} \rangle ,$ with ${{\textbf {C}}}_{{i}} \in {\mathbb {F}}_{q}^{m\times n}$ linearly independent. Two matrix codes ${\mathcal {C}}, {\mathcal {D}}\subset {\mathcal {M}}_{m,n}(q)$ are said to be equivalent if there exists an isometry $\mu$ with $\mu ({\mathcal {C}}) = {\mathcal {D}}$.

An isometry from ${\mathcal {C}}$ to ${\mathcal {D}}$ is always of the form ${{\textbf {M}}} \mapsto {\textbf {AMB}}$, ${{\textbf {M}}} \mapsto {\textbf {M}}^{\top }$ or a composition of these two, where ${{\textbf {A}}}\in {\text {GL}}_m(q)$ and ${{\textbf {B}}}\in {\text {GL}}_n(q)$ [33, 52]. We restrict our attention to the isometries of the first form and we will say that two matrix codes are equivalent if there exists a map ${{\textbf {C}}}\mapsto {\textbf {ACB}}$ from ${\mathcal {C}}$ to ${\mathcal {D}}$ where ${{\textbf {A}}}\in {\text {GL}}_m(q)$ and ${{\textbf {B}}}\in {\text {GL}}_n(q)$. We will denote this map as a pair $({{\textbf {A}}},{{\textbf {B}}})$. When $n=m$, If there exists a map $({{\textbf {A}}},{{\textbf {A}}}^\top ): {{\textbf {C}}}\mapsto {\textbf {ACA}}^{\top }$ from ${\mathcal {C}}$ to ${\mathcal {D}}$, where ${{\textbf {A}}}\in {\text {GL}}_m(q)$, we will say that the codes ${\mathcal {C}}$ and ${\mathcal {D}}$ are congruent. This is a direct generalization of the notion of congruent matrices. An automorphism of a code is a map $({{\textbf {A}}},{{\textbf {B}}}):{\mathcal {C}}\rightarrow {\mathcal {C}}$, i.e. for each ${{\textbf {C}}}\in {\mathcal {C}}$, we get ${{\textbf {A}}}{{\textbf {C}}}{{\textbf {B}}}\in {\mathcal {C}}$. The automorphism group of ${\mathcal {C}}$ contains all the automorphisms of ${\mathcal {C}}$. If the automorphism group contains only the maps $(\lambda {{\textbf {I}}},\nu {{\textbf {I}}})$ for scalars $\lambda ,\nu \in {\mathbb {F}}_{q}^*$, we say the automorphism group is trivial.

The main focus of this article will be the Matrix Code Equivalence(MCE) problem which is formally defined as follows:

Problem 4

MCE $(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$:

Input: Two k-dimensional matrix codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathcal {M}}_{m, n}(q)$

Question: Find—if any—a map $({{\textbf {A}}},{{\textbf {B}}})$, where ${{\textbf {A}}}\in {\text {GL}}_m(q),{{\textbf {B}}}\in {\text {GL}}_n(q)$ such that for all ${{\textbf {C}}}\in {\mathcal {C}}$, it holds that ${\textbf {ACB}} \in {\mathcal {D}}$.

This is the computational version of MCE which, similarly to its counterpart in the Hamming metric [4, 5, 11], seems to be more interesting for cryptographic applications than its decisional variant. We will thus be interested in evaluating the practical hardness only of MCE, and present algorithms only for MCE and not its decisional variant. It is also interesting to consider the following variant of MCE:

Problem 5

MCE base$(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$:

Input: The bases $({{\textbf {C}}}^{(1)},\dots ,{{\textbf {C}}}^{(k)})$ and $({{\textbf {D}}}^{(1)},\dots ,{{\textbf {D}}}^{(k)})$ of two k-dimensional matrix codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathcal {M}}_{m, n}(q)$

Question: Find—if any—a map $({{\textbf {A}}},{{\textbf {B}}})$, where ${{\textbf {A}}}\in {\text {GL}}_m(q),{{\textbf {B}}}\in {\text {GL}}_n(q)$ such that for all ${{\textbf {C}}}^{(i)}$, it holds that ${{\textbf {AC}}}^{({{i}})}{{\textbf {B}}} = {{{\textbf {D}}}}^{(i)}$.

Intuitively, MCE base seems easier than MCE, and as a matter of fact, we will show later that most random instances are solvable in polynomial time. Another variant of the MCE problem is the Matrix Codes Right Equivalence problem (MCRE) (left equivalence could be defined similarly):

Problem 6

MCRE $(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$:

Input: Two k-dimensional matrix codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathcal {M}}_{m, n}(q)$

Question: Find—if any—${{\textbf {B}}}\in {\text {GL}}_n(q)$ such that for all ${{\textbf {C}}}\in {\mathcal {C}}$, it holds that ${\textbf {CB}} \in {\mathcal {D}}$.

It has been shown in [17] that MCE is at least as hard as code equivalence in the Hamming metric, Hamming Code Equivalence (HCE), also known as Linear or Monomial Equivalence. Interestingly, the same paper shows that MCRE is actually easy and can always be solved in probabilistic-polynomial time.

For vector rank codes ${\mathcal {C}}\subset {\mathbb {F}}_{q^m}^n$, isometries are similar to the case of matrix codes. We get the Vector Rank Code Equivalence (VRCE) problem.

Problem 7

VRCE $(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$:

Input: Two k-dimensional vector rank codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathbb {F}}_{q^m}^n$

Question: Find—if any—a matrix ${{\textbf {B}}}\in {\text {GL}}_n(q)$ such that for all ${{\textbf {c}}} \in {\mathcal {C}}$, it holds that ${\textbf {cB}} \in {\mathcal {D}}$.

Given a vector rank code ${\mathcal {C}}\subset {\mathbb {F}}_{q^m}^n$ and a basis $\Gamma$ for ${\mathbb {F}}_{q^m}$ over ${\mathbb {F}}_{q}$, each vector ${{\textbf {c}}} \in {\mathcal {C}}$ can be expanded to a matrix $\Gamma ({{\textbf {c}}}) \in {\mathcal {M}}_{m,n}(q)$, giving rise to a matrix code $\Gamma ({\mathcal {C}})$. For any two bases $\Gamma$ and $\Gamma '$, an equivalence between two vector rank codes ${\mathcal {C}}$ and ${\mathcal {D}}$ implies an equivalence between the matrix codes $\Gamma ({\mathcal {C}})$ and $\Gamma '({\mathcal {D}})$ [31], so VRCE is trivially a subproblem of MCE. However, using the ${\mathbb {F}}_{q^m}$-linearity of vector rank codes, VRCE reduces non-trivially to MCRE[17].

2.2 Systems of quadratic polynomials

Let ${\mathcal {P}}=(p_1,p_2,\dots ,p_k): {\mathbb {F}}_{q}^N \rightarrow {\mathbb {F}}_{q}^k$ be a vectorial function of k quadratic polynomials in N variables $x_1, \ldots , x_N$, where

$$\begin{aligned} p_s(x_1,\ldots ,x_N)=\sum \limits _{1\leqslant i\leqslant j\leqslant N}{\gamma }_{ij}^{(s)}x_ix_j + \sum _{i=1}^N{\beta }_i^{(s)}x_i + {\alpha }^{(s)}, \end{aligned}$$

with ${\gamma }_{ij}^{(s)},{\beta }_i^{(s)}, {\alpha }^{(s)}\in {\mathbb {F}}_{q}$ for $1\leqslant s\leqslant k$.

It is common to represent the quadratic homogeneous part of the components of ${\mathcal {P}}$ using symmetric matrices, but unfortunately, a natural correspondence only exists for finite fields of odd characteristic. For the case of even characteristic, we will adopt a technical representation that is a common workaround in the literature of multivariate cryptography and will still be good for our purposes.

Let $p(x_1,\ldots ,x_N)=\sum \limits _{1\leqslant i\leqslant j\leqslant N}{\gamma }_{ij}x_ix_j$ be a quadratic form over ${\mathbb {F}}_{q}$. Then, for fields of odd characteristic, we can associate to p a symmetric matrix ${{\textbf {P}}}={\overline{{{\textbf {P}}}}}+{\overline{{{\textbf {P}}}}}^\top$, where ${\overline{{{\textbf {P}}}}}$ is an upper triangular matrix with coefficients ${\overline{{{\textbf {P}}}}}_{ij}={\gamma }_{ij}/2$ for $i\leqslant j$. Clearly, there is a one-to-one correspondence between quadratic forms and symmetric matrices, since for ${{\textbf {x}}}=(x_1,\ldots ,x_N)$ it holds that

$$\begin{aligned} p(x_1,\ldots ,x_N)={{\textbf {x}}}{{\textbf {P}}}{{\textbf {x}}}^\top . \end{aligned}$$

(1)

Now, all operations on quadratic forms naturally transform into operations on matrices since the one-to-one correspondence between quadratic forms and symmetric matrices is in fact an isomorphism. Note that, in matrix form, a change of variables (basis) works as:

$$\begin{aligned} p({{\textbf {x}}}{{\textbf {S}}})={{\textbf {x}}}{{\textbf {S}}}{{\textbf {P}}}{{\textbf {S}}}^\top {{\textbf {x}}}^\top . \end{aligned}$$

(2)

In what follows, we will interchangeably work with both the quadratic form p and its matrix representation ${{\textbf {P}}}$.

Over fields ${\mathbb {F}}_{q}$ of even characteristic, the relation (1) does not hold, since for a symmetric matrix ${{\textbf {P}}}$ we have $({{\textbf {P}}}_{ij}+{{\textbf {P}}}_{ji})x_ix_j=2{{\textbf {P}}}_{ij}x_ix_j=0$. The nice correspondence between quadratic forms and symmetric matrices is broken, but we would still like to be able to use some sort of matrix representation for quadratic forms. Thus, in even characteristic we associate to p a symmetric matrix ${{\textbf {P}}}={\overline{{{\textbf {P}}}}}+{\overline{{{\textbf {P}}}}}^\top$, where ${\overline{{{\textbf {P}}}}}$ is an upper triangular matrix with coefficients ${\overline{{{\textbf {P}}}}}_{ij}={\gamma }_{ij}$ for $i\leqslant j$.

This representation can also be used in odd characteristic when it comes to linear operations and changes of basis, as the correspondence $p\mapsto {{\textbf {P}}}$ is a homomorphism. However, it is not a bijection, since all the quadratic forms in the set $\{\sum \limits _{1\leqslant i< j\leqslant N}{\gamma }_{ij}x_ix_j + \sum \limits _{1\leqslant i\leqslant N}{\gamma }_{ii}x_i^2 \mid \gamma _{ii}\in {\mathbb {F}}_{q}\}$ map to the same symmetric matrix (note that it has zeros on the diagonal). In practical, cryptographic applications, this typically does not pose a problem, and can be overcome. The same holds for our purpose of solving equivalence problems for systems of quadratic polynomials.

2.2.1 Differential of quadratic functions

Given a non-zero ${{\textbf {a}}} \in {\mathbb {F}}_{q}^N$, an object directly related to the symmetric matrix representation of quadratic forms is the differential of ${\mathcal {P}}$ at ${{\textbf {a}}}$ (see [22, 28]):

$$\begin{aligned} D_{{\textbf {a}}} {\mathcal {P}}: {\mathbb {F}}_{q}^N \rightarrow {\mathbb {F}}_{q}^k, \quad {{\textbf {x}}}\mapsto {\mathcal {P}}({{\textbf {x}}}+ {{\textbf {a}}}) - {\mathcal {P}}({{\textbf {x}}}) - {\mathcal {P}}({{\textbf {a}}}). \end{aligned}$$

Note that the differential of a quadratic function is closely related to the bilinear form $\beta ({{\textbf {x}}},{{\textbf {y}}})=q({{\textbf {x}}}+{{\textbf {y}}})-q({{\textbf {x}}})-q({{\textbf {y}}})$ associated to a quadratic form q. In this work we are especially interested in the kernel of $D_{{\textbf {a}}} {\mathcal {P}}$, as $D_{{\textbf {a}}} {\mathcal {P}}({{\textbf {x}}}) = 0$ implies ${\mathcal {P}}({{\textbf {x}}}+ {{\textbf {a}}}) = {\mathcal {P}}({{\textbf {x}}}) + {\mathcal {P}}({{\textbf {a}}})$, that is, ${\mathcal {P}}$ acts linearly on the kernel of $D_{{\textbf {a}}} {\mathcal {P}}$.

2.3 Isomorphism of polynomials

The Isomorphism of Polynomials (IP) problem (or Polynomial Equivalence (PE) [24]) was first defined by Patarin in [43] for the purpose of designing a “graph isomorphism”-like identification scheme and a digital signature scheme using the Fiat–Shamir transform [27]. It is defined as follows.

Problem 8

IP $(N,k,{\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k\times {\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k)$:

Input: Two k-tuples of multivariate polynomials ${\mathcal {F}}=(f_1,f_2,\ldots ,f_k),\ {\mathcal {P}}=(p_1,p_2,\ldots , p_k) \in {\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k$.

Question: Find—if any—$({{\textbf {S}}},{{\textbf {s}}})\in {\text {AGL}}_N(q),({{\textbf {T}}},{{\textbf {t}}}) \in {\text {AGL}}_k(q)$ such that

$$\begin{aligned} {\mathcal {P}}({{\textbf {x}}})={\mathcal {F}}({{\textbf {x}}}{{\textbf {S}}}+{{\textbf {s}}}){{\textbf {T}}}+{{\textbf {t}}}. \end{aligned}$$

(3)

The variant of the problem where $({{\textbf {T}}},{{\textbf {t}}})$ is trivial is known as the Isomorphism of Polynomials with one secret (IP $1{\mathcal {S}}$), whereas if ${\mathcal {P}}$ and ${\mathcal {F}}$ are quadratic and both ${{\textbf {s}}}$ and ${{\textbf {t}}}$ are the null vector, the problem is known as Quadratic Maps Linear Equivalence (QMLE) problem.

The decisional version of IP is not $\mathcal{N}\mathcal{P}$-complete [42], but it is known that even IP $1{\mathcal {S}}$ is at least as difficult as the Graph Isomorphism problem [42]. The IP problem has been investigated by several authors, initially for the security of the $C^*$ scheme [42]. In [44] it was shown that the IP $1{\mathcal {S}}$ is polynomially solvable for most of the instances with $k \ge N$, and Bouillaguet et al. [13] gave an algorithm with running time of ${\mathcal {O}}(N^6)$ for random instances of the IP $1{\mathcal {S}}$ problem, thus fully breaking Patarin’s identification scheme [43]. The authors of [42] gave an algorithm for solving the general IP, called To-and-Fro, that runs in time ${\mathcal {O}}(q^{2N})$ for $q > 2$ and ${\mathcal {O}}(q^{3N})$ for $q = 2$. It was noted in [14] that the algorithm is only suited for bijective mappings ${\mathcal {F}}$ and ${\mathcal {P}}$. Getting rid of the bijectivity constraint has been explored in [15] with the conclusion that the proposed workarounds either have a non-negligible probability of failure or it is unclear how greatly they affect the complexity of the algorithm.

Regarding QMLE, the linear variant of IP, an empirical argument was given in [24] that random inhomogeneous instances are solvable in ${\mathcal {O}}(N^9)$ time, but a rigorous proof for this case still remains an open problem. Under this assumption, the same paper provides an algorithm of complexity ${\mathcal {O}}(N^9q^N)$ for the homogeneous case which is considered the hardest, that was subsequently improved to ${\mathcal {O}}(N^9q^{2N/3})$ in [14]. Both works reduce a homogenous instance to an inhomogenous instance and assume the obtained inhomogeneous instance behaves as a random instance. This, however, is a wrong assumption which questions the claimed complexity of the algorithm.

In this work, we will be interested in the homogeneous variant of QMLE, that we denote hQMLE, as the hardest and most interesting instance of QMLE. Formally, the hQMLE problem is defined as follows.

Problem 9

hQMLE $(N,k,{\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k\times {\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k)$:

Input: Two k-tuples of homogeneous multivariate polynomials of degree 2

$$\begin{aligned} {\mathcal {F}}=(f_1,f_2,\ldots ,f_k),\ {\mathcal {P}}=(p_1,p_2,\ldots ,p_k) \in {\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k. \end{aligned}$$

Question: Find—if any—a map $({{\textbf {S}}},{{\textbf {T}}})$ where ${{\textbf {S}}}\in {\text {GL}}_N(q), {{\textbf {T}}}\in {\text {GL}}_k(q)$ such that

$$\begin{aligned} {\mathcal {P}}({{\textbf {x}}})=({\mathcal {F}}({{\textbf {x}}}{{\textbf {S}}})){{\textbf {T}}}. \end{aligned}$$

(4)

Interestingly, the case of $k=1$, which we will call Quadratic Form Equivalence (QFE) has been completely solved for more than 80 years already in the works of Witt [49] and Arf [50]. It is known that every quadratic form is equivalent to a unique canonical diagonal (for odd characteristic) or block diagonal (for even characteristic) form which can be obtained in time ${\mathcal {O}}(N^3)$. Thus, QFE can also be solved in time ${\mathcal {O}}(N^3)$ by first calculating the transformations to the canonical forms of the two quadratic forms. If the canonical forms are the same, by composition, one can find the equivalence. If the canonical forms are not the same, the two quadratic forms are not equivalent.

In this work, we also consider a variant of QMLE where ${\mathcal {F}}$ and ${\mathcal {P}}$ are bilinear forms. We call this problem Bilinear Maps Linear Equivalence (BMLE). In this variant, ${\mathcal {F}}$ and ${\mathcal {P}}$ are k-tuples of homogeneous polynomials of degree 2 in two sets of variables $[x_1, \dots , x_n]$ and $[y_1, \ldots , y_m]$, where each monomial is of the form $x_i y_j$. Formally, the BMLE problem is defined as follows.

Problem 10

BMLE $(n,m,k,{\mathbb {F}}_{q}[x_1,\ldots ,x_n,y_1, \ldots , y_m]^k\times {\mathbb {F}}_{q}[x_1,\ldots ,x_n,y_1, \ldots , y_m]^k)$:

Input: Two k-tuples of bilinear forms

$$\begin{aligned} {\mathcal {F}}=(f_1,f_2,\ldots ,f_k),\ {\mathcal {P}}=(p_1,p_2,\ldots ,p_k) \in {\mathbb {F}}_{q}[x_1,\ldots ,x_n,y_1, \ldots , y_m]^k \end{aligned}$$

Question: Find—if any—a triplet $({{\textbf {S}}}_1,{{\textbf {S}}}_2,{{\textbf {T}}})$ where ${{\textbf {S}}}_1\in {\text {GL}}_n(q), {{\textbf {S}}}_2\in {\text {GL}}_m(q)$, ${{\textbf {T}}}\in {\text {GL}}_k(q)$ such that

$$\begin{aligned} {\mathcal {P}}({{\textbf {x}}},{{\textbf {y}}})=({\mathcal {F}}({{\textbf {x}}}{{\textbf {S}}}_1,{{\textbf {y}}}{{\textbf {S}}}_2)){{\textbf {T}}}. \end{aligned}$$

(5)

The inhomogenous versions of QMLE and BMLE will be referred to as inhQMLE and inhBMLE respectively. We write $\textsf {inh(Q/B)MLE}$ when it does not matter if we are referring to the quadratic or the bilinear version.

3 How hard is MCE?

In this section we investigate the relation of the MCE problem to other known problems that we notably split in two groups—equivalence problems for systems of multivariate quadratic polynomials and equivalence problems for codes.

3.1 Relations to equivalence problems for qaudratic polynomials

We start with establishing a straightforward link between MCE and polynomial equivalence problems by proving that the MCE and BMLE problems are equivalent.

Theorem 11

The MCE problem is at least as hard as the BMLE problem.

Proof

In order to prove our claim, we need to show that an oracle ${\mathcal {A}}$ solving any instance of the MCE problem can be transformed in polynomial time to an oracle ${\mathcal {B}}$ solving any instance of the BMLE problem.

Suppose ${\mathcal {B}}$ is given an instance ${\mathcal {I}}_{{\textsf {BMLE}}}({\mathcal {F}},{\mathcal {P}})$ of BMLE $(n,m,k,{\mathbb {F}}_{q}[{{\textbf {x}}},{{\textbf {y}}}]^k\times {\mathbb {F}}_{q}[{{\textbf {x}}},{{\textbf {y}}}]^k)$, where ${\mathcal {F}}=(f_1,f_2,\ldots ,f_k)$, ${\mathcal {P}}=(p_1,p_2,\ldots ,p_k) \in {\mathbb {F}}_{q}[{{\textbf {x}}},{{\textbf {y}}}]^k$ are k-tuples of bilinear forms. Without loss of generality, we assume $f_1,f_2,\ldots ,f_k$ (respectively $p_1,p_2,\ldots ,p_k$) to be linearly independent. ${\mathcal {B}}$ can efficiently construct an instance of the MCE problem as follows.

${\mathcal {B}}$ represents the components $f_s$ and $p_s$, $s\in \{1,\dots ,k\}$ of the mappings ${\mathcal {F}}$ and ${\mathcal {P}}$ as $m\times n$ matrices ${{\textbf {F}}}^{(s)}$ and ${{\textbf {P}}}^{(s)}$, where ${{\textbf {F}}}^{(s)}_{i,j}$ equals the coefficient of $x_iy_j$ in $f_s$ and ${{\textbf {P}}}^{(s)}_{i,j }$ equals the coefficient of $x_iy_j$ in $p_s$. Taking $({{\textbf {F}}}^{(1)},\dots ,{{\textbf {F}}}^{(k)})$ to be a basis of a matrix code ${\mathcal {C}}$ and $({{\textbf {P}}}^{(1)},\dots ,{{\textbf {P}}}^{(k)})$ a basis of a matrix code ${\mathcal {D}}$, ${\mathcal {B}}$ obtains an instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ of MCE $(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$.

${\mathcal {B}}$ gives the instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ as an input to ${\mathcal {A}}$. ${\mathcal {A}}$ outputs either a solution $({{\textbf {A}}},{{\textbf {B}}})$ to the MCE instance (in the case it was a positive instance) or outputs that there is no solution (in the case it was a negative instance). In the latter case, ${\mathcal {B}}$ immediately outputs: no solution. In the former case, ${\mathcal {B}}$ constructs the matrices ${{\textbf {R}}}^{(s)}={{\textbf {A}}}{{\textbf {F}}}^{(s)}{{\textbf {B}}}\in {\mathcal {D}}$ and solves the following system of equations in the variables $t_{i,j}$:

$$\begin{aligned} \sum _{j=1}^{k}t_{j,i}\cdot {{\textbf {R}}}^{(j)}={{\textbf {P}}}^{(i)}, \forall i\in \{1,\dots ,k\} \end{aligned}$$

(6)

The system has always a solution, since $({{\textbf {R}}}^{(1)}, \dots , {{\textbf {R}}}^{(k)})$ is a basis of the code ${\mathcal {D}}$.

${\mathcal {B}}$ sets ${{\textbf {T}}}=\left( t_{i,j}\right)$, and outputs $({{\textbf {A}}},{{\textbf {B}}}^\top ,{{\textbf {T}}})$ as the solution to ${\mathcal {I}}_{{\textsf {BMLE}}}({\mathcal {F}},{\mathcal {P}})$. ${\mathcal {B}}$ succeeds whenever ${\mathcal {A}}$ succeeds and the reduction runs in time ${\mathcal {O}}(k^6)$. $\square$

Theorem 12

BMLE is at least as hard as MCE.

Proof

We proceed similarly as in the other direction—Given an oracle ${\mathcal {A}}$ solving any instance of BMLE, we can construct in polynomial time an oracle ${\mathcal {B}}$ with access to ${\mathcal {A}}$ that can solve any instance of MCE.

Suppose ${\mathcal {B}}$ is given an instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ of MCE $(n, m, k, {\mathcal {M}}^{[k]}_{m, n}(q))$. ${\mathcal {B}}$ takes arbitrary bases $({{\textbf {C}}}^{(1)},\dots ,{{\textbf {C}}}^{(k)})$ and $({{\textbf {D}}}^{(1)},\dots ,{{\textbf {D}}}^{(k)})$ of the codes ${\mathcal {C}}$ and ${\mathcal {D}}$ respectively. For each of the matrices ${{\textbf {C}}}^{(s)}$, ${\mathcal {B}}$ constructs the bilinear forms $c_s({{\textbf {x}}},{{\textbf {y}}})=\sum \limits _{1\leqslant i\leqslant m, 1\leqslant j\leqslant n}{{{\textbf {C}}}_{ij}^{(s)}x_iy_j}$ and for the matrices ${{\textbf {D}}}^{(s)}$ the bilinear forms $d_s({{\textbf {x}}},{{\textbf {y}}})=\sum \limits _{1\leqslant i\leqslant m, 1\leqslant j\leqslant n}{{{\textbf {D}}}_{ij}^{(s)}x_iy_j},\forall s, 1\leqslant s\leqslant k$. Taking ${\mathcal {F}}=(c_1,c_2,\dots ,c_k)$ and ${\mathcal {P}}=(d_1,d_2,\dots ,d_k)$ we obtain an instance ${\mathcal {I}}_{{\textsf {BMLE}}}({\mathcal {F}},{\mathcal {P}})$ of BMLE $(n,m,k,{\mathbb {F}}_{q}[{{\textbf {x}}},{{\textbf {y}}}]^k\times {\mathbb {F}}_{q}[{{\textbf {x}}},{{\textbf {y}}}]^k)$.

${\mathcal {B}}$ queries ${\mathcal {A}}$ with the instance ${\mathcal {I}}_{{\textsf {BMLE}}}({\mathcal {F}},{\mathcal {P}})$ and ${\mathcal {A}}$ outputs a solution $({{\textbf {S}}}_1,{{\textbf {S}}}_2, {{\textbf {T}}})$ to the BMLE instance, or no solution if there isn’t any. In the first case, this immediately gives a solution $({{\textbf {S}}}_1,{{\textbf {S}}}_2^\top )$ to the MCE instance. In the second case, there is no solution to the MCE instance. $\square$

In order to prove the connection of MCE to the more general problem hQMLE we first need to establish some properties of matrix codes.

Lemma 13

Let ${\mathcal {C}}$ and ${\mathcal {D}}$ be matrix codes generated by the bases $=({{\textbf {C}}}_1,\dots ,{{\textbf {C}}}_k)$ and $({{\textbf {D}}}_1,\dots ,{{\textbf {D}}}_k)$ of (skew-)symmetric matrices, and assume that ${\mathcal {C}}$ and ${\mathcal {D}}$ have trivial automorphism groups. Then ${\mathcal {C}}$ is equivalent to ${\mathcal {D}}$ if and only if ${\mathcal {C}}$ is congruent to ${\mathcal {D}}$.

Proof

Clearly, by definition if ${\mathcal {C}}$ is congruent to ${\mathcal {D}}$, then ${\mathcal {C}}$ is equivalent to ${\mathcal {D}}$.

For the opposite direction, let ${\mathcal {C}}$ be equivalent to ${\mathcal {D}}$. Then there exist nonsingular matrices ${{\textbf {A}}}$, ${{\textbf {B}}}$ and ${{\textbf {T}}}$ such that

$$\begin{aligned} \sum _{i=1}^{k}t_{j,i}{{\textbf {D}}}_i = {{\textbf {A}}}{{\textbf {C}}}_j{{\textbf {B}}}\end{aligned}$$

Since ${{\textbf {C}}}_i$ and ${{\textbf {D}}}_i$ are (skew-)symmetric the last rewrites as

$$\begin{aligned} \sum _{i=1}^{k}t_{j,i}{{\textbf {D}}}_i = {{\textbf {B}}}^\top {{\textbf {C}}}_j{{\textbf {A}}}^\top \end{aligned}$$

Combining the two, and since ${{\textbf {A}}}$ and ${{\textbf {B}}}$ are non-singular, we obtain

$$\begin{aligned} {{\textbf {C}}}_j={{\textbf {A}}}^{-1}{{\textbf {B}}}^\top {{\textbf {C}}}_j{{\textbf {A}}}^\top {{\textbf {B}}}^{-1} \end{aligned}$$

The automorphism group being trivial implies ${{\textbf {A}}}=\lambda {{\textbf {B}}}^\top$ for some $\lambda \in {\mathbb {F}}_{q}$ which in turn implies that ${\mathcal {C}}$ is congruent to ${\mathcal {D}}$. $\square$

Remark 14

The result of Lemma 13 has already been known for algebraically closed fields of non-even characteristic [6, 47]. Since finite fields are not algebraically closed, this result is not useful in our context. On the other hand, requiring a trivial automorphism group for the codes is not a huge restriction, and we typically expect the automorphism group to be trivial for randomly chosen matrix codes. Specifically for cryptographic purposes with regards to MCE, one wants the orbit of ${\mathcal {C}}$ to be maximal under the action of suitable isometries, which happens when the automorphism group of ${\mathcal {C}}$ is trivial. Similar requirements for trivial or small automorphism groups occur in the Hamming metric, where it is known that without this requirement there might exist weak keys [25, 26].

Theorem 15

Let ${\mathcal {T}}$ denote the subset of ${\mathcal {M}}^{[k]}_{m, n}(q)$ of k-dimensional matrix codes of symmetric matrices with trivial automorphism groups. Further, let ${\mathcal {T}}'$ denote the subset of ${\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k$ of k-tuples of polynomials with trivial automorphism groups.

The MCE $({\mathcal {T}})$ problem is at least as hard as the hQMLE $({\mathcal {T}}')$ problem

Proof

We perform the reduction in a similar manner as previously.

Suppose ${\mathcal {B}}$ is given an instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ of hQMLE $(N,k,{\mathcal {T}}')$, where ${\mathcal {F}}=(f_1,f_2,\ldots ,f_k)$, ${\mathcal {P}}=(p_1,p_2,\ldots ,p_k) \in [x_1,\ldots ,x_N]$ are k-tuples of linearly independent quadratic forms from ${\mathcal {T}}'$. ${\mathcal {B}}$ can efficiently construct an instance of the MCE $(N,N,k,{\mathcal {T}})$ problem as follows.

${\mathcal {B}}$ forms the $N\times N$ symmetric matrices ${{\textbf {F}}}^{(s)}$ and ${{\textbf {P}}}^{(s)}$ associated to the components $f_s$ and $p_s$, $s\in \{1,\dots ,k\}$ of the mappings ${\mathcal {F}}$ and ${\mathcal {P}}$. Taking $({{\textbf {P}}}^{(1)},\dots ,{{\textbf {P}}}^{(k)})$ to be a basis of a matrix code ${\mathcal {D}}$ and $({{\textbf {F}}}^{(1)},\dots ,{{\textbf {F}}}^{(k)})$ a basis of a matrix code ${\mathcal {C}}$, ${\mathcal {B}}$ obtains an instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ of MCE. Per assumption, the matrix codes ${\mathcal {C}}$ and ${\mathcal {D}}$ have trivial automorphism groups, hence the instance is from MCE $(N,N,k,{\mathcal {T}})$.

${\mathcal {B}}$ queries ${\mathcal {A}}$ with the instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$., ${\mathcal {A}}$ answers with a solution $({{\textbf {A}}},{{\textbf {B}}})$ to the MCE instance if it is positive, and no solution otherwise. In the former case, from Lemma 13, since the matrices are symmetric, ${{\textbf {A}}}={{\textbf {B}}}^\top$. Now, ${\mathcal {B}}$ applies the change of variables ${{\textbf {x}}}{{\textbf {A}}}$ to ${\mathcal {F}}$ and obtains ${\mathcal {R}}({{\textbf {x}}})={\mathcal {F}}({{\textbf {x}}}{{\textbf {A}}})$. It then solves the system

$$\begin{aligned} \sum _{j=1}^{k}t_{j,s}\cdot r_j=p_s, \forall s\in \{1,\dots ,k\} \end{aligned}$$

(7)

The system has a solution if ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ is a positive instance. This is always the case in odd characteristic, because there is a one-to-one correspondence between polynomials and their symmetric matrix representation. Over characteristic 2, it may happen that the ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ is not a positive instance while its symmetric matrix representation ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ is. In this case, the system (7) does not have a solution and ${\mathcal {B}}$ outputs no solution.

If the system has a solution, ${\mathcal {B}}$ sets ${{\textbf {T}}}=\left( t_{i,j}\right)$, and outputs $({{\textbf {A}}},{{\textbf {T}}})$ as the solution to ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$. ${\mathcal {B}}$ succeeds whenever ${\mathcal {A}}$ succeeds and the reduction takes time ${\mathcal {O}}(k^6)$. $\square$

For the following theorem, we define the symmetric matrix representation of a matrix code ${\mathcal {C}}$ as the code $\{ \ \left[ \begin{matrix} {{\textbf {0}}} &{} {{\textbf {C}}}^\top \\ {{\textbf {C}}} &{} {{\textbf {0}}} \end{matrix}\right] \ |\ {{\textbf {C}}}\in {\mathcal {C}}\}$.

Theorem 16

Let ${\mathcal {T}}_s$ denote the subset of ${\mathcal {M}}^{[k]}_{m, n}(q)$ of k-dimensional matrix codes whose symmetric matrix representation has a trivial automorphism group. Similarly, let ${\mathcal {T}}_s'$ denote the subset of ${\mathbb {F}}_{q}[x_1,\ldots ,x_N]^k$ of k-tuples of polynomials with trivial automorphism groups.

The hQMLE $({\mathcal {T}}_s')$ problem is at least as hard as the MCE $({\mathcal {T}}_s)$ problem.

Proof

We show that given any oracle ${\mathcal {A}}$ that solves the hQMLE $({\mathcal {T}}_s')$ problem there exists an oracle ${\mathcal {B}}$ running in polynomial time that solves the MCE $({\mathcal {T}}_s)$.

Suppose ${\mathcal {B}}$ is given an instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ of MCE $(n,m,k,{\mathcal {T}}_s)$. ${\mathcal {B}}$ can efficiently construct an instance of the hQMLE $(n+m,k,{\mathcal {T}}'_s)$ problem as follows.

${\mathcal {B}}$ fixes bases $({{\textbf {D}}}^{(1)},\dots ,{{\textbf {D}}}^{(k)})$ of the code ${\mathcal {D}}$ and $({{\textbf {C}}}^{(1)},\dots ,{{\textbf {C}}}^{(k)})$ of the code ${\mathcal {C}}$. For each of the matrices ${{\textbf {C}}}^{(s)}$, ${\mathcal {B}}$ constructs the quadratic forms $c_s({{\textbf {x}}})=\sum \limits _{1\leqslant i\leqslant m, m+1\leqslant j\leqslant m+n}{{{\textbf {C}}}_{ij}^{(s)}x_ix_j}$ and for the matrices ${{\textbf {D}}}^{(s)}$ the quadratic forms $d_s({{\textbf {x}}})=\sum \limits _{1\leqslant i\leqslant m, m+1\leqslant j\leqslant m+n}{{{\textbf {D}}}_{ij}^{(s)}x_ix_j},\forall s, 1\leqslant s\leqslant k$, where ${{\textbf {x}}}=(x_1,\dots ,x_{m+n})$. Taking ${\mathcal {F}}=(c_1,c_2,\dots ,c_k)$ and ${\mathcal {P}}=(d_1,d_2,\dots ,d_k)$ ${\mathcal {B}}$ obtains an instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ of hQMLE $(n+m,k,{\mathcal {T}}'_s)$.

${\mathcal {B}}$ queries ${\mathcal {A}}$ with the instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ which outputs a solution $({{\textbf {S}}}, {{\textbf {T}}})$ to the hQMLE instance.

We argue that this solution can be transformed to a solution to the MCE instance, if it is a positive instance. The symmetric matrix representation of the codes ${\mathcal {C}}$ and ${\mathcal {D}}$ is given by

$$\begin{aligned} \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(i)})^\top \\ {{\textbf {D}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right] \text { and } \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {C}}}^{(i)})^\top \\ {{\textbf {C}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right] , i\in \{1,\dots ,k\}. \end{aligned}$$

(8)

The solution $({{\textbf {S}}},{{\textbf {T}}})$ means

$$\begin{aligned} \sum _{}{\tilde{t}}_{i,j} \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(j)})^\top \\ {{\textbf {D}}}^{(j)} &{} {{\textbf {0}}} \end{matrix}\right] = {{\textbf {S}}}\left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {C}}}^{(i)})^\top \\ {{\textbf {C}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right] {{\textbf {S}}}^\top , i\in \{1,\dots ,k\}. \end{aligned}$$

(9)

If the given MCE instance is positive, then there exist matrices ${{\textbf {A}}},{{\textbf {B}}},{{\textbf {L}}}$ such that ${{\textbf {A}}}{{\textbf {C}}}_i{{\textbf {B}}}=\sum _{j}l_{i,j}{{\textbf {D}}}_j$. This implies

$$\begin{aligned} \sum _{}l_{i,j} \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(j)})^\top \\ {{\textbf {D}}}^{(j)} &{} {{\textbf {0}}} \end{matrix}\right] = \left[ \begin{matrix} {{\textbf {B}}}^\top &{} {{\textbf {0}}}\\ {{\textbf {0}}} &{} {{\textbf {A}}}\end{matrix}\right] \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {C}}}^{(i)})^\top \\ {{\textbf {C}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right] \left[ \begin{matrix} {{\textbf {B}}}&{} {{\textbf {0}}}\\ {{\textbf {0}}} &{} {{\textbf {A}}}^\top \end{matrix}\right] , i\in \{1,\dots ,k\}. \end{aligned}$$

(10)

The last two imply

$$\begin{aligned} \sum _{}\lambda _{i,j} \left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(j)})^\top \\ {{\textbf {D}}}^{(j)} &{} {{\textbf {0}}} \end{matrix}\right] = \left[ \begin{matrix} {{\textbf {B}}}^\top &{} {{\textbf {0}}}\\ {{\textbf {0}}} &{} {{\textbf {A}}}\end{matrix}\right] {{\textbf {S}}}^{-1}\left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(i)})^\top \\ {{\textbf {D}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right] {{\textbf {S}}}^{-\top }\left[ \begin{matrix} {{\textbf {B}}}&{} {{\textbf {0}}}\\ {{\textbf {0}}} &{} {{\textbf {A}}}^\top \end{matrix}\right] , i\in \{1,\dots ,k\}. \end{aligned}$$

(11)

By assumption, the automorphism group of the $\left[ \begin{matrix} {{\textbf {0}}} &{} ({{\textbf {D}}}^{(i)})^\top \\ {{\textbf {D}}}^{(i)} &{} {{\textbf {0}}} \end{matrix}\right]$ matrices is trivial, which means ${{\textbf {S}}}$ necessarily equals $\left[ \begin{matrix} {{\textbf {B}}}^\top &{} {{\textbf {0}}}\\ {{\textbf {0}}} &{} {{\textbf {A}}}\end{matrix}\right]$ up to scalar multiplication. For such an ${{\textbf {S}}}$, the MCE solution can immediately be extracted. ${\mathcal {B}}$ then outputs the extracted solution.

If on the other hand, ${{\textbf {S}}}$ is not of such block-diagonal form, ${\mathcal {B}}$ outputs no solution, as this implies the instance is not positive. $\square$

Remark 17

Using the above reduction between MCE and hQMLE, we can reduce the MCE base problem to and from a special case of IP known as IP $1{\mathcal {S}}$. Interestingly, Perret [44] shows IP $1{\mathcal {S}}$ is polynomially solvable for most instances $k \ge N$, and later work [13] gives an algorithm with running time of ${\mathcal {O}}(N^6)$ for most random instances, although no rigorous proof that bounds the complexity of the problem to polynomial was given. This nevertheless implies that the MCE base problem can practically be solved in polynomial time for most cryptographically interesting parameters.

3.2 Relations to equivalence problems for linear codes

In this section, we show that MCE is at the heart of various code equivalence problems. Equivalence problems for different metrics, such as the Hamming metric or the sum-rank metric, reduce to MCE, making the hardness analysis of MCE the more exciting.

3.2.1 Hamming code equivalence

Codes ${\mathcal {C}}\subset {\mathbb {F}}_{q}^n$ equipped with the Hamming metric have isometries of the form

$$\begin{aligned} \tau : (c_1, \ldots , c_n) \mapsto (\alpha _1 c_{\pi ^{-1}(1)}, \ldots , \alpha _n c_{\pi ^{-1}(n)}), \quad \alpha _i \in {\mathbb {F}}_{q}^*,\ \pi \in S_n. \end{aligned}$$

(12)

From this, we define Hamming code equivalence (HCE) as the problem of finding an isometry between two Hamming codes ${\mathcal {C}}$ and ${\mathcal {D}}$.

Problem 18

$\textsf {HCE}(k, n)$:

Input: Two k-dimensional Hamming codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathbb {F}}_{q}^n$

Question: Find—if any—$\alpha \in {{\mathbb {F}}_{q}^*}^n, \pi \in S_n$ such that $\alpha \pi ({{\textbf {c}}}) \in {\mathcal {D}}$ holds for all ${{\textbf {c}}} \in {\mathcal {C}}$.

The subproblem where $\alpha$ is trivial is called the monomial equivalence problem.

It is easy to turn an HCE instance into a MCE instance [17], given the description of isometries in Eq. (12). First, define $\Phi : {\mathbb {F}}_{q}^n \rightarrow {\mathcal {M}}_n({\mathbb {F}}_{q})$ by

$$\begin{aligned} {{\textbf {x}}} = (x_1, \ldots , x_n)&\mapsto \begin{pmatrix} x_1 &{}&{}\\ &{} \ddots &{}\\ &{}&{} x_n \end{pmatrix}. \end{aligned}$$

The map $\Phi$ is an isometry from the Hamming metric to the rank metric: codewords with weigh t are mapped to matrices of rank t. From this, we quickly get the reduction: Writing $\pi$ as a matrix ${{\textbf {P}}} \in {\text {GL}}_n(q)$, $\Phi$ translates a Hamming isometry $\tau$ to a rank-metric isometry by

$$\begin{aligned} \Phi (\tau ): \Phi ({{\textbf {x}}}) \mapsto {{\textbf {P}}}^{-1} \Phi ({{\textbf {x}}}) {{\textbf {A}}} {{\textbf {P}}}, \quad \text {where } {{\textbf {A}}} = \begin{pmatrix} \alpha _1 &{}&{}\\ &{} \ddots &{}\\ &{}&{} \alpha _n \end{pmatrix} \in {\text {GL}}_n(q). \end{aligned}$$

A second reduction from HCE to MCE is given later in [17], which concerns the search variant of the problem, and is more explicit. Both reductions, however, do not help with solving HCE in practice: both the permutational (${{\textbf {A}}}$ is trivial) and the linear variant of code equivalence in the Hamming metric have algorithms [4, 45] that perform much better for an HCE instance $\tau$ than the algorithms we propose for solving $\Phi (\tau )$ as an MCE instance.

3.2.2 Sum-rank code equivalence

The sum-rank metric [41] is a metric that is gaining in popularity in coding theory. It is commonly given as a generalization of the vector-rank metric, but one can also define a variant that generalizes matrix-rank metric. We will reduce both vector and matrix sum-rank equivalence problems to MCE. The idea is the same as for HCE, we find the right isometry from sum-rank metric to rank metric to get the reduction.

Definition 19

Let n be partitioned as $n = n_1 + \ldots + n_\ell$. Let ${{\textbf {v}}}^{(i)} = (v^{(i)}_1, \ldots , v^{(i)}_{n_i}) \in {\mathbb {F}}_{q^m}^{n_i}$ and ${{\textbf {v}}} = ({{\textbf {v}}}^{(1)}, \ldots , {{\textbf {v}}}^{(\ell )}) \in {\mathbb {F}}_{q^m}^n$. Let $\Gamma$ be a basis for ${\mathbb {F}}_{q^m}$ over ${\mathbb {F}}_{q}$. Then the vector sum-rank of ${{\textbf {v}}}$ is defined as

$$\begin{aligned} {{\,\textrm{SumRank}\,}}({{\textbf {v}}}):= \sum _{i =1}^\ell {{\,\textrm{Rank}\,}}\Gamma ({{\textbf {v}}}^{(i)}). \end{aligned}$$

Let m be partitioned as $m = m_1 + \ldots + m_\ell$. Let ${{\textbf {V}}}^{(i)} \in {\mathcal {M}}_{m_i \times n_i}({\mathbb {F}}_{q})$ and ${{\textbf {V}}} = ({{\textbf {V}}}^{(1)}, \ldots , {{\textbf {V}}}^{(\ell )})$. Then the matrix sum-rank of ${{\textbf {V}}}$ is defined as

$$\begin{aligned} {{\,\textrm{SumRank}\,}}({{\textbf {V}}}) = \sum _{i = 1}^\ell {{\,\textrm{Rank}\,}}{{\textbf {V}}}^{(i)}. \end{aligned}$$

The sum-rank generalizes both the Hamming metric and the rank metric: taking $\ell = n$ gives the Hamming metric, whereas $\ell = 1$ gives the rank metric. We define isometries again as maps that preserve the sum-rank. Sum-rank isometries are simple generalisations of rank isometries (see Problem 7).

Proposition 20

[1, Thm. 3.7] Isometries with respect to the vector sum-rank metric are given by vector rank isometries $\mu ^{(i)}: {{\textbf {x}}}^{(i)} \mapsto \alpha ^{(i)} {{\textbf {x}}}^{(i)} {{\textbf {B}}}^{(i)}$ per ‘block’ with $\alpha ^{(i)} \in {\mathbb {F}}_{q^m}^*$ and ${{\textbf {B}}}^{(i)} \in {\text {GL}}_{n_i}(q)$, and suitable permutations $\pi$ of such blocks if $n_i = n_j$ for $i \ne j$, so

$$\begin{aligned} \mu : ({{\textbf {x}}}^{(1)}, \ldots , {{\textbf {x}}}^{(\ell )}) \mapsto ( \alpha ^{(1)} {{\textbf {x}}}^{\pi ^{-1}(1)} {{\textbf {B}}}^{(1)}, \ldots , \alpha ^{(\ell )} {{\textbf {x}}}^{\pi ^{-1}(\ell )} {{\textbf {B}}}^{(\ell )} ) \end{aligned}$$

is a general description of a vector sum-rank isometry.

Generalizing to matrix sum-rank codes is achieved by simply replacing $\alpha ^{(i)} \in {\mathbb {F}}_{q^m}^*$ with ${{\textbf {A}}}^{(i)} \in {\text {GL}}_{m_i}(q)$ [39, Prop. 4.25]. This gives us the Vector Sum-Rank Code Equivalence (VSRCE) and Matrix Sum-Rank Code Equivalence (MSRCE) problems.

Problem 21

VSRCE(n, m, k):

Input: Two k-dimensional vector sum-rank codes ${\mathcal {C}},{\mathcal {D}}\subset {\mathbb {F}}_{q^m}^n$

Question: Find—if any—$\alpha ^{(i)} \in {\mathbb {F}}_{q^m}^*, {{\textbf {B}}}^{(i)} \in {\text {GL}}_{n_i}(q)$ and a permuation $\pi$ such that for all ${{\textbf {c}}} \in {\mathcal {C}}$, it holds that $\mu ({{\textbf {c}}}) \in {\mathcal {D}}$.

Problem 22

$\textsf {MSRCE}(n, m, k)$:

Input: Two k-dimensional matrix sum-rank codes ${\mathcal {C}},{\mathcal {D}}\subset \left( {\mathcal {M}}_{m_i \times n_i}({\mathbb {F}}_{q})\right) _i$

Question: Find—if any—${{\textbf {A}}}^{(i)} \in {\text {GL}}_{m_i}(q), {{\textbf {B}}}^{(i)} \in {\text {GL}}_{n_i}(q)$ and a permuation $\pi$ such that for all ${{\textbf {C}}} \in {\mathcal {C}}$, it holds that $\mu ({{\textbf {C}}}) \in {\mathcal {D}}$.

In order to give a reduction to MCE, we use the same idea as for HCE. First, we define a ‘nice’ map $\Psi : {\mathbb {F}}_{q}^n \rightarrow {\mathcal {M}}_{\ell \cdot m \times n}({\mathbb {F}}_{q})$ by

$$\begin{aligned} {{\textbf {x}}} = ({{\textbf {x}}}^{(1)}, \ldots , {{\textbf {x}}}^{(\ell )}) \mapsto \begin{pmatrix} {\text {Mat}}({{\textbf {x}}}^{(1)}) &{}&{}\\ &{} \ddots &{}\\ &{}&{} {\text {Mat}}({{\textbf {x}}}^{(\ell )}) \end{pmatrix}. \end{aligned}$$

It is clear that $\Psi$ is an isometry from the vector sum-rank metric to the rank metric, as it preserves the weight. We get the following reduction.

Theorem 23

Let ${\mathcal {T}}$ denote the subset of ${\mathcal {M}}^{[k]}_{m, n}(q)$ of k-dimensional matrix codes with trivial automorphism groups. Let $\mathcal {T'}$ denote the subset of k-dimensional vector sum-rank codes that are in the preimage $\Psi ^{-1}({\mathcal {T}})$. Then MCE $({\mathcal {T}})$ is at least as hard as VSRCE $(\mathcal {T'})$.

Proof

Suppose ${\mathcal {B}}$ is given an instance ${\mathcal {I}}_{{\textsf {VSRCE}}}({\mathcal {C}}, {\mathcal {D}})$ of VSRCE $(n, m, k, {\mathcal {T}}')$, where ${\mathcal {C}}$ and ${\mathcal {D}}$ are k-dimensional vector sum-rank codes. ${\mathcal {B}}$ can efficiently construct an instance of the MCE $({\mathcal {T}})$ problem as follows. By writing the permutation $\pi$ of the ‘blocks’ by a matrix representation ${{\textbf {P}}}$, ${\mathcal {B}}$ can translate a vector sum-rank isometry $\mu$ into a matrix code isometry $\Psi (\mu )$ by

$$\begin{aligned} \Psi (\mu ): \Psi ({{\textbf {x}}}) \mapsto {{\textbf {P}}}^{-1} {{\textbf {A}}} \Psi ({{\textbf {x}}}) {{\textbf {B}}} {{\textbf {P}}} \quad \text {where } {{\textbf {A}}} = \begin{pmatrix} \alpha ^{(1)} &{}&{}\\ &{} \ddots &{}\\ &{}&{} \alpha ^{(\ell )} \end{pmatrix}, {{\textbf {B}}} = \begin{pmatrix} {{\textbf {B}}}^{(1)} &{}&{}\\ &{} \ddots &{}\\ &{}&{} {{\textbf {B}}}^{(\ell )} \end{pmatrix} \end{aligned}$$

with ${{\textbf {A}}} \in {\text {GL}}_\ell (q^m)$, ${{\textbf {B}}} \in {\text {GL}}_n(q)$. Hence, $\Psi (\mu )$ is an instance of MCE $(n, m, k, {\mathcal {T}})$, with which ${\mathcal {B}}$ queries ${\mathcal {A}}$. ${\mathcal {A}}$ outputs a solution $({{\textbf {A}}}', {{\textbf {B}}}')$ to this MCE $({\mathcal {T}})$ instance. As the automorphism group is trivial, ${\mathcal {B}}$ computes $\lambda {{\textbf {A}}}' = {{\textbf {P}}}^{-1} {{\textbf {A}}}$ and $\lambda {{\textbf {B}}}' = {{\textbf {B}}} {{\textbf {P}}}$ for $\lambda \in {\mathbb {F}}_{q}$, and therefore solves the ${\mathcal {I}}_{{\textsf {VSRCE}}}$ instance. $\square$

From vector sum-rank code equivalence to matrix sum-rank code equivalence is only a small step. Given a partition $m = m_1 + \ldots + m_\ell$, the map we need is only slightly different from $\Psi$, namely ${\tilde{\Psi }}: \left( {\mathcal {M}}_{m_i \times n_i}({\mathbb {F}}_{q})\right) _i \rightarrow {\mathcal {M}}_{m \times n}({\mathbb {F}}_{q})$ by

$$\begin{aligned} {{\textbf {X}}} = ({{\textbf {X}}}^{(1)}, \ldots , {{\textbf {X}}}^{(\ell )}) \mapsto \begin{pmatrix} {{\textbf {X}}}^{(1)} &{}&{}\\ &{} \ddots &{}\\ &{}&{} {{\textbf {X}}}^{(\ell )} \end{pmatrix}. \end{aligned}$$

Theorem 24

Let ${\mathcal {T}}$ denote the subset of ${\mathcal {M}}^{[k]}_{m, n}(q)$ of k-dimensional matrix codes with trivial automorphism groups. Let $\mathcal {T'}$ denote the subset of k-dimensional matrix sum-rank codes that are in the preimage ${\tilde{\Psi }}^{-1}({\mathcal {T}})$. Then MCE $({\mathcal {T}})$ is at least as hard as MSRCE $(\mathcal {T'})$.

Proof

This is a simple generalization of Theorem 23: Replace $\alpha ^{(i)}$ by ${{\textbf {A}}}^{(i)} \in {\text {GL}}_{m_i}(q)$ so that ${{\textbf {A}}} \in {\text {GL}}_m(q)$. Then again, for a matrix sum-rank $\mu$ we get ${\tilde{\Psi }}(\mu )$ by $\Psi ({{\textbf {x}}}) \mapsto {{\textbf {P}}}^{-1} {{\textbf {A}}} \Psi ({{\textbf {x}}}) {{\textbf {B}}} {{\textbf {P}}}$ as an MCE $({\mathcal {T}})$ instance. $\square$

The link between such MCE instances $\Psi (\mu )$ coming from vector sum-rank and ${\tilde{\Psi }}(\mu )$ coming from matrix sum-rank is given by a representation $\rho : {\mathbb {F}}_{q^m}^* \rightarrow {\text {GL}}_m(q)$. We map a vector sum-rank instance to a matrix sum-rank instance by ${{\textbf {A}}}^{(i)} = \rho (\alpha ^{(i)})$, so that ${{\textbf {A}}} \in {\text {GL}}_{\ell \cdot m}(q)$.

To show the equivalences between the rank and sum-rank instances, we need to show that an MCE instance is also an MSRCE instance. But this is trivial: the sum-rank metric generalizes the rank metric, thus an MCE instance is an MSRCE instance with $\ell = 1$. Hence, we get the following theorem for free.

Theorem 25

MSRCE is at least as hard as MCE.

4 Solving Matrix Code Equivalence

In this section, we analyze the complexity of solving an instance of MCE(n, m, k). We start by establishing a useful lemma.

Lemma 26

An MCE(n, m, k) instance can in polynomial time be turned into an MCE $(\sigma (n),\sigma (m),\sigma (k))$ instance for any permutation $\sigma$ on the set $\{n,m,k\}$. Furthermore, they are either both positive, or both negative instances.

Proof

Let ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ be a given MCE(n, m, k) instance. Let $({{\textbf {C}}}^{(1)},\dots ,{{\textbf {C}}}^{(k)})$ and $({{\textbf {D}}}^{(1)},\dots ,{{\textbf {D}}}^{(k)})$ be bases of the codes ${\mathcal {C}}$ and ${\mathcal {D}}$ respectively. Without loss of generality, we will turn this instance into an MCE(m, k, n) instance (the rest can be done analogously). We set $\bar{{{\textbf {C}}}}^{(i)}_{j,t}={{\textbf {C}}}^{(t)}_{i,j}$, $\bar{{{\textbf {D}}}}^{(i)}_{j,t}={{\textbf {D}}}^{(t)}_{i,j}$ and we take $(\bar{{{\textbf {C}}}}^{(1)},\dots ,\bar{{{\textbf {C}}}}^{(n)})$ and $(\bar{{{\textbf {D}}}}^{(1)},\dots ,\bar{{{\textbf {D}}}}^{(n)})$ to be the bases of the codes $\bar{{\mathcal {C}}}$ and $\bar{{\mathcal {D}}}$ respectively. Clearly, $\bar{{\mathcal {C}}}$ and $\bar{{\mathcal {D}}}$ are equivalent if and only if ${\mathcal {C}}$ and ${\mathcal {D}}$ are equivalent. $\square$

Without loss of generality, and with Lemma 26 in mind, we assume $m=\min \{m,n,k\}$.

As a baseline we have a straightforward algorithm that uses a result from [17] that MCRE can be solved in polynomial time. By enumerating either ${{\textbf {A}}}$ or ${{\textbf {B}}}$, we obtain an instance of MCRE. This means the dominating complexity is the enumeration resulting in an overall complexity of $\tilde{{\mathcal {O}}}(q^{m^2})$ for MCE.

The approach we outline in the section makes use of the reduction of MCE to hQMLE (see Theorem 16). This means that we use techniques already applied for solving hQMLE, but generalize and improve them by making use of the specific structure that MCE instances show when viewed as hQMLE instances.

4.1 Solving MCE as QMLE

At Eurocrypt 2013, Bouillaguet et al. [14] proposed an algorithm for solving hQMLE using techniques from graph theory. Their main idea was to reduce the homogeneous case to the inhomogeneous case, which they assume is efficiently solvable (e.g. using the heuristic algebraic approach of [24]). Starting from an instance of hQMLE, they build two exponentially-large graphs that correspond to the given maps ${\mathcal {F}}$ and ${\mathcal {P}}$ such that, finding an isomorphism between the two graphs is equivalent to finding an isomorphism between the two quadratic maps. Since the graphs are exponentially large, a technique is provided to walk through the graphs without constructing them. Walking through the graphs consists of finding adjacent vertices and computing the degree of a vertex, both in polynomial time. The algorithm consists in finding pairs of vertices from the first and the second graph that have the same degree and making queries to an inhomogenous QMLE solver. If the solver finds an isomorphism by which two vertices are related, then the isomorphism between the two graphs, and thus the isomorphism between the two quadratic maps, is found.

4.2 First algorithm for solving MCE

The algorithm for solving hQMLE from [14] considers a graph arising from the differential of a given polynomial map—a vertex ${{\textbf {a}}}$ is connected to all the vertices that vanish at the differential at ${{\textbf {a}}}$. It is, however, not entirely clear how the property we choose to construct such graphs impacts the complexity of the algorithm. We revisit the algorithm, and show how it can be generalized, i.e. abstracted from the property used in [14], under certain conditions. In this section we present this generalization—a birthday-based algorithm for finding an isomorphism between two objects when a specific solver exists. In this form, it can be applied to a broader type of equivalence problems, using more general invariants, here implemented as a predicate ${\mathbb {P}}$.

Let $S_1$ and $S_2$ be subsets of a universe U of equal size N. Algorithm 1 finds an equivalence function $\phi : S_1\rightarrow S_2$. We assume there exists a predicate ${\mathbb {P}}:U\rightarrow \{\top ,\bot \}$ that can be computed in polynomial time, and we denote the cost $C_{\mathbb {P}}$. We assume ${\mathbb {P}}$ is invariant under the equivalence $\phi$, i.e. ${\mathbb {P}}(x)=\top \leftrightarrow {\mathbb {P}}(\phi (x))=\top$. Let $U_{\top } = \{ x \in U \ |\ {\mathbb {P}}(x) = \top \}$, and $d=\vert U_\top |/ |U |$. We will call d the density of the predicate ${\mathbb {P}}$ and we assume the density on $S_1$ and $S_2$ is approximately equal to d. We further assume the existence of an algorithm FindFunction, that given $x\in S_1, y\in S_2$ returns $\phi$ if $y=\phi (x)$ and $\bot$ otherwise. We denote the cost of a query to FindFunction by $C_\textsc {FF}$.

Algorithm 1

General Birthday-based Equivalence Finder

Lemma 27

For a fixed success probability of $1-1/e$, Algorithm 1 performs on average ${\mathcal {O}}( \sqrt{N/d} )$ operations in SampleSet, queries FindFunction at most $d \cdot N$ times.

The optimal value for d, up to a polynomial factor, is $d = N^{-1/3} \cdot C_\textsc {FF}^{-2/3}$, for which the total time complexity of the algorithm is ${\mathcal {O}}(N^{\frac{2}{3}}\cdot C_\textsc {FF}^\frac{1}{3})$ and the memory complexity is ${\mathcal {O}}(N^{\frac{1}{3}}C_\textsc {FF}^{-\frac{1}{3}})$. If FindFunction runs in polynomial time, this reduces to time complexity of $\tilde{{\mathcal {O}}}(N^{\frac{2}{3}})$ and memory complexity of ${\mathcal {O}}(N^{\frac{1}{3}})$.

Proof

First note that the expected number of elements in $S_1$ and $S_2$ such that ${\mathbb {P}}(x)$ holds is equal to dN by the definition of the density d. By the birthday paradox, it is enough to take lists of size $\ell =\sqrt{d\cdot N}$, to be sure that with probability of $1 - \frac{1}{e}$ FindFunction returns a solution [51]. With this length of the lists, the number of queries to FindFunction is dN. On the other hand, the number of samples needed to build the list $L_1$ (resp. $L_2$) of elements $a \in S_1$ (resp. $b\in S_2$) such that ${\mathbb {P}}(a)$ (resp. ${\mathbb {P}}(b)$) holds is $\ell /d$, which gives a complexity of ${\mathcal {O}}( \sqrt{N/d} )$ to build these lists $L_i$.

The total running time is optimal when these two quantities $\sqrt{N/d}$ and $d\cdot N \cdot C_\textsc {FF}$ are equal, which holds when $d = N^{-1/3} \cdot C_\textsc {FF}^{-2/3}$. Such a density gives complexity of ${\mathcal {O}}( N^{\frac{2}{3}} \cdot C_\textsc {FF}^{\frac{1}{3}})$ for SampleSet and at most $N^{\frac{2}{3}}$ queries to FindFunction. If $C_\textsc {FF}$ is polynomial, this gives a total time complexity of $\tilde{{\mathcal {O}}}(N^{\frac{2}{3}})$. The memory requirements of the algorithm correspond to the size of the lists $L_i$. This results in a memory complexity of ${\mathcal {O}}(N^{\frac{1}{3}}C_\textsc {FF}^{-\frac{1}{3}})$, or ${\mathcal {O}}(N^{\frac{1}{3}})$ if $C_\textsc {FF}$ is polynomial. $\square$

Remark 28

The success probability in Lemma 27 is chosen rather arbitrarily, mostly for practical verification of the algorithm’s correctness. It can be increased to any value $1-1/c$ for a positive constant c by appropriately building lists that are larger only by a constant factor compared to the case treated in Lemma 27. The overall complexity only differs by a constant factor, i.e., does not change asymptotically.

As said earlier, the algorithm presented in [14] is a special case of Algorithm 1. Their algorithm can be seen as an instantiation of Algorithm 1 by defining $G_{{\mathcal {F}}}$ (resp. $G_{{\mathcal {P}}}$) to be the linearity graph of ${\mathcal {F}}$ (resp. ${\mathcal {P}}$), where a vertex ${{\textbf {a}}}$ is connected to all vertices ${{\textbf {x}}}$ such that $D_{{\textbf {a}}} {\mathcal {F}}({{\textbf {x}}})=0$ (resp. $D_{{\textbf {a}}} {\mathcal {P}}({{\textbf {x}}})=0$), taking the predicate ${\mathbb {P}}_\kappa ({{\textbf {a}}}): \dim \ker D_{{\textbf {a}}} {\mathcal {F}} = \kappa$ on the universe ${\mathcal {M}}_{k, N}(q)$, and taking for FindFunction the assumed polynomial-time solver from [24] for inhQMLE. Finding a collision $(\alpha , \beta )$ such that $\beta = \alpha S$ makes the instance ${\mathcal {P}}({{\textbf {x}}} + \alpha ) = {\mathcal {F}}({{\textbf {x}}}S + \beta ){{\textbf {T}}}$ an inhomogeneous instance by defining ${\mathcal {P}}'({{\textbf {x}}}) = {\mathcal {P}}({{\textbf {x}}} + \alpha )$ and ${\mathcal {F}}'({{\textbf {x}}}) = {\mathcal {F}}({{\textbf {x}}} + \beta )$. Running FindFunction on ${\mathcal {P}}'$ and ${\mathcal {F}}'$ then returns ${{\textbf {S}}}$ and ${{\textbf {T}}}$. In this case, Lemma 27 gives the precise result from [14, Thm. 1], which we present as a corollary to our Lemma 27, for completeness.

Corollary 29

Assuming a polynomial-time solver for the inhomogenous case of QMLE, an hQMLE(N, N) instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ over ${\mathbb {F}}_{q}$ can be solved with complexity and number of queries equal to $\tilde{{\mathcal {O}}}( q^{\frac{2}{3}N} )$ with a success probability of $1 - 1/c$ for any $c > 0$ and a memory complexity of ${\mathcal {O}}(q^{\frac{1}{3}N})$.

Proof

Let $G_{{\mathcal {F}}}$ (i.e. $G_{{\mathcal {P}}}$) be the linearity graph of ${\mathcal {F}}$ (i.e. ${\mathcal {P}}$), where a vertex ${{\textbf {a}}}$ is connected to all ${{\textbf {x}}}$ such that $D_{{\textbf {a}}} {\mathcal {F}}({{\textbf {x}}})=0$ (i.e. $D_{{\textbf {x}}} {\mathcal {P}}({{\textbf {a}}})=0$). We use the predicate ${\mathbb {P}}_\kappa ({{\textbf {a}}}): \dim \ker D_{{\textbf {a}}} {\mathcal {F}} = \kappa$ we have that $\deg ({{\textbf {a}}}) = q^\kappa$. The density of the predicate $d_\kappa$ in the universe of $N \times N$ matrices is independent of ${\mathcal {F}}$ and ${\mathcal {P}}$, and is therefore the same as the density of linear maps with kernel of dimension $\kappa$. Thus, $d_\kappa$ is approximately a monotonic decreasing function in $\kappa$, going from 1 to 0. Hence, by Lemma 27, there exists some optimal $\kappa$ for which we get that $d_\kappa \approx |G_{{\mathcal {P}}}|^{-1/3} = q^{-N/3}$, which gives a total time complexity of $q^{\frac{2}{3}N}$ and a memory complexity of $q^{\frac{1}{3}N}$. $\square$

Remark 30

The assumption on a polynomial-time solver in [14] turns out to be too strong: such a solver exists for random instances, however, for inhQMLE instances as obtained in Corollary 29 the running time is probably not polynomial [15]. Nevertheless, the algorithm and result are valid, but require a different rebalancing depending on $C_{\textsc {FF}}$. Section 5 analyzes $C_\textsc {FF}$ in detail for different instances.

To apply this approach to MCE instances, we need to generalize to the case of N not necessarily equal to k. For an MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}}, {\mathcal {D}})$, we get an hQMLE $(n + m, k)$ instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}}, {\mathcal {P}})$ by Theorem 16. We take again the predicate ${\mathbb {P}}_\kappa ({{\textbf {a}}}): \dim \ker D_{{\textbf {a}}} {\mathcal {F}} = \kappa$, but this time on the universe ${\mathcal {M}}_{k, n + m}(q)$, where $D_{{\textbf {a}}} {\mathcal {F}}$ lives. To get a similar result to Corollary 29, we need to show two things. (a), that this predicate satisfies the assumptions required for Algorithm 1. (b), that there is a $\kappa$ such that the density $d_\kappa$ of ${\mathbb {P}}_\kappa$ is optimal as described in Lemma 27. If both are satisfied, we get a complexity of ${\mathcal {O}}( q^{\frac{2}{3}(n+m)} C_\textsc {FF}^\frac{1}{3})$, hence $\tilde{{\mathcal {O}}}(q^{\frac{2}{3}(n+m)})$ when the solver is polynomial, with a success probability of $1 - 1/c$ for any $c > 0$ for an MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}}, {\mathcal {D}})$. We start with a).

Lemma 31

The predicate ${\mathbb {P}}_\kappa (D_{{\textbf {a}}} {\mathcal {F}}): \dim \ker D_{{\textbf {a}}} {\mathcal {F}} = \kappa$ is a suitable predicate for Algorithm 1, as i) ${\mathbb {P}}_\kappa$ can be computed in polynomial time, ii) is invariant under equivalence, iii) and $d_\kappa$ does not depend on ${\mathcal {F}}$.

Proof

 

• 1. The cost $C_{{\mathbb {P}}_\kappa }$ is the cost of computing $\dim \ker D_{{\textbf {a}}} {\mathcal {F}}$, i.e. computing the kernel of a $k \times (n + m)$ matrix over ${\mathbb {F}}_{q}$. This can be done in polynomial time.

• 2. Let ${\mathcal {P}}({{\textbf {x}}}) = {\mathcal {F}}({{\textbf {x}}}{{\textbf {S}}}){{\textbf {T}}}$ be the equivalence. If ${{\textbf {x}}} \in \ker D_{{\textbf {a}}} {\mathcal {P}}$ then ${{\textbf {x}}}{{\textbf {S}}} \in \ker {\mathcal {F}}_{{{\textbf {a}}}{{\textbf {S}}}}$ and vice versa, as ${{\textbf {T}}}$ does not affect the kernel. As ${{\textbf {S}}}$ is invertible, we get a one-to-one correspondence ${{\textbf {x}}} \mapsto {{\textbf {x}}}{{\textbf {S}}}$ between the kernels, so ${\mathbb {P}}_\kappa (D_{\textbf {aS}} {\mathcal {F}}) = {\mathbb {P}}_\kappa (D_{{\textbf {a}}} {\mathcal {P}})$.

• 3. For ${\mathcal {F}}$ coming from an MCE instance, we always have ${-{\textbf {a}}} \in \ker D_{{{\textbf {a}}}} {\mathcal {F}}$. We want to show that the distribution of the rank of $D_{{\textbf {a}}} {\mathcal {F}}$ follows the ranks of linear maps vanishing at ${-{\textbf {a}}}$. This is given by [22, Thm. 2] for even characteristic and easily adapted to odd characteristic, which shows $d_\kappa$ is independent of ${\mathcal {F}}$.

$\square$

We now continue with (b): we show that there is a $\kappa$ such that $d_\kappa$ is optimal. For now, existence of $\kappa$ is enough to derive a complexity on MCE. We will explicitely compute $\kappa$ later, in Sect. 5, when we have a detailed view of $C_\textsc {FF}$ for specific parameter sets (k, n, m).

The general density $d_\kappa$ for the predicate ${\mathbb {P}}_\kappa$ is given by the following lemma, taking $a = k$ and $b = n + m$ to avoid confusion with regards to n, m and $n+m$.

Lemma 32

Define the predicate ${\mathbb {P}}_\kappa : \dim \ker {{{\textbf {M}}}} = \kappa$ for ${{\textbf {M}}} \in U = {\mathcal {M}}_{a, b}(q)$ with $a \geqslant b$. Then the density of the predicate ${\mathbb {P}}_\kappa$ is $d_\kappa = 1/\Theta (q^{(\kappa ^2 + \kappa \cdot (a - b))})$.

Proof

There are $|U |= q^{ab}$ matrices in ${\mathcal {M}}_{a, b}(q)$, out of which

$$\begin{aligned} \prod _{i = 0}^{r - 1} \frac{(q^a - q^i)(q^b - q^i)}{q^r - q^i} = \Theta \left( q^{(a+b-r)r}\right) \end{aligned}$$

have rank r [34]. We have $\kappa = b - r$ and so $d_\kappa ^{-1} = \frac{|U |}{|U_\top |} = \Theta (\frac{q^{ab}}{q^{- (a + b - r)r}}) =\Theta ( q^{\kappa ^2 + \kappa (a - b)}).$ Specifically when the matrix is square, $d_\kappa ^{-1} =\Theta (q^{\kappa ^2})$. $\square$

From Lemma 32 we can conclude that for some $\kappa$, the density $d_\kappa$ is optimal. This means we satisfy both (a) and (b) and we can apply Lemma 27.

In conclusion, we get our first result on the hardness of MCE, which significantly improves straightforward enumeration. This requires that such a $\kappa$ exists, which happens when $k \leqslant 2(n+m)$, by Lemma 32. Note that, in contrast to [14, Thm. 1], we do not assume a polynomial-time solver for the inhomogeneous case of QMLE. Instead, we write this cost as $C_\textsc {FF}$ and explore the precise cost in Sect. 5.

Theorem 33

An MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}}, {\mathcal {D}})$ over ${\mathbb {F}}_{q}$ with $k \leqslant 2(n+m)$ can be solved using Algorithm 1 with time complexity equal to ${\mathcal {O}}(q^{\frac{2}{3}(n + m)} \cdot C_{\textsc {FF}}^{\frac{1}{3}} \cdot (C_{{\mathbb {P}}_\kappa } + 1))$, memory complexity equal to ${\mathcal {O}}(q^{\frac{1}{3}(m+n)}C_\textsc {FF}^{-\frac{1}{3}})$ and with success probability of $1 - 1/c$ for any $c > 0$, where $C_{\textsc {FF}}$ denotes the cost of a single query to FindFunction.

We will show in Sect. 5 that, even though $C_\textsc {FF}$ is not polynomial-time, the complexity of Algorithm 1 is still $\tilde{{\mathcal {O}}}(q^{\frac{2}{3}(n + m)})$ for some optimal $\kappa$.

When $k > 2(n+m)$, we can no longer assume elements with $\dim \ker D_{{\textbf {a}}} {\mathcal {F}} > 1$ exist, as practically all differentials $D_{{\textbf {a}}} {\mathcal {F}}$ will have only the trivial kernel spanned by $-{{\textbf {a}}}$. In such a scenario, we have two alternatives:

• Take a single element ${{\textbf {a}}}$ and run FindFunction on $({{\textbf {a}}}, {{\textbf {b}}})$ for all ${{\textbf {b}}} \in {\mathbb {F}}_{q}^{n+m}$ until we find the isometry. This deterministic process has a time complexity of ${\mathcal {O}}(q^{(n + m)} \cdot C_{\textsc {FF}})$. The memory requirements of this algorithm are negligible, since we do not build lists of elements;

• Alternatively, note that in this case $n\leqslant 2(k+m)$. Thus, we can also use the result of Lemma 26, and instead of an MCE(n, m, k) instance, we can solve an MCE(k, m, n) instance using Algorithm 1. In this case we end up with a complexity of $\tilde{{\mathcal {O}}}(q^{\frac{2}{3}(k + m)})$. However, for the given regime of parameters, this is always larger than $\tilde{{\mathcal {O}}}(q^{(n + m)})$, so the first deterministic approach is better.

4.3 Second algorithm

The algorithm that we presented in the previous section does not take advantage of the bilinear structure of an instance of MCE when viewed as hQMLE. In such a case, the differential $D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}}$ of a k-dimensional bilinear form admits a special structure.

Lemma 34

Let ${\mathcal {F}}({{\textbf {x}}}, {\textbf {y)}}$ be a k-dimensional bilinear form with ${{\textbf {x}}} \in {\mathbb {F}}_{q}^m$ and ${{\textbf {y}}} \in {\mathbb {F}}_{q}^n$. Let ${{\textbf {F}}}_{{\textbf {a}}}$ denote the $k\times n$ matrix of the linear map ${\mathcal {F}}({{\textbf {a}}}, -): {\mathbb {F}}_{q}^n \rightarrow {\mathbb {F}}_{q}^k$ for a fixed ${{\textbf {a}}} \in {\mathbb {F}}_{q}^m$. Similarly let ${{\textbf {F}}}_{{\textbf {b}}}$ denote the $k\times m$ matrix of the linear map ${\mathcal {F}}(-, {{\textbf {b}}}): {\mathbb {F}}_{q}^m \rightarrow {\mathbb {F}}_{q}^k$ for a fixed ${{\textbf {b}}} \in {\mathbb {F}}_{q}^n$. Then

$$\begin{aligned} D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}}({{\textbf {x}}}, {{\textbf {y}}}) = \left( \ {{\textbf {F}}}_{{\textbf {b}}} \ {{\textbf {F}}}_{{\textbf {a}}} \ \right) \begin{pmatrix} {{\textbf {x}}}^\top \\ {{\textbf {y}}}^\top \end{pmatrix}. \end{aligned}$$

Proof

By bilinearity, $D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}}({{\textbf {x}}}, {{\textbf {y}}}):= {\mathcal {F}}({{\textbf {x}}} + {{\textbf {a}}}, {{\textbf {y}}} + {{\textbf {b}}}) - {\mathcal {F}}({{\textbf {x}}}, {{\textbf {y}}}) - {\mathcal {F}}({{\textbf {a}}},{{\textbf {b}}})$ equals ${\mathcal {F}}({{\textbf {a}}}, {{\textbf {y}}}) + {\mathcal {F}}({{\textbf {x}}}, {{\textbf {b}}}) = {\textbf {F}}_{\textbf{a}}{{\textbf {y}}}^\top + {\textbf {F}}_{\textbf{b}}{{\textbf {x}}}^\top .$ $\square$

Similarly for ${\mathcal {P}}$, we use the notation ${\textbf {P}}_{\textbf{a}}$ and ${\textbf {P}}_{\textbf{b}}$. The equivalence in such a case becomes ${\mathcal {P}}({{\textbf {x}}}, {{\textbf {y}}}) = {\mathcal {F}}({{\textbf {x}}}{{\textbf {A}}}, {{\textbf {y}}}{{\textbf {B}}}^\top ) {{\textbf {T}}}$, with ${{\textbf {A}}}, {{\textbf {B}}}$ precisely the matrices from the MCE instance. Then, as ${\mathcal {F}}$ and ${\mathcal {P}}$ are bilinear, one can see SampleSet in Algorithm 1 as sampling both ${{\textbf {a}}}\in {\mathbb {F}}_{q}^n$ and ${{\textbf {b}}}\in {\mathbb {F}}_{q}^m$ at the same time as one $({{\textbf {a}}}, {{\textbf {b}}}) \in {\mathbb {F}}_{q}^{n+m}$, until $D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}}$ has a kernel of dimension $\kappa$. However in the bilinear case, ${{\textbf {a}}}$ influences only the matrix ${{\textbf {F}}}_{{\textbf {a}}}$, and ${{\textbf {b}}}$ influences only ${{\textbf {F}}}_{{\textbf {b}}}$. Hence, we can sample ${{\textbf {a}}} \in {\mathbb {F}}_{q}^m$ and ${{\textbf {b}}} \in {\mathbb {F}}_{q}^n$ separately. This hints that we can apply ideas from Algorithm 1 to the smaller universes $U_{{\textbf {a}}} = {\mathcal {M}}_{k, n}(q)$ and $U_{{\textbf {b}}} = {\mathcal {M}}_{k, m}(q)$, where ${{\textbf {F}}}_{{\textbf {a}}}$ and ${{\textbf {F}}}_{{\textbf {b}}}$ live. By finding well-chosen predicates in these smaller universes, we hope to find collisions faster.

We first analyse the properties of ${\textbf {F}}_{\textbf{a}}$ and ${\textbf {F}}_{\textbf{b}}$ a bit more. Let ${\mathfrak {F}}_a$ be the set of elements ${{\textbf {a}}}$ for which $\dim \ker {\textbf {F}}_{\textbf{a}}$ is non-trivial, and ${\mathfrak {F}}_b$ similarly, i.e.

$$\begin{aligned} {\mathfrak {F}}_a = \{ {{\textbf {a}}} \in {\mathbb {F}}_{q}^m \ |\ \dim \ker {\mathcal {F}}({{\textbf {a}}}, -)> 0 \}, \quad {\mathfrak {F}}_b = \{ {{\textbf {b}}} \in {\mathbb {F}}_{q}^n \ |\ \dim \ker {\mathcal {F}}(-, {{\textbf {b}}}) > 0 \}. \end{aligned}$$

For ${\mathcal {P}}$, we define ${\mathfrak {P}}_a$ and ${\mathfrak {P}}_b$ similarly. For isomorphic bilinear forms ${\mathcal {F}}$ and ${\mathcal {P}}$, these sets have special properties.

Lemma 35

Let $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {T}}}): {\mathcal {F}} \rightarrow {\mathcal {P}}$ be an isomorphism between two k-tuples of bilinear homogenous quadratic polynomials ${\mathcal {F}}$ and ${\mathcal {P}}$, such that ${\mathcal {P}}({{\textbf {x}}}, {{\textbf {y}}})={\mathcal {F}}({{\textbf {x}}}{{\textbf {A}}}, {{\textbf {y}}}{{\textbf {B}}}^\top ){{\textbf {T}}}$. We have the following properties:

• 1. Given ${{\textbf {a}}} \in {\mathfrak {F}}_a$ and any ${{\textbf {b}}} \in \ker {\textbf {F}}_{\textbf{a}}$, we get ${\mathcal {F}}({{\textbf {a}}}, {{\textbf {b}}}) = 0$.

• 2. ${\mathfrak {F}}_b$ is completely determined by ${\mathfrak {F}}_a$, as ${\mathfrak {F}}_b = \bigcup _{{{\textbf {a}}} \in {\mathfrak {F}}_a} \ker {\textbf {F}}_{\textbf{a}}$.

• 3. For ${{\textbf {a}}} \in {\mathbb {F}}_{q}^n$ and ${{\textbf {y}}} \in {\mathbb {F}}_{q}^{m}$, we have ${\textbf {P}}_{\textbf{a}}({{\textbf {y}}}) = {\textbf {F}}_{\textbf{aA}} ({{\textbf {y}}}{{\textbf {B}}}^\top ) {{\textbf {T}}}$.

• 4. For ${{\textbf {a}}} \in {\mathbb {F}}_{q}^n$, we get $\ker {\textbf {P}}_{\textbf{a}} = \ker {\mathcal {F}}_{{\textbf {aA}}} \cdot {{{\textbf {B}}}}^\top$.

• 5. The isomorphism $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {T}}})$ induces the bijections

  $$\begin{aligned} {\mathfrak {P}}_a \rightarrow {\mathfrak {F}}_a: {{\textbf {a}}} \mapsto {\textbf {aA}}, \quad {\mathfrak {P}}_b \rightarrow {\mathfrak {F}}_b: {{\textbf {b}}} \mapsto {{\textbf {b}}}{{\textbf {B}}}^\top . \end{aligned}$$

Proof

 

• 1. ${{\textbf {b}}} \in \ker {\textbf {F}}_{\textbf{a}}$ is equivalent by definition to ${\textbf {F}}_{\textbf{a}}{{\textbf {b}}}^\top = {\mathcal {F}}({{\textbf {a}}}, {{\textbf {b}}}) = {{\textbf {0}}}$.

• 2. This follows directly from 1.: ${{\textbf {b}}} \in {\mathfrak {F}}_b$ only if there exists an ${{\textbf {a}}}\in {\mathfrak {F}}_a$ such that ${\mathcal {F}}({{\textbf {a}}}, {{\textbf {b}}}) = {{\textbf {0}}}$. But then ${{\textbf {b}}} \in \ker {\textbf {F}}_{\textbf{a}}$ for this specific ${{\textbf {a}}}$.

• 3. Per definition ${\textbf {P}}_{\textbf{a}}({{\textbf {y}}}) = {\mathcal {P}}({{\textbf {a}}}, {{\textbf {y}}}) = {\mathcal {F}}({\textbf {aA}}, {{\textbf {y}}}{{\textbf {B}}}^\top ){{\textbf {T}}} = {\textbf {F}}_{\textbf{aA}}({\textbf {yB}}^{\top }){{\textbf {T}}}$.

• 4. This follows directly from 3.: as ${{\textbf {T}}}$ is invertible, it does not affect the kernels, so ${{\textbf {y}}} \in \ker {\textbf {P}}_{\textbf{a}}$ if and only if ${\textbf {yB}}^{\top } \in \ker {\textbf {F}}_{\textbf{aA}}$

• 5. This follows directly from 4.: Given ${{\textbf {a}}} \in {\mathfrak {P}}_a$ we get ${\textbf {aA}} \in {\mathfrak {F}}_a$ and vice versa as ${{\textbf {A}}} \in {\text {GL}}_m(q)$. A similar argument gives ${\mathfrak {F}}_b \rightarrow {\mathfrak {P}}_b$.

$\square$

Lemma 35 shows that ${{\textbf {a}}} \in {\mathfrak {F}}_a$ and ${{\textbf {b}}} \in {\mathfrak {F}}_b$ describe all non-trivial roots $({{\textbf {a}}}, {{\textbf {b}}})$ of a given ${\mathcal {F}}$. For an instance $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {T}}}): {\mathcal {F}} \rightarrow {\mathcal {P}}$, Item 5 shows that non-trivial roots are mapped bijectively by $({{\textbf {A}}},{{\textbf {B}}},{{\textbf {T}}})$. Such non-trivial roots can be used to find collisions more easily between ${\mathcal {F}}$ and ${\mathcal {P}}$. However, this requires that instances ${\mathcal {F}} \rightarrow {\mathcal {P}}$ have non-trivial roots. We can get an estimate on the sizes of ${\mathfrak {F}}_a$, ${\mathfrak {F}}_b$, ${\mathfrak {P}}_a$, and ${\mathfrak {P}}_b$ for given parameters n, m, and k, in the following way.

Lemma 36

When $k \geqslant n$, $|{\mathfrak {F}}_a |= |{\mathfrak {P}}_a |\approx q^{2n - k - 1}$ and $|{\mathfrak {F}}_b |= |{\mathfrak {P}}_b |\approx q^{2\,m - k - 1}$.

Proof

By Lemma 35, we get $|{\mathfrak {F}}_a |= |{\mathfrak {P}}_a |$. Then, using Lemma 32, we see that the size of these sets is dominated by elements ${{\textbf {a}}}$ with $\kappa = \dim \ker {\textbf {F}}_{\textbf{a}} = 1$ (a one-dimensional kernel). From the same lemma, the density of $\kappa = \dim \ker {\textbf {F}}_{\textbf{a}} = 1$ elements is $d_1 = q^{-(1 + 1\cdot (k - n))}$. Hence we expect $d_1 \cdot q^n = \Theta (q^{2n - k - 1})$ such elements. A similar argument gives $|{\mathfrak {F}}_b |= |{\mathfrak {P}}_b |$ as $\Theta ( q^{2\,m - k - 1})$. $\square$

Summarizing, this implies

Corollary 37

Assuming $n = m$ as the hardest case, a random MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {F}},{\mathcal {P}})$ over ${\mathbb {F}}_{q}$ has an expected value ${\mathcal {E}}_{n,m,k,q}$ of non-trivial roots

• when $k < 2n$, with ${\mathcal {E}}_{n,m,k,q} = \Theta (q^{2n - k - 1})$,

• when $k = 2n$, with ${\mathcal {E}}_{n,m,k,q} = \Theta (\frac{1}{q})$,

• when $k > 2n$, with ${\mathcal {E}}_{n,m,k,q} = \Theta (\frac{1}{q^{k - 2n + 1}})$.

From these results, we can expect non-trivial roots for an MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {F}},{\mathcal {P}})$ over ${\mathbb {F}}_{q}$ with $k \leqslant n+m$. These non-trivial roots can be seen as a suitable predicate on the smaller universes $U_{{\textbf {a}}}$ and $U_{{\textbf {b}}}$: we search for collisions $({{\textbf {a}}}, {{\textbf {b}}}) \times ({{\textbf {a}}}{{\textbf {A}}}, {{\textbf {b}}}{{\textbf {B}}}^\top )$, where $({{\textbf {a}}}, {{\textbf {b}}})$ is a non-trivial root of ${\mathcal {P}}$, and $({{\textbf {a}}}{{\textbf {A}}}, {{\textbf {b}}}{{\textbf {B}}}^\top )$ of ${\mathcal {F}}$. Given such a collision, we proceed as in Sect. 4.2.

The following result shows that we always find such a collision if ${\mathcal {F}}$ and ${\mathcal {P}}$ have non-zero roots.

Lemma 38

Let $m\leqslant n$ and $k \leqslant n+m$. Let ${\mathfrak {F}}_a$, ${\mathfrak {F}}_b$ and ${\mathfrak {P}}_a$, ${\mathfrak {P}}_b$ describe the non-trivial roots of an MCE(n, m, k) instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {F}},{\mathcal {P}})$ over ${\mathbb {F}}_{q}$. Let ${{\textbf {x}}} = ({{\textbf {a}}}, {{\textbf {b}}}) \in {\mathfrak {F}}_a \times {\mathfrak {F}}_b$, then looping over ${{\textbf {y}}} \in {\mathfrak {P}}_a \times {\mathfrak {P}}_b$ gives a collision $({{\textbf {x}}}, {{\textbf {y}}})$ with certainty.

Proof

This follows quickly from Lemma 35. We have ${{\textbf {x}}} = ({{\textbf {a}}}, {{\textbf {b}}})$ and two bijections ${\mathfrak {F}}_a \rightarrow {\mathfrak {P}}_a$ and ${\mathfrak {F}}_b \rightarrow {\mathfrak {P}}_b$, so ${{\textbf {x}}}$ is mapped to some ${{\textbf {y}}} \in {\mathfrak {P}}_a \times {\mathfrak {P}}_b$. As this set is finite, we can loop over it in a finite number of steps until we find the collision. $\square$

Therefore, as soon as we have non-trivial roots, we can use a single one of them to find a collision. This leads to the following pseudo-algorithm:

• 1. compute ${\mathfrak {F}}_b$ by computing $\ker {\textbf {F}}_{\textbf{b}}$ for all ${{\textbf {b}}} \in {\mathbb {F}}_{q}^m$,

• 2. if ${\mathfrak {F}}_b$ is non-empty, compute ${\mathfrak {F}}_a$ using Lemma 35-2. Same for ${\mathfrak {P}}_a$ and ${\mathfrak {P}}_a$.

• 3. sample a single ${{\textbf {x}}} \in {\mathfrak {F}}_a \times {\mathfrak {F}}_b$

• 4. loop over ${{\textbf {y}}} \in {\mathfrak {P}}_a \times {\mathfrak {P}}_b$ with FindFunction$({{\textbf {x}}}, {{\textbf {y}}})$ until the solver finds $\mu$.

Corollary 39

Let $m\leqslant n$ and $k \leqslant n+m$. The above algorithm terminates successfully and has a total complexity of ${\mathcal {O}}(q^m \cdot C_{{\mathbb {P}}_\kappa } + q^{2(n + m - k - 1)} \cdot C_{\textsc {FF}})$, where $C_{{\mathbb {P}}}$ denotes the cost of computing $\ker {\textbf {F}}_{\textbf{b}}$ and $C_{\textsc {FF}}$ denotes the cost of a single query to FindFunction.

Proof

Building ${\mathfrak {F}}_b$ and ${\mathfrak {P}}_b$ has a complexity of ${\mathcal {O}}(q^m \cdot C_{{\mathbb {P}}_\kappa })$, and these give us ${\mathfrak {F}}_a$ and ${\mathfrak {P}}_a$ by Lemma 35. Then for every step in the loop we get a query to FindFunction. By Lemma 36, the size of ${\mathfrak {P}}_a \times {\mathfrak {P}}_b$ is at most ${\mathcal {O}}(q^{2(n + m - k - 1)})$. $\square$

We will see later in Sect. 5 that the dominating complexity is $q^m \cdot C_{{\mathbb {P}}_\kappa }$ as for specific parameters (k, n, m) the number of queries z can be reduced so that $z \cdot C_\textsc {FF} < q^m$. As $C_{{\mathbb {P}}_\kappa }$ is polynomial, we get a complexity of $\tilde{{\mathcal {O}}}(q^m)$ for such instances.

For efficiency, one can decrease further the number of queries to FindFunction by applying other, secondary predicates. For example, the sets ${\mathfrak {F}}_a \times {\mathfrak {F}}_b$ and ${\mathfrak {P}}_a \times {\mathfrak {P}}_b$ can be split into zeros ${\mathfrak {F}}^{{\textbf {0}}}=\{{{\textbf {x}}}\in {\mathbb {F}}_{q}^{n+m} |{\mathcal {F}}({{\textbf {x}}}) = {{\textbf {0}}}\}$ and non-zeros ${\mathfrak {F}}={\mathfrak {F}}_a \times {\mathfrak {F}}_b{\setminus }{\mathfrak {F}}^{{\textbf {0}}}$, which reduces the collision search to each of these sets. Another secondary predicate is to only use elements ${{\textbf {a}}}$ with $\dim \ker {\textbf {F}}_{\textbf{a}} = \kappa$ for some specific value $\kappa >0$.

We summarize the MCE solver for instances with roots in Algorithm 2. Practically, since the algorithm is deterministic, we do not need to build and store the list ${\mathfrak {F}}$. We only need to find one element from it. However, for iterating through the list ${\mathfrak {P}}$, $S_{a}$ and $S_{b}$ need to be stored. The estimated size of these lists is $q^{n+m-k-1}$.

Algorithm 2

Bilinear MCE-Solver, assuming $n \geqslant m$.

The next theorem summarises the conditions and cost of Algorithm 2 for solving MCE.

Theorem 40

Assuming a solver for the inhomogenous case of QMLE with cost $C_{\textsc {FF}}$, an MCE(n, m, k) instance over ${\mathbb {F}}_{q}$ with $m\leqslant n$ and $k \leqslant n+m$ (in which case roots exist for ${\mathcal {F}}$ and ${\mathcal {P}}$ with overwhelming probability) can be solved using Algorithm 2 with ${\mathcal {O}} \left( q^m \cdot C_{{\mathbb {P}}_\kappa } \right)$ operations in SampleZeros and z queries to the solver. This amounts to a total time complexity of ${\mathcal {O}}\left( q^m \cdot C_{{\mathbb {P}}_\kappa } + z \cdot C_\textsc {FF} \right)$. The memory complexity of the algorithm is ${\mathcal {O}}(q^{n + m -k-1})$.

We will show in Sect. 5 that, even though $C_\textsc {FF}$ is not polynomial-time, the dominating factor in this complexity is still $q^m \cdot C_{{\mathbb {P}}_\kappa }$, where $C_{{\mathbb {P}}_\kappa }$ is the cost to compute the kernel of an $m \times k$ matrix.

The regime of operation of Theorem 40 seems to imply that we can use it only if $k \leqslant n+m$. However, note that if $k>n+m$ then $n \leqslant k+m$. Hence, by Lemma 26, we can turn the given MCE(n, m, k) instance into an MCE(k, m, n) instance and solve this instance using Algorithm 2. This results in a complexity of $\tilde{{\mathcal {O}}}(q^m)$. Recall that we assume $m=\min \{m,n,k\}$, thus, we obtain the following general theorem which is our main result about the practical complexity of solving MCE.

Theorem 41

An MCE(n, m, k) instance over ${\mathbb {F}}_{q}$ can be solved using Algorithm 2 in time $\tilde{{\mathcal {O}}} \left( q^{\min \{m,n,k\}}\right)$.

5 Filling the gaps in the complexity analysis

The cost $C_{\mathbb {P}}$ is polynomial in all of the cases because it either requires computing the rank of a linear map or sampling a random element from a set. The FindFunction in Algorithms 1 and 2 checks whether a given pair of vectors is a collision, and if so, it returns the solution to the MCE instance. This is done by solving an instance of the inhBMLE that has the same solutions as the input MCE instance. Thus, to estimate the value of $C_\textsc {FF}$, we analyse the complexity of inhBMLE on these instances, by relying on algorithms that have been developed for the inhQMLE case with $N=k$.

5.1 Algorithms for inhQMLE

The two algorithms described in this section have been used for tackling the inhQMLE problem within the birthday-based algorithm for hQMLE[14, 15]. Their analysis is thus important to estimate $C_\textsc {FF}$. In Sect. 5.2 we adapt this analysis for the inhBMLE case with arbitrary k and N and we see how this affects Algorithms 1 and 2 for different parameter sets.

5.1.1 The Gröbner bases attack

The algebraic attack on the inhQMLE problem starts by reducing ${\mathcal {P}}({{\textbf {x}}}){{\textbf {T}}}^{-1}={\mathcal {F}}({{\textbf {x}}}{{\textbf {S}}})$, with ${{\textbf {S}}}$ and ${{\textbf {T}}}$ unknown, to a system of polynomial equations. By rewriting the problem in matrix form we obtain the following constraints

$$\begin{aligned} \sum \limits _{1\leqslant r\leqslant k}{\widetilde{T}}_{rs}{{\textbf {P}}}^{(r)} = {{\textbf {S}}}{{\textbf {F}}}^{(s)}{{\textbf {S}}}^\top ,&\ \ \forall s, 1\leqslant s\leqslant k, \nonumber \\ {{\textbf {P}}}^{[1]}{{\textbf {T}}}^{-1}={{\textbf {S}}}{{\textbf {F}}}^{[1]},&\nonumber \\ {{\textbf {P}}}^{[0]}{{\textbf {T}}}^{-1}={{\textbf {F}}}^{[0]},&\end{aligned}$$

(13)

where ${{\textbf {F}}}^{[1]} \in {\mathbb {F}}_q^{N \times k}$ and ${{\textbf {P}}}^{[1]} \in {\mathbb {F}}_q^{N \times k}$ describe the degree-1 homogeneous part of an $\textsf {inh(Q/B)MLE}$ instance and ${{\textbf {F}}}^{[0]} \in {\mathbb {F}}_q^{k}$ and ${{\textbf {P}}}^{[0]} \in {\mathbb {F}}_q^{k}$ describe the degree-0 part. We will denote the subsystem of equations derived from the degree-d homogeneous part as ${\mathcal {S}}_d$. The resulting system can be solved using Gröbner basis algorithms and this is referred to as the Gröbner attack [24]. The observation that ${{\textbf {S}}}$ and ${{\textbf {T}}}$ are common solutions to homogeneous parts of separate degrees of an inhQMLE instance (also proven in [13, Lemma 1]) and the idea that moving ${{\textbf {T}}}$ to the other side of the equality results in a lower degree system where we solve for ${{\textbf {T}}}^{-1}$ originate from this work.

The complexity of Gröbner basis algorithms depends foremost on the degree of regularity, which is usually hard to estimate, but it can sometimes be observed through experimental work. Such experiments applied to inhQMLE instances imply that the system is solved at degree three. A degree-three linearized system in n variables is represented by a matrix of size roughly $n^3$ and thus, Gaussian Elimination on such a system is performed in ${\mathcal {O}}(n^{3\omega })$ operations, where $\omega$ is the linear algebra constant. This reasoning leads to the assumption that there exists a polynomial-time solver for the inhomogeneous case of QMLE. Another empirical observation made in [24] is that the time to construct the system exceeds the time of the Gröbner basis computation. Since the generation of the system is known to be polynomial, this suggests that the Gröbner basis computation is performed in polynomial time as well. However, these experiments are performed on random inhomogeneous instances of the QMLE problem.

In the birthday-based approach for solving QMLE, ${{\textbf {F}}}^{[1]}$, ${{\textbf {P}}}^{[1]}$, ${{\textbf {F}}}^{[0]}$ and ${{\textbf {P}}}^{[0]}$ are obtained from a collision [14]. Specifically, if we have a collision on ${{\textbf {x}}} \in {\mathbb {F}}_q^N$ and ${{\textbf {y}}} \in {\mathbb {F}}_q^N$ such that ${{\textbf {y}}}={{\textbf {x}}}{{\textbf {S}}}$, they are obtained as

$$\begin{aligned}&{{\textbf {F}}}^{[1]} = D_{{\textbf {y}}} {\mathcal {F}},&{{\textbf {P}}}^{[1]} = D_{{\textbf {x}}} {\mathcal {P}}, \\&{{\textbf {F}}}^{[0]} = {\mathcal {F}}({{\textbf {y}}}),&{{\textbf {P}}}^{[0]} = {\mathcal {P}}({{\textbf {x}}}). \end{aligned}$$

Instances of inhQMLE derived from a collision are, on average, harder to solve than random inhQMLE instances. Recall that in Algorithm 1 the instances of inhQMLE are chosen such that $\dim \ker D_{{\textbf {y}}} {\mathcal {F}} = \dim \ker D_{{\textbf {x}}} {\mathcal {P}}=\kappa$. Hence, the number of linearly independent equations in ${\mathcal {S}}_1$ is exactly $k(N-\kappa )$, instead of the expected kN on average. The size of ${\mathcal {S}}_0$ can also depend on the predicate that we choose for the birthday-based algorithm. For instance, when we use the predicate of searching for a collision between the non-trivial roots of ${\mathcal {P}}$ and ${\mathcal {F}}$, we obtain no equations in ${\mathcal {S}}_0$. Additionally, since ${{\textbf {F}}}^{[1]}$ (i.e. ${{\textbf {P}}}^{[1]}$) and ${{\textbf {F}}}^{[0]}$ (i.e. ${{\textbf {P}}}^{[0]}$) are obtained respectively from computing the differential of and evaluating ${\mathcal {F}}$ (i.e ${\mathcal {P}}$) at a given point, ${\mathcal {S}}_1$ and ${\mathcal {S}}_0$ are not as independent from ${\mathcal {S}}_2$ as they would be in the random case. It is difficult to estimate the complexity of solving these instances compared to solving random instances with the same structure. Figure 2 shows experiments confirming our intuition that the complexity of collision-derived instances is worse than that of random ones. This implies that we can not rely on the experimental observations in [24] to estimate the complexity of these specific instances. We conclude that, in contrast with the literature, we can not assume that $C_\textsc {FF}$ is polynomial when the Gröbner attack is used.

Fig. 2

Comparison of runtime for solving random and collision-derived $\textsf {inh(Q/B)MLE}$ instances using the Gröbner attack. Results are averaged over 50 runs

5.1.2 The matrix-pencil attack

The matrix-pencil attack was proposed in Bouillaguet’s thesis [15] and used for the implementation of the birthday-based attack [14]. This algorithm has a complexity of ${\mathcal {O}}(N^6)$ with non-negligible probability for random inhQMLE instances where $N=k$. Its complexity for inhQMLE instances derived from a collision attack depends strongly on the parameter $\kappa$. We give a general description of the approach. For details on how it relates to the matrix pencil equivalence problem, we refer to [15, Ch. 14].

The first step is to retrieve a basis of the solution space V of the subsystem of linear equations ${\mathcal {S}}_1$. Let $\ell =\dim V$ and let $({{\textbf {S}}}^{[1]},{{\textbf {T}}}^{[1]}),\ldots , ({{\textbf {S}}}^{[\ell ]},{{\textbf {T}}}^{[\ell ]})$ be a basis of V. Once the solution space of ${\mathcal {S}}_1$ is known, in order to find the solution space of the overall system one rewrites ${\mathcal {S}}_2$ as a system in $\ell$ variables. Concretely, this is done by replacing ${{\textbf {S}}}$ and ${{\textbf {T}}}$ by $\sum _{i=1}^\ell x_i {{\textbf {S}}}^{[i]}$ and $\sum _{i=1}^\ell x_i {{\textbf {T}}}^{[i]}$ in Eq. (13) and then looking for solutions in variables $x_1, \ldots , x_\ell$. This standard approach is also described in [13]. A key idea in the matrix-pencil attack is to use the knowledge of ${{\textbf {F}}}^{[1]}$/${{\textbf {P}}}^{[1]}$ and ${{\textbf {F}}}^{[0]}$/${{\textbf {P}}}^{[0]}$ to find a (second) collision and double the number of linear equations in ${\mathcal {S}}_1$. Supposing that there exists ${\textbf {x'}}$ such that ${\textbf {x'}}{{\textbf {P}}}^{[1]}={{\textbf {P}}}^{[0]}$, we infer that there also exists ${\textbf {y'}}$ such that ${\textbf {y'}}{{\textbf {F}}}^{[1]}={{\textbf {F}}}^{[0]}$ and that ${\textbf {y'}}={\textbf {x'}}{{\textbf {S}}}$. We can thus append the equations obtained from $(D_{\textbf {x'}} {\mathcal {P}}){{\textbf {T}}}^{-1}={{\textbf {S}}}(D_{\textbf {y'}} {\mathcal {F}})$ to ${\mathcal {S}}_1$. After applying this technique, the resulting system is usually highly overdetermined and can be solved through direct linearization. The most favorable case is when ${\textbf {x'}}$ and ${\textbf {y'}}$ are uniquely identified. However, if $\dim \ker {{\textbf {F}}}^{[1]}=\kappa >1$, then ${\textbf {x'}}$ is chosen arbitrarily and we loop through the $q^\kappa$ possible values for ${\textbf {y'}}$. The complexity of the algorithm is ${\mathcal {O}}(q^\kappa \ell ^2 N^4)$, under the condition that $\ell (\ell +1)/2 \le |{\mathcal {S}}_2 |$. Another condition for the success of this approach is that ${\mathcal {P}}({{\textbf {x}}}) \ne 0$ and there is an ${{\textbf {x}}}$ such that ${{\textbf {x}}}D_{{\textbf {x}}} {\mathcal {P}}={\mathcal {P}}({{\textbf {x}}})$, because this assumption is used to find the second collision. As per the analysis in [15], the probability that the condition for success is met is $1-1/q + 1/q^3 +{\mathcal {O}}(1/q^6)$.

5.2 The complexity of inhBMLE

In the following analysis, we use the matrix-pencil algorithm as the inhBMLE solver, as it seems to outperform the Gröbner attack and we have a better understanding of its complexity for these specific instances.

5.2.1 The case $k \le n+m$

Based on the analysis in Sect. 4.3 for the purpose of usage in Algorithm 2 we can assume without loss of generality that $k \le n+m$ and $m=\min \{m,n,k\}$.

The complexity of Algorithm 2 is dominated by the SampleZeros function, as long as the complexity of the inhBMLE solver does not surpass ${\mathcal {O}}(q^m)$. In the matrix-pencil algorithm, we can not use the zero subsets ${\mathfrak {F}}^{{\textbf {0}}}$ and ${\mathfrak {P}}^{{\textbf {0}}}$, as this contradicts its condition for success ${\mathcal {P}}({{\textbf {x}}}) \ne 0$. The non-zeros subsets ${\mathfrak {F}}$ and ${\mathfrak {P}}$ can be used with a small adjustment to the algorithm: after finding a basis of the solution space of ${\mathcal {S}}_1$, we rewrite and solve through linearization the system comprised of both ${\mathcal {S}}_2$ and ${\mathcal {S}}_0$. Note that ${\mathfrak {F}}$ and ${\mathfrak {P}}$ are non-empty only when the instance has at least two roots. Since in Algorithm 2 we do not restrict the value of $\kappa$, we will approximate to the one that has the highest probability, which for the case of $k \le n+m$ is $\kappa =(m+n)-k$. Hence, $C_\textsc {FF}$ is approximated to

$$\begin{aligned} {\mathcal {O}}(q^{m+n-k}\cdot (m+n)^6). \end{aligned}$$

When $k \ge m$, this is always smaller than ${\mathcal {O}}(q^m)$.

5.2.2 The case $n+m< k < 2(n+m)$

This case is not relevant for Algorithm 2, but it is for  Algorithm 1. Since the complexity of the inhBMLE solver contains a non-negligible factor of $q^\kappa$, the choice of $\kappa$ needs to be adapted, so that the running times of SampleSet and CollisionFind are equal. Let $N=n+m$ and let $r=N-k$. The optimal $\kappa$ is chosen such that

$$\begin{aligned} q^{\frac{N-(\kappa ^2 + \kappa r)}{2}} \cdot q^{\kappa ^2+\kappa r} \approx q^{N-(\kappa ^2 + \kappa r)} \cdot q^\kappa . \end{aligned}$$

This gives us $\kappa = \frac{k - (n + m + \sqrt{\delta })}{2} + \frac{1}{3},$ with $\delta = (k - (n + m))^2 + \frac{4}{3} (k + \frac{1}{3})$. The complexity of the overall algorithm with this optimal choice for $\kappa$ is then

$$\begin{aligned} q^{\frac{n + m}{2} + \frac{k-\sqrt{\delta }}{6} + \frac{1}{9}}. \end{aligned}$$

We get that $\sqrt{\delta } \geqslant |k - (n + m) |$ and so for all values of k between $n + m$ and $2(n + m)$, the term $k - \sqrt{\delta }$ is bounded by $n + m$, and hence this gives a bound on the complexity by ${\mathcal {O}}(q^{\frac{2}{3}(n + m) + \frac{1}{9}})$. The term $\frac{1}{9}$ adds a few bits at most to this complexity, but is negligible for most cryptographic purposes.

5.2.3 The case $k \ge 2(n+m)$

When $k \ge 2(n+m)$, as per Lemma 32, the probability that there exist elements with $\dim \ker D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}}> 1$ is extremely small, which is why we can not define a distinguishing predicate for Algorithm 1 and $\kappa =1$ with overwhelming probability. In this case, the complexity of the matrix-pencil algorithm is

$$\begin{aligned} {\mathcal {O}}(q \cdot (m+n)^6), \end{aligned}$$

as with random $\textsf {inhBMLE}$ instances.

6 Experimental results

To confirm our theoretical findings, we solved randomly generated positive instances of the MCE problem, using the two approaches presented in this paper. First, we implement the birthday-based Algorithm 1 in three steps. (1) We randomly generate a positive instance ${\mathcal {I}}_{{\textsf {MCE}}}({\mathcal {C}},{\mathcal {D}})$ of MCE(n, m, k) and reduce it to an instance ${\mathcal {I}}_{{\textsf {hQMLE}}}({\mathcal {F}},{\mathcal {P}})$ of hQMLE $(m+n,k)$. (2) We build the two sample sets for a predefined predicate ${\mathbb {P}}$ and we combine them to create pairs of potential collisions. (3) For each pair we create an inhQMLE instance and we query an inhQMLE solver until it outputs a solution for the maps ${{\textbf {S}}}$ and ${{\textbf {T}}}$. Our implementation is built on top of the open source birthday-based hQMLE solver from [15], which is implemented in MAGMA [12].

Table 1 shows running times for solving the MCE problem using Algorithm 1. The goal of this first experiments was to confirm that there is a parameter choice where the probability of success of the algorithm surpasses $1-1/e$ and that our running times are comparable to the ones given in [14]. These experiments are done with the parameter $q=2$ and all results are an average of 50 runs.

Table 1 Experimental results on solving the MCE problem using Algorithm 1

The second approach, described in Sect. 4.3 uses the bilinear structure of hQMLE instances derived from MCE instances to have an improved algorithm for building the sample sets and a more precise predicate that results in fewer queries to the inhQMLE solver. The consequence of these two improvements to the runtime can be observed in Table 2 where we show experimental results of Algorithm 2 using the non-zeros subsets. Recall that, this approach can be used only when there exist at least two roots of ${\mathcal {F}}$ and ${\mathcal {P}}$. Otherwise, the sampled sets contain only the trivial root and the instance is solved using Algorithm 1. Table 2 shows results of the case when the sets are non-trivial and the probability of this case for the given parameters is shown in the last column. For efficiency, we take the minimal subset with a common dimension of the kernel of ${\textbf {F}}_{\textbf{b}}$, and when looking for collisions, we are careful to skip pairs $({\textbf {ab}},{\textbf {a}}^{\prime }{} {\textbf {b}}^{\prime }$) where $\dim \ker {\textbf {F}}_{\textbf{b}}=\dim \ker {\textbf {P}}_{{\textbf{b}}^{\prime }}$ but $\dim \ker D_{({{\textbf {a}}}, {{\textbf {b}}})}{\mathcal {F}} \ne \dim \ker D_{({\textbf {a}}^{\prime }, {\textbf {b}}^{\prime })}{\mathcal {P}}$. In these experiments, $q=3$ and all results are an average of 50 runs.

Table 2 Experimental results on solving the MCE problem using the non-zeros-subsets variant of Algorithm 2

Our experiments confirm that Algorithm 2 outperforms Algorithm 1 for solving MCE instances with non-trivial roots.