Identifying Overly Restrictive Matching Patterns in SMT-based Program Verifiers (Extended Version)

4.1 Input Formula

To simplify our algorithm, we pre-process the inputs (i.e., the proof obligations or the axioms of a verifier): we Skolemize existential quantifiers and transform all propositional formulas into negation normal form (NNF), where negation is applied only to literals and the only logical connectives are conjunction and disjunction; we also apply the distributivity of disjunction over conjunction and split conjunctions into separate formulas. These steps preserve satisfiability and the semantics of patterns (Section 6 addresses scalability issues). The resulting formulas follow the grammar from Figure 6. Literals L may include interpreted and uninterpreted functions, variables, and constants. Free variables are nullary functions. Quantified variables can have interpreted or uninterpreted types, and we ensure that their names are globally unique. We assume that each quantifier is equipped with a pattern P (if none is provided, we run the solver to infer one). Patterns are combinations of uninterpreted functions and must mention all quantified variables. Since there are no existential quantifiers after Skolemization, we use the term quantifier to denote universal quantifiers.

Fig. 6.

View Figure

4.2 Algorithm

The pseudo-code of our algorithm is given in Algorithm 1. It takes as input an SMT formula \(\texttt {I}\) (defined in Figure 6), which we treat in a slight abuse of notation as both a formula and a set of conjuncts. Three other parameters allow us to customize the search strategy and are discussed later. The algorithm yields a triggering term that enables the unsat proof, or None if no term was found. We assume here that \(\texttt {I}\) contains no nested quantifiers and present those at the end of this subsection.

The algorithm iterates over each quantified conjunct F of \(\texttt {I}\) (Algorithm 1, line 3) and checks if it is individually unsatisfiable (for depth = 0). For complex proofs, this is usually not sufficient, as \(\texttt {I}\) is typically inconsistent due to a combination of conjuncts (\(F_0 \wedge F_1\) in Figure 5). In such cases, the algorithm proceeds as follows:

Step 1: Clustering. It constructs clusters of formulas similar to F (Algorithm 2, line 4), based on their Jaccard similarity index. Let \(F_i\) and \(F_j\) be two arbitrary formulas, and \(S_i\) and \(S_j\) their respective sets of uninterpreted function symbols (from their bodies and the patterns of the quantifiers). The Jaccard similarity index is defined as (3) \(\begin{equation} J(F_i, F_j) =\frac{|S_i \cap S_j|}{|S_i \cup S_j|} . \end{equation}\) That is, the number of common uninterpreted functions divided by the total number. For the two formulas from Figure 5, \(S_0 =\lbrace \mathtt {f}\rbrace\), \(S_1 =\lbrace \mathtt {f, g} \rbrace\), therefore \(J(F_0, F_1) = \frac{|\lbrace \mathtt {f}\rbrace |}{|\lbrace \mathtt {f, g} \rbrace |} = 0.5\).

Our algorithm explores the search space by iteratively expanding clusters to include transitively-similar formulas up to a maximum depth (parameter \(\delta\) in Algorithm 1). For two formulas \(F_i, F_j \in \texttt {I}\), we define the similarity function as (4) \(\begin{equation} \texttt {sim}_\texttt {I}^{\delta }(F_i, F_j, \sigma)= \left\lbrace \begin{array}{ll} J(F_i, F_j) \ge \sigma , &\: \: \delta = 1 \\ \exists F_k: \texttt {sim}_{\texttt {I}\setminus \lbrace F_i\rbrace }^{\delta -1}(F_i, F_k, \sigma) \text{and} J(F_k, F_j) \ge \sigma , &\:\:\delta \gt 1 \end{array} \right. , \end{equation}\) where \(\sigma \in [0,1]\) is a similarity threshold used to parameterize our algorithm and J is defined in (3).

The initial cluster (for \(\texttt {depth} = 1\)) includes all the conjuncts of \(\texttt {I}\) that are directly similar to F. Each subsequent iteration adds the conjuncts that are directly similar to an element of the cluster from the previous iteration, that is, transitively similar to F. This search strategy allows us to gradually strengthen the formulas G (used to synthesize candidate terms in step 3) without overly constraining them (an over-constrained formula is unsatisfiable, and has no models).

Step 2: Syntactic unification. Next (Algorithm 2, line 7), we identify rewritings, i.e., constraints under which two similar quantified formulas share terms. (Section 4.4 presents the quantifier-free case.) We obtain the rewritings by performing a simplified form of syntactic term unification, which reduces their number to a practical size. Our rewritings are directed equalities. For two formulas \(F_i\) and \(F_j\) and an uninterpreted function \(\mathtt {f}\) they have the following shape: (5) \(\begin{equation} x_m = {rhs}_n , \end{equation}\) where \(m = i\) and \(n = j\) or \(m = j\) and \(n = i\), \(x_m\) is a quantified variable of \(F_m\), \(F_m\) contains a term \(\mathtt {f}(x_m)\), \(F_n\) contains a term \(\mathtt {f}({rhs}_n)\), and \({rhs}_n\) is a constant \(c_n\), a quantified variable \(x_n\), or a composite function \((\mathtt {f} \circ \mathtt {g}_0 \circ \cdots \circ \mathtt {g}_p)(\overline{c_n}, \overline{x_n})\) occurring in the formula \(F_n\); \(\mathtt {g}_0, \ldots , \mathtt {g}_p\) are arbitrary (interpreted or uninterpreted) functions. We thus determine the most general unifier [14] only for those terms that have uninterpreted functions as the outer-most functions and quantified variables as arguments. The unification algorithm is standard (except for the restricted shape), so it is not shown explicitly.

In Figure 5, \(F_1\) is similar to \(F_0\) for any \(\sigma \le 0.5\). We then compute the rewritings for all the quantified variables of \(F_0\) that appear in its body as arguments to some common uninterpreted functions (in this case, only \(x_0\)). Unifying the terms \(\mathtt {f}(x_0)\) and \(\mathtt {f}(\mathtt {g}(x_1))\) generates the rewriting \(x_0 = \mathtt {g}(x_1)\), which has the shape defined in (5).

Since a term may appear more than once in F, or F unifies with multiple similar formulas through the same quantified variable, we can obtain alternative rewritings for a quantified variable. In such cases, we either duplicate or split the cluster, such that in each cluster-rewriting pair, each quantified variable is rewritten at most once (see Algorithm 2, line 11). For example, in Figure 7, both \(F_1\) and \(F_2\) are similar to \(F_0\) (all three formulas share the uninterpreted function symbol \(\mathtt {f}\)). As the unification produces alternative rewritings for \(x_0\) (\(x_0 = x_1\) and \(x_0 = x_2\)), the procedure clustersRewritings returns the pairs \(\lbrace (\lbrace F_1\rbrace , \lbrace x_0 = x_1\rbrace), (\lbrace F_2\rbrace , \lbrace x_0 = x_2\rbrace)\rbrace\). Step 3: Identifying candidate terms. From the clusters and the rewritings (identified before), we then derive quantifier-free formulas G (Algorithm 1, line 10), and, if they are satisfiable, construct the candidate triggering terms from their models (Algorithm 1, line 15). Each formula G consists of: (1) \(\lnot F_{\approx }\) (defined in (1), which is equivalent to \(\lnot F^{\prime }\), since F has the shape \(\forall \overline{x} :: F^{\prime }\) from Algorithm 1, line 3), (2) the instantiations (defined in (2)) of all the similar formulas from the cluster, and (3) the corresponding rewritings R. (As we assume that all the quantified variables are globally unique, we do not perform variable renaming when computing the instantiations).

Fig. 7.

View Figure

If a similar formula has multiple disjuncts \(D_k\), the SMT solver may use short-circuiting semantics when generating the model for G. That is, if it can find a model that satisfies the first disjunct, it may not consider the remaining ones. To obtain more diverse models, we synthesize formulas that cover each disjunct, i.e., make sure that it evaluates to \(\mathtt {true}\) at last once. We thus compute multiple instantiations of each similar formula, of the form: \((\bigwedge _{0 \le j \lt k} \lnot D_j) \wedge D_k, \forall k:0 \le k \le n\) (see Algorithm 1, line 7). To consider all the combinations of disjuncts, we derive the formula G from the Cartesian product of the instantiations (Algorithm 1, line 9). (For presentation purposes, we also store \(\lnot F^{\prime }\) in the instantiations map (Algorithm 1, line 8), even if it does not represent the instantiation of F.)

In Figure 8, \(F_1\) is similar to \(F_0\) and \(R = \lbrace x_0 = x_1\rbrace\). \(F_1\) has two disjuncts and thus two possible instantiations: \(\textsf {Inst[}F_1\textsf {]} = \lbrace x_1 \ge 1, (x_1 \lt 1) \wedge (\mathtt {f}(x_1) = 6) \rbrace\). The formula \(G = (x_0 \gt -1) \wedge (\mathtt {f}(x_0) \le 7) \wedge (x_1 \ge 1) \wedge (x_0 = x_1)\) for the first instantiation is satisfiable, but none of the values the solver can assign to \(x_0\) (which are all greater or equal to 1) are sufficient for the unsatisfiability proof to succeed. The second instantiation adds additional constraints: instead of \(x_1 \ge 1\), it requires (\(x_1 \lt 1) \wedge (\mathtt {f}(x_1) = 6)\). The resulting G formula has a unique solution for \(x_0\), namely 0, and the triggering term \(\mathtt {f}(0)\) is sufficient to prove unsat.

Fig. 8.

View Figure

The procedure \(\textsf {candidateTerm}\) in Algorithm 3 synthesizes a candidate triggering term T from the models of G and the rewritings R. We first collect all the patterns of the formulas from the cluster C (Algorithm 3, line 2), i.e., of F and of its similar conjuncts (see Algorithm 1, line 15). Then, we apply the rewritings, in an arbitrary order (Algorithm 3, lines 3–6). That is, we substitute the quantified variable x from the left-hand side of the rewriting with the right-hand side term rhs and propagate this substitution to the remaining rewritings. This step allows us to include in the synthesized triggering terms additional information, which cannot be provided by the solver. Then (Algorithm 3, lines 7–8) we substitute the remaining variables with their constant values from the model (i.e., constants for built-in types, and fresh, unconstrained variables for uninterpreted types). For interpreted, user-defined types (such as a type IList for representing a List of Int, where List and Int are both interpreted types), the solver generates constants for each type component, or a sequence of operations required to construct them. For instance, insert(0, nil) (i.e., a singleton list containing the constant 0) is a possible model provided by the SMT solver Z3 [24] for a variable of type IList. The resulting triggering term is wrapped in an application to a fresh, uninterpreted function dummy to ensure that conjoining it to \(\texttt {I}\) does not change \(\texttt {I}\)’s satisfiability.

Step 4: Validation. We validate the candidate triggering term T by checking if \(\texttt {I}\wedge T\) is unsatisfiable, i.e., if these particular interpretations for the uninterpreted functions generalize to all interpretations (Algorithm 1, line 16). If this is the case, then we return the minimized triggering term (Algorithm 1, line 18). The \(\mathtt {dummy}\) function has multiple arguments, each of them corresponding to one pattern from the cluster (Algorithm 3, line 9). This is an over-approximation of the required triggering terms (once instantiated, the formulas may trigger each other), so minimized removes redundant (sub-)terms. If T does not validate, we re-iterate its construction up to a bound \(\mu\) and strengthen the formula G to obtain a different model (Algorithm 1, lines 19 and 11). The parameter \(\mu\) allows us to deal with other sources of incompleteness, as we explain next.

Let us consider the formula from Figure 9, which was part of an axiomatization with 2,495 axioms. F axiomatizes the uninterpreted function \(\mathtt {\_div}:\mathrm{Int} \times \mathrm{Int} \rightarrow \mathrm{Int}\) and is inconsistent, because there exist two integers whose real division (“/”) is not an integer. The model produced by Z3 for the formula \(G = \lnot F^{\prime }\) is \(x = -1, y = 0\). \(-1/0\) is defined (“/” is a total function [18]), but its result is not specified. Thus, the solver cannot validate this model (i.e., it returns unknown). In such cases, we ask the solver for a different model. In Figure 9, if we simply exclude previous models, we can obtain a sequence of models with different values for the numerator, but with the same value (0) for the denominator. There are infinitely many such models; all of them fail to validate for the same reason.

Fig. 9.

View Figure

There are various heuristics one can employ to guide the solver’s search for a new model and our algorithm can be parameterized with different ones. In our experiments, we interpret the conjunct \(\lnot \mathtt {model}\) from Algorithm 1, line 19 as \((\bigwedge _{x\in \overline{x}} x \ne \mathtt {model}(x)) \wedge (\bigwedge _{x_i, x_j\in \overline{x}, \; i \ne j, \; \mathtt {model}(x_i) = \mathtt {model}(x_j)} x_i \ne x_j)\). This allows us to synthesize the triggering term \(\mathtt {dummy}(\mathtt {\_div}(1, 2))\) and expose the error from Figure 9.

The first component (\(\bigwedge _{x\in \overline{x}} x \ne \mathtt {model}(x)\)) requires all the variables to have different values than before. This requirement may be too strong for some variables, but as we use only soft constraints, the solver may ignore some constraints if it cannot generate a satisfying assignment. The second part (i.e., \(\bigwedge _{x_i, x_j\in \overline{x}, \; i \ne j, \; \mathtt {model}(x_i) = \mathtt {model}(x_j)}\) \(x_i \ne x_j\)) requires models from different equivalence classes, where an equivalence class includes all the variables that are equal in the model. For example, if the model is \(x_0 = x, x_1 = x\), where x is a value of the corresponding type, then \(x_0\) and \(x_1\) belong to the same equivalence class. Considering equivalence classes is particularly important for variables of uninterpreted types. The solver cannot provide actual values for them, thus it assigns fresh, unconstrained variables. However, different fresh variables do not lead to diverse models.

Nested quantifiers. Our algorithm also supports nested quantifiers. Nested existential quantifiers in positive positions and nested universal quantifiers in negative positions are replaced in NNF by new, uninterpreted Skolem functions. Step 2 is also applicable to them: Skolem functions with arguments (the quantified variables from the outer scope) are unified as regular uninterpreted functions; they can also appear as rhs in a rewriting, but not as the left-hand side (we do not perform higher-order unification). In such cases, the result is imprecise: the unification of \(\mathtt {f}(x_0, \mathtt {skolem}())\) and \(\mathtt {f}(x_1, 1)\) produces only the rewriting \(x_0 = x_1\).

After pre-processing, the conjunct F and the similar formulas may still contain nested universal quantifiers. F is always negated in G, thus it becomes, after Skolemization, quantifier-free. To ensure that G is also quantifier-free (and the solver can generate a model), we extend the algorithm to recursively instantiate similar formulas with nested quantifiers when computing the instantiations.

4.3 Extensions

Next, we describe various extensions of our algorithm that enable complex unsatisfiability proofs.

Combining multiple candidate terms. In Algorithm 1, each candidate term is validated separately. To enable proofs that require multiple instantiations of the same formula, we developed an extension that validates multiple triggering terms at the same time. In such cases, the algorithm returns a set of terms that are necessary and sufficient to prove unsat. Figure 10 presents a simple example from SMT-COMP 2019 pending benchmarks [3]. (The files in this category are not guaranteed to comply with the SMT-LIB standard, but our benchmarks selection algorithm described in Section 5.2 checks this automatically.) The input \(\texttt {I}= F_0 \wedge F_1\) is unsatisfiable, as there does not exist an interpretation for the uninterpreted function \(\mathtt {U}\) that satisfies all the constraints: \(F_1\) requires \(\mathtt {U}(\mathtt {s})\) to be \(\mathtt {true}\); if \(F_0\) is instantiated for \(x_0 = \mathtt {s}\), the solver learns that \(\mathtt {U}(\mathtt {il})\) must be \(\mathtt {true}\) as well; however, if \(x_0 = \mathtt {il}\), then \(\mathtt {U}(\mathtt {il})\) must be \(\mathtt {false}\), which is a contradiction. Exposing the inconsistency thus requires two instantiations of \(F_0\), triggered by \(\mathtt {f}(\mathtt {s})\) and \(\mathtt {f}(\mathtt {il})\), respectively. We generate both triggering terms, but in separate iterations (independently, both fail to validate). However, by validating them simultaneously (i.e., conjoining both of them to \(\texttt {I}\), as arguments to the fresh function \(\mathtt {dummy}\)), our algorithm identifies the required triggering term \(T = \mathtt {dummy}(\mathtt {f}(\mathtt {s}), \mathtt {f}(\mathtt {il}))\).

Fig. 10.

View Figure

Unification across multiple instantiations. The clusters constructed by our algorithm are sets (see Algorithm 2, line 11), so they contain a formula at most once, even if it is similar to multiple other formulas from the cluster. We thus consider the rewritings for multiple instantiations of the same formula separately, in different iterations. To handle cases that require multiple (but boundedly many) instantiations, we extend the algorithm with a parameter \(\Phi\), which bounds the maximum frequency of a quantified conjunct within the formulas G. That is, it allows a similar quantified formula, as well as F itself, to be added to a cluster (now represented as a list) more than once (after performing variable renaming, to ensure that the names of the quantified variables are still globally unique). This results in an equisatisfiable formula for which our algorithm determines multiple triggering terms. Inputs whose unsatisfiability proofs require an unbounded number of instantiations typically contain a matching loop, thus, we do not consider them here. Figure 11 presents an example, which consists of a single inconsistent formula F. Our regular algorithm from Algorithm 2 does not identify any rewritings. However, with this extension, F unifies with itself for any \(\Phi \gt 1\), and one possible rewriting is \(x^{\prime } = 7\) (where \(x^{\prime }\) is a fresh variable representing the second instantiation of F). The corresponding triggering term, \(T = \mathtt {dummy}(\mathtt {g}(7))\), allows E-matching to prove unsat. Note that the uninterpreted function \(\mathtt {g}\) is used only as a pattern. If the pattern would have been \(\mathtt {f}(x)\), any triggering term \(\mathtt {f}(c)\), where c is an integer constant, would have been sufficient to complete the proof: (1) for \(c = 7\), the contradiction would have been exposed directly; (2) for \(c \ne 7\), the term \(\mathtt {f}(7)\), obtained from the first instantiation of F, would have triggered its second instantiation and case (1) would have then applied. Type-based constraints. The rewritings of the form \(x_i = x_j\) can be too imprecise (especially for quantified variables of uninterpreted types), as they do not constrain the rhs. In Figure 12, the solver cannot provide concrete values of the uninterpreted type \(\mathrm{U}\) for \(e_1\) and \(op_1\), it can only assign fresh, unconstrained variables (e.g., e and op). However, the triggering terms \(\mathtt {some}(e)\) and \(\mathtt {get}(op)\), which can be obtained from these fresh variables, are not sufficient to prove unsat; one would additionally need the rewriting \(e_1 = \mathtt {get}(op_1)\), which cannot be identified by our unification from Section 4.2. To address such scenarios, we extend the unification to also consider as rhs a constant or an uninterpreted function from the body of the similar formulas, which has the same type as the quantified variable from the left-hand side. For Figure 12, it will thus generate the rewritings \(R = \lbrace e_0 = \mathtt {get}(op_2), e_1 = \mathtt {get}(op_2), op_1 = \mathtt {none}, op_2 = \mathtt {none}\rbrace\) (this is one of the alternatives). These type-based constraints allow us to synthesize the triggering term \(T = \mathtt {dummy}(\mathtt {some}(\mathtt {get}(\mathtt {none})))\), which exposes the unsoundness from Gobra’s [52] option types axiomatization.

Fig. 11.

View Figure

Fig. 12.

View Figure

Unification for sub-terms. Figure 13 shows an example which cannot be solved by any extension discussed so far, since it requires semantic reasoning: by applying \(\mathtt {f}\) on both sides of the equality, one can learn from \(F_1\) that \(\mathtt {f}(\mathtt {g}(2020)) = \mathtt {f}(\mathtt {g}(2021))\). From \(F_0\) though, \(\mathtt {f}(\mathtt {g}(2020)) = 2020\) and \(\mathtt {f}(\mathtt {g}(2021)) = 2021\), which implies that \(2020 = 2021\), i.e., \(\mathtt {false}\). Our extended algorithm synthesizes the required triggering term \(T = \mathtt {dummy}(\mathtt {f}(\mathtt {g}(2020)), \mathtt {f}(\mathtt {g}(2021)))\) by applying the unification also to sub-terms; due to our restrictive shape of the rewritings, the sub-terms can only be applications of uninterpreted functions. In Figure 13, trying to unify \(\mathtt {f}(\mathtt {g}(x_0))\) does not produce any rewritings, as \(F_1\) does not contain \(\mathtt {f}(\mathtt {g})\). We thus unify the sub-term \(\mathtt {g}(x_0)\) with \(\mathtt {g}(2020)\) and \(\mathtt {g}(2021)\) and obtain the rewritings \(R = \lbrace x_0 = 2020, x_0 = 2021\rbrace\). Together with the extension for combining multiple candidate terms described above, these rewritings provide sufficient information for the unsat proof to succeed. This unification is syntactic, but produces the triggering terms that would be obtained if the solver would apply some uninterpreted function present in the input on a learned predicate (the solver performs semantic reasoning automatically, but without generating new triggering terms). Alternative triggering terms. Our algorithm returns the first candidate term that successfully validates (Algorithm 1, line 18). However, it might also be useful to synthesize alternative triggering terms for the same input, as they may correspond to different completeness or soundness issues. Our tool provides this option and can also return all the triggering terms found within the given timeout.

Fig. 13.

View Figure

All these extensions (individually or together with other extensions) allow us to complete the refutation proofs for particular benchmarks. Section 5 evaluates the impact of a few configurations of our technique, which can be obtained by enabling some of the extensions or by setting certain values for some of the additional parameters. Automatically determining which is the most suited configuration for a particular input is left as future work.

4.4 Additional Examples

In this section, we illustrate our algorithm on various examples (including those from Figure 1 and Figure 2, and an example with nested quantifiers). We also explain how the algorithm supports quantifier-free formulas, synonym functions as patterns, multi-patterns, and alternative patterns.

Nested quantifiers. Our algorithm handles inputs with nested quantifiers as described in Section 4.2. We illustrate this aspect on the formulas from Figure 14, which axiomatize operations over lists of integers. The axioms \(F_3\) and \(F_4\) set contradictory constraints on \(\mathtt {indexOf}\) when the element is not contained in the list. According to Algorithm 2, one of the clusters generated for \(F_3\) is \(C=\lbrace F_2, F_0\rbrace\), with the rewritings \(R=\lbrace l_3 = l_2, el_3 = el_2, l_2 = l_0\rbrace\). The algorithm then computes the instantiations for \(F_0\) and \(F_2\); as \(F_2\) contains nested quantifiers, we remove both of them and obtain: \(\textsf {Inst[}F_2\textsf {]} = \lbrace \lnot \mathtt {isEmpty}(l_2), \mathtt {isEmpty}(l_2) \wedge \lnot \mathtt {has}(l_2, el_2)\rbrace\), \(\textsf {Inst[}F_0\textsf {]} = \lbrace \lnot (l_0 = \mathtt {EmptyList}), l_0 = \mathtt {EmptyList} \wedge \mathtt {isEmpty}(l_0) \rbrace\). The model of the corresponding G formula and R allow us to synthesize the required triggering term \(T= \mathtt {dummy}(\mathtt {isEmpty}(\mathtt {EmptyList}), \mathtt {has}(\mathtt {EmptyList}, 0))\). Quantifier-free formulas. Our algorithm iterates only over quantified conjuncts but leverages the additional information provided by quantifier-free formulas and includes them in the clusters even if the unification cannot find a rewriting (Algorithm 2, line 8). Since quantifier-free conjuncts can be seen as already instantiated formulas, we only have to cover all their disjuncts (Algorithm 1, line 7). Boogie example. Figure 15 shows the example from Figure 1 encoded in our input format. The quantifier-free formula \(F_3\) (i.e., the negation of the verification condition) is similar to \(F_0\) (they share the function symbol \(\mathtt {len}\)) and unifies through the rewritings \(R = \lbrace x_0 = 7\rbrace\). We obtained the required triggering term \(T = \mathtt {dummy}(\mathtt {len}(\mathtt {nxt}(7)))\) from the model of the formula \(G = \lnot F_0^{\prime } \wedge \textsf {Inst[}F_3\textsf {][}0\textsf {]} \wedge \bigwedge {R} = (\mathtt {len}(x_0) \le 0) \wedge (\mathtt {len}(7) \le 0) \wedge (x_0 = 7)\).

Fig. 14.

View Figure

Fig. 15.

View Figure

Dafny example. Our algorithm can synthesize various triggering terms that expose the unsoundness from Figure 2, depending on the values of its parameters. We explain here one, obtained for \(\sigma = 0.1\). For \(\texttt {depth} = 0\), the algorithm checks each formula \(F_0\)–\(F_4\) in isolation. As they are all individually satisfiable, it continues with \(\texttt {depth} = 1\). To avoid redundant explanations, we present only the iteration for \(F_3\). \(F_3\) shares at least two uninterpreted function symbols with each of the other formulas, so there are various alternative rewritings: \(s_3 = \mathtt {Empty}(t_1)\), \(s_3 = s_4\), \(s_3 = \mathtt {Build}(s_4, i_4, v_4, l_4)\), and so on. As we consider clusters-rewritings pairs in which each quantified variable has maximum one rewriting, one such pair is \((C = \lbrace F_4\rbrace , R = \lbrace s_3 = \mathtt {Build}(s_4, i_4, v_4, l_4)\rbrace)\). \(F_4\) has two disjuncts, therefore its instantiations are: \(\textsf {Inst[}F_4\textsf {]} = \lbrace \mathtt {typ}(s_4) = \mathtt {Type}(\mathtt {typ}(v_4),\) \(\lnot (\mathtt {type}(s_4) = \mathtt {Type}(\mathtt {typ}(v_4)) \wedge (\mathtt {Len}(\mathtt {Build}(s_4,\) \(i_4, v_4, l_4)) = l_4)\rbrace\). From these instantiations and the rewritings R, we derive two formulas: \(G_0 = \lnot F_3^{\prime } \wedge \textsf {Inst[}F_4\textsf {][}0\textsf {]} \wedge \bigwedge R\), with the model \(s_3 = s\), \(s_4 = s^{\prime }\), \(i_4 = 0\), \(v_4 = v\), \(l_4 = 1\) and \(G_1 = \lnot F_3^{\prime } \wedge \textsf {Inst[}F_4\textsf {][}1\textsf {]} \wedge \bigwedge R\), with the model \(s_3 = s\), \(s_4 = s^{\prime }\), \(i_4 = 0\), \(v_4 = v\), \(l_4 = -1\), where s, \(s^{\prime }\), and v are fresh variables of type \(\mathrm{U}\). (We use indexes for the G formulas to refer to them later.) We then construct the candidate triggering terms from the patterns of the formulas \(F_3\) and \(F_4\). We replace \(s_3\) by its rhs in the rewriting, i.e., \(\mathtt {Build}(s_4, i_4, v_4, l_4)\), and all the other quantified variables by their constants from the model. The result after removing redundant terms is: \(T_0 = \mathtt {dummy}(\mathtt {Len}(\mathtt {Build}(t, 0, v, 1)))\) and \(T_1 = \mathtt {dummy}(\mathtt {Len}(\mathtt {Build}(t, 0, v, -1)))\). Since the validation step fails for both \(T_0\) and \(T_1\), we continue with the other \((C, R)\) pairs, the remaining quantified conjuncts and their similarity clusters.

If no candidate term is sufficient to prove unsat, our algorithm expands the clusters. To scale to real-world axiomatizations, it efficiently reuses the results from the previous iterations; i.e., it prunes the search space if a previously synthesized formula G is unsatisfiable and it strengthens G if it is satisfiable. The pair \((C = \lbrace F_4\rbrace , R = \lbrace s_3 = \mathtt {Build}(s_4, i_4, v_4, l_4)\rbrace)\) can be extended to \((C = \lbrace F_4, F_1\rbrace , R = \lbrace s_3 = \mathtt {Build}(s_4, i_4, v_4, l_4), s_4 = \mathtt {Empty}(t_1), t_1 = \mathtt {typ}(v_4)\rbrace)\), as \(F_1\) is similar to \(F_4\) through the rewritings \(R = \lbrace s_4 = \mathtt {Empty}(t_1), t_1 = \mathtt {typ}(v_4)\rbrace\). We thus conjoin the instantiation of \(F_1\) and the two additional rewritings to the formulas \(G_0\) and \(G_1\) from the previous iteration. This is equivalent to recomputing the similarity cluster, the rewritings, and the combinations of instantiations. We then obtain: \(G_0^{\prime } = G_0 \wedge (\mathtt {type}(\mathtt {Empty}(t_1)) = \mathtt {Type}(t_1)) \wedge (s_4 = \mathtt {Empty}(t_1)) \wedge (t_1 = \mathtt {typ}(v_4))\), which is unsatisfiable, and \(G_1^{\prime } = G_1 \wedge (\mathtt {type}(\mathtt {Empty}(t_1)) = \mathtt {Type}(t_1)) \wedge (s_4 = \mathtt {Empty}(t_1)) \wedge (t_1 = \mathtt {typ}(v_4))\) with the model: \(s_3 = s\), \(s_4 = s^{\prime }\), \(i_4 = 0\), \(v_4 = v\), \(l_4 = -1\), \(t_1 = t\), where s, \(s^{\prime }\), v, and t are fresh variables of types \(\mathrm{U}\) and \(\mathrm{V}\), respectively. From this model and the rewritings, we construct the triggering term \(T = \mathtt {dummy}(\mathtt {Len}(\mathtt {Build}(\mathtt {Empty}(\mathtt {typ}(v)), 0, v, -1)))\), which is sufficient to expose the inconsistency between \(F_3\) and \(F_4\). VCC/HAVOC example. Figure 16 presents a fragment of an unsatisfiable benchmark generated by VCC/HAVOC [22, 47], which cannot be refuted by neither E-matching, nor MBQI. (Section 5 provides additional experimental results and a detailed comparison between our algorithm and alternative refutation techniques.) F, which was part of a set of 160 formulas, is inconsistent by itself: when \(size = 0\), it evaluates to \(\mathtt {false}\) for any integer values a, b, such that \(a \le b\). Our algorithm synthesizes a triggering term for E-matching in \(\approx\)7 s because it initially considers each quantified conjunct in isolation. The formula \(G = \lnot F^{\prime } = \mathtt {both\_ptr}(a, b, size) * size \gt a - b\) is satisfiable and the simplest models the solver can provide (without assigning an interpretation to the uninterpreted function \(\mathtt {both\_ptr}\)) all include \(size = 0\) and some values for a and b, such that \(a \le b\) (e.g., \(a = -2\), \(b = -1\)). Synonym functions as patterns. For the examples discussed so far, the functions used as patterns were also present in the body of the quantifiers. However, to have better control over the instantiations, one can also write formulas where the patterns are additional uninterpreted functions, which do not appear in the bodies. Such patterns are not uncommon in proof obligations. Figure 17 shows an example, which uses the synonym functions technique [33] to avoid matching loops. \(\mathtt {sum}\) and \(\mathtt {sum\_syn}\) compute the sum of the elements of a sequence, between a lower and an upper bound. The two functions are identical (according to \(F_0\)), but only \(\mathtt {sum}\) is used as a pattern. For equal bounds, \(F_1\) and \(F_2\) set contradictory constraints on the interpretation of \(\mathtt {sum\_syn}\). \(\mathtt {seq.nth}\) returns the nth element of the sequence. Using the information from the quantifier-free formula \(F_3\), our algorithm generates the triggering term \(T=\mathtt {dummy}(\mathtt {sum}(\mathtt {empty}, 0, 0), \mathtt {sum}(\mathtt {empty}, 0+1, 0))\). The term “\(0+1\)” comes from the rewriting \(l_0 = l_2 +1\). The addition is a built-in function but is used as an argument to the uninterpreted function \(\mathtt {sum\_syn}\), thus, it is supported by our unification. Our algorithm is syntactic, so it does not perform arithmetic operations, it just substitutes \(l_2\) with its value from the model. The solver then performs theory reasoning and concludes unsat.

Fig. 16.

View Figure

Fig. 17.

View Figure

Multi-patterns and alternative patterns. SMT solvers allow patterns to contain multiple terms, all of which must be present to perform an instantiation. \(F_1\) in Figure 18 has such a multi-pattern and can be instantiated only when triggering terms that match both \(\lbrace \mathtt {g}(b_1)\rbrace\) and \(\lbrace \mathtt {f}(x_1)\rbrace\) are present in the SMT run. Our algorithm directly supports multi-patterns, as the procedure \(\textsf {candidateTerm}\) instantiates all the patterns from the given cluster (see Algorithm 3, line 9). For the example from Figure 18, our technique synthesizes the triggering term \(T = \mathtt {dummy}(\mathtt {f}(7), \mathtt {g}(b))\) from the rewritings \(R=\lbrace x_0 = x_1\rbrace\) and the model of the formula \(G = \lnot F_0^{\prime } \wedge \textsf {Inst[}F_1\textsf {][}1\textsf {]} \wedge \bigwedge R = (\mathtt {f}(x_0) = 7) \wedge (\lnot \mathtt {g}(b_1) \wedge \mathtt {f}(x_1) = x_1) \wedge (x_0 = x_1)\). Here b is a fresh, unconstrained variable of the uninterpreted type \(\mathrm{B}\).

Fig. 18.

View Figure

Formulas can also contain alternative patterns. For example, the quantified formula \(\forall x:\mathrm{Int} :: \lbrace \mathtt {f}(x)\rbrace \: \lbrace \mathtt {h}(x)\rbrace \; (\mathtt {f}(x) \ne 7) \vee (\mathtt {h}(x) = 6)\) is instantiated only if there exists a triggering term that matches \(\lbrace \mathtt {f}(x)\rbrace\) or one that matches \(\lbrace \mathtt {h}(x)\rbrace\). Our algorithm does not differentiate between multi-patterns and alternative patterns, thus it always synthesizes the arguments for all the patterns of a cluster. For alternative patterns, this results in an over-approximation of the set of necessary triggering terms. However, the minimization step (performed before returning the triggering term that successfully validates), removes the unnecessary terms.

5.1 Effectiveness on Verification Benchmarks with Known Triggering Issues

First, we used manually-collected benchmarks with known triggering issues from four state-of-the-art verifiers: Dafny [32], F* [49], Gobra [52], and Viper [38]. We reconstructed 4, respectively 2 inconsistent axiomatizations from Dafny and F*, based on the changes from the repositories and the messages from the issue trackers; we obtained 11 inconsistent axiomatizations of arrays and option types from Gobra’s developers and collected 15 incompleteness issues from Viper’s test suite [1], with at least one assertion needed only for triggering (we removed these assertions from the benchmarks, as our work is expected to find the triggering terms automatically). The Viper files contain algorithms for arrays, binomial heaps, binary search trees, and regression tests. The input sizes (minimum–maximum number of formulas or quantifiers) are shown in Table 1, columns \(\mathbf {\#F}\)–\(\mathbf {\#\forall }\).

Configurations. We ran our tool with five configurations, to also analyze the impact of its parameters (see Algorithm 1 and Section 4.3). The default configuration C0 has: \(\sigma = 0.3\) (similarity threshold), \(\beta =64\) (batch size, i.e., the number of candidate terms validated together), \(\lnot\)type (no type-based constraints), \(\lnot\)sub (no unification for sub-terms). The other configurations differ from C0 in the parameters shown in Table 1. All configurations use \(\delta = 2\) (maximum transitivity depth), \(\mu = 4\) (maximum number of different models), and 600 s timeout per file.

Results. Columns C0–C4 in Table 1 show the number of files solved by each configuration, Our work summarizes those solved by at least one. Overall, we found suited triggering terms for 65.6%, including all F* and Gobra benchmarks. An F* unsoundness exposed by all configurations in \(\approx\)60 s is given in Figure 9. It required two developers to be manually diagnosed based on a bug report [4]. A simplified Gobra axiomatization for option types is shown in Figure 12; the entire axiomatization (considered in Table 1) was solved only by C4 in \(\approx\)13 s. Gobra’s team spent one week to identify some of the issues. As our triggering terms for F* and Gobra were similar to the manually-written ones, we believe they could have reduced the human effort in localizing and fixing the errors.

Our algorithm synthesized missing triggering terms for 7 Viper files, including the array maximum example [2], for which E-matching could not prove that the maximal element in a strictly increasing array of size 3 is its last element. Our triggering term loc(a,2) (loc maps arrays and integers to heap locations) can be added by a user of the verifier to their postcondition. A developer can fix the root cause of the incompleteness by including a generalization of the triggering term to arbitrary array sizes: len(a) != 0 ==> x == loc(a, len(a)-1).val (val allows one to access the value at the corresponding heap location). Both fixes result in E-matching refuting the proof obligation in under 0.1 s. We also exposed another case where Boogie (which is used by Viper) is sound only modulo patterns (as in Figure 3), i.e., the unsoundness is visible only at the SMT level.

As Table 1 shows, configurations with smaller \(\sigma\) (C1 and C4) were particularly important for some of the F* and Gobra benchmarks. Our algorithm starts with the given \(\sigma\) and if it does not find the required triggering terms, it decreases \(\sigma\) by 0.1 and reiterates. Thus C0 also covers the case \(\sigma = 0.1\), if the overall timeout is large enough. However, always starting with a small \(\sigma\) may prevent our algorithm from synthesizing the triggering terms, since the number of rewritings it has to explore is considerably high. The extensions for unifying sub-terms (C4) and identifying type-based constraints (C3) were also needed for one, respectively, two input files.

5.2 Effectiveness on SMT-COMP Benchmarks

Next, we considered 61 SMT-COMP [5] benchmarks from Spec# [16], VCC [47], Havoc [22], Simplify [25], and the Bit-Width-Independent (BWI) encoding [39]. These were selected automatically using a filtering algorithm that we designed (described below) and are summarized in Table 2.

Benchmarks selection. We collected all 27,716 benchmarks from SMT-COMP 2020 (single query track) [5], with ground truth unsat and at least one pattern (as this suggests they were designed for E-matching). We then ran Z3 to infer the missing patterns and to transform the formulas into NNF and removed all benchmarks for which the inference or the transformation did not succeed within 600 s per file and 4 s per formula. We also removed the files with features not yet supported by PySMT [27], the parsing library used in our experiments (e.g., sort signatures in datatypes declarations), but we did extend PySMT to handle, e.g., patterns and overloaded functions. This filtering resulted in 6,481 benchmarks. We then ran E-matching and kept only those 61 examples that could not be solved within 600 s due to incompleteness in instantiating quantifiers (our work only targets this incompleteness, but the SMT-COMP suite also contains other solving challenges).

Results. The results are shown in Table 2, which follows the structure of Table 1. Our algorithm enabled E-matching to refute 47.5% of the files, most of them from Spec# and VCC/Havoc. We manually inspected some BWI benchmarks (for which the algorithm had worse results) and observed that the validation step times out even with a much higher timeout. This shows that some candidate terms trigger matching loops and explains why C2 (which validates them individually) solved one more file. Extending our algorithm to avoid matching loops, by construction, is left as future work. The other configurations did not prove to be better than C0 for these SMT-COMP inputs.

5.3 Comparison with Unsatisfiability Proofs

As an alternative to our work, the developers of program verifiers could try to manually identify triggering issues from refutation proofs. In this experiment, we considered three state-of-the-art provers that rely on different solving strategies and could generate such proofs: Z3 (4.8.10, the same version as for our tool) with MBQI [28] (a model based quantifier instantiation technique), CVC4 (1.6) [17] with enumerative instantiation [43] (an algorithm based on E-matching, used when E-matching saturates), and the first-order theorem prover Vampire (4.4) [31], using Z3 for ground theory reasoning [42] and the CASC [48] portfolio mode with competition presets. Note that the only approach comparable with ours is enumerative instantiation; MBQI and Vampire do not consider patterns, thus they solve a different problem. We are not aware of any other work that synthesizes triggering terms for E-matching.

The last three columns in Tables 1 and 2 show the number of unsatisfiability proofs produced by each of these alternatives. CVC4 failed for all examples (it cannot construct proofs for quantified logics), Vampire refuted most of them. Our algorithm enabled E-matching to solve more inputs than MBQI for F* and Gobra and had similar results for Dafny, Spec#, and VCC/Havoc. All our five configurations solved two VCC/Havoc files not refuted by MBQI (Figure 16 presents one).

In terms of complexity, our triggering terms are much simpler than the proofs and directly highlight the root cause of the issues. The term loc(a,2) generated for Viper’s array maximum example from Section 5.1 is easier to understand than MBQI’s proof (which has 2,135 lines and over 700 reasoning steps) and than Vampire’s proof (with 348 lines and 101 inference steps). Other proofs have similar sizes. Therefore, determining the source of the inconsistency from such proofs requires expert knowledge of the tool-specific proof format and significant manual effort.

Most deductive verifiers [10, 13, 16, 22, 26, 32, 50] employ E-matching for discharging their proof obligations because E-matching is the most efficient SMT algorithm for program verification [28] (the vast majority of the SMT-COMP benchmarks we initially collected were also directly refuted by E-matching). It is thus important to help the developers use the algorithm of their choice and return sound results even if they rely on patterns for soundness (as in Figure 3).

As our algorithm accepts as input an SMT formula, it can also produce triggering terms required only at the SMT level, but which cannot be encoded into the input language of the verifier (e.g., Boogie), since they are rejected by the type system. However, such triggering terms can be filtered out, as lifting them to the input language is mostly straightforward (we performed this step manually in our experiments, to identify the cases of soundness modulo patterns; automating this process is a possible future extension). However, this is not the case for refutation proofs, whose back translation to the source language is an open research problem. To enable the developers debug the axiomatizations or fix the incompleteness more efficiently, our tool can also generate multiple triggering terms (as explained in Section 4.3). It can thus reveal multiple triggering issues for the same input formula, information which cannot be directly obtained from unsatisfiability proofs.

5.4 Threats to Validity

We identified the following two threats to the validity of our experiments:

Non-determinism. The SMT solvers use randomized algorithms, which can cause non-determi- nism. To mitigate this problem, we fixed all the available random seeds and used the same seeds in all the phases of our evaluation (i.e., for inferring the patterns, pre-filtering via E-matching, running our tool and MBQI).

Benchmarks selection. We relied on Z3’s E-matching algorithm to select examples with incompleteness in instantiating quantifiers. An implementation of E-matching from another solver could have led to different files. To avoid biases, we used Z3 in all the experiments.