Modelling neural probabilistic computation using vector symbolic architectures

Abstract

Distributed vector representations are a key bridging point between connectionist and symbolic representations in cognition. It is unclear how uncertainty should be modelled in systems using such representations. In this paper we discuss how bundles of symbols in certain Vector Symbolic Architectures (VSAs) can be understood as defining an object that has a relationship to a probability distribution, and how statements in VSAs can be understood as being analogous to probabilistic statements. The aim of this paper is to show how (spiking) neural implementations of VSAs can be used to implement probabilistic operations that are useful in building cognitive models. We show how similarity operators between continuous values represented as Spatial Semantic Pointers (SSPs), an example of a technique known as fractional binding, induces a quasi-kernel function that can be used in density estimation. Further, we sketch novel designs for networks that compute entropy and mutual information of VSA-represented distributions and demonstrate their performance when implemented as networks of spiking neurons. We also discuss the relationship between our technique and quantum probability, another technique proposed for modelling uncertainty in cognition. While we restrict ourselves to operators proposed for Holographic Reduced Representations, and for representing real-valued data. We suggest that the methods presented in this paper should translate to any VSA where the dot product between fractionally bound symbols induces a valid kernel.

Appendix 1 Model complexity analysis

We presented an algebraic interpretation of VSA operations and the results for spiking neural implementations of these algorithms. Next, we present an analysis of the complexity of these networks. We frame in terms of the number of synaptic operations, which would be simple additions in a spiking neural network, or a multiply and accumulate operation in the case of implementation in a graphics processor or general-purpose CPUs. Because we implemented these networks using spiking rectified linear neural networks, we do not account for the complexity of neural dynamics in this analysis. To estimate the complexity, we require the quantities laid out in Table 1. The summary of the analyses is denoted in Big-O notation in Table 2.

Table 1 Quantities used in computing the complexity of the proposed neural networks

To produce probability estimates using single neurons we have an input of dimension d, and a neural population size of 1, meaning that estimating the probability of a single observation is d synaptic operations. To estimate a probability density using a population of neurons that represent $n_\text{sampling}$ sampled points for each input dimension, the number of synaptic operations is $d(n_\text{sampling})^{m}$.

Pre-rectification marginalization requires one linear mapping in SSP space, which is a $d\times d$ operation. If the SSP is represented as a population of neurons, then we require mapping from the neural population to the SSP latent space, which requires $n_\text{dim}$ synaptic operations for each SSP dimension, d. Hence, the pre-marginal rectification requires $d^{2} +d n_\text{dim}$ synaptic operations. Computation of the marginalizing matrix can be computed off-line for each dimension and is not included in the analysis.

Post-rectification marginalization requires first computing a sampled probability distribution, which is $d(n_\text{sampling})^{m}$, then there must follow the summation over the marginalized dimensions, $m_\text{marg}$. This requires, for each point in the unmarginalized dimensions, $(n_\text{sampling})^{m-m_\mathrm{marg}}$, computing $(n_\text{sampling})^{m_\mathrm{marg}}$ sums. This results in a synaptic complexity of $d(n_\text{sampling})^{m} + (n_\text{sampling})^{m}$.

Conditioning requires computing the binding operator from the HRR VSA, which is circular convolution. In this work we use the default Nengo implementation of binding between two d -dimensional vectors, a and b, which produces a new d-dimensional vector, $c = \textbf{a}\circledast \textbf{b}$. The circular convolution is implemented by a series of rotated dot products, defined:

$$\begin{aligned} c_i = \sum _{j=1}^{d}a_{i}b_{((i-j) \text{mod} d)} \quad i \in \{1,\ldots ,d\}. \end{aligned}$$

(9)

The multiplication of individual vector elements, $a_{i}b_{((i-j) \text{mod}\,\, d)}$, is computed using a product network (Gosmann 2015), which requires $3n_\text{prod}$ synaptic operations. Computing the entire circular convolution requires computing d products for all d elements of the vectors, resulting in a complexity of $d^{2}\times 3n_\text{prod}$ synaptic operations.

Computing entropy as described in this paper again requires first constructing a sampling of the distribution, which is $O(d (n_\text{sampling})^{m})$, followed by computing $-p\log p$ for every neuron in the distribution, which we implement in a single hidden layer neural network, which contains $n_\text{log}$ neurons. This requires $(n_\text{sampling})^{m}\times 2n_\text{log}$ synaptic operations. This is followed by a population of $n_\text{ent}$ neurons to represent the sum $\sum _{i} -p_{i}\log p_{i}$. Consequently, the entropy calculation requires $2n_\text{ent}(n_\text{sampling})^{m}$ synaptic operations. The function $p\log p$ can be difficult to compute, and requires substantial neural resources, so we assume in the worst case that $n_\text{ent} = n_\text{log}$. The total cost to compute the entropy of a distribution is $4n_\text{log}(n_\text{sampling})^{m} + O(d (n_\text{sampling})^{m})$.

To compute mutual information we must first sample the joint probability distribution, which can be over two vector-valued variables. If we have two variables, $X_1 \in \mathbb {R}^{m_1}$ and $X_2 \in \mathbb {R}^{m_2}$, we will assume that $m = max \{m_{1},m_{2}\}$. Then the initial distribution representation requires $d(n_\text{sampling})^{2m}$ synaptic operations.

The joint distribution is then marginalized twice, assuming post-rectification for increased accuracy. Exploiting the initial distribution representation permits a complexity of $2(n_\text{sampling})^{2m}$. We then must compute the entropy of the joint and two marginal distributions, which requires $4n_\text{log}(n_\text{sampling})^{2m}$ and $8n_\text{log}(n_\text{sampling})^{m}$, respectively. This results in a final complexity of $(d + 4n_\text{log} + 2)(n_\text{sampling})^{2\,m} + 8n_\text{log}(n_\text{sampling})^{m}.$

Finally, we consider the cost to update an SSP representation of a distribution. To update the distribution we take the neural population whose latent space is representing the distribution using SSPs and project it into the SSP space, this requires $dn_\text{dim}$ synaptic operations. Then we add it and the new observation together and store them in the neural population, this requires 2d multiplies ($O(< b^{2})$ for b-bit numbers, depending on the implementation) to compute the running average and $2dn_\text{dim}$ synaptic operations to update the population representing the distribution. We note that for more biologically plausible implementations the running average may be replaced by a low pass filter, which has constant multiplication terms that can be integrated into synaptic weights directly.

Table 2 A summary of the complexity of the proposed operations in terms of synaptic operations

The above analysis looks at the synaptic operations required for computing the probabilistic operations. This is a measure of resource requirement to construct networks, as well as the total volume of computation that must be executed. However, many of these operations can be parallelized, and on the right computing framework the time between an input being presented to a network to compute these operations can be improved significantly.