1 Introduction

Crowdsourcing projects such as OpenStreetMap (OSM), a freely editable global map, represent an invaluable source of geographic information and a platform for citizen engagement (Cochrane et al. 2017; Lauriault and Mooney 2014). Since its inception in 2004, OSM has since grown to become one of the largest repositories of volunteered geographic information (VGI). It has played a pivotal role in shedding light on previously unmapped communities (Mahabir et al. 2018, 2016), and its geospatial data has been relied upon to provide crucial support for humanitarian aid and crisis response efforts (Soden and Palen 2014). Currently, the OSM platform has more than 10 million registered users globally, with the number of users continuing to grow (Stats). As a commons-based peer production effort, OSM is developed through the collaboration of many individuals who work cooperatively to produce data. However, compared to more formal systems, such as commercial and government organizations, these activities are typically less structured (Benkler and Nissenbaum 2006). The process is also decentralized, with participants largely acting independently, resulting in generated content that reflects the various individual skills, motivations, and experiences of volunteers (Sehra et al. 2013). Correspondingly, the reliability of the data produced through such means has been an important area of concern (Calazans et al. 2017; Goodchild and Li 2012).

In terms of assessing the quality of VGI, Linus’ Law, a mantra within the software development community, has been frequently considered for open-source projects generally (Goodchild and Li 2012). Linus’ Law asserts that given enough eyes on an open-source project, almost every issue can be converged upon and fixed (Raymond 1999). Applied to open-source mapping projects, this would suggest that as the number of contributors increases, there is a resulting increase in the quality of the mapped content as well (Haklay et al. 2010). Conversely, the longer a feature remains unchanged, the higher the likelihood that it corresponds to reality (Muttaqien et al. 2018).

However, some researchers dispose of this law as a fallacy, citing a lack of evidence and that data quality does not scale linearly with the number of users (Glass 2003; Haklay et al. 2010). Others, such as Wang and Carroll (2011), further state that users’ experiences must also be considered when commenting on the quality of such production activities. As the concept underlying Linus’s Law has typically been relied upon to help explain the efforts of VGI contributors, such discrepancies within the open-source community warrants further investigation.

To assess the validity of Linus’ Law as applied to open-source mapping projects and to better understand the underlying mechanisms driving emergent patterns of data creation, we employ an abstract agent-based model (ABM) to simulate the co-production of spatial data. Collaborative knowledge production is a complex process, and an ABM is apt for simulating the heterogeneous interaction between populations required to produce such dynamic spatial content (Crooks and Heppenstall 2012; Whitenack and Mahabir 2022). This paper investigates various mechanisms, such as the level and variability of knowledge among a community of mappers, as well as the impact of editing prioritization on data production and quality.

2 Related works

Data quality refers to how well-suited a dataset is to serve its specific purpose, be it for operations, decision making, or planning. In the context of VGI projects, numerous data quality measures have been extensively studied. These encompass evaluations of spatial coverage (Mahabir et al. 2017), positional (Haklay 2010), attribute (Borkowska and Pokonieczny 2022), temporal, logical, thematic accuracy (Girres and Touya 2010), and various other dimensions of quality. However, while informative, the results of such measurements depend largely on high-quality reference datasets, which are not always accessible, especially in the context of developing countries (Mahabir et al. 2017). Furthermore, results can vary from one location to another due largely to the heterogeneous nature of VGI (Calazans et al. 2017). The growing recognition of these factors has prompted the need for incorporating intrinsic data quality measures in the assessment of VGI (Barron et al. 2014). However, intrinsic measures such as user behavior, data trust, and data history (Goodchild and Li 2012; Kesler and Groot 2013; Zhou 2018), have received comparatively limited attention in the existing literature.

Linus’ Law, in particular, has been viewed as a possible measure for intrinsically evaluating VGI quality. Although a high number of contributors and feature versions (i.e., number of updates) can be a positive indicator of feature trustworthiness (Kesler and Groot 2013; Muttaqien et al. 2018), the complete applicability of Linus’ Law in the VGI context remains to be understood, and its implications may be misleading. Haklay et al. (2010), for example, empirically tested the validity of Linus’ Law in OSM and found little improvement in the positional accuracy of road networks mapped in England beyond 15 contributors. Other research also suggest that Linus’ Law does not apply well to VGI because errors in obscure features are naturally more likely to go unnoticed (Calazans et al. 2017; Goodchild and Li 2012). These findings underscore the need for continued research to evaluate the extent to which Linus’ Law is applicable to VGI quality assessment.

Additionally, few studies have delved further into the nature of the behavioral habits of contributors. Included is an examination of more in-direct participation measures, such as contributor proficiency and prioritization, which the current work addresses. To this end, more complex-centered approaches are valuable. These should consider individual users, their preferences and contributions, as well as their interactions with other users and their environment. Previous research has highlighted the advantages of using ABM to examine such complex systems (Crooks and Heppenstall 2012). Thus, the utility of ABM in the study of VGI is anticipated to yield valuable insights into the underlying processes and the multitude of factors influencing emergent production patterns.

Several studies have utilized ABM to examine behavioral patterns within various collaborative knowledge production communities. This includes work by Panchal (2009), who studied contribution patterns in commons-based production projects, and Blythe et al. (2019), that developed a multi-agent simulation of the evolution of contribution patterns in GitHub based on historical data. Further, Blythe et al. (2019) used agent-based simulations to understand co-production activities within Wikipedia and community reaction to vandalism (Xu et al. 2008). Specific to VGI, Arsanjani et al. (2015) used a cellular automata model to predict user contributions in OSM. That study emphasized the necessity of gaining a deeper understanding of the behavioral patterns exhibited by OSM mappers within a collaborative project. To the best of the authors’ knowledge, this research pioneers the use of an ABM to investigate behavioral characteristics underlying OSM user collaborations.

Our goal is to contribute to the VGI literature by (1) showing how VGI mapping activities, as a complex system, can be modeled and used to understand emergent production patterns, (2) to further expand on previous seminal work by Haklay et al. (2010), specifically to understand the nonlinear nature of such activities, and finally (3) how these factors impact the quality of VGI.

3 Model design

The purpose of this model is to explore how the number of contributors, along with their decisions, impact the overall data quality of geographic features in a local region. Applying Epstein and Axtell and Epstein (1994) classification of agent-based models, our model is intended to be operationalized at Level 1, which is to be in qualitative agreement with empirical macro-structures. Primarily, we reference the empirical work of Haklay et al. (2010) concerning Linus’ Law in VGI, along with prior work on data history and trust (Kesler and Groot 2013; Muttaqien et al. 2018) when evaluating model outputs.

The model’s design is based on an environment, or world, of mappable cells that a set of agents can choose to interact with and edit over multiple time steps. The model is relatively simple, with an abstracted environment and agents. We implemented a pattern-oriented modeling approach to optimize the model in terms of complexity and to help ensure a balance between simplicity and explainability (Crooks and Heppenstall 2012). This entailed testing the model’s sensitivity to parameter uncertainties, where in Sect. 3.4, we identify and retain the most relevant patterns in the system.

3.1 Model implementation and output

The ABM was built using NetLogo (Wilensky 1999), a multi-agent programmable modeling environment that provides a framework to incorporate agent transitions and interactions, widgets for user interactivity, adjustable model parameters, and various options for visualizing model outputs. In our case, model outputs include a distribution of mapped cell errors, time series of the number of cells mapped, average cell error, and average cell versions. The model world - which is a grid of cells representing mappable geographic features - is also used to help visualize the dynamic distribution of spatial data created and its change in quality over time. Our experiments focus on the final values for the number of cells mapped, their average error, and average version number. The source code and related information to reproduce our results are provided in the Notes section.

3.2 Environment

The model environment is a grid composed of A×A cells (see Fig. 1), where each cell represents a single mappable geographic feature in a given space. Cells have two attributes: error and version number. These attributes are used to track the quality and quantity of agents’ interactions with each cell. The error represents the positional error in meters of a geographic feature. This metric, and its range, were chosen to correspond to the work of Haklay et al. (2010), where they define a positional error as the offset between OSM road segments and an authoritative data source. In this case, 0 m represents a perfect agreement, while larger values represent less accurate OSM data. This definition of error is arbitrary, but since our goal is to come to a qualitative agreement with empirical observations, the approach used to assess error in this study was found to be suitable. A cell’s error is visualized by a gradient of white (offset of 0 m) to red (offset of 40 m). A number in each cell represents its current version number, representing the number of times the cell has been updated by an agent. At initialization, all cells begin at version 0. Any cells with version 0 are greyed out, indicating that the cell has yet to be mapped.

Fig. 1
figure 1

Model layout and example of the final state of a run with 4 agents (blue and green agents are identical). (Color figure online)

Full size image

3.3 Agents

Throughout the simulation, agents react to a changing environment, where their agent-environment interactions lead to indirect agent-agent interactions. Agents represent the contributors of OSM content (i.e., mappers) who create and edit geographical features. They are initialized at random start locations within the gridded environment, and at each time step, they select a cell within their Moore neighborhood of 24 cells to move to and edit (see Fig. 2). The user specifies the number of agents and the frequency at which they move and edit features within a time step. Agents’ behavior is described in probabilistic terms to account for the distribution of different volunteers and their likelihood of choosing where and how to contribute to VGI. Blythe et al. (2019) and Dawson et al. (2011) used similar probabilistic approaches.

Fig. 2
figure 2

Flow diagram of agents’ decision-making within a time step

Full size image

If priority-based selection is toggled ON, agents move to new cells based on a weighted randomization of the attractiveness of all cells in their neighborhood. The attractiveness of a cell is computed using a power law of its error, \(0.01^{(1-err/40)}\), where unmapped cells are by default weighted at 1.0, given a maximum error of 40 m. The creation priority parameter allows the user to set unmapped cells’ attractiveness (0 to 1.0) in the weighted randomization where larger values result in higher attractiveness (i.e., agents are more likely to map new cells rather than edit existing cells). Overall, this selection prioritization emulates how geographic features’ absence or obvious positional inaccuracies may tend to attract more attention from contributors (Goodchild and Li 2012).

Once an agent moves to a new cell, they have the option to edit and update its error. In this model, an agent will only update the error and version of a cell if it is an improvement. The quality of an agent’s contribution is determined by their proficiency, a randomized gamma distribution characterized by the parameters - mean and variance - set by the user. A gamma distribution was chosen as it most closely approximated the distribution of positional error in OSM derived by Haklay et al. (2010).

3.4 Sensitivity analysis

Local sensitivity tests were performed to identify the most significant model elements based on their impact on results. This approach was also used to check the robustness of the main qualitative conclusions (Borgonovo et al. 2022). Specifically, we varied parameters in isolation and observed their impact on the relationship between the number of contributors and the final mean error of the world, paying special attention as well to how parameters influenced each other.

Starting with the smallest effect, an agent’s neighborhood size only impacted the final qualitative results presented when considering a very small Moore Neighborhood. Specifically, the typical Moore Neighborhood of 8 cells was found to increase the sensitivity of agents’ initial positions on the results, leading to high variability between runs. In this case, the limited size of the neighborhood hindered the agents’ mobility and diminished the opportunities for collaboration between agents. Considering this, a Moore Neighborhood consisting of 24 cells was used to strike a balance between providing agents sufficient mobility and minimizing the computational requirements necessary for running simulations.

Similarly, we ran the model for a range of contributors while independently varying the world size, the run time, and an agent’s frequency of edits (see Fig. 3). All three parameters were found to have a similar impact on the results. Note the similarity between the 40 × 40 and 25 steps results in Fig. 3, where, compared to the base case of 20 × 20 and 100 steps, the edits per feature are scaled by a quarter. The model’s main assumptions can partly explain these results: (1) error is improved with increasing version number, and (2) regions with more contributors also have more edits. Considering these assumptions, all three parameters impact the potential number of edits per feature, reflected via the version number of features. These parameters effectively scale the results without altering the qualitative conclusions derived from the model.

Fig. 3
figure 3

Impact of world size and run time on error given the number of contributors

Full size image

4 Experiments and results

The developed ABM was utilized to conduct two experiments, aiming to evaluate the applicability of Linus’ Law in VGI projects and delve into potential factors underlying spatial data collection that could impact the observed outcomes. The experiments assess the influence of contributors’ behavior and knowledge, specifically regarding prioritization and proficiency, on the relationship between the number of contributors and data quality.

The run time, frequency of edits, and the number of cells are kept constant in the experiments. This decision was based on the results of the prior sensitivity tests, which showed that while these parameters affect the rate of convergence of error versus the number of contributors, they do not impact the final qualitative results. Their values were determined to best fit with the positional error versus the number of contributors results from Haklay et al. (2010) to create a relatable reference point for the experiments. Our study area is a 20 × 20 gridded environment, representative of a local community. The simulations were conducted for a run time of 100 time steps and agents were set to act at a frequency of 5. The individual values of these static parameters are not as important as the combination of them to develop a relatable reference point. The final output values were recorded as the average of 10 runs, with the number of initialized casual agents ranging from 1 to 33. Error bars in the results represent the standard deviation of 10 runs.

4.1 Experiment one: influence of prioritization

In the first experiment, we observed the impact of agent prioritization on the relationship between error and the number of contributors by varying the value of the creation priority parameter and toggling priority-based selection. Agents’ proficiency distributions were set to a mean of 16 m and a variance of 50 m to best fit the distribution of positional error in OSM derived by Haklay et al. (2010). When testing the impact of priority-based selection, the creation priority parameter was set to 1% (or 0.01 in the NetLogo interface). Figure 4 shows the experiment results.

Fig. 4
figure 4

Impact of prioritization on the relationship between error and the number of contributors. Change in the weight assigned to creating features (left) and whether agents prioritize their efforts at all (right)

Full size image

With priority-based selection and a creation priority of 1%, our results match the trend from Haklay et al. (2010), \(R^2 = 0.93\), although, as we discuss later, our model does not capture the same variability. The general agreement between our model and empirical data suggests that at least the nonlinearity between spatial data quality and the number of contributors can be in part explained by the simple mechanisms driving the ABM. In particular, as the positioning of geographic features becomes more accurate, or at least meets the standard accuracy expected for a specific application, it is less likely to be noticed by contributors and further positionally corrected. Additional tests (results not presented) using a uniform distribution to describe the proficiency of agents also result in nonlinearity between spatial data quality and the number of contributors.

Priority-based selection has little impact on the results except when incorporating prioritization for creating cells. Varying creation priority impacts the trend of the results, where a lower creation priority results in greater linearity. With a higher creation priority, agents will focus on mapping the entire environment before returning to edit and improve existing data. Therefore, the extent to which agents prioritize data creation holds greater significance in environments with fewer contributors. Essentially, when provided with a limited number of actions, agents make a trade-off decision between either improving data expansiveness or improving data quality. We suggest that it is necessary to empirically examine the impact of the number of contributors on the completeness of VGI and investigate potential negative correlations between completeness and positional accuracy. Further, our model shows a very close association between the average version and the average error. Although feature version is a positive indicator of trustful VGI (Kesler and Groot 2013), in reality, the relationship between object version and quality may not be one-to-one as presented here.

Our model does not capture the same variability as seen in empirical results. Based on sensitivity tests, the model’s variability was slightly impacted by run time and frequency. Additionally, it is possible that our model does not account for the same level of variability due in part to differences in sampling. In the case of Haklay et al. (2010), model variability may be largely influenced by smaller sample sizes of regions with larger number of contributors. Another possibility is the existence of a randomized element in knowledge production within VGI, which our model may not capture.

4.2 Experiment two: influence of proficiency

The second experiment aims to investigate the impact of contributors’ proficiency on data quality, considering a range of number of contributors. This was accomplished by independently varying the mean and variance parameters that describe the proficiency distribution. For this experiment, the creation priority parameter was set to 1%.

Fig. 5
figure 5

Impact of the mean (blue lines) and variance (orange lines) of agent proficiency on the relationship between error and the number of contributors. (Color figure online)

Full size image

The results from experiment two suggest that the number of contributors alone is insufficient for determining data quality within a region. Instead, one must also consider other intrinsic measures, such as the experience of contributors, as previously suggested by Wang and Carroll (2011). Figure 5 shows that both improved mean and greater variance in proficiency result in a lower error. From a practical standpoint, an improved mean can be achieved by providing training to OSM contributors within organizations and volunteer communities on the appropriate technology and best practices (Senaratne et al. 2017). It can also be achieved by drawing more from local communities where local familiarity correlates with the quality of contribution (Muttaqien et al. 2018). Such suggestions are in line with the findings of De Leeuw et al. (2011), where authors discovered that when volunteers with local knowledge classified roads, they were over 92% accurate on average. Regarding the variance in proficiency, greater variance corresponds to greater diversity within a mapping community, where contributors may have specific knowledge of certain features in an area to occasionally provide higher-quality contributions. This aligns with existing literature, where a diverse community of reviewers in open-source software projects is paramount to Linus’ Law’s effectiveness in improving code quality (Wang et al. 2015).

5 Summary and future work

In this paper, an ABM was developed to understand how co-editing amongst volunteers impacts the overall quality of spatial data. Overall, our results show that OSM data production patterns of quality concerning the number of contributors can evolve from simple agent-environment interactions using an ABM. In the process, we corroborated previous research on the validity of Linus’ Law, which suggests that the relationship between the quality of VGI and the number of contributors is not linear (Glass 2003; Haklay et al. 2010).

The model enabled an in-depth exploration of the underlying processes that contribute to this nonlinearity. Within the bounds of the model, the results are consistent with the hypothesis that contributors to VGI are more likely not to notice obscure or subtle errors (Goodchild and Li 2012), unlike with bug catching in open-source software projects - thus limiting the effectiveness of Linus’ Law in VGI. We suggest further investigation with an empirical analysis of how completeness and version correlate with the positional accuracy of OSM objects to further aid in understanding the impact of the number of contributors on VGI quality. Furthermore, our findings demonstrate that while the number of contributors can serve as a positive indicator of data quality, it cannot substitute for greater proficiency and diversity among contributors.

Additionally, this study has identified several limitations that present opportunities for future research and exploration. First, while the ABM results were comparable to findings of prior work, many factors affect mapping quality, such as contributors’ socio-economic and demographic backgrounds (Antoniou and Skopeliti 2015). A more advanced model should take these into account. Second, OSM features such as buildings, roads, and green areas should be investigated instead of generic spatial features used in this work. We would also suggest that these features mirror existing land cover and land use for the specific study area. Third, similar to prior work (Mooney and Corcoran 2014; Neis and Zipf 2012), the different types of OSM mappers and their editing dispositions should be investigated. Finally, future work may also incorporate more complex collaboration constructs between agents such as competition (Girres and Touya 2010) and social networks (Anderson and Dragićević 2020).

Our study highlights a significant need for and demonstrates the immense potential of applying agent-based modeling to gain insights into the behavioral dynamics that drive collaboration and knowledge production in VGI projects like OSM. Especially given the complexity of collaborative knowledge production, we expect that advances in this regard can help researchers and practitioners to understand the dynamics of collaboration, identify factors that contribute to successful knowledge production, and to identify potential challenges and bottlenecks that may hinder effective collaboration in VGI crowdsourcing projects.

Moreover, we assert that ABM models can equip urban planners and decision-makers with valuable tools to leverage the collective power of the crowd in addressing and bridging existing data gaps in infrastructure. In our case, by understanding the specific number of contributors needed to achieve an acceptable level of mapping accuracy in various locations, urban planners can strategically target and incentivize fewer individuals to help with mapping efforts at those locations. This approach would prove particularly beneficial in developing countries, where accessing current and reliable data can be challenging (Mahabir et al. 2016).

Overall, by integrating agent-based modeling with the discovery of volunteer proficiency and diversity’s positive impact on data quality, future communities can refine their crowdsourcing strategies, design more effective training programs, and develop governance frameworks. Such opportunities will further enable communities to harness the power of VGI to assist with more informed decision-making and better sustainable development practices.