Abstract
Awareness has been a valuable concept in collaborative systems since its formation, an essential part of groupware. Awareness research followed the evolution of the whole field over the last decades. We can see the progress in a mutual understanding of awareness and developing concepts and technology of awareness support. An efficient awareness mechanism ensures a better understanding and, consequently, a better projection of future actions; in contrast, the lack of these mechanisms undermines comprehension and prevents participants from projecting their work accordingly. Few works present methods or processes that assist in providing awareness in groupware systems; most common strategies focus on the design/development stages or are ad-hoc evaluation models. Furthermore, there are no standardized tests for awareness assessment; thus, measures must be established to assess awareness and identify the criteria for achieving awareness indicators. This work establishes an assessment model for collaborative interfaces by analyzing the awareness mechanisms provided from the participant’s viewpoint. In this model, we consider the participant’s skill in understanding the awareness and the difficulty involved, providing advances toward designing, developing, and evaluating groupware systems. The proposed assessment model allows us to measure the awareness support provided considering the collaboration, workspace, and contextual awareness perspectives. Assuming a plural collaborative environment, where different participants with different skills, knowledge, and wisdom meet and interact, this model seeks to build a more faithful representation of these existing profiles across a broad spectrum of individual abilities.
Similar content being viewed by others
Availability of data and materials
All the steps in this research, including the planning, extraction, analysis, and synthesis, were documented, and the dataset is available at [60].
References
Agogo, D., Hess, T.J.: “How does tech make you feel?’’ A review and examination of negative affective responses to technology use. Eur. J. Inf. Syst. 27(5), 570–599 (2018). https://doi.org/10.1080/0960085X.2018.1435230
Altenburger, T., Guerriero, A., Vagner, A., et al.: Groupware requirements modelling for adaptive user interface design. In: eWork and eBusiness in Architecture, Engineering and Construction—Proceedings of the European Conference on Product and Process Modelling 2012, ECPPM 2012, vol. 1, no. 1, pp. 825–832 (2012). https://doi.org/10.1201/b12516-131
Antunes, P., Sapateiro, C., Pino, J.A., et al.: Awareness checklist: reviewing the quality of awareness support in collaborative applications. Collab. Technol. 1(1), 202–217 (2010). https://doi.org/10.1007/978-3-642-15714-1_16
Antunes, P., Herskovic, V., Ochoa, S.F., et al.: Structuring dimensions for collaborative systems evaluation. ACM Comput. Surv. 44(2), 1–28 (2012)
Antunes, P., Herskovic, V., Ochoa, S.F., et al.: Reviewing the quality of awareness support in collaborative applications. J. Syst. Softw. 89(1), 146–169 (2014). https://doi.org/10.1016/j.jss.2013.11.1078
Bailey, K.D.: Typologies and taxonomies: An introduction to classification techniques, vol. 102. Sage, London (1994)
Baker, F.B.: The Basics of Item Response Theory. ERIC, Washington (2001)
Baker, F.B., Kim, S.H.: Item Response Theory: Parameter Estimation Techniques. CRC Press, Boca Raton (2004)
Baker, F.B., Kim, S.H.: The Basics of Item Response Theory Using R. Springer, New York (2017). https://doi.org/10.1007/978-3-319-54205-8
Barbosa, S., Silva, B.: Interação humano-computador. Elsevier, Brasil (2010)
Basili, V.: Software modeling and measurement: The goal/question/metric paradigm. Technical Report (1992)
Beecham, S., Hall, T., Britton, C., et al.: Using an expert panel to validate a requirements process improvement model. J. Syst. Softw. 76(3), 251–275 (2005). https://doi.org/10.1016/j.jss.2004.06.004
Belkadi, F., Bonjour, E., Camargo, M., et al.: A situation model to support awareness in collaborative design. Int. J. Hum. Comput Stud. 71(1), 110–129 (2013). https://doi.org/10.1016/j.ijhcs.2012.03.002
Bodemer, D., Dehler, J.: Group awareness in CSCL environments. Comput. Hum. Behav. 27(3), 1043–1045 (2011). https://doi.org/10.1016/j.chb.2010.07.014. (group Awareness in CSCL Environments)
Bødker, S., Christiansen, E.: Computer support for social awareness in flexible work. Comput. Support. Coop. Work 15(1), 1–28 (2006). https://doi.org/10.1007/s10606-005-9011-y
Brezillon, P., Borges, M., Pino, J., et al.: Context-based awareness in group work. In: Proceeding of the 17th International FLAIRS Conference. AAAI Press, Miami (2004)
Brown, T.A.: Confirmatory Factor Analysis for Applied Research. Guilford Publications, New York (2015)
Chalmers, R.P.: mirt: a multidimensional item response theory package for the R environment. J. Stat. Softw. (2012). https://doi.org/10.18637/jss.v048.i06
Collazos, C.A., Gutierrez, F.L., Gallardo, J., et al.: Descriptive theory of awareness for groupware development. J. Ambient. Intell. Humaniz. Comput. 10(12), 4789–4818 (2019)
Cronbach, L.J.: Coefficient alpha and the internal structure of tests. Psychometrika 16(3), 297–334 (1951)
Cruz, A., Correia, A., Paredes, H. et al.: Towards an overarching classification model of cscw and groupware: a socio-technical perspective. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Raesfeld, Germany, pp. 41–56 (2012). https://doi.org/10.1007/978-3-642-33284-5_4
de Andrade, D.F., Tavares, H.R., da Cunha, V.R.: Teoria da resposta ao item: conceitos e aplicações. ABE, Sao Paulo (2000)
De Araújo, L.P., Mantau, M.J., Citadin, J.R., et al.: Heuristic evaluation for mobile groupware: evaluating two audience response systems. J. Appl. Comput. Res. 3(2), 64–77 (2014)
De Souza, L.G., Barbosa, S.D.J.: Evaluating the molicc notation using the cognitive dimensions of notations framework. In: Proceedings of the 14th Brazilian Symposium on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, IHC ’15, pp. 1–10 (2015). https://doi.org/10.1145/3148456.3148474
DeVellis, R.F.: Scale Development: Theory and Applications, Applied Social Research Methods, vol. 26, 4th edn. Sage Publications Inc, London (2016)
Do Espirito Santo, D.A., De Oliveira Rodrigues, B.R., Hadad Zaidan, F., et al.: Quality assessment of awareness support in agile collaborative tools. In: Proceedings—2018 44th Latin American Computing Conference, CLEI 2018, Sao Paulo, Brazil, pp. 21–30 (2018). https://doi.org/10.1109/clei.2018.00013
Dourish, P., Bellotti, V.: Awareness and coordination in shared workspaces. In: Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work. Association for Computing Machinery, New York, NY, USA, CSCW ’92, pp. 107–114 (1992). https://doi.org/10.1145/143457.143468
Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Massachussetts Institute of Techonology, Ohio, USA (2001). https://doi.org/10.7551/mitpress/7221.001.0001
El Helou, S., Tzagarakis, M., Gillet, D., et al.: Participatory design for awareness features: Enhancing interaction in communities of practice. In: Proceedings of the 6th International Conference on Networked Learning, vol 1, no. 1, pp 23–530 (2008)
Ellis, C., Gibbs, S., Rein, G.: Groupware some issues and experiences. Commun. ACM 34(1), 38–58 (1991). https://doi.org/10.1145/99977.99987
Engelmann, T., Dehler, J., Bodemer, D., et al.: Knowledge awareness in CSCL: a psychological perspective. Comput. Hum. Behav. 25(4), 949–960 (2009). https://doi.org/10.1016/j.chb.2009.04.004. (including the Special Issue: The Use of Support Devices in Electronic Learning Environments)
for Standardization IO.: Systems and software engineering—systems and software quality requirements and evaluation (square)—evaluation process. Standard, International Organization for Standardization, Geneva, CH (2011)
Fornell, C., Larcker, D.F.: Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 18(1), 39–50 (1981)
Gallardo, J., Molina, A.I., Bravo, C., et al.: An ontological conceptualization approach for awareness in domain-independent collaborative modeling systems: application to a model-driven development method. In: Expert Systems with Applications, pp. 1099–1118 (2011). https://doi.org/10.1016/j.eswa.2010.05.005
Gallardo, J., Bravo, C., Molina, A.I.: A framework for the descriptive specification of awareness support in multimodal user interfaces for collaborative activities. J. Multimodal User Interfaces 12(2), 145–159 (2018)
Gerosa, M., Fuks, H., Lucena, C.: Suporte a percepção em ambientes digitais de aprendizagem. Revista Brasileira de Informá1tica na Educação. Soc. Bras. Comput. 11, 75–85 (2003)
Gold, R., Mascolo, C.: Use of context-awareness in mobile peer-to-peer networks. In: Proceedings Eighth IEEE Workshop on Future Trends of Distributed Computing Systems. FTDCS 2001, pp. 142–147 (2001)
Greenberg, S.: Collaborative interfaces for the web. Hum. Factors Web Dev. 18, 241–254 (1997)
Gross, T.: Supporting effortless coordination: 25 years of awareness research. Comput. Support. Coop. Work 22(4–6), 425–474 (2013). https://doi.org/10.1007/s10606-013-9190-x
Gutwin, C., Barjawi, M., Pinelle, D.: The emergence of high-speed interaction and coordination in a (formerly) turn-based groupware game. In: Proceedings of the 19th International Conference on Supporting Group Work. Association for Computing Machinery, New York, NY, USA, GROUP ’16, pp. 277–286, (2016). https://doi.org/10.1145/2957276.2957315
Gutwin, C., Bateman, S., Arora, G. et al.: Looking away and catching up: dealing with brief attentional disconnection in synchronous groupware. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. Association for Computing Machinery, New York, NY, USA, CSCW ’17, pp. 2221–2235 (2017). https://doi.org/10.1145/2998181.2998226
Gutwin, C., Stark, G., Greenberg, S.: Support for workspace awareness in educational groupware. In: The First International Conference on Computer Support for Collaborative Learning. L. Erlbaum Associates Inc., USA, CSCL ’95, pp. 147–156 (1995). https://doi.org/10.3115/222020.222126
Gutwin, C., Greenberg, S.: A descriptive framework of workspace awareness for real-time groupware. Comput. Support. Coop. Work 11(3–4), 411–446 (2002). https://doi.org/10.1023/a:1021271517844
Haddaway, N.R., Page, M.J., Pritchard, C.C., et al.: Prisma 2020: an r package and shiny app for producing Prisma 2020-compliant flow diagrams, with interactivity for optimised digital transparency and open synthesis. Campbell Syst. Rev. 18(2), e1230 (2022). https://doi.org/10.1002/cl2.1230
Hair, J.F., Black, W.C., Babin, B.J. et al.: Análise multivariada de dados. Bookman editora (2009)
Hakkila, J., Mantyjarvi, J.: Developing design guidelines for context-aware mobile applications. In: Proceedings of the 3rd International Conference on Mobile technology, Applications and Systems. ACM, New York (2006)
Herskovic, V., Ochoa, S.F., Pino, J.A., et al.: The iceberg effect: behind the user interface of mobile collaborative systems. JUCS J. Univ. Comput. Sci. 17(2), 183–201 (2011)
Hinkelmann, K., Kempthorne, O.: Design and Analysis of Experiments: Advanced Experimental Design. Wiley-Interscience, New York (2005). https://doi.org/10.1002/0471709948
Hummel, D., Schacht, S., Maedche, A.: Determinants of multi-channel behavior: exploring avenues for future research in the services industry. In: ICIS (2016)
Izquierdo, I., Olea, J., Abad, F.J.: Exploratory factor analysis in validation studies: uses and recommendations. Psicothema 26(3), 395–400 (2014)
Janssen, J., Erkens, G., Kirschner, P.A.: Group awareness tools: it’s what you do with it that matters. Comput. Hum. Behav. 27(3), 1046–1058 (2011). https://doi.org/10.1016/j.chb.2010.06.002
Jonasson, N.: Designing for collaboration: interface guidelines for improving co-located collaboration on tabletop displays. In: Proceedings of Umea’s 14th Student Conference in Computing Science—USCCS 2010, Umea University, Sweeden, pp. 61–71 (2010)
Khalilijafarabad, A., Helfert, M., Ge, M.: Developing a data quality research taxonomy-an organizational perspective. In: ICIQ, pp. 176–186 (2016)
Kim, K.T., Javed, W., Williams, C., et al.: Hugin: a framework for awareness and coordination in mixed-presence collaborative information visualization. In: ACM International Conference on Interactive Tabletops and Surfaces, ITS, vol. 2010, pp. 231–240 (2010)
Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. University of Durham, Durham, UK, Technical Report (2007)
Kreijns, K., Kirschner, P.A.: The social affordances of computer-supported collaborative learning environments. In: 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193), FIE ’01, vol 1. IEEE Computer Society, pp. T1F–12 (2001). https://doi.org/10.1109/FIE.2001.963856
Ledesma, R.D., Valero-Mora, P.: Determining the number of factors to retain in EFA: an easy-to-use computer program for carrying out parallel analysis. Pract. Assess. Res. Eval. 12(1), 2 (2019)
Lopez, G., Guerrero, L.A.: Awareness supporting technologies used in collaborative systems: a systematic literature review. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. Association for Computing Machinery, New York, NY, USA, CSCW ’17, pp. 808–820 (2017). https://doi.org/10.1145/2998181.2998281
Mantau, M.J., Benitti, F.B.V.: Awareness support in collaborative system: reviewing last 10 years of CSCW research. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 564–569. IEEE (2022). https://doi.org/10.1109/cscwd54268.2022.9776091
Mantau, M.J., Benitti, F.B.V.: The awareness assessment model repository. (2023). https://doi.org/10.5281/zenodo.8298950
Mantau, M.J., Benitti, F.B.V.: Towards an awareness taxonomy. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 495–500. IEEE (2022). https://doi.org/10.1109/cscwd54268.2022.9776129
Mantau, M.J., Berkenbrock, C.D.M., Berkenbrock, G.R.: Visualization and filtering awareness information in mobile groupware: an action research approach. In: Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 563–568 (2014). https://doi.org/10.1109/cscwd.2014.6846906
Mantau, M.J., Berkenbrock, C.D.M., Berkenbrock, G.R.: Análise de requisitos de percepção em um groupware móvel síncrono. Rev. Sist. Inf. FSMA 1(19), 16–33 (2017)
Ml, B., Karahoca, D., Karahoca, A.: A measurement and structural model for usability evaluation of shared workspace groupware. Int. J. Hum. Comput. Interact. 34(1), 35–56 (2018)
Molina, A.I., Gallardo, J., Redondo, M.Á., et al.: Assessing the awareness mechanisms of a collaborative programming support system. Dyna 82(193), 212–222 (2015). https://doi.org/10.15446/dyna.v82n193.53497
Nickerson, R.C., Varshney, U., Muntermann, J.: A method for taxonomy development and its application in information systems. Eur. J. Inf. Syst. 22(3), 336–359 (2013). https://doi.org/10.1057/ejis.2012.26
Niemantsverdriet, K., Essen, H.V., Pakanen, M., et al.: Designing for awareness in interactions with shared systems: the DASS framework. ACM Trans. Comput. Hum. Interact. 26(6), 1–41 (2019). https://doi.org/10.1145/3338845
Nunnally, J.C., Bernstein, I.: Psychometric Theory, 3rd edn. McGraw-Hill Series in Psychology, McGraw Hill Higher Education, Maidenhead, England (1993)
Osborne, J.W.: Best practices in exploratory factor analysis. CreateSpace Independent Publishing Platform (2014)
Pasquali, L.: TRI-Teoria de resposta ao item: Teoria, procedimentos e aplicacoes. Editora Appris (2020)
Pasquali, L., Primi, R.: Fundamentos da teoria da resposta ao item: Tri. Aval. Psicol. Interam. J. Psychol. Assessment 2(2), 99–110 (2003)
Petersen, K., Vakkalanka, S., Kuzniarz, L.: Guidelines for conducting systematic mapping studies in software engineering: An update. Inf. Softw. Technol. 64, 1–18 (2015). https://doi.org/10.1016/j.infsof.2015.03.007
Petticrew, M., Roberts, H.: Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, Oxford (2006). https://doi.org/10.1002/9780470754887
Poulovassilis, A., Xhafa, F., O’Hagan, T.: Event-based awareness services for p2p groupware systems. Informatica (Netherlands) 26(1), 135–157 (2015). https://doi.org/10.15388/informatica.2015.42
Prouzeau, A., Bezerianos, A., Chapuis, O.: Awareness techniques to aid transitions between personal and shared workspaces in multi-display environments. In: ISS 2018—Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, Tokyo, Japan, pp. 291–304 (2018). https://doi.org/10.1145/3279778.3279780
Reckase, M.D.: Unifactor latent trait models applied to multifactor tests: results and implications. J. Educ. Stat. 4(3), 207–230 (1979). https://doi.org/10.2307/1164671
Reinhardt, W., Mletzko, C., Sloep, P.B., et al.: Understanding the meaning of awareness in research networks. In: CEUR Workshop Proceedings, pp. 13–29. Saarbrucken, Germany (2012)
Richardson, R.J.: Pesquisa social: Métodos e técnicas. Editora Atlas, S.A. (2017)
Rocker, C.: Universal access to awareness information: using smart artefacts to mediate awareness in distributed teams. Univ. Access Inf. Soc. 11(3), 259–271 (2012)
Rogers, Y., Sharp, H., Preece, J.: Design de interação. Bookman Editora (2013)
Samejima, F.: Estimation of latent ability using a response pattern of graded scores. Psychometrika monograph supplement (1969). https://doi.org/10.1007/bf03372160
Santos, V., Tedesco, P., Salgado, A.: Percepçãoo e contexto. In: Pimentel, M., Fuks, H. (eds.) Sistemas Colaborativos, vol. 1, 1st edn., pp. 157–172. Elsevier, São Paulo (2011) . (chap 10)
Schmidt, K., Randall, D.: Preface to the special issue on ‘reconsidering “awareness’’ in cscw’. Comput. Support. Coop. Work (CSCW) 25(4), 229–233 (2016). https://doi.org/10.1007/s10606-016-9257-6
Schmitt, N.: Uses and abuses of coefficient alpha. Psychol. Assess. 8(4), 350 (1996). https://doi.org/10.1037/1040-3590.8.4.350
Seebach, C., Beck, R., Pahlke, I.: Situation awareness through social collaboration platforms in distributed work environments. In: International Conference on Information Systems 2011, ICIS 2011, pp. 1–22. Shanghai, China (2011)
Singh, J.: Tackling measurement problems with item response theory: principles, characteristics, and assessment, with an illustrative example. J. Bus. Res. 57(2), 184–208 (2004)
Spencer, S.G.: The strength of multidimensional item response theory in exploring construct space that is multidimensional and correlated. Brigham Young University (2004)
Sternberg, R., Mio, J.: Cognitive Psychology, 5th edn. Cengage Learning, Belmount (2006)
Sternberg, R.J., Sternberg, K., Mio, J.: Cognitive Psychology. Wadsworth. Cengage Learning, Belmont (2012)
Strasser, A.: Delphi method variants in information systems research: taxonomy development and application. Electron. J. Bus. Res. Methods 15(2), 120–133 (2017)
Szopinski, D., Schoormann, T., Kundisch, D.: Because your taxonomy is worth it: towards a framework for taxonomy evaluation. In: ECIS (2019)
Talaei-Khoei, A., Vichitvanichphong, S., Solvoll, T., et al.: A methodology to develop awareness in computer supported collaborative work using policies. J. Comput. Syst. Sci. 80(7), 1323–1338 (2014). https://doi.org/10.1016/j.jcss.2014.03.003
Tang, T.Y., Winoto, P., Leung, H.: A usability study of an educational groupware system: supporting awareness for collaboration. J. Educ. Comput. Res. 50(3), 379–402 (2014). https://doi.org/10.2190/ec.50.3.e
Tavakol, M., Wetzel, A.: Factor analysis: a means for theory and instrument development in support of construct validity. Int. J. Med. Educ. 11, 245–247 (2020). https://doi.org/10.5116/ijme.5f96.0f4a
Tenenberg, J., Roth, W.M., Socha, D.: From i-awareness to we-awareness in CSCW. Comput. Support. Coop. Work (CSCW) 25(4), 235–278 (2016). https://doi.org/10.1007/s10606-014-9215-0
Thissen, D., Wainer, H.: Test Scoring. Routledge, London (2001)
Thompson, B.: Exploratory and confirmatory factor analysis: Understanding concepts and applications. In: Washington, DC, vol 10694, no. 3 (2004)
Tripathi, A.: A transaction model with multilevel consistency for shared data in distributed groupware systems. In: Proceedings—2016 IEEE 2nd International Conference on Collaboration and Internet Computing, IEEE CIC 2016, Pittsburgh, PA, United states, pp. 236–245. https://doi.org/10.1109/cic.2016.041(2016)
Trochim, W.M., Donnelly, J.P.: Research Methods Knowledge Base, vol. 2. Atomic Dog Publishing Macmillan Publishing Company, New York (2001)
Urbina, S.: Fundamentos da testagem psicológica. Artmed Editora (2009)
Usman, M., Britto, R., Borstler, J., et al.: Taxonomies in software engineering: a systematic mapping study and a revised taxonomy development method. Inf. Softw. Technol. 85, 43–59 (2017). https://doi.org/10.1016/j.infsof.2017.01.006
Van Solingen, R., Berghout, E.W.: The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development. McGraw-Hill, London (1999)
Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. Association for Computing Machinery, New York, NY, USA, EASE ’14, pp. 1–10 (2014). https://doi.org/10.1145/2601248.2601268
Wohlin, C., Runeson, P., Höst, M., et al.: Experimentation in Software Engineering. Springer, New York (2012). https://doi.org/10.1007/978-3-642-29044-2
Yang, C.L., Yuan, C.W.T., Wang, T.Y., et al.: Knowing that you know what I know helps? Understanding the effects of knowledge transparency in online knowledge transfer. Proc. ACM Hum. Comput. Interact. 2(CSCW), 1–21 (2018). https://doi.org/10.1145/3274458
Yin, R.K.: Case Study Research: Design and Methods, vol. 5. Sage, Thousand Oaks (2009)
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that the research was conducted without any commercial or financial relationships that could be construed as a potential conflict of interest.
Authors’ contributions
All authors contributed to the study’s conception and design, including the material preparation, data collection, and analysis. The manuscript was written by MJM and revised by FBVB. All authors contributed to the manuscript revision and read and approved the submitted version.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Mantau, M.J., Benitti, F.B.V. The awareness assessment model: measuring awareness and collaboration support over participant’s perspective. Univ Access Inf Soc (2024). https://doi.org/10.1007/s10209-024-01110-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s10209-024-01110-5