From [[http://cpsquare.org CPsquare]], the community of practice on communities of practice.

Communities of Practice Bibliography books and articles on theory, practice, examples, and so on

Project Editors Category

Nature doesn't measure, nature recognises patterns .. -- Peter Senge

Contents 1 Evaluating Community Cultivation Efforts 2 Assessing impact of communities in organizational settings 3 A Range of relevant & adjacent Methodologies 4 Measuring community elements and strategies

Evaluating Community Cultivation Efforts

There is a fuzzy boundary between evaluating communities and evaluating community cultivation efforts.

• Etienne Wenger, Beverly Trayner, and Maarten de Laat, Promoting and assessing value creation in communities and networks: a conceptual framework Rapport 18, 978-90-358-1808-8, Open Universiteit rdmc.ou.nl. 2011. http://www.bevtrayner.com/base/2011/05/monitoring-the-value-of-communities-and-networks/

• Part of (http://www.cdc.gov/phcommunities/resourcekit/resources.html) This template helps you start-off your evaluation process by aligning it with CDC's standard evaluation for CoPs. The template contains sample evaluation objectives and sample questions. Each CoP can modify the template by modifying or adding additional questions that are specific to its goals and objects. http://www.cdc.gov/phcommunities/docs/Evaluate_Core_Template.doc

Assessing impact of communities in organizational settings

• Richard McDermott. Measuring the impact of communities of practice http://www.kunnskapsnettverk.no/C14/C10/CoP/Document%20Library/Measuring%20the%20Impact%20of%20Communities.pdf Short but concise article with a framework of where to expect the impact of communities of practice and how to monetize this.

• Garton, L., Haythornthwaite, C. and Wellman, B. (1997), Studying Online Social Networks. Journal of Computer-Mediated Communication, 3: 0. doi: 10.1111/j.1083-6101.1997.tb00062.x

• Evaluating Training Programs 3rd edition by James and Donald Kirkpatrick 2006. used by Wenger, Trayner and de Laat as a poitn of departure

• Turpin, Wagenaar and Hulsebosch Monitoring and evaluating knowledge management strategies. This paper describes monitoring and evaluation issues in the development sector. Some of the lessons may have a wider application.

• Turpin, Wagenaar and Hulsebosch Monitoring and evaluating knowledge management strategies. http://www.google.nl/url?sa=t&source=web&cd=4&ved=0CDUQFjAD&url=http%3A%2F%2Fwiki.ikmemergent.net%2Ffiles%2F901130_IKM_Background_Paper_Monitoring_and_evaluating_knowledge_management_strategies.pdf&rct=j&q=IKM%20monitoring%20and%20evaluation%20hulsebosch&ei=2V_fTdWSK4fsObSq8f8J&usg=AFQjCNHSQ0cchhCabAbMYzYZ1AdD7ZvjTQ&sig2=Y4hIc3QH3RYDE73VY-jkWw&cad=rja. This paper describes monitoring and evaluation issues in the development sector. Some of the lessons may have a wider application.

• Development of Communities of Practice to Facilitate Quality Improvement Initiatives in Surgical Oncology Michael Fung-Kee-Fung, Elena Goubanova, Karen Sequeira, Arifa Abdulla, Rose Cook, Claire Crossley, Bernard Langer, Andrew J. Smith2, Hartley Stern1, http://journals.lww.com/qmhcjournal/pages/articleviewer.aspx?year=2008&issue=04000&article=00010&type=abstract Quality Management in Health Care: April/June 2008 - Volume 17 - Issue 2 - p 174-185

A Range of relevant & adjacent Methodologies

There are several methodologies to measure the impact of communities of practice, including:

• Ethnographic studies: Assessment as practice: notes on measures, tests and targets by Brigitte Jordan and Peter Putz. Human Organisation http://www.lifescapes.org/Papers/031030%20assessment%20to%20HO.htm
• Social Return on Investment (see http://www.proveandimprove.org/new/tools/sroi.php for instance)
• Positive Deviance (see http://www.positivedeviance.org/ or google it)
• Most Significant Change ([on MSC])
• Learning histories (http://www.solonline.org/res/wp/18004.html)
• Assessment as practice: notes on measures, tests and targets by Brigitte Jordan and Peter Putz. Human Organisation (forthcoming)
  http://www.lifescapes.org/Papers/031030%20assessment%20to%20HO.htm
  
• SQUIRM-CP http://www.robotegg.com/data/CommEVAL21-60.html
  
• Department of the Navy Building Cops:
  http://openacademy.mindef.gov.sg/openacademy/central/html%20folder/km/bcp/buildcop_Measuring.htm
  
• NeLH http://www.nelh.nhs.uk/knowledge_management km2/toolkit.asp
  
• Efios measuring value in CoP http://www.efios.com/pdf/Value_efios.pdf
  
  
• De Jong and Kessels. Human Resource Development for Social Capital: an intricate process of knowing http://www.jyu.fi/en/congress/soca07/abstracts/abtrack4/t416/ introduces the idea of two approaches to ROI, focusing either on economic output or on knowledge productivity/innovation.
• Creswell JW, Research design: qualitative, quantitative and mixed methods approaches. 2009. Sage Publications Inc.

Measuring community elements and strategies

• International Network for Social Network Analysis http://www.insna.org/sna/what.html
• Cross, R., T. Laseter, A. Parker and G. Velasquez. (2004) Assessing and Improving Communities of Practice with Organizational Network Analysis. In The Network Roundtable at the University of Virginia. Virginia: University of Virginia, Stanford University, Halliburton. Available at http://www.robcross.org/pdf/roundtable/formalizing_communities_of_practice.pdf

• Marc Coenders and John Smith developed a CoP barometer http://www.learningalliances.net/download/LPPBarometer.pdf

• Society of organisational learning. Assessing to learn, Learning to assess, http://www.solonline.org/repository/download/ProgRep98.html?item_id=443229 Very interesting report which hightlights major dilemmas and offers a way to distinguish various types of assessments. 

• http://headrush.typepad.com/creating_passionate_users/2005/09/subvert_from_wi.html Guide to measure online communities

• Miguel Cornejo, Measuring activities and usefulness of communities http://www.knowledgeboard.com/item/2674 with the Macarium set of parameter to measure online community

• Roy McDermott?

In contrast to traditional educational assessment, which focuses on the individual, assessment of communities of practice is an assessment of the work of a group, or more accurately a community. The notion of individual measurement comes from psychology and the notion of individual educational measurement from educational psychology. It is useful to ignore these fields and instead start with a very simple model of teaching and learning so we might see where assessment fits in the teaching and learning process. The model is essentially a circular one, where we start with some ideas about a curriculum (what is to be learned), which serves as the basis for planning teaching (how we will organize to facilitate learning), which if the job has been done competently creates an environment where learning occurs. The model is a cycle which feeds back on itself: each stage informs the next and the next iteration of the cycle. When this has all been planned with care, assessment easily follows. Assessment is often a problem in education because it has not been planned and thought about in advance or because it has not been thought about as part of this cycle. Curriculum or competence based assessment is typically assessment grounded and driven by the curriculum and teaching stages.Standardized testing on the other hand typically has an acquaintance with the curriculum, but is not directly grounded and driven by the curriculum. It typically ignores the teaching phase: thus the common occurrence of people doing poorly on standardized tests, which is taken as a reflection on the individuals, but is in reality because the material in the test was not taught or more fundamentally was simply not a part of the curriculum.

Curriculum->Teaching->Learning->Assessment->Curriculum->Teaching->....

Perhaps if we are to transfer this model of educational assessment to the world of Communities of Practice, we can come up with a similarly model of a cycle which allows each stage to drive the next and subsequent rounds of the cycle. If we define Community of Practice as the accumulation of existing learning about an area of practice, held by a community of practice, we might define Leadership as a future vision of what the community could grow to become. Note that this is not an individual but rather a group or community based notion of leadership. Providing this future vision is subscribed to by the community as a whole, then assessment simply becomes the act of determining whether or not this has been achieved or an act of marking progress towards this goal. The complexity is less in the assessment than it is in the definition of a rich future vision of the community, which has been defined in terms which are amendable to assessment.

Community of Practice->Leadership->Assessment->Community of Practice-Leadership...

Measurement is a secondary issue and there is always a tendency to gravitate towards quantitative measurement. This tendency should be recognized and it can thus be moderated in discussions about assessment. There are a rich range of qualitative assessments in the literature on educational assessment and when assessment is thought about broadly, a mix of qualitative and quantitative assessments can be chosen. The key question about a measurement to ask is: does this measurement give us information which will help guide us in what to do next i.e. modification of existing goals for the community or the development of subsequent goals for the community. Beyond this, it is useful to recognize that several measurements of something are often needed in education (different types of measurements and measurement at several time periods or stages); they each act as checks and validation on each other. Finally, recognizing that the best measure of something is the thing itself e.g. if I want to measure how fast someone can run 100 meters, I might measure many of their physical attributes. Ultimately the best measure is how fast they run 100 meters, timed.