Notes from the 6/10/11 & 6/11/11 MacArthur Documenting Learning Meeting.

Summary of meeting discussions
Notes from 6/10/11
Notes from 6/11/11



Meeting Notes

Summary of meeting discussions

Jay Lemke

At our first meeting, held in San Francisco and hosted both at the Golden Gate Club in the Presidio and at the nearby Exploratorium, from Friday afternoon and dinner through Saturday lunch (June 10-11, 2011), we opened up discussion of our core topics very broadly for advice and input from invited participants.

More detailed notes and the references to the literature made during the meeting are provided separately by Robert Lecusay.

A key issue is that of the unit of assessment. I believe the consensus of the group was that this unit is a system over time, which includes individual learners, but also other participants, mediating tools and semiotic media, and local conditions directly relevant to and supportive of the learning activities. It needs to be extended to wider contexts that make the setting of learning possible institutionally, but with decreasing detail as relevance to the specifics of learning trajectories decreases.

This was later applied to the question of meta-data or backstories for video records of learning activities. A key question, answers to which may vary by type of setting or type of activity, is:

How much and what kinds of information are relevant to identifying valued learning and the specific aspects of activity that support it in the video?

A suggestion for estimating the role of background information is to have a group view the video first without background, and then again with partial background, and a third time with much more complete background.

It was suggested that in any records of learning activity over time, some valued learning may be more readily visible, to more observers and with less detailed analysis or less experienced professional vision, while other instances may be less readily visible.

It was also agreed that different kinds of learning may become visible when records of learning activities are studied over longer vs. shorter time periods.

A key proposal was to identify and survey research and assessment in and across a variety of types of settings, including:

Learning in the home through everyday activities or activities not specified by the requirements of some other educational institution (example: doing and discussing mathematics during home remodeling)

After-school programs, where activities are not directly meant to serve school-based academic functions (example: playing an educational computer game and making innovative use of it for fun, with ancillary learning)

Community center programs, where activities are negotiated between learners and providers, and which may have specific learning objectives but changing approaches to the goal (example: telementoring and use of computer simulation of electric circuits, together with an onsite coach familiar with the student, but not responsible for the content)

Museum-based programs, where visitors can choose to manipulate hands-on materials in the context of questions and explanations of phenomena observed/produced (example: young visitors connecting a battery to various electric devices to see the results of completing a circuit, with a coach, and showing the results to a parent; a group of young visitors extracting insects from a bag to feed to a pet as part of a longer term project, and one overcoming a reluctance to touch the insects)

Online communities and Forums, where participants ask and answer questions on a specific area of competence or expertise and evaluate one another’s answers or contributions, and where they may also engage in joint activity in a virtual space or mediated by tools and social interactions in that space (example: learning to build in Second Life; “theory-crafting” to identify technical characteristics of computer games by systematically playing many options within them; modding in World of Warcraft; raiding as joint play for a goal).

Team sports, both live action and “fantasy” or virtual-world mediated

Crafting communities, online and offline, such as Ravelry (knitting, etc.)

Cooking communities, online and offline.

Issues of Value and Significance

By what criteria do we decide that some learning is valuable?

It was recommended that the first criterion be that there is evidence of value for the participants, e.g. through the length of time they focus on a task or activity, their reluctance to leave or end the activity, displays of intense or positive affect during the activity, comments on the activity during and after, explicit elicited evaluations.

Beyond this there are additional criteria which may be applied, primarily: the judgment of expert educators or others, such as parents, regarding what is of value to the learners and/or to society; evidence of consequentiality of learning for other, conventional academic activities (e.g. increased interest, increased participation, more positive affect, more effective completion of tasks, ability to teach content and skills to others, ability to solve problems collaboratively, etc.)

Issues of Documentation

Video alone is often not sufficient documentation of learning activity because of the inferences that need to be made to identify valued learning. Ideally video should be supplemented by fieldnotes from participant observation or observant participation, interviews with participants, relevant histories of the setting and of the participants.

The unit of analysis for documentation should be activity in a setting over times long enough to show: origins of participation, evolution of the activity, learning in the activity, consequentiality of learning for some other activity.

An individual episode captured on video may be significant as part of such a longer trajectory of learning and development, and/or as an instance of a frequently repeated pattern of learning and activity across other cases in the same or similar settings.

Issues of Definition and Scope

“Informal learning” is at best a shorthand for a more complex combination of specific features which need to be identified. These features may in principle occur in both school and classroom-based learning and in other settings, but in different combination and to different degrees. Each setting and perhaps each kind of learning activity will tend to have a particular combination and degree of each feature.

Some of the features noted as relevant include:

Voluntary participation
Enjoyment of the learning activity for its own sake
Intense engagement with tasks
Flexibility in goals and re-purposing resources
Unpredictability of some significant learning outcomes
Improvisation and innovation within and concerning the activity
Commitment over time
Continuing voluntary participation despite setbacks
Relatively equitable power relations in negotiating goals and means

Various literatures may name activities or settings where these features are present, dominant, constitutive, or highly significant “interest-based learning” “free-choice learning” “nonformal learning” “ learning in passion communities” etc. as well as making distinctions among these based on role relationships or types of institutional goals and constraints.

The scope of our review may need to be limited in order to be useful, given the breadth and diversity of settings in which learning with some or all of these features occurs.

It was suggested that we could focus on learning in which digital media play a significant role.

It was also proposed that we initially survey a number of different types of settings, such as those listed above, try to identify key features of learning, and then determine a limited set of such types or domains on which to focus in order to be able to say something about common features/problems and unique features/problems of documentation and assessment of learning.

It was suggested that we seek out samples of video and other documentation from a relatively wide range of such settings as part of our analysis.

Invited participants in the meeting agreed to send us references to relevant literature with which they were familiar.

Future Meetings

We also briefly discussed ideas for the second and third meetings. One possibility for the second meeting is to focus on the domain of digital media for non-school learning. For the third meeting we might invite researchers familiar with each of a number of different setting types or domains and attempt to identify similarities and differences in effective approaches to documenting and assessing learning.

It was also agreed that at least by the third meeting we should have a draft of key issues drawn from a review of the literature to distribute before the meeting and for which we would invite specific comments, additions, criticisms, etc.

An offer was made to host the next meeting at Northwestern University.

Notes from 6/10/11 (top of page)


P1 – P11 stand for the 11 participants who attended the meeting

P10 Introduction

MacArthur has funded research on everyday uses of digital media by youth study (e.g. Ito et al. Hanging out book). The question policy makers and funders are now asking: What are youth learning as a result? The charge of this workshop is to explore what it means to document learning in these situations particularly in the context of research that has been conducted in this area over the past decade.

P10 Presentation

Slide: Setting - Continuum: Casual spaces -- Designed spaces -- School

Slide: How to approach it?: By setting, technology, or by method or technique

Key question: on whose terms is the assessing happening?

P7 provides overview of the workshop structure:
First meeting: big issues; Second meeting: technologies; Third Meeting: whatever was missed in the first two meetings.

What we want: Examples of documentation and assessment, relevant literature and contacts

P7 presentation

Review of [[file/view/MacDoc Springboard June2011.pdf|Springboard Document]] accessible here:

[[file/view/MacDoc Springboard June2011.pdf|]]

Examples and Discussions

What prior and in-progress work of your own, your students, and nearer and more distant colleagues would you recommend that:

  1. Provides examples of the kinds of challenges for documenting learning in informal and media-rich settings noted above?

  1. Offers partial solutions or promising approaches to meeting these challenges?

  1. Formulates potentially useful conceptual models or methodological approaches to meeting these challenges?

We would also welcome references to research that has met with largely insurmountable difficulties in documenting learning in such settings.

P7 Presentation (cont.): Documenting and Assessing Dynamic Learning

Slides available here:


P1 raises two policy points
  1. Not ranking of individuals but of program type.
  2. The question of out-of-school time as a form of support of the school agenda (which has direct implications for measurement).

P7: Can new technologies can be engaged through different methods to get at those school approaches?

Discussion of what different foundations are orienting towards - P9: Gates Foundation focuses on “deeper learning;” P3: design thinking ; P10: MacArthur seeking consistent outcomes that can be drawn from youth uses of media like facebook, etc.

P10 notes that Mimi’s team was assessing four areas:
  1. systems thinking
  2. collaboration
  3. time management
  4. social engagement

P1: common metric: are they going to college?

P7: This suggests another dimension - valued outcomes of the activities in and of themselves vs. supporting more standard academic outcomes

P9 and P7: It’s clear that we know how to measure in terms of grades, tests, but do we know how to establish a connection between school and informal learning? Establishing the connection is a research design problem vs. th e fact that interesting learning is happening in these informal settings, but it is difficult to document and analyze it in a way to make this informal-formal connection (the primary problem)

P1 recommends: Wai, J., Lubinski, D., Benbow, C. P., & Steiger, J. H. (2010). Accomplishment in science, technology, engineering, and mathematics (STEM) and its relation to STEM educational dose: A 25-year longitudinal study. Journal of Educational Psychology, 102, 860-871.

P7: Some methodologies: 1. shadowing people beyond the activity, across settings. 2. video records in the setting. 3. cognitive ethnographies – field notes from different points of view

P2: Summarizes two approaches he’s heard in the conversation so far:
-Indices of valued behaviors in activity that we can document and validate against some kind of criterion.
- Valued behaviors not assessed according to external criteria (e.g. children value, parents value)

P10 offers three ways to look at assessment
  1. create an activity that can be evaluated
  2. observing actiivity in situ
  3. Collaborative activity in which participants assess one another as part of the activity (i.e. what is locally, culturally valued)
P9: Raises the question of embedded assessment

P7: self-documenting vs. self-assessing. E.g. kids taking over the video camera and using familiar genres and adapting them to make records of what their friends were doing and what they were doing. This documentation can potentially be (re)used by others as assessment resources. This raises the question: Can the same forms of documentations be useful to different audiences?

P10: warns against the risks of misuse of these kinds of youth-produced products

P1: Motivates discussion about the need for starting at an abstract level, of having a coherent, overarching theory to inform selection of activities, ways of measuring as we try to define what it is we are studying because the nature of learning in out-of-school time is so varied. P7 - Yes we need a framework to drive inquiry but a theory of what? Of learning, of emotional development . . . what do we feel are the missing pieces? E.G. role of emotions in learning is one (still in an exploratory phase).

P2: Conversation is too broad – we need to get tractable. Should we narrow it down to electronically mediated things?

P7: Reminds all that second workshop is focused on technology.

P8: generating orienting questions, for example:
  1. Do people get more learning done with theories of learning. Consider societies where learning is not an institution, where learning is accomplished primarily by doing (Deweying).
  2. Conversation so far seems focused on individual learning. Take Roger Barker for example. Focus should also be about how people get to the situations where learning happens.
  3. Weighing how much learning is done at home vs. school

P7: Notes that these points motivate the question of what is the UNIT OF ASSESSMENT (e.g.
What about Gee’s proposal of assessing communities that assess their members)

P9: Introduces example of youtube videos and the comments (assessments) people make.
P2 points out that these forms of commenting are built into the modes of communities, they’re reified. What about phenomena that is more ephemeral – how to mark it?

P1: You can document and assess products, but what about developing ways of helping people in the flow to attend to these things? 1:31:00

P7: re: documenting learner’s helping one another – the question of competence and willingness to help others.

P5: Study of kids helping activity as a way of indexing trajectories of kid’s expertise, enjoyment etc.

P10: notes shift in conversation from individual to setting; so should it be a question of what participation means when in a particular setting. 1:35:00

P7 makes connection to legitimate participation model – tracking changes in learner participation in a community. P5 notes that this approach allows for an emphasis on the ensemble

P2: transitions to conversation to use of personal use of digital technologies, e.g. APPs

P3: Boggle as example. “individual” games are social in the sense that proxues for the social are built into them). P7: makes the connection to how kids enjoy showing others their accomplishments on solo games. This is generative of social interaction, more involved interaction. P3: notes that this is also connected to the fact that you can learn about other games through social networking. How participation in one domain can lead you to explore another domain

P11: Raises question of funders seeking something concrete, tradition, something that will persuade them.

P1: Warns against dichotomizing individual and social. P7 notes that we have to be mindful of the fact that funders and policy makers think in these dichotomous terms.

P1: Recommends work of Brigid Barron:

P5: Raises point of the different ways that children represent their learning. P7 – notion of demonstrating activities.

P3: Brings up viral example: the Alien-themed APP for calculating XY coordinates. P7 – the difference between this and non-Alien framed activity: the Alien activity was highly socio-emotional, there was social motivation to participate.

P10: Reorients conversation to try and articulate goals for the project

Video file: MDL-Workshop-061011b

P2: Mimi’s problem vs. Macarthur’s problem; Advocates emphasis on Mimi’s interests. Macarthur interests may lead us to looking at grades. P5: focus on helping Mimi think about understanding the field for opportunities for learning.

P3: emphasis on letting people know that what they are doing is not slacking – what you do with your family contributes to your kids learning – you don’t need to put your kids in a million activities.

P4: people often ask if activities elective (implicitly less valuabe) How do we turn choice into an asset; the fact that people choose to learn these things is valuable.

P7: proposes identifying in cases some of the most valuable kinds of learning in these settings and distinguishing the most and least visible. There are many layers of learning taking place in informal settings vs. formal settings where generally the focus is on one aspect. Proposes discussing concrete examples

P1 presents video examples. Girl building circuits at the Exploratorium, followed by
Palo Alto girl scout summer camp – taking care of animals.

P7: things extractable from video: role relations, orientations, friendship role, social environment role in changing affordances of participation. Discusses literature on science and squeemishness, fear of electricity in boys . . . What are the circumstances under which they would engage in activities they normally would not want to engage in? What is the social milieu that makes this possible. Related to this P1 notes that science as a social process, is a context for social interaction.

P4: different people will see of different things of value in this video . . . lets develop a taxomony of value. P2 suggests exercise of asking colleagues for video examples + their notions of what is valuable about what these videos capture. P7: do this as proof of concept. P5: and turn this into a protocol. P7: as means of thinking about criteria of what seems documentable.

P2: returns to question of internal and external criteria of assessment. External: Intel example. Internal (Milbray), practices internal to the activity, the setting, that satisfy internally, culturally valued things and simultaneously serve as documentation that people on the outside can draw on for assessment 41:00

P1: example girls riding horses become more confident example – these are things that are not so easy to look for (i.e. video of girl riding horse doesn’t show confidence in the future). P2: but are we going to document the varieties of outside connections to the activity? P7: yes, if we’re interested in the activity system.

P3: how do we value the quality of one touch (in the squeamish scenario)? We have to look at more than one touch. P7: fallacy of the causal model – she touched the cricket once, she is now a phd biologist. The need for assessment at larger time scales

P10: suggests making a list of what is being indexed in videos – use this as a way to organize our call to the field for examples. P9: informal institutions have formal trajectories of participation.

P8 brings up learning typology from Understanding Learning Environments class:

1. Learning by socialization/acquisition of and by abilities
2. Learning by association and imitation
3. Learning and play
4. Learning, categorization, and scripts
5. Learning, scaffolding and self-regulation
6. Learning by inquiry
7. Learning by reflection, description, and narratives
8. Learning by interaction and embodiment
9. Learning by participation
10. Learning by resistance and confrontation

Notes from 6/11/11 (top of page)

P1 – P11 stand for the 11 participants who attended the meeting

P2: The question of value – whose value? School related value? Activities valuable in and of themselves? Who is valuing these activities and for what reason? What about the emphasis of some funding institutions on simply keeping kids off the street? Also, the problem of access to so volume of media that kids get into things that the adults don’t want them to get into.

P10: suggests we not worry about the link to school, that we think of activities in and of themselves. P7: Worry about this issue at the end of the paper.

Question of different standards and values shaping design and assessment of activties (“value criteria that orient our looking”). Those of the:
  1. participants
  2. organizers of activities (practitioners)
  3. researchers
  4. funders (P9)
  5. caregivers

Presentation video of Physics Telementoring activity at UCSD after-school site

P4 raises questions about the importance of the backstory. P7: there are things that can be understood from just watching the video and things that require the backstory (but what backstory information do you need?). P11: you always need backstory to some degree.

P4: Does the student in video assign personal significance to what we assign significance to? P6: Student evidence that she valued the activity: desire to continue the activity even when P6 had to leave; desire to engage in the activity on her own. P2: this points to the importance of long-term observations.

P1: STEM individual, notable achievements vs. longitudinal cases

P7: How long should the longitudinal be in order to make warranted claims about changes in individuals, in an activity, in a program?

P3: Example of work in homes/math. Over 300 instances of math activity at home, each of which has potential for study . . .

P7: significance of event increased by seeing it as an instance of frequent type of event vs. increasing significance of event by seeing how it fits into longer timescale 45:00

P11: who are we trying to convince?

P7: what questions are significant questions in order to do better research?
What current literature is out there to answer these questions? What research should be done in the future to answer these questions? (EG –what kind of back story, what moments to study in a longitudinal context)

P8: relative to P6 physics example, how many people are involved? What is the level organization? How far out do we go? P7: What is the system context (in addition to longitudinal context, ) frequency of occurrence? who is involved? what are the connections involved? What are the affordances? There is more expectation for improvisational activity in informal settings

P10: The expectation of an argument here, of values of the different institutions, there has to be a stance. 53:00

P7: what kind of things should be documented about a setting or an activity when evaluating it for its potential to contribute to significant informal learning – what are some of the features of good informal learning activities have that should be on your checklist of activities before the fact, and after the fact.

P9: are we expanding scope of the charge here? But aren’t we looking for good evidence for learning in these settings? P7: there is a loop here . . . good features of learning settings have implications for what you would assess . . . it orients your research.

P4: we should also pay attention to how many resources were necessary to do what is done. Could you do it with different levels, degrees of resources? how do you marshal the resources?

P7: which costs are assignable to a particular benefit? E.G. Schools – relative cost for school districts for classroom work, these economics are not very favorable compared to informal learning activities.

BL: raises question of a spectrum of activities ordered on cost – online communities as having virtually no cost. P7: but what is the actual system that enables an online community to exist and operate (e.g. online gaming communities, costs of development). P11: Modding communities example, virtually no cost.

P2: is this a research question for this group? Aren’t we being asked to document learning in these contexts without dealing with the cost, the heaviness 1;00;00

P7: this meeting is about expanding out and then narrowing down. P11: lets get action points to start to orient. P7: how far out we go depends on the unit of assessment (individual, activity, community, institution).

P3: raises questions about P6’s video. We need to consider the density of resources available to the learner in an activity. P2: is this the question that we’re supposed to be addressing?

P1: density of experience: many informal learning experiences are highly scaffolded. Like parenting – think about density here (duration, frequency, intensity of interaction). P4: it’s density and boundedness. How do we talk about the system while remaining true to the things we care about? P7: what happens when you can’t separate outcomes from processes – then the person has to be bigger than the moment. 1:05:00

P1: Funders will ask if we take one element out (resources), will it change the activity significantly? P7: part of documenting effectiveness of activity requires saying something about the role of the elements of the activity.

P10: NRC volume from a few years back includes table of learning forms based on the continuum that P10 presented earlier (Casual spaces Û Designed spaces Û School) and each of these spaces was then categorized in terms of, for example, what is the nature of the interaction, where is agency located, what’s the nature of assessment in these spaces. P10 suggests this as something we could leverage, add to.

5 minute break

Video file: MDL-Workshop-061111b

Presentation of P3’s math/home video examples

Clip 1. Talking about, story about. Discussion among family members at home about using math to redesign bathroom & kitchen.

Clip 2. Event. – Family members collaborating to determine dimensions of kitchen counter top; debating how to determine the area of a triangle (8:30 end video)

P7: Video shows usual progression of setting up the problem, coming up with solution, high emotion; however it isn’t until halfway through the video that you get recognition of a contradiction between the first and second solutions. P9: it’s goal directed activity, you see how the formal learning, the procedure gets misapplied – Good example of the contrast between the informal and formal learning

P7: Question of backstory: who are the participants? What are their backgrounds? What is the goal? P3: goal – buying and fitting granite counter top. At stake: thousands of dollars that one of the sisters in the video borrowed off of her law school loans. 13:00 P8: rich backstory, but we don’t need that much backstory – do we? Vs. school where problems are just dropped on the table – uninteresting. P2: no need for backstory in this example to identify certain kinds of learning that are present. P7: But can you identify the elements that contribute to this learning? P3: Background (defined in terms of race, income) would be one of the first things that NSF would ask about if shown this video

16:00 P1: What is being learned in this example? What we see in the video is knowing, the application of knowledge. P7: learning activity vs. demonstrating activity, both are taking place in this example. P10: self-correction and self-evaluation in the video is evidence of learning, but we also need to know what this is a case of, what type of situation, setting, but no need for long backstory.

P7: What analytic lens can we apply to this video: literature on small group learning in classrooms – significance here: the roles taken on by participants in a group. What are people’s relationships to one another? Who trusts who? Who challenges whom? Who is seen as the expert? This is where backstpry becomes relevant

P11: What is the standard of evidence here?

P9: Part of what we’re struggling with: we haven’t agreed on audience and purpose.
We’re having trouble with standards and scope. P7: his sense of scope and audience: MacArthur-Mimi want to be able to agree on the standards by which assessment of informal learning activities should be done, what is the relevant kind of evidence for demonstrating learning in the kinds of research that Mimi is doing. Let’s get people together to define this. P7 compares current project to the NSF video analysis white paper he worked on before: developing guidelines for good practice.

25:00 P7: review literature to identify what new research needs to be done (white paper genre) – framed by identifying what are the key questions that need to be answered in this area. P4: we would like to understand what kinds of evidence we could collect about these activities that are important in and of themselves. This is the core concern. P7: how far out should we build from this core?

P2: We seem to be recognizing that there is creation of cultural capital outside of school, and in those different domains how does one document the different kinds of learning in which this goes on (home, AS) and in these domains what kinds of documentable things go on there.

P10: What are some promising places where indices of learning show up? P11: Second life programming; modding in WoW; Theory crafting: math behind the game mechanics (engineers posit, but nivices read these forums) happening in the value system of being a more effective player. P1: learning about learning (learning about identifying and obtaining the tools you need) P2: easily accessible, track-able data: online forums will show a history of group conversation about practice (group learning). P7: notes distinction between people seeking canonical answers vs. studying the history that gave birth to this canonical answer. Information literacy (e.g. librarians research on how undergrads use information resources.) P11: changes in games force learning in gamers.

P4: the question transfer across game environments. P11: The is research, but evidence is informal.

P1: Online gaming communities provide a space for identity development, not just centered on expertise of the game, but around other topics that emerge from participating in these communities . For example, in online forums users interact with people of different backgrounds and consequently can get in to conversations about topics other than those for which the forum is organized. P1 offers example of her son getting into arguments about evolution with other members of the forum. 3830 P4: importance of this cross-class contact. Sparse examples of this happening easily. P1: this also connects to need for a longitudinal perspective – you can track individual changes over time.

Video file: MDL-Workshop-061111c

P4: Intergenerational question. WoW as an example of an activity in which people of different ages collaborate and learn. P7: Quest Atlantis example – cultural models for these kinds of intergenerational relationships are unusual. Buddies ad kids at UDA pioneering ideas of what a relationship between a 25 year old and 10 year old can be like.

P1: documentation suggestion: purposeful appropriation of tools, knowledge in order to achieve valued goals – this could be shown in someone’s online engagement in a game. Also in how a person draws upon density of prior personal experiences.

P2: Reads P6 email to undergrad who was a former participant in UCSD-LCHC after school practicum. Email describes how over time kids at the after school center take on leadership roles as a result of expertise developed from prior participation in activities at the center. Example highlights the need for longitudinal work. 7:30

P7: Important to make a case for the longitudinal approach, to demonstrate the plausibility that looking at longer timescales you will see new evidence of learning and its consequentiality. You can’t evaluate just in the moment. P7: In schools the tendency is for short term evidence of learning (tests), and no long-term evidence of the consequentiality. In informal settings it would be nice to show that in the short term it may be difficult to nail down the consequential learning, but as you look over longer and longer timescales its clear that there is this consequential learning.

P8: Need for alternatives to the term “informal” (unformed!). Learning happens in relationships and it moves forward in these relationships. P7: learning is consequentiality – not learning if it has no effect on the future.

P7: Asks for other candidate domains. P11: – online knitting communities. P1: citizen science. P3: go players online.

P1: Returns to the issue of alternatives to the term “informal.” P7: Free choice learning, voluntary learning, learning in passion communities.

P8: Recommends 1983 Charles Fracke paper on formality (notion of a gate or border that people have to pass through to move between formality and informality. P7: this is an anthropological theme: formal informal across different societies.

P7: Notes the unpredictability element, improvisation element in informal settings, and concommitant expectations about learning.

P7: Raises issue of comfort as a quality of informal learning. P1: informal learning is not comfortable either though (e.g. parents requiring their kids to attend after-school)

P9: Suggests self-directed learning as an alternative, but this is something that can happen in both formal and informal settings

P8: Frackes definition – you call it formal because you invoke arbitrary criteria for entry. Example of jazz – mistakes as opportunities. Art Tatum and Fats Waller example.

P1: Kinds of Audiences: Browsers, Extended learners, Professional audiences – all can coexist in the same space. Review of existing typologies show that all defining with respect to schools – dead end.

P2 and P7 suggest:

P2: potential for people to change their own activity is critical. P1: pushback – power of learning in self-directed ways.

Definitions of formal, informal, non-formal from Helen Colley, Phil Hodkinson & Janice Malcolm.: Formal – institution provided, structure, leading to certification, intentional from learners persective; Non-formal: not provide by institution, no certification, yes intentional; Informal – daily activities, not structure, not certification, may be intentional. GROUP: general dissatisfaction with this characterization

P9: Better instead to focus on features of informal settings P10: we could identify settings, types of games . . . that there are places were the interaction is such that participants can index their own knowledge of learning. So . . . map out the spaces, what are the dimensions that seem to matter, then pick an example and drill down. (Mimi did do drill down in some areas). Identify and study examples of evaluative moments (EG: kid beats the boss in a game by hacking) mini typology: positive assessments, asking for how this was done, advising others on specific steps on how to beat the boss, critiques of the technique, direct discussions of one’s status)

P2: Team sports as a domain to look at. One thing to ask about settings: to what extent do settings include practices that make learning visible. P9: post game debrief with the coach – coach asking kids for self-critique. Balancing motivation and learning. Also being mindful of the public/private in these settings – kids critiquing themselves or others, coach critiquing privately or infrint of the group. P9: study critiques online vs offline in one of these critique groups.

P8: textbook learning example – class he teaches on learning with Roy Pea: learning by association, play, categorization, self-regulation, scores, planning embodiment, inquiry, participation, resistance. P7: are there different kinds of documentation procedures for those different kinds of learning.

10 min break

Video file: MDL-Workshop-061111d

P10: suggests 15 minutes to have particiapants send P6 literature. Orienting categories for selecting literature – work that examines:
1. What is the nature of the spaces/setting: What do we want to focus on in these spaces?
2. What are the norms or practices within the setting around learning (showing up, using it) that setting?
3. What are potential indices that are noticeable and documentable?

P1 shows slide presentation of work done to elaborate a shared set of design principles for informal science learning activities. Presentation accessible here:


(10:40) Following presentation P7 discusses post-meeting stage: developing a checklist of the kinds of observations and codes discussed in P1 presentation, one that cumulates across domains. Then the task is to whittle down the list to key observations. Also, looking at existing literature of assessment in informal settings

P4: What about competitions as sites of evaluation (for the researchers and for the kid themselves)

P1: The thread is purpose, what drives participant purpose in these settings. P1: motivates discussion of fields of literature. P1: environmental psyche. P2: encyclopedia of informal learning. P1: chapter in encyclopedia of the sciences of learning. P10: serious games P2: sports and learning P7: Lave & communities of practice. Service learning literature? Check discussion of how UGs are assessed in service learning programs

19:40 P7: We’re looking for methodologies of assessment, criteria of evidence, patterns of argument of evidence to conclusion. P1: Suggests Mahoney et al. Organized activities as contexts of development. Meredith Honig as a resource. P2: National collaboration for youth competency assessment tools. P1: Assessing young childrens learning at disneys kids discovery clubs - Koncanjer

28:00 Discussion of the next two workshops

P2 – idea of asking folks to send video with two pages of back story.

Other recommendations: Herbert Ginsberg, Fred Erickson, Journal of Everyday Learning