Concept&Go

1. Positioning: What problem is addressed through which solution to what confidence? 

         Problem: State and characterize the problem.

• Problem statement: How can concept mapping be supported for helping teachers to find the causes of the students’ misconceptions?

• Context description: Teachers can provide several sources of knowledge related to the subject matter that can assist learners to achieve their learning aims. One of the ways to assist a student's concept mapping process is providing texts associated with the topic or focus question. The concept mapping task carried out by the students is followed by the evaluation and validation by the teacher. The latter allows teachers to note missing or irrelevant concepts, and trivial or incorrect linking phrases. It is not odd for teachers to wonder what is behind these misunderstandings. Yet, using the concept map as the unique resource to perform the validation allows the detection of students' misunderstandings, but not to trace their origin.  Here, the problem is that the grounding reading that supports the concept construct is not explicit but in the students’ heads.

• Goodness description: Usefulness of design principles that helps teachers find the causes of the students’ misconceptions supporting a concept map activity that can help teachers be trace for evaluation

  
  

         Solution: Outline and characterize the solution.

• Solution essence: we proposed a concept mapping activity based on an annotation practice.Our premise is that rephrasing the concept map activity as an annotation activity can help. Annotations provide a link between the reading material and the developed concept map so that teachers can have a deeper insight into students’ misconceptions.

• Representation description: we developed a web annotation tool named Concept&Go, which is a web extension that supports a concept mapping activity based on Web annotations. 

• Process description: Design of the steps to conduct the concept mapping activity using web annotations (Annotation-based concept map construction) and instantiation of an artifact that allows performing each step (Concept&Go).

  
  

         Evaluation: Describe evaluation activities and results.

• We have not been able to carry out an evaluation of the project because we have not had the opportunity to evaluate it with real professionals in a real environment. We are looking for researchers who can help us evaluate the tool

2. Grounding: What knowledge is informing the design? 

        Search Process: Which search strategy was applied?

• Review of literature about which are the benefits and how a concept mapping assignment by students based on reading materials and identify which are the problems involved in this practice. Analyzing which are the necessary steps for conducting a concept map task, we considered The Standard Concept Map Construction Method [4]. Design of solution (adapt the process to an annotation practice and development of the software artifact). Finally, we try to find a conference where the tool can be presented in order to receive feedback and find practitioners in education willing to help us evaluate it [5].

       Search Results:

• Kernel theory (Ω-knowledge): Theory on complex systems and IT complexity (no integral theory yet, structural and dynamic models); the focus on this kernel theory was preceded by a focus on knowledge about IT standardization and IT heterogeneity in earlier cycles. 
• Design knowledge (λ-knowledge): The practice of annotation has been shown to be an efficient method to facilitate the understanding and interpretation of the content that is being read.

3. Aligning: How do the design activities contribute to creating the DK? 

          Design process documentation: What activities were conducted in which sequence? 

• Exploration and identification of the problem (when assessing Concept maps, finding the causes of the misconception is difficult). Identification of requirements (traceability), design (development of the solution space adjusting the practice of annotation to concept mapping and developing an IT artifact). Evaluation is needed

         Design process rationale: Why were the activities conducted in this sequence?

• First, as we did not have the necessary knowledge about the context (concept mapping), we needed to read and learn how it is performed in the practice. We perceived that web annotation could support the concept mapping activity when it is based on reading materials. We believe that concept maps can be enriched with linked resources that can complement them and allow teachers to have more information to assess and find students’ causes of misconceptions. 

4. Advancing: How does the DK chunk provided advance?

• Compared to existing DK. The annotation practice has been applied in different settings to solve problems of traceability. Different annotation practices can be found in Social Sciences and Humanities, Journalism investigation, or Medical and Biological Sciences. 

We depart from web annotation tools that were used to support different practices. 

• Highlight&Go [1] is a web extension that supports the web annotation practice to support researchers doing Systematic Literature Reviews.

• Mark&Go [2] is a web extension that supports the web annotation practice to support teachers to assess assignments based on rubrics.

• Review&Go [3] is a web extension that supports the web annotation practice to support reviewers reviewing papers.

We conducted the projectability of the use of web annotations to adjust it to our own domain (i.e. concept mapping assignments) and stakeholders (teachers and students). In fact, the Concept&Go was developed by extending and reusing the source code of Review&Go.

• Anticipated advancements of current DK. 

• Anticipated projectability: can this solution be linked to a different problem space?

As shown above, the annotation practice can be applied in multiple settings. Reusing the IT artifact, as we did previously with another one, can help to accelerate the research of the application of web annotation in other settings and contexts that can be supported with it. For example, if the users of the tool were teachers instead of students, teachers could use Concept&Go to prepare concept maps that group the learning materials by web annotations within concepts and relationships. In this case, the concept map can serve as a resources hub, instead of an element to be assessed.

• Anticipated fitness: what changes can be made in the solution for this problem?

Concept&Go is a first software prototype that implements the solution. It would be interesting to assess the solution with experts in the area of concept maps. We have more knowledge about the technical part of the solution than about the context of the concept mapping tasks. Therefore, more experience with concept maps could help us to provide a better solution to support traceability, or improve the application of web annotations

• Anticipated confidence: what evaluations can be conducted for current DK?

It is the most important advancement to consider. This project has not yet an evaluation performed in a real-setting. Therefore, we are looking forward to finding collaborators that can help us to evaluate it in a real setting.

5. Resourcing: which are the available resources?

          About Problem: see attached pptx file

          About Solution: Artifact in Chrome Store; Source code; User manual

          About evaluation: ---

Support&Collaboration:

           - xabier.garmendiad@ehu.eus

          - haritz.medina@ehu.eus

REFERENCES

[1] Díaz, O., Medina, H., & Anfurrutia, F. I. (2019). Coding-Data Portability in Systematic Literature Reviews: a W3C's Open Annotation Approach. In Proceedings of the Evaluation and Assessment on Software Engineering (pp. 178-187).

[2] https://chrome.google.com/webstore/detail/markgo/kjedcndgienemldgjpjjnhjdhfoaocfa?hl=es

[3] Díaz, O., Contell, J. P., & Medina, H. (2019, June). Performant Peer Review for Design Science Manuscripts: A Pilot Study on Dedicated Highlighters. In International Conference on Design Science Research in Information Systems and Technology (pp. 61-75). Springer, Cham.

[4] Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. Report to the Chief of Naval Education and Training, 1-108.

[5] https://www.cmapconference.org/

PROBLEM STATEMENT

When assessing Concept maps, finding the causes of the misconception is difficult.

SUPPORTING EVIDENCE

- "Before examining the quality of a concept map by applying it for the purpose it has been generated, though, evidence needs to be given regarding its validity of the knowledge in question. This means it has to be  determined whether the concept map constitutes a valid model" [1]

- "The concept map evaluation will influence the validity of the assessment by affecting the quality of the information extracted from the concept maps. To some extent, this will be influenced by the nature of the concept mapping task. If the procedures for creating a map are not well specified, the variation in students’ maps may make interpretation difficult"[2].

- "In analyzing the factors that influence the effect of teaching, the teacher can determine a student’s knowledge structure and highlight misconceptions by inspecting the concept maps and logs." [3]

ASCERTAIN CONSECUENCIES

- Difficulties for detecting students' misconceptions [4,5]

- Not being able to adjust learning materials [4,6,7]

ASCERTAIN CAUSES

- Multi-perspective [1,3,8]

- No evidence of what is in the student's mind at creating the map [3]

- Students have difficulty in the concept mapping task [5,9]

REFERENCES

[1] Albert, D., & Steiner, C. M. (2005, September). Representing domain knowledge by concept maps: How to validate them. In The 2nd Joint workshop of cognition and learning through media-communication for advanced e-learning (pp. 169-174). Berlin: Japanisch-Deutsches Zentrum.

[2] McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 36(4), 475-492.

[3] Strautmane, M. (2014, June). Increasing the flexibility of automated concept map-based knowledge assessment. In Proceedings of the 15th International Conference on Computer Systems and Technologies (pp. 349-356).

[4] Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. Report to the Chief of Naval Education and Training, 1-108.

[5] Atapattu, T., Falkner, K., & Falkner, N. (2017). A comprehensive text analysis of lecture slides to generate concept maps. Computers & Education, 115, 96-113.

[6] Cheng, S. Y., Lin, C. S., Chen, H. H., & Heh, J. S. (2005). Learning and diagnosis of individual and class conceptual perspectives: an intelligent systems approach using clustering techniques. Computers & Education, 44(3), 257-283.

[7] Brinda, T., Napierala, S., Tobinski, D., & Diethelm, I. (2019). Student strategies for categorizing IT-related terms. Education and Information Technologies, 24(3), 2095-2125.

[8] Novak, J. D., & Cañas, A. J. (2008). The theory underlying concept maps and how to construct and use them.

[9] Atapattu, T., Falkner, K., & Falkner, N. (2014, April). An evaluation methodology for concept maps mined from lecture notes: An educational perspective. In International Conference on Computer Supported Education (pp. 68-83). Springer, Cham.

REQUIREMENTS

Annotations are commonly made highlighting text fragments during reading activities. The practice of annotation has been shown to be an efficient method to facilitate the understanding and interpretation of the content that is being read [1]. As its name explains, Web annotations are online annotations made externally over a web resource, e.g. a web page or PDF document. This particular type of annotation is displayed over a layer on top of the Web. The annotation layer is usually provided by a Web annotation client, which allows end-users to add, modify or remove information from a Web resource without modifying the resource itself.

The annotation process does not only consist of highlighting text, but also includes activities such as adding comments and classifying the content for giving meaning to annotations. The fact of being able to write down ideas and be able to give meaning to them, made us think it could be interesting to apply them in a concept mapping activity. Rephrasing the concept mapping as an annotation activity leads to a concept map construction based on the concepts and relationships that are annotated during a reading activity (see Fig.1).

Figure 1.  Overview of the idea put forward concept mapping based on annotations.

The Standard Concept Map Construction Method explained by Cañas et al. [2], the concept map construction involves: 1. definition of focus question 2. identification of concepts 3. order concepts from most general to the most specific 4. add links & linking phrases 5. add cross-links 6. and map review. Next, we propose a similar method for annotation-based concept mapping, which follow some of these steps.

Definition of focus question. The reading activity allows students to discover new concepts and ideas from an unknown topic. This step involves defining which will be the topic of the concept map and collecting documents associated with it. It is important to properly select which documents will be used for the annotation activity because they are going to be the only knowledge source of students.

Concept handling. Annotations can serve for capturing text fragments that give rise to the conception of a new concept. When a new concept appears, the concept mapper should have the opportunity to capture it by highlighting and assigning a name to it. As an example, in Fig. 1 the concepts “photosynthesis” and “plants” are captured while reading in Wikipedia. 

Relationships handling. In addition to finding new concepts, reading also allows finding how these are related to each other. As well as concepts, relationships (links & cross-links) can also be captured by annotations, but it involves one addition. In this case, apart from giving the name of the linking word, it is also necessary to include in the annotation the ‘from’ and ‘to’ concepts that link it. As an example, in Fig. 1 the linking word “used by” is captured while reading in Wikipedia for “photosynthesis” and “plants”.

The Map is reviewed. During the annotation activity, annotations are made to capture concepts and relationships among them. However, annotations themselves do not form a concept map. Therefore, a further step is required. Gathered concepts and relationships must be moved to a concept map, and students can refine and manipulate the knowledge that has been acquired from lecture notes. The result of this step is the concept map and the set of annotations that teachers use to validate it. 

Benefits are twofold. First, annotations provide physical support for the reading material managed by students. Second, annotations provide the link between the reading material and the concept map, so that teachers can have a deeper insight about student misconceptions that do not stop at the concept map but the ground reading that roots the misconception. On the other hand, performing that annotation-based activity by hand and without support adds too much effort to the activity. Therefore, we have developed a web annotation client that supports the annotation-based concept mapping process.

REFERENCES

[1] Kawase, R., Herder, E., & Nejdl, W. (2009). A comparison of paper-based and online annotations in the workplace. In European Conference on Technology Enhanced Learning (pp. 240-253). Springer, Berlin, Heidelberg.

[2] Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. Report to the Chief of Naval Education and Training, 1-108

CONCEPT&GO

Concept&Go is a web browser extension that extends and tailors a web browser interface to add functionality over reading resources with the aim of supporting the students' concept map construction process explained above. It also allows generating automatically the concept map with the annotations that sustain the concepts and relationships to facilitate the validation performed by teachers. When Concept&Go is activated over a reading material opened in the web browser, it shows a sidebar on the left part of the document (see Fig. 2). This sidebar contains the collected concepts and the required tools for developing the annotation-based concept map (e.g. buttons to annotate concepts and relationships) (see Fig.2).

Figure 2.  A PDF augmented by activating Concept&

Concept&Go does not support the whole process by itself. It needs to make use of two external applications to perform the whole annotation-based concept mapping process. On the one hand, Hypothes.is is used as persistent storage to save and retrieve created annotations when necessary. It is a widely-used annotation server that requires to have a user account to use it. On the other hand, Concept&Go also uses CmapCloud (the cloud service of CmapTools)to depict the developed concept map with its annotations. Next, we briefly explain how to perform the steps of concept mapping in Concept&Go.

Focus question definition. When the extension is activated for the first time over a reading material, it asks to introduce the topic of the new concept map. Once the topic is defined, the topic concept is established as the root node of the concept map and it is displayed as a button in the sidebar. In addition, Concept&Go provides the opportunity to change the focus question anytime and to set any of the already annotated concepts as the root node of the concept map.

Concept handling. The next step is to start identifying concepts. When the student identifies a new possible concept during the reading process, he/she has to select the text fragment on the document and click the ‘new concept’ button.  Once the concept is created, an annotation is created over the selected text and a new button is added within the sidebar in order to annotate more evidence for that concept.

Relationship handling. The next step is to create relationships between concepts. Relationships are made up of two concepts that are linked by a linking word. As well as with concepts, users can find ‘concept-link-concept’ triples propositions while reading. These relationships can be found written explicitly, implicitly, or collected on a table. Therefore, the user has to be able to capture the relationship by annotating the linking word in the same way as with concepts. To complete a relationship, it is necessary to provide the "from" and "to" concepts together with the linking word.

Map is reviewed. After capturing map elements that are relevant for the student, the student has to export the annotated content as a concept map to review the result. Concept&Go provides the option to export a concept map with the annotations to a CmapCloud account on the top part of the sidebar. The user has to provide his/her CmapCloud user credentials to associate Concept&Go with the CmapCloud account. Once the association is done, students are able to export the concept map, which is exported in a folder that is automatically generated in the CmapCloud home folder with the name of the concept map and its version. The folder contains the file that represents the concept map and the web annotation resources (URLs) that are referenced within the concept map (see 1 in Fig 3). The resources attached to the concepts and relationships of the concept map, allow to navigate to the annotation that created when the map element was captured (see 2 in Fig 3). Concepts maps can be shared with the teacher, and this brings the opportunity to evaluate the task with the concept map and the annotations to trace the origin of the elements.

TODO

Until now it has been evaluated with colleagues, but we could not perform a consistent evaluation. We are trying to finde practitioners (i.e, teachers that use concept maps) that could collaborate with us to evaluate the idea and the instantiation of the solution (i.e, Concept&Go).

• Users: those whose aims are to be achieved. 
  

  Enactors: perform actions as part of the mechanisms that are used to accomplish the aim.

  Context: Boundary conditions, implementation setting, a further user characteristic

  Mechanisms: the means to achieve the ends envisioned by the design principles

  Rationale: a  justification for believing that the mechanisms will lead to achieving the aim.