ISERN Agenda

Welcome and New Introductions

Context: ISERN is open to academic and industrial groups world-wide that are active in experimental software engineering research and willing to adopt the experimental research framework. ISERN members are pairs of organization and contact person. If the contact person leaves the organization, the organization must reapply for membership. Interested organizations may apply by sending an electronic proposal to “isern at informatik.uni-kl.de” describing their past experience in experimental software engineering research as well as their expectations from a future ISERN membership. Candidates will be invited to observe the ISERN Meeting following their application.

Goal of the session: To facilitate the membership application process by giving an opportunity for candidates to present their research and for observers to introduce themselves.

Session Format: Membership is granted according to a 3-step procedure:

1. Attending as invited observer at an annual ISERN meeting.
2. Attending as invited candidate at the following ISERN meeting giving a presentation. Membership is granted if a two-thirds majority of current members approve the application in an email voting after the meeting.
3. Attending as a full ISERN member all following meetings.

Vic Basili: Introduction of members, candidates, and observers.

Current members present 1-minute contact/affiliation changes:

• Forrest Shull for FCMD
• Letiza Jaccheri for NTNU

Candidates give a 5 min presentation each:

• Emilia Mendes, University of Auckland
• Martin Solari, Universidad ORT
• Ricardo Valerdi, MIT
• Guenther Ruhe, University of Calgary

Observers give 2 min introduction without a presentation:

• Dr. M. Ali Babar, Lero, The Irish Software Engineering Research Centre
• Yoshiki Mitani, Software Engineering Center (SEC) at the Information Technology Promotion Agency (IPA), Japan
• Barbara Russo, Free University of Bolzano-Bozen
• Lorin Hochstein, University of Nebraska-Lincoln
• Mark Müller, BOSCH
• Sandro Morasca, Università degli Studi dell'Insubria
• Dr. Stefan Wagner, Technische Universität München
• Dr. Michel R.V. Chaudron, Eindhoven University of Technology

Reports from the 2006 Session Chairs

• Dieter Rombach: Empirical Research, Experiments
• Marv Zelkowitz: Data Ownership
• Andreas Jedlitischka: Experimentation and Decision making in Software Engineering: How they are related? and ISERN Experience Factory
• Jyrki Kontio: Good practices for empirical research with industry
• Dag Sjøberg: Generalization from empirical studies
• Stefan Biffl: Value-Based Empirical Research

Announcements

• Forrest Shull: New IEEE Column

Roadmap

Chair: Rick Selby

Goals:

1. Summarize the Research Roadmap for Empirical Software Engineering defined at the Dagstuhl Workshop held in June 2006;
2. Gather feedback from the ISERN community members regarding the roadmap itself and progress along its dimensions;
3. Update the roadmap to reflect ISERN community consensus; and
4. Define near-term ideas, collaborations, projects, etc. that will catalyze and accelerate research progress inspired by the roadmap criteria.

Context:

In 1992, a Dagstuhl Workshop was held to share ideas and experiences on "Experimental Software Engineering". Fourteen years later in 2006, many of the same researchers gathered again at Dagstuhl to discuss the tremendous progress since the first workshop and to broaden the focus to be "Empirical Software Engineering". At the 2006 Dagstuhl Workshop, we aggregated everyone's ideas and formulated them into an overall roadmap that defined research dimensions and progress criteria along each dimension. The roadmap consists of four categories that organize and group together nine dimensions, and they are as follows with dimensions indented below categories:

• Maturity
  • Cohesiveness of field
  • Research methodology
• Coverage
  • Process/technique/phase
  • Problem domain
  • Artifact scale
  • Subject expertise level
• Understanding
  • Evidence
• Impact
  • Science/engineering
  • Industry

Each of the nine dimensions has several “signposts” or indicators that signify progress along the dimension, and each dimension has a final progress indicator that defines the ultimate goal for the dimension. The ordering of these progress indicators suggests a logical path or maturation along a dimension. The 2007 gathering of the ISERN Community in Madrid provides the ideal forum for critique and improvement of this Empirical Software Engineering roadmap as well as assessment of progress relative to its signpost criteria. As background reading, we have attached two short articles that introduce and summarize the roadmap in its current form.

• Reading from Dagstuhl: Introduction
• Reading from Dagstuhl: Discussion and Summary

Facilitators:

After a brief plenary session, we will break into small parallel working groups to discuss, assess, and evolve the roadmap. We will recruit 3-4 facilitators to help lead the working groups. If you are interested in volunteering to help lead a working group, please send email to Rick.Selby@NGC.com. We look forward to your active participation in this session in Madrid!

Lunch

Roadmap wrap up

Breakout session intros:

ISERN Experience Factory [Andreas Jedlitschka]

VB Software Process Framework [Nilay Oza/Jurki Kontio]

VB Software Process Framework

Chairs: Jyrki Kontio, Dr. Nilay Oza

Goals:

• Explore how value-perspectives are included in SPI models and SW development models, using (i) EF and (ii) Scrum as examples.
• Develop definitions and metrics for value for the functions of (a) software development, (b) software process assessment, and (c) software process improvement.
• Propose different empirical study designs for studying value considerations, given the two example "method frameworks" (i and ii) and the three functions (a, b, c).

Context:

VBSE was included in a session in 2006 and this session will build upon those results.

Facilitators:

Stefan Biffl (additional facilitators TBD).

ISERN Experience Factory

Chair: Andreas Jedlitschka

Goals:

Motivate the community to use the portal as a means for communication and exchange. As well as potential place for building the empirical body of SE. Within the working group we will discuss issues regarding the portal management and concrete actions that should be taken to facilitate the collaboration. We envision to have sub-groups to work on different tasks.

Context:

Last year, we have collected some requirements for a community portal. We have envisioned how ISERN could systematically collect the common empirical knowledge. This years focus is on presenting a first prototype of both, the community portal for collaboration and a repository for collecting evidence.

Facilitators:

Marcus Ciolkowski, Muhammad Ali Barbar.

Break out session wrap up and plans for the next day

A Checklist for Case Studies: A Practical Evaluation

Session Chairs: Martin Höst, Per Runeson, Claes Wohlin

Goals:

At this session a first version of a checklist for doing case study research will be evaluated by the participants by using it while reviewing a set of research papers. The goal is that the methodology for conducting case study research should be discussed and that the session participants should give their input to the further work with the checklist.

Context:

Previous ISERN meetings have discussed guidelines for conducting empirical research. For example, at the ISERN meeting last year (2006), guidelines focusing on conducting controlled experiments were presented and evaluated by reviewing example studies. The checklist that will be evaluated in this session is focused on case study research.

Breakout session intros:

Glossary of terms [Guilherme Travassos]

Evaluation in industry [Haruka Nakao, Daniel Port, Jürgen Münch]

Lunch

Glossary of terms

Session Chairs: Guilherme Travassos, Mike Barker

Goals:

For the ISERN community to review, discuss and agree upon an Experimental Software Engineering – Glossary of Terms (http://lens-ese.cos.ufrj.br/wikiese) for those interested in experimental software engineering worldwide to share and use.

Context:

The ISERN manifesto states that “… software engineering research needs to be performed in an experimental context that allows us to observe and experiment with the technologies in use, understand their weaknesses and strengths, tailor the technologies for the goals and characteristics of particular projects, and package them together with empirically gained experience to enhance their reuse potential in future projects. … Several academic and industrial research groups world-wide made the paradigm shift to an experimental software engineering view. Each of these groups has produced and will continue to produce software engineering models valid within their local laboratory environments. In order to take the next step towards building the basic models and components of the entire software engineering discipline, we have to be able to abstract from the characteristics of specific environments. No single research group is able to provide the laboratory environment necessary to learn about variations in the effects of technologies across multiple environments and influential factors.”

Experimental initiatives in software engineering are increasing yearly. Research groups around the world have used different types of primary studies and more recently secondary studies to provide evidence and build a body of knowledge about software engineering.

Motivated by the importance of the field to advance technology, some of them have tried to adapt concepts and definitions to their own local perspectives sometimes differently from those usually used by the ISERN community. This movement reinforces the need to have an explicit ISERN common terminology defined in multiple languages to communicate and share knowledge, as previously identified in the International Workshop at Dasgthul Castle, Germany, 2006. It will represent an important step towards a taxonomy (perhaps someday an ontology) of the experimental software engineering field.

An initial set of concepts (English, Portuguese, Spanish) merging works from ISERN – Basic Terminology (1995) and ESELAW – Experimental Software Engineering Latin American Workshop (2006), is available at http://lens-ese.cos.ufrj.br/wikiese. It uses a wiki-based facility to support the necessary review and discussion. The definition of terms and fixing of defects are on-going. Feel free to browse it and comment!

Facilitators:

We are contacting some ISERN members. If you believe that you could help us with this discussion or help with reviewing or moderating a small discussion group, please, let us know (send us a message – ght@cos.ufrj.br or mbarker@MIT.EDU – Title: [ISERN section] <topic>). Volunteers are really welcome!

Guidelines for Model and Methodology Evaluation in Industrial Environments

Session Chairs: Jürgen Münch, Haruka Nakao, Daniel Port

Context:

In previous industry-oriented sessions at ISERN 2006 and at the Dagstuhl Workshop on Empirical SE recommendations on how to attract industry for collaboration with research organizations and practical guidelines for conducting such collaboration were collected. This session will be based on these findings and focus specifically on how to perform evaluations in industrial environments.

Goals:

The goal of the session is to collect guidelines on how to perform model and methodology evaluations in industrial environments.

Questions to be addressed include:

• What are the benefits and cost of industrial technology evaluations?
• What are the challenges of evaluating innovative techniques in industry and how to overcome them?
• How can industry implement a technology evaluation process (e.g., as part of an innovation process or a risk assessment process)? How could this look like?
• How can academia support industry in performing technology evaluation?

Break

Breakout session wrap up

ISERN Business

Dieter Rombach

Open Discussion