ISERN Agenda

Welcome and New Introductions

Context: ISERN is open to academic and industrial groups world-wide that are active in experimental software engineering research and willing to adopt the experimental research framework. ISERN members are pairs of organization and contact person. If the contact person leaves the organization, the organization must reapply for membership. Interested organizations may apply by sending an electronic proposal to "isern at informatik.uni-kl.de" describing their past experience in experimental software engineering research as well as their expectations from a future ISERN membership. Candidates will be invited to observe the ISERN Meeting following their application.

Goal of the session: To facilitate the membership application process by giving an opportunity for candidates to present their research and for observers to introduce themselves.

Session Format: Membership is granted according to a 3-step procedure:

1. Attending as invited observer at an annual ISERN meeting.
2. Attending as invited candidate at the following ISERN meeting giving a presentation. Membership is granted if a two-thirds majority of current members approve the application in an email voting after the meeting.
3. Attending as a full ISERN member all following meetings.

Dieter Rombach: Introduction of members, candidates, and observers.

Current members present 1-minute contact/affiliation changes:

• Jeffrey Carver
• Teresa Baldassarre

Candidates give a 5 min presentation each:

• IPA/SEC Japan
• BOSCH
• LERO
• Università degli Studi dell'Insubria
• Free University of Bolzano-Bozen
• Leiden University
• Technische Universität München
• Naval Postgraduate School

Observers give 2 min introduction without a presentation:

• Institute of Software Chinese Academy of Sciences(ISCAS), Lab for Internet Software Technology, Ye Yang
• Vrije Universiteit, Amsterdam Rahul Premraj

Reports from the 2007 Session Chairs

• Roadmap: Rick Selby
• VB Software Process Framework: Jyrki Kontio, Dr. Nilay Oza
• ISERN Experience Factory: Andreas Jedlitschka
• A Checklist for Case Studies: A Practical Evaluation: Martin Höst, Per Runeson, Claes Wohlin
• Guidelines for Model and Methodology Evaluation in Industrial Environments: Jürgen Münch, Haruka Nakao, Daniel Port
• " Glossary of terms: Guilherme Travassos, Mike Barker

Coffee Break

ESE Methods: Aggregating results from Experiments

Chair: N. Juristo

Goals:Aggregation should be taken to mean the combination of the results of more than one experiment to generate pieces of knowledge that can be used in practice to develop software. The fact that one experiment yields certain results should not be taken as evidence enough to consider these results as a proven fact. Taken separately, experiments provide partial results, while conclusive results or evidence can be gained by accumulating partial results

Context:In its early days, Empirical SE (ESE) focused on studying the application of the principles of the laboratory and experiment to SE. Twenty years later, running lab experiments in ESE is a fairly well understood task. But running isolated experiments is just one step of the experimental paradigm. Other principles of experimentalism remain to be analysed and adapted to SE: Experiment reporting; Replication; Systematic reviews; Aggregation, etc.

Facilitators:Marcus Ciolkowski, Forrest Shull

Readings:

Lunch

Open Space

• Resolved: you can't make safe software using agile methods!!!

Chair: M. Barker

Goals:the goal of this session is to provoke discussion about the use of agile methods in developing safe software. I expect people will talk about security, but safety can also include other elements.

Context:we usually have something about agile methods, and security and operational software are often touchstones for empirical software engineering.

Outcomes:a key outcome should be providing everyone with a better understanding of the trade-offs involved in using agile methods. A secondary outcome may be reminding people about the effectiveness of debating as a method of investigation and decision-making.

Facilitators:right now I think it's just me. I need to have at least four debate members (two per side) and would like to have up to eight (four per side).

Readings:this would depend on the debate members. I don't think there is any particular preparation readings -- the debate presentations should cover it.

• We will provide a place to announce topics to be discussed. Whenever the group feels it important enough, it could go into a session.

Application of ESE: Applying Empirical Software Engineering to Software Architecture

Chair: G. Cantone

Goals: Gathering information regarding software architecture from the point of view of the ISERN members in the context of an ISERN session by focusing on (1) How a study of software architecture is similar to/different from a study on any other software engineering topic, (2) Measurement models for goodness of software architecture, and possibly (3) How we teach software architecture around the different countries.

Context:empirical software engineering scientists with some experience in conducting experiments on software architecture topics usually with students. Limited participation of software architecture practitioners.

Facilitators:M. AliBabar, J. Carver, D. Cruzes, D. Falessi, O. Pastor, F. Shull, and G. H. Travassos.

Readings:

• Introduction

Coffee Break

Roadmap: Aligning ISERN 2010 Milestones and Associated Key Tasks

Chairs: N. Oza, S. Biffl

Goals:Main goal of the session is to develop a community-level understanding of ISERN's key milestones by 2010, and a set of achievable key tasks we need to carry out as a community to achieve these milestones. The session will foster understanding of the task candidates and their impact on the ISERN milestones. The session will also help to develop consensus oriented understanding in the community in ISERN's goals and means to achieve those goals. The session is facilitated by the session chairs with customized version of an SPI-Stakeholder method and tool support developed earlier.

Context:Recent ISERN meeting sessions discussed goals for the ISERN research roadmap and value-based methods for empirical research. The ISERN community is growing and emerging as an important contributor to Software Engineering research with particular focus on Empirical Methods. The community representatives gathering in Kaiserslautern is a good opportunity to understand what each member feels for community development and goals. The session will address it by aggregating collective understanding which can then be used to identify or refine the actions at the community level.

Facilitators:Barry Boehm, Dietmar Winkler

Readings:

• Paper: N. Oza, S. Biffl, et al. "Reducing the Risk of Misalignment between Software Process Improvement Initiatives and Stakeholder Values" Proc. EuroSPI 2008 (to appear).

Wrap-up and plan for Tuesday

Meeting of the ISERN Steering Committee (invitation)

ISERN Dinner

ESE Methods: Guidelines for conducting and reporting qualitative research - Subtitle: Stories from the trenches

Chair: C. Seaman

Goals:

• mistakes we've made
• unexpected obstacles
• stupid things we've seen others do
• common shortcomings of qualitative papers we've reviewed
• ethical dilemmas
• design tradeoffs
• hints for using analysis software
• things we haven't figured out how to do yet
• Leave ego at the door and laugh at ourselves

Context:By my count, since 1995 (my first ISERN meeting), we have discussed qualitative methods at least 5 times, covering basic concepts, validity concerns, and guidelines for reporting. Now that there is a critical mass of ISERN researchers who have at least tried some qualitative research, it's time to start sharing experiences.

Facilitators:Carolyn Seaman (again!), but this time with help from some friends. Let me know if you have experiences you'd like to share, or just bring them to the session!

Readings:None

Coffee Break

Roadmap

Chair: R. Selby

Goals: At ISERN 2007, we discussed and critiqued the proposed roadmap for Empirical Software Engineering (ESE). The goal for ISERN 2008 is to use the ESE roadmap to help identify collaboration opportunities and accelerate progress toward achieving broad ESE goals.

Context: We have 90 minutes for this roadmapping session, and we plan to use the time as follows:

• Give brief overview of the roadmapping results from last year (10 minutes)
• Describe a roadmapping exercise where people pair up and discuss potential research collaborations based on the roadmap (5 minutes)
• Pair-up people and discuss potential collaborations to accelerate progress (30 minutes)
• Map/populate these potential collaborations visually onto the roadmap graphic, which will reveal gaps and synergies/overlaps (30 minutes)
• Example of how our roadmapping influenced particular researchers (10 minutes)
• Closing discussion and actions for ISERN 2009 (5 minutes)

Facilitators:Rick Selby and Laurie Williams

Readings:

• Introduction
• Discussion and Summary

Lunch

ESE Methods: Empirical Investigation Using Simulation

Chair: Dan Port

Goals:Our objective for this session is to raise awareness and initiate the development of guidelines for appropriate application of simulation for acquiring empirical evidence and validation of inaccessible software engineering research conjectures, claims, and hypothesis. It is not our intent to explicitly discuss the details for design of simulation studies (perhaps a future session) although it is inevitable that some discussion of this will occur. We wish to make best use of the collection of experience and expertise of ISERN members in this very limited amount of time available.

Context:For many software engineering methods verifying the benefit of following a particular approach is a challenge. Intangible values such as benefit are difficult to measure directly, industry and student/classroom based experimental studies are generally impractical to use for large numbers of experiments to achieve statistical significance, and access to industry data is often limited. Simulation offers an attractive approach to generate empirical evidence for validating and investigating fundamental software engineering properties.

Facilitators:Jeff Carver, Juergen Muench, Dietmar Pfahl

Readings:

• Introduction

Coffee Break

Community

Chair: A. Jedlitschka

Goals:ISERN is a healthy, productive social organization that is stuck in 1990's technology (email + simple static website), and even that technology does not appear to be used all that well. Nevertheless, we get things done somehow. In the past few years, a great number of innovative social network technologies have appeared, including: MySpace, FaceBook, Twitter, FriendFeed, BlogSpot, MediaWiki, etc. The goal of this session is to ask two kinds of questions:

• (1) Could we improve the efficiency/effectiveness of ISERN by adopting one or more of these technologies? What would it mean to improve the "efficiency" or "effectiveness" of ISERN? How could we measure that? What would it mean to "adopt" one or more of these technologies?
• (2) Could adoption of these technologies improve software development processes and products? How could we study that?
• Social Networking Technologies for Software [Developers | Researchers]
• Repository of Experiments
• ISERN Portal "A walk-through"

Context: ISERN Portal Requirements

Facilitators:

Readings: The ISERN Web site

ISERN Business

Dieter Rombach

Open Discussion