14th International Software Engineering Research Network Annual Meeting

(ISERN 2006)

September 18-19


ISERN 2006 is part of, and co–located with
The 2006 Experimental Software Engineering International Week (ESEIW 2006)

ISERN Co-Chairs

Prof. Markku Oivo

University of Oulu, Finland

Prof. Dag Sjøberg

Simula Research Laboratory and University of Oslo, Norway


ISERN Meeting Registration

List of Attendees

ISERN is a community that believes software engineering research needs to be performed in an experimental context. ISERN annual meetings are open for ISERN members, candidates and invited observers only (see meetings at ISERN Manifesto). ISERN meetings are not conference style with refereed papers and presentations. Instead, meetings build on previous meetings and each session is supposed to foster collaboration, encourage discussions and contribute in building the knowledge in experimental software engineering. The continuous knowledge building will be formalized in ISERN Experience Factory. The sessions in this year’s meeting are categorized in methodology oriented and industry oriented sessions.

Please note that each session will have reading material that participants are expected to read in advance. See session descriptions below.



Monday, Setember 18th




S1: New participant introductions

(Chair: Markku Oivo)

Context: ISERN is open to academic and industrial groups world-wide that are active in experimental software engineering research and willing to adopt the experimental research framework. ISERN members are pairs of organization and contact person. If the contact person leaves the organisation, the organisation must reapply for membership.  Interested organisations may apply by sending an electronic proposal to “isern at informatik.uni-kl.de” describing their past experience in experimental software engineering research as well as their expectations from a future ISERN membership. Candidates will be invited to observe the ISERN Meeting following their application.


Goal of the session: To facilitate the membership application process by giving an opportunity for candidates to present their research and for observers to introduce themselves.


Session Format:  Membership is granted according to a 3-step procedure:

1.    Attending as invited observer at an annual ISERN meeting.

2.    Attending as invited candidate at the following ISERN meeting giving a presentation. Membership is granted if  2/3 majority of  current members approve the application in an email voting after the meeting.

3.    Attending as a full ISERN member all following meetings.


Candidates give a 5 min presentation each:

Microsoft Research, Nachiappan Nagappan

Osaka University, Shinji Kusumoto

Politecnico di Torino, Maurizio Morisio


Observers give 2 min introduction without a presentation:

University of Auckland, Emilia Mendes

Universidad ORT, Martin Solari


S2: Empirical Research

(Chair: Dieter Rombach)

Context: Previous ISERN sessions at ISERNs have focused on Guidelines for Documenting Empirical Results, but these guidelines have not been accepted and applied broadly. At the same time we still face problems getting empirical results published in journals within the larger software engineering community ”

Goal of the session: Identify guidelines for getting empirical studies published. Input are the existing guidelines from previous ISERN 2005, the discussion that happened at a recent Dagstuhl Workshop on Empirical Software Engineering (2006), and an invited presentation by Basili & Elbaum (Better Empirical Science for Software Engineering) at ICSE 2006 in Shanghai. In light of this input we will then conduct group work aimed at applying the guidelines from ISERN 2005 to already published or pre-published studies. These experiences will be fed into a rework of these guidelines.

 Session format:

- Introduction on the goals of the session

- Short summaries of the Dagstuhl and ICSE Feedback.

- Group work on applying the guidelines to example studies

- Discussion of findings from group work & rework of guidelines

- (Summarize contributions from the session for ISERN experience factory) – post meeting

Reading Materials:

“Slides (Basili & Elbaum) from ICSE 2006”

(MORE later)



Coffee Break


S2: Empirical Research

(Chair: Dieter Rombach)





S3: Data ownership

(Chair: Marv Zelkowitz)

Context: Previous ISERN sessions in 2004 and 2005 discussed issues concerning data ownership and how experimental software engineering data can be shared. Since ISERN 2005 a series of protocols have been developed for understanding data ownership issues. Both creators of empirical data and users of empirical data need a set of guidelines on the use of this data. No one set of rules works in all situations. Properties such as data ownership, privacy concerns, transfer to a third party, publication rights, and costs for using or maintaining the data all need to be spelled out in order to create a market in empirical software engineering data.

Goal of the session: In this laboratory session, attendees will apply the data ownership protocols to test their applicability in a variety of settings. The goal is to uncover any remaining attributes still undefined by the protocols.


Session format:

- Introduction on the goals of the session.

- Brief overview of the data ownership attributes.

- Group will divide into 5 or 6 subgroups. Each subgroup will apply the data ownership attributes to a different data set

- Each subgroup will give a short presentation to the larger group on how effective the data ownership attributes were to describing data usage rules.

- General discussion on the effectiveness of the data ownership rules to the community at large.


Reading Materials:

Protocols in the use of Empirical Software Engineering Artifacts” by M. Zelkowitz, V. Basili, D. Sjøberg, P. Johnson and T. Cowling.










S4: Experimentation and Decision making in Software Engineering: How they are related?

(Chair: Andreas Jedlitschka)

Context: Experimentation is always a means to better understand, control and improve software processes, products and technologies. Decisions are the crystallization point is being successful in software projects. This session is addressing the relationship between experimentation and decision-making and how synergies can be made.

Emphasis is made to look at this relationship form both a research and industry perspective.


Goal of the session:

-   Identify the relationship between experimentation and decision-making from an industrial and research perspective.

-   Provide examples how questions of decision-making have helped to run more focused experiments and how results of experimentation have helped to make better decisions.

-   Add to the guidelines for empirical research.


Session format:

- Introduction and goals of the session

- Panel discussion with initial position statements of

Natalia Juristo (Univ Politecnica de Madrid)

Nachi Nagappan (Microsoft)

Forrest Shull (Fraunhofer-Center Maryland)

Claes Wohlin (Blekinge Institute of Technology)

- Open discussion

- Summary and outlook


Reading Material:

The topic is based and linked to some former and two forthcoming ISERN sessions:

- 2003 session titled: Empirical Study of Decisions in Software Engineering and Project Management (organized by Stefan Biffl)

- 2004/05: Guidelines for empirical research (addressed decisions to some extend).

- 2006: link to the sessions

S6: Good practices for empirical research with industry (Jyrki Kontio) and

S8: Value-Based Empirical Research (Stefan Biffl)

 Paper “Software Engineering Decision Support and Empirical Investigations – A Proposed Marriage”, WSESE 2003

(b) Reading material

(c) Slides



Coffee Break


Wrap-up of parallel sessions


S5: ISERN Experience Factory

(Chair: Marcus Ciolkowski)

Context: The community has recognized the importance of having a common experience factory.


Goal of the session: discuss (and agree on)  experience factory processes and identify community contribution


Session format:
-Introduction on the goals of the session
-Short presentation of what we have, the ISERN EF vision, and summary of issues
-Group work on identifying how the community can / is willing to contribute
-Discussion of findings from group work
-Summarize contributions from the session for ISERN experience factory

Reading Material: ISERN Experience Factory set of slides




Closing and planning for day 2

20:00-23:00 ISERN DINNER (Barra Brasa Leblon - Transportation will be available for ISERN participants )

Tuesday, Setember 19th


S6: Good practices for empirical research with industry

(Chair: Jyrki Kontio)

Context: Cooperation with industry is a central element in empirical software engineering research. Due to time and cost constraints, it is challenging to attract industry to cooperate in empirical research. The empirical software engineering community will benefit from sharing good practices in establishing and mainintaing such cooperation.


Goal of the session: identify and refine good strategies and practices in industry university cooperation in empirical software engineering.


Session format:

- conclusions from a panel at CSEET (http://www.sbl.tkk.fi/CSEET06panel/) are presented

- participants are divided into teams to evaluate and refine the proposed good practices

- results are summarized and reviewed

- ownership of writing a "Good practice guide on empirical research with industry" is allocated


Reading material to be posted later.


10:45-11:15 Coffee Break

S7: Generalization from empirical studies

(Chair: Dag Sjøberg)

Context: Previous ISERN sessions at ISERN 2004 (“Empirical theory”) and 2005  on “Generalization from empirical studies”,

Goal of the session: Discuss the complex issues of generalising from empirical studies and hopefully agree on certain strategies or directions for future work

 Session format:

- Introduction on the challenges and goals of the session and short description of the topics to be discussed in each of the three discussion groups (15 min)

- Group work on respectively theory (moderator: Tore Dybå), statistical generalization (moderator: Dag Sjøberg) and aggregation of evidence (moderator: Marcus Ciolkowski) (45 min)

- Summarize discussion from the group work in the session (30 min)


Reading Materials:

“Generalization Session ISERN06 Preliminary Slides”

 M. Jørgensen and D. I.K. Sjøberg. Generalization and Theory-Building in Software Engineering Research, In: Empirical Assessment in Software Engineering (EASE2004). IEE Proceedings, pages 29–36, 2004.


Previous ISERN session slides:

 ISERN 2004 Empirical theory

 ISERN 2005: Generalization from empirical studies







S8: “Value-Based Empirical Research” (with industry focus)

(Chair: Stefan Biffl)

Context: Previous ISERN sessions at ISERNs 2003 to 2005  on “Need-driven Empirical Research”, “Value of empirical work” , and “Value-based Empirical Studies”

Goal of the session: Identify typical industry stakeholders and their expectations of value contributions from empirical studies; look at recent empirical studies and their value contributions. From the group work and discussion researchers in the audience should become more aware on the potential value contribution of their empirical studies and on risks to consider when planning an empirical study targeting industry stakeholders.


Session format:

- Introduction on the goals of the session

- Short issue summaries of senior researchers on their experience with empirical research with industry stakeholders and typical value propositions.

- Group work on identifying stakeholders in empirical research and value contribution (with industry focus)

- Discussion of findings from group work; simplifiers and complicators in aligning empirical research with the industrial stakeholders’ value propositions

- Summarize contributions from the session for ISERN experience factory and a roadmap (conclusions, guidelines, lessons learned).


Reading Materials:

“Value-Based Empirical Research Session ISERN06 Preliminary Slides ”]

"Value-based Software Engineering" (VBSE) book chapter 6 (key elements of VBSE, get started with section 6.9)

- Previous ISERN session slides :

“Need-driven Empirical Research” ISERN 2003

“Value of empirical work” ISERN 2004

“Value-based Empirical Studies” ISERN 2005 


12:45-13:45 Lunch

Wrap-up of parallel sessions


S9: ISERN business

(Dieter Rombach)

This session addresses issues related to future activities (e.g., next meetings, cooperation projects) and further improvements of ISERN. Concrete proposals are a presentation by next year's organizers of the meeting in Spain, the updating of our interrelation matrix, and a critical review of the 2006 meeting. In addition, the decisions of the Steering Committee will be presented.  Additional issues may be suggested by ISERN members.


S10: Open Forum

16:00 - 17:00

ISERN 2006 Adjourned with refreshments (Happy hour)