Call for Artifacts

One of the challenges in the SEAMS community has been the scarcity of community resources (e.g., exemplars or model problems) that can be used to compare existing approaches or be built upon by other researchers. There is a community repository at http://self-adaptive.org/exemplars that contains some examples that have been discussed in prior SEAMS, but there is a need to populate this repository with additional examples.

To address this, SEAMS 2017 will continue the effort in previous editions by having a special artifacts session. Resources of interest include:

  • Exemplars, Case Studies, and Benchmarks, which are implementations of systems that can be used with multiple self-adaptive approaches.
  • Model Problems, which are descriptions of problems that pose and highlight fundamental or characteristic challenges in this community, and that self-adaptive systems should address.
  • Data repositories, which are data (e.g., logging data, system traces, survey raw data) useful in other studies.
  • Frameworks, which are tools and services illustrating new approaches to self-adaptation that could be used by other researchers in different contexts.

We solicit artifacts that can be used by researchers in our community in two modalities:

  • Stand-alone modality. Requires the submission of an artifact paper (up to 6 pages) including a synopsis or description of the problem that is being addressed, a description of the context(s) in which the resource would be useful, a list of the challenges that it poses for self-adaptation, and examples of its use in at least one area of self-adaptive systems. Accepted papers and artifacts will be included in the proceedings, and authors will be given an opportunity to present at SEAMS.
  • Research paper modality. The artifact complements a long research paper and does not require a separate paper submission. In this modality, the evaluation of the artifact and the review process for the research paper will be carried out independently, and the outcome of each of the processes will not affect that of the other.

In both modalities, submissions should also contain links (anonymous Dropbox link or direct download link) to the artifacts being discussed, which will be downloaded by the artifacts committee evaluators. Papers and accompanying artifacts will be evaluated for suitability for being used by the wider community.

Submission link: https://easychair.org/conferences/?conf=seamsicse2017

Quality Seal and Best Artifact Award

Papers with artifacts that meet or exceed expectations set will be awarded with a quality seal by the artifacts committee.

Moreover, there will be a best artifact award recognizing the work of authors who contribute the most useful artifact to the community.

Artifact submissions for quality seals and the best artifact award will be selected based on the insight, timeliness, usefulness, and usability of artifacts (see section on selection criteria).

Packaging Guidelines
When packaging your artifact, please keep in mind: a) how accessible you are making your artifact to other researchers, and b) the fact that the SEAMS evaluators will have very limited time in which to make an assessment of each artifact. The setup for your artifact should take less than 30 minutes or it is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it. To expedite the evaluation process, it may be appropriate to include a virtual machine image as part of your package.

You should make your artifact available as a single archive file using a widely available compressed archive format such as ZIP (.zip), tar and gzip (.tgz), or tar and bzip2 (.tbz2).

The archive must:

  1. be self-contained (with the exception of pointers to external tools or libraries; which we will not consider being part of the evaluated artifact, but which we will try to use when evaluating the artifact);
  2. contain a HTML file called index.html that fully describes the artifact and includes (relative) links to the files (included in the archive) that constitute the artifact;
  3. include a Getting Started Guide (a section within index.html, see below);
  4. include Step-by-Step Instructions (another section within index.html) for how you propose to evaluate your artifact;
  5. where appropriate, include descriptions of and links to files (included in the archive) that represent expected outputs (e.g., the log files expected to be generated by your tool on the given inputs).

The artifact may include, but is not limited to, code, executables, data, a virtual machine image, and documents. Please use open formats for documents and we prefer experimental data to be submitted in csv format.

Within your index.html, you must include a section with a basic Getting Started guide. Reviewers will follow all steps in the Getting Started guide at the beginning of the evaluation period and if necessary can relay questions if they run into difficulties. You should write your Getting Started guide to be as simple and straightforward as possible, and yet it should stress the key elements of your artifact. If well written, anyone who has successfully completed the Getting Started guide should not have any technical difficulties with the rest of your artifact. The Getting Started guide is your only opportunity to allow “debugging” of your artifact. Once the reviewers have completed this phase, there will be no further opportunity for interaction with you, the authors.

Selection Criteria
The artifact will be evaluated in relation to the expectations set by the paper. Thus, in addition to just running the artifact, the evaluators will read the paper and may try to tweak provided inputs and create new ones, to test the limits of the system.


Artifacts will be scored using the following criteria:

  • Insightful: Does the artifact address/identify a gap in previous work?
  • Timely: Does the artifact address a problem that is current/pressing?
  • Useful: Does it serve a useful purpose? Does it serve a purpose that would otherwise be tedious, prolonged, awkward, or impossible?
  • Usable: Is it easy to understand? Is it accompanied by tutorial notes/videos and other documentation? If the artifact is executable, is it easy to download, install, or execute? Does it include source code? (source code is desirable but not required). Is it available in a virtual machine image? Is it available online? Is it supported by configuration management tools to permit easy updates?

Submission
Note that there can be different kinds of artifacts, such as software tools, data sets (such as raw data from experiments), services, or anything else that may be relevant for community use. You may want to include multiple artifacts (e.g., a tool and the data set used in its evaluation) in your submission. Please bundle everything into a single archive file. Papers submitted in this track should contain a link for accessing the artifacts. After acceptance, authors will be encouraged to add an entry to https://www.hpi.uni-potsdam.de/giese/public/selfadapt/exemplars/.

Authors will also be invited to archive their accepted artifacts on the new Dagstuhl Artifacts Series (DARTS) published in the Dagstuhl Research Online Publication Server (DROPS). Each artifact will be assigned a DOI, separate from the companion paper, allowing the community to cite artifacts on their own.

Please perform your artifact submission via EasyChair, following the submission guidelines above. The submissions deadlines for abstracts and papers also apply to artifact paper submissions.

Optionally, authors are encouraged to submit a link to a short video (YouTube, max. 5 minutes) demonstrating the artifact.