==+== ===================================================================== ==+== Begin Review ==+== Reviewer: Gurbinder Gill email@example.com
==+== Paper #93 ==-== Title: High Performance Stencil Code Generation with LIFT
==+== A. Artifact publicly available?
==-== Are all artifacts related to this paper are publicly available?
==-== Note that it is not obligatory to make artifacts publicly
==-== The author-created artifacts relevant to this paper will receive ==-== an ACM "artifact available" badge only if they have been placed ==-== on a publically accessible archival repository. ==-== Choices: ==-== 1. Publicly available ==-== 2. Not publicly available ==-== Enter the number of your choice:
==+== B. Artifact functional?
==-== * Package complete?
==-== * All components relevant to evaluation are included in the ==-== package? ==-==
==-== * Well documented? ==-==
==-== * Enough to understand, install and evaluate artifact? ==-==
==-== * Exercisable? ==-==
==-== * Includes scripts and/or software to perform appropriate ==-== experiments and generate results? ==-==
==-== * Consistent? ==-==
==-== * Artifacts are relevant to the associated paper and contribute in ==-== some inherent way to the generation of its main results? ==-==
==-== The artifacts associated with the paper will receive an "Artifacts ==-== Evaluated - Functional" badge only if they are found to be ==-== documented, consistent, complete, exercisable, and include ==-== appropriate evidence of verification and validation. ==-== Choices: ==-== 1. Exceeded expectations ==-== 2. Met expectations ==-== 3. Fell below expectations ==-== Enter the number of your choice:
==+== C. Artifact customizable and reusable?
==-== Can this artifact and experimental workflow be easily reused and
==-== customized? For example, can it be used on a different platform,
==-== with different benchmarks, data sets, compilers, tools, under
==-== different conditions and parameters, etc.?
==-== The artifacts associated with the paper will receive an "Artifact ==-== Evaluated - Reusable" badge only if they are of a quality that ==-== significantly exceeds minimal functionality. That is, they have ==-== all the qualities of the Artifacts Evaluated - Functional level, ==-== but, in addition, they are very carefully documented and ==-== well-structured to the extent that reuse and repurposing are ==-== facilitated. In particular, norms and standards of the research ==-== community for artifacts of this type are strictly adhered to. ==-== Choices: ==-== 1. Exceeded expectations ==-== 2. Met expectations ==-== 3. Fell below expectations ==-== Enter the number of your choice:
==+== D. Results validated?
==-== Can all main results from the paper be validated using provided
==-== Report any unexpected artifact behavior (depends on the type of
==-== artifact such as unexpected output, scalability issues, crashes,
==-== performance variation, etc).
==-== The artifacts associated with the paper will receive a "Results ==-== replicated" badge only if the main results of the paper have ==-== been obtained in a subsequent study by a person or team other than ==-== the authors, using, in part, artifacts provided by the author. ==-==
==-== Note that variation of empirical and numerical results is ==-== tolerated. In fact it is often unavoidable in computer systems ==-== research - see "how to report and compare empirical results?" in ==-== AE FAQ! ==-== Choices: ==-== 1. Exceeded expectations ==-== 2. Met expectations ==-== 3. Fell below expectations ==-== Enter the number of your choice:
==+== E. Comments for author ==-== Markdown styling and LaTeX math supported.
- Some of the path variables were hardcoded in the scripts, for example, DATADIR and OpenCL paths in the benchmarks were hard coded.
==+== F. Comments for PC ==-== Hidden from authors. ==-== Markdown styling and LaTeX math supported.
I used the following configuration for the artifact evaluation:
Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
GPU: Tesla K80
The documentation provided with the artifact was thorough and made the evaluation process very easy.
Authors did a very good job specifying the workflow, downloading input data set, setting up the required 3rd party platforms and running experiments to produce the results and plots used in the paper.
Results were consistent with those reported in the paper.
==+== Scratchpad (for unsaved private notes)
==+== End Review