About xBed
Reproducibility is one of the main principles of the scientific method,
and refers to the ability of a test or experiment to be accurately
reproduced, or replicated, by someone else working independently. The
purpose of the xBed Project
is to facilitate planning and execution of experiments with
combinatorial solvers -- experiments that are reproducible in the
context of the definition above. Typical measures of solver
performance are runtime and/or correlated combinatorial counts.
The reproducibility of the such experiments is ensured by repeatedly
invoking the solvers on data sets that represent classes of
isomorphs, where instances from the same isomorph class have been
derived from the same reference instance.
The principle schemas that create an xBed
environment include:
- a data sets
schema that organizes data sets into user-defined
hierarchy of
families, groups, categories, etc. -- but with the leaf instances
organized into two distinct but related subdirectories, with names that
are also xBed keywords: _REFERENCES
and _ISOMORPHS. Any instance
under _REFERENCES/ may induce a subdirectory of related isomorphs under
the subdirectory _ISOMORPHS/; click here
for an illustration. In addition, the illustration also displays
a subdirectory _RANDOM
(another xBed keyword) that is reserved to group a set of instances for
which we really do not how how they relate, except that we may find
them of the same 'size' or some other similar and easy-to-derive
parameters. For an example of an actual hierarchy of data sets, click here.
- a results schema that
organizes experimental results with combinatorial
solvers almost identically to the hierarchy that organizes
data sets on which the solvers were invoked. However, in place of
data sets, we now find new subdirectories that archive results
generated by specific solvers on these data sets under
_REFERENCES/,
_ISOMORPHS/, and _RANDOM/ ; click
here for an illustration of this schema. For an example of an
actual hierarchy of experimental results, click
here.
- a software
schema that hierarchically organizes generic xBed
software
utilities, combinatorial solver-specific encapsulation scripts, and
executables of combinatorial solvers themselves such that either
- each component may be invoked as a standalone program via a
command line, or
- a universal xBed client may
be invoked on a user-defined configuration file and execute a
series of comprehensive experiments in a sequence of well-defined
steps. Here, the
user-defined configuration file references any data sets, any subset
of component command-lines, and any location of subdirectories
where the results of experiments are being archived for each solver.
For additional context, click here.
This document has been initiated on Sat Feb 17 10:50:47 EST 2007.