%************************************************
\chapter{Experiment Resources}%
\label{ch:benchmarks}
%************************************************

\noindent{}All experiments included in this thesis were conducted on a dedicated node in a computational cluster.
The machine operates using a \textbf{Intel Xeon 8260} \gls{cpu}, which has 24 non-hyperthreaded cores, and has access to \textbf{268.55 GB} of \gls{ram}.
Each experimental test was given exclusive access to a single \gls{cpu} core and access to sufficient \gls{ram}.

\section{Experimental Design}%
\label{sec:bench-env}

To promote the verification and recreation of our experiments, we have publish our experimental design in the form of several bundles.
Each bundle contains the exact versions of the software, models, and data that were used.
They are accompanied with instructions to perform the experiments.

The experiments are split into three bundles, each testing a different system.
Each part is published as a \gls{git} repository on GitHub.

\paragraph{MiniZinc Prototype} The design for the experiments of the \minizinc{} prototype implementation can be found in:

\begin{center}
	\url{file://TODO}
\end{center}

This repository contains the flattening, recursive function, and the incremental flattening benchmarks. 

\paragraph{Half Reification} The design for all the \gls{half-reif} experiments is located in:

\begin{center}
	\url{https://github.com/Dekker1/half-reif-benchmarks}
\end{center}

This repository contains both the benchmark for the \gls{chuffed} propagators of \gls{half-reif} constraints and the \minizinc{} Challenge 2019 \& 2020 benchmark.

\paragraph{Restarting Meta-Search} Finally, the design of the experiments for restart based \gls{meta-search} can be found in:

\begin{center}
	\url{https://github.com/Dekker1/on-restart-benchmarks}
\end{center}

This repository contains the benchmarks for both \gls{gecode} and \gls{chuffed}.


\section{Software}%
\label{sec:bench-soft}

Central to the experiments in this thesis are two kinds of programs: programs that rewrite a high-level \constraint{} model into a \solver{} specification and the \solvers{} that given a specification search for solutions.
We generally test the rewriting programs, in particular the \minizinc{} flattener, but use the \solvers{} to evaluate the effect of changes to these rewriting programs.

\subsection{MiniZinc Flattener}

In this thesis we use three different versions of the \minizinc\ flattening tool:

\begin{itemize}

	\item The officially released MiniZinc flattener (version 2.5.5) \autocite{minizinc-2021-minizinc}.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/MiniZinc/libminizinc/tree/2.5.5}
		\end{center}

	\item A prototype for the language architecture designed as part of this thesis.
		The source code for this version can be found here:
		\begin{center}
			\url{file://TODO}
		\end{center}

	\item The official MiniZinc flattener (version 2.5.5) \autocite{minizinc-2021-minizinc}, adjusted with the restart-based mechanisms.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/libminizinc/tree/feature/on_restart}
		\end{center}

\end{itemize}

\subsection{MiniZinc Solvers}

In this thesis we use the following \solvers{}:

\paragraph{Chuffed} is an open source \gls{lcg} \solver{} \cite{chuffed-2021-chuffed}.
In this thesis we use three different versions of the \gls{chuffed} \solver{}:

\begin{itemize}

	\item The official development version of \gls{chuffed}. 
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/chuffed/chuffed/tree/develop}
		\end{center}
	
	\item An extension of \gls{chuffed} that includes propagators for \glspl{half-reif} of the \mzninline{all_different} and \mzninline{element} \constraints{}.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/chuffed/tree/feature/imp_globals}
		\end{center}

	\item An extension of \gls{chuffed} that includes propagators for restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/chuffed/tree/feature/on_restart}
		\end{center}

\end{itemize}

\paragraph{Coin-OR Branch-and-Cut} is an open source \gls{mip} \solver{} \autocite{forrest-2020-cbc}.
In this thesis we use version 2.10.5 of this solver.
Its source code is available at

		\begin{center}
			\url{https://github.com/coin-or/Cbc/tree/releases/2.10.5}
		\end{center}

\paragraph{Gecode} is an open source \gls{cp} solver \autocite{gecode-2021-gecode}.
In this thesis we use four different versions of the \gls{gecode} solver:
\begin{itemize}

	\item The official release of \gls{gecode}, version 6.3.0.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Gecode/gecode/tree/release-6.3.0}
		\end{center}
	
	\item An extension of \gls{gecode} that includes propagators for restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart}
		\end{center}

	\item An extension of \gls{gecode} that is able to record the search process of restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart_record}
		\end{center}

	\item An extension of \gls{gecode} that is able to replay the search process of restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart_replay}
		\end{center}

\end{itemize}

\paragraph{IBM CPLEX} is a proprietary \gls{mip} \solver{} \autocite{cplex-2020-cplex}.
In this thesis we use \gls{cplex} version 20.1 under an academic license in our experiments.

\paragraph{OpenWBO} is a open source \gls{maxsat} \solver{} \autocite{martins-2014-openwbo}.
In this thesis we use version \jip{xxx} in our experiments.
The source code for this version can be found here:

\begin{center}
	\url{https://github.com/sat-group/open-wbo}
\end{center}


\section{MiniZinc Models}%
\label{sec:bench-models}

In the following paragraphs, we introduce the origins of the models used in the experiments of this thesis.

\paragraph{MiniZinc Challenge} Most of the models used in the experiments in this thesis stem from the \minizinc{} Challenge \autocite{stuckey-2010-challenge,stuckey-2014-challenge}.
The \minizinc{} Challenge is a yearly competition, held since 2008, in which \minizinc{} \solvers{} compete against each other in solving 100 \minizinc{} instances.
\Glspl{solver} are scored based on their performance solving each instance: time to solution, quality of solution, and proof of optimality.
The 100 instances stem from 20 \minizinc{} models, 10 of which are guaranteed to have never been used in earlier challenges.

A repository of all \minizinc{} models previously used in the challenges and accompanying data files can be found here:

\begin{center}
	\url{https://github.com/minizinc/minizinc-benchmarks}
\end{center}

The flattening experiment presented in \cref{sec:rew-experiments} use a selection of seventeen models originating from the \minizinc{} challenge.
These models can identified using the following labels:

\begin{multicols}{3}
	\ttfamily
	\begin{itemize}
		\item accap
		\item amaze
		\item city-position
		\item community-detection
		\item depot-placement
		\item freepizza
		\item groupsplitter
		\item kidney-exchange
		\item multi-knapsack
		\item nonogram
		\item nside
		\item rcpsp-wet
		\item road-cons
		\item roster
		\item stack-cuttingstock
		\item steelmillslab
		\item triangular
	\end{itemize}
\end{multicols}

The large scale \gls{half-reif} experiment presented in \cref{sec:half-experiments} uses the \minizinc{} instances from 2019 and 2020.
The 2019 \minizinc{} Challenge included the models identified by the following labels:

\begin{multicols}{3}
	\ttfamily
	\begin{itemize}
		\item accap
		\item amaze
		\item code-generator
		\item fox-geese-corn 
		\item groupsplitter
		\item hrc
		\item kidney-exchange
		\item {\scriptsize{} liner-sf-repositioning}
		\item lot-sizing
		\item median-string
		\item multi-knapsack
		\item nside
		\item ptv
		\item rcpsp-wet-diverse
		\item rotating-workforce
		\item stack-cuttingstock
		\item steelmillslab
		\item stochastic-vrp
		\item triangular
		\item zephyrus
	\end{itemize}
\end{multicols}

\noindent{}The 2020 \minizinc{} Challenge included the models identified by the following labels:

\begin{multicols}{3}
	\ttfamily
	\begin{itemize}
		\item bnn-planner
		\item cable\_tree\_wiring
		\item code-generator
		\item {\scriptsize{} collaborative-construction}
		\item gbac
		\item hoist-benchmark
		\item is
		\item lot-sizing
		\item {\scriptsize{} minimal-decision-sets}
		\item p1f-pjs
		\item pentominoes
		\item pillars-and-planks
		\item racp
		\item radiation
		\item sdn-change 
		\item skill-allocation
		\item soccer-computational
		\item stable-goods
		\item tower\_challenge
		\item whirlpool
	\end{itemize}
\end{multicols}

The incremental experiments presented in \cref{sec:inc-experiments} are based on four models from the \minizinc{} Challenge:

\begin{multicols}{2}
	\ttfamily
	\begin{itemize}
		\item gbac
		\item radiation
		\item steelmillslab
		\item rcpsp-wet
	\end{itemize}
\end{multicols}


\paragraph{Prize Collecting Path} This problem was introduced in by Feydy et al.\ in the original paper on \gls{half-reif} \autocite{feydy-2011-half-reif}.
In this thesis it is used to benchmark the implementation of a propagator for the \gls{half-reif} of the \mzninline{element} constraint.
This experiment is shown in \cref{sec:half-experiments}.
\Cref{lst:bench-prize} shows a \minizinc{} model that can be used to solve the problem.
The data and original model are available at: 

\begin{center}
	\url{https://people.eng.unimelb.edu.au/pstuckey/half/}
\end{center}

\begin{listing}
	\mznfile{assets/mzn/prize.mzn}
	\caption{\label{lst:bench-prize} A \minizinc{} model for the Prize Collecting Path problem}
\end{listing}

\paragraph{QCP-Max} This problem was introduced in by Feydy et al.\ in the original paper on \gls{half-reif} \autocite{feydy-2011-half-reif}.
In this thesis it is used to benchmark the implementation of a propagator for the \gls{half-reif} of the \mzninline{all_different} constraint.
This experiment is shown in \cref{sec:half-experiments}.
\Cref{lst:bench-qcpmax} shows a \minizinc{} model that can be used to solve the problem.
The data and original model are available at: 

\begin{center}
	\url{https://people.eng.unimelb.edu.au/pstuckey/half/}
\end{center}

\begin{listing}
	\mznfile{assets/mzn/qcp_max.mzn}
	\caption{\label{lst:bench-qcpmax} A \minizinc{} model for the QCP-Max problem}
\end{listing}