%************************************************
\chapter{Experiment Resources}%
\label{ch:benchmarks}
%************************************************

\noindent{}All experiments included in this thesis were conducted on a dedicated node in a computational cluster.
The machine operates using a \textbf{Intel Xeon 8260} \gls{cpu}, which has 24 non-hyperthreaded cores, and has access to \textbf{268.55 GB} of \gls{ram}.
Each experimental test was given exclusive access to a single \gls{cpu} core and access to sufficient \gls{ram}.

\section{Experimental Design}%
\label{sec:bench-env}

To promote the verification and recreation of our experiments, we have publish our experimental design in the form of several bundles.
Each bundle contains the exact versions of the software, models, and data that were used.
They are accompanied with instructions to perform the experiments.

The experiments are split into three bundles, each testing a different system.
Each part is published as a \gls{git} repository on GitHub.

\paragraph{MiniZinc Prototype} The design for the experiments of the \minizinc{} prototype implementation can be found in:

\begin{center}
	\url{file://TODO}
\end{center}

This repository contains the flattening, recursive function, and the incremental flattening benchmarks. 

\paragraph{Half Reification} The design for all the \gls{half-reif} experiments is located in:

\begin{center}
	\url{https://github.com/Dekker1/half-reif-benchmarks}
\end{center}

This repository contains both the benchmark for the \gls{chuffed} propagators of \gls{half-reif} constraints and the \minizinc{} Challenge 2019 \& 2020 benchmark.

\paragraph{Restarting Meta-Search} Finally, the design of the experiments for restart based \gls{meta-search} can be found in:

\begin{center}
	\url{https://github.com/Dekker1/on-restart-benchmarks}
\end{center}

This repository contains the benchmarks for both \gls{gecode} and \gls{chuffed}.


\section{Software}%
\label{sec:bench-soft}

Central to the experiments in this thesis are two kinds of programs: programs that rewrite a high-level \constraint{} model into a \solver{} specification and the \solvers{} that given a specification search for solutions.
We generally test the rewriting programs, in particular the \minizinc{} flattener, but use the \solvers{} to evaluate the effect of changes to these rewriting programs.

\subsection{MiniZinc Flattener}

In this thesis we use three different versions of the \minizinc\ flattening tool:

\begin{itemize}

	\item The officially released MiniZinc flattener (version 2.5.5) \autocite{minizinc-2021-minizinc}.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/MiniZinc/libminizinc/tree/2.5.5}
		\end{center}

	\item A prototype for the language architecture designed as part of this thesis.
		The source code for this version can be found here:
		\begin{center}
			\url{file://TODO}
		\end{center}

	\item The official MiniZinc flattener (version 2.5.5) \autocite{minizinc-2021-minizinc}, adjusted with the restart-based mechanisms.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/libminizinc/tree/feature/on_restart}
		\end{center}

\end{itemize}

\subsection{MiniZinc Solvers}

In this thesis we use the following \solvers{}:

\paragraph{Chuffed} is an open source \gls{lcg} \solver{} \cite{chuffed-2021-chuffed}.
In this thesis we use three different versions of the \gls{chuffed} \solver{}:

\begin{itemize}

	\item The official development version of \gls{chuffed}. 
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/chuffed/chuffed/tree/develop}
		\end{center}
	
	\item An extension of \gls{chuffed} that includes propagators for \glspl{half-reif} of the \mzninline{all_different} and \mzninline{element} \constraints{}.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/chuffed/tree/feature/imp_globals}
		\end{center}

	\item An extension of \gls{chuffed} that includes propagators for restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/chuffed/tree/feature/on_restart}
		\end{center}

\end{itemize}

\paragraph{Coin-OR Branch-and-Cut} is an open source \gls{mip} \solver{} \autocite{forrest-2020-cbc}.
In this thesis we use version 2.10.5 of this solver.
Its source code is available at

		\begin{center}
			\url{https://github.com/coin-or/Cbc/tree/releases/2.10.5}
		\end{center}

\paragraph{Gecode} is an open source \gls{cp} solver \autocite{gecode-2021-gecode}.
In this thesis we use four different versions of the \gls{gecode} solver:
\begin{itemize}

	\item The official release of \gls{gecode}, version 6.3.0.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Gecode/gecode/tree/release-6.3.0}
		\end{center}
	
	\item An extension of \gls{gecode} that includes propagators for restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart}
		\end{center}

	\item An extension of \gls{gecode} that is able to record the search process of restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart_record}
		\end{center}

	\item An extension of \gls{gecode} that is able to replay the search process of restart-based meta-search.
		The source code for this version can be found here:
		\begin{center}
			\url{https://github.com/Dekker1/gecode/tree/feature/on_restart_replay}
		\end{center}

\end{itemize}

\paragraph{IBM CPLEX} is a proprietary \gls{mip} \solver{} \autocite{cplex-2020-cplex}.
In this thesis we use \gls{cplex} version 20.1 under an academic license in our experiments.

\paragraph{OpenWBO} is a open source \gls{maxsat} \solver{} \autocite{martins-2014-openwbo}.
In this thesis we use version \jip{xxx} in our experiments.
The source code for this version can be found here:

\begin{center}
	\url{https://github.com/sat-group/open-wbo}
\end{center}


\section{MiniZinc Models}%
\label{sec:bench-models}

\paragraph{MiniZinc Challenge} 

\paragraph{Prize Collecting Path} This problem was introduced in by Feydy et al.\ in the original paper on \gls{half-reif} \autocite{feydy-2011-half-reif}.
In this thesis it is used to benchmark the implementation of a propagator for the \gls{half-reif} of the \mzninline{element} constraint.
This experiment is shown in \cref{sec:half-experiments}.
\Cref{lst:bench-prize} shows a \minizinc{} model that can be used to solve the problem.
The data and original model are available at: 

\begin{center}
	\url{https://people.eng.unimelb.edu.au/pstuckey/half/}
\end{center}

\begin{listing}
	\mznfile{assets/mzn/prize.mzn}
	\caption{\label{lst:bench-prize} A \minizinc{} model for the Prize Collecting Path problem}
\end{listing}

\paragraph{QCP-Max} This problem was introduced in by Feydy et al.\ in the original paper on \gls{half-reif} \autocite{feydy-2011-half-reif}.
In this thesis it is used to benchmark the implementation of a propagator for the \gls{half-reif} of the \mzninline{all_different} constraint.
This experiment is shown in \cref{sec:half-experiments}.
\Cref{lst:bench-qcpmax} shows a \minizinc{} model that can be used to solve the problem.
The data and original model are available at: 

\begin{center}
	\url{https://people.eng.unimelb.edu.au/pstuckey/half/}
\end{center}

\begin{listing}
	\mznfile{assets/mzn/qcp_max.mzn}
	\caption{\label{lst:bench-qcpmax} A \minizinc{} model for the QCP-Max problem}
\end{listing}

