Dekker1/ResearchMethods
Archived
1
0
Fork 0
Code

Merge remote-tracking branch 'origin/master'

Browse Source
This commit is contained in:
Jip J. Dekker 2018-05-25 12:14:13 +10:00
commit 7d635447d1
3 changed files with 175 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
mini_proj/report/references.bib
View File

@ -1,3 +1,10 @@
@misc{openData,
title={Open Database License (ODbL) v1.0},
url={https://opendatacommons.org/licenses/odbl/1.0/},
journal={Open Data Commons},
year={2018},
month={Feb}
}
@techreport{knn,
title={Discriminatory analysis-nonparametric discrimination: consistency properties},
author={Fix, Evelyn and Hodges Jr, Joseph L},

140
mini_proj/report/waldo.tex
View File

@ -6,8 +6,8 @@
\usepackage[justification=centering]{caption} % Used for captions
\captionsetup[figure]{font=small} % Makes captions small
\newcommand\tab[1][0.5cm]{\hspace*{#1}} % Defines a new command to use 'tab' in text
% Math package
\usepackage{amsmath}
\usepackage[comma, numbers]{natbib} % Used for the bibliography
\usepackage{amsmath} % Math package
% Enable that parameters of \cref{}, \ref{}, \cite{}, ... are linked so that a reader can click on the number an jump to the target in the document
\usepackage{hyperref}
%enable \cref{...} and \Cref{...} instead of \ref: Type of reference included in the link
@ -99,6 +99,16 @@
architectures, as this method is currently the most used for image
classification.
\textbf{
\\A couple of papers that may be useful (if needed):
- LeNet: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
- AlexNet: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks
- General comparison of LeNet and AlexNet:
"On the Performance of GoogLeNet and AlexNet Applied to Sketches", Pedro Ballester and Ricardo Matsumura Araujo
- Deep NN Architecture:
https://www-sciencedirect-com.ezproxy.lib.monash.edu.au/science/article/pii/S0925231216315533
}
\subsection{Classical Machine Learning Methods}
The following paragraphs will give only brief descriptions of the different
@ -130,6 +140,12 @@
\subsection{Neural Network Architectures}
\todo{Did we only do the three in the end? (Alexnet?)}
Yeah, we implemented the LeNet architecture, then improved on it for a fairly standar convolutional neural network (CNN) that was deeper, extracted more features, and condensed that image information more. Then we implemented a more fully convolutional network (FCN) which contained only one dense layer for the final binary classification step. The FCN added an extra convolutional layer, meaning the before classifying each image, the network abstracted the data more than the other two.
\begin{itemize}
\item LeNet
\item CNN
\item FCN
\end{itemize}
\paragraph{Convolutional Neural Networks}
@ -139,61 +155,95 @@
\section{Method} \label{sec:method}
\tab
In order to effectively utilize the aforementioned modelling and classification techniques, a key consideration is the data they are acting on.
A dataset containing Waldo and non-Waldo images was obtained from an Open Database\footnote{``The Open Database License (ODbL) is a license agreement intended to allow users to freely share, modify, and use [a] Database while maintaining [the] same freedom for others"\cite{openData}}hosted on the predictive modelling and analytics competition framework, Kaggle.
The distinction between images containing Waldo, and those that do not, was providied by the separation of the images in different sub-directories.
It was therefore necessary to preprocess these images before they could be utilised by the proposed machine learning algorithms.
\subsection{Image Processing}
\tab
The Waldo image database consists of images of size 64$\times$64, 128$\times$128, and 256$\times$256 pixels obtained by dividing complete Where's Waldo? puzzles.
Within each set of images, those containing Waldo are located in a folder called `waldo', and those not containing Waldo, in a folder called `not\_waldo'.
Since Where's Waldo? puzzles are usually densely populated and contain fine details, the 64$\times$64 pixel set of images were selected to train and evaluate the machine learning models.
These images provide the added benefit of containing the most individual images of the three size groups.
\\
\par
Each of the 64$\times$64 pixel images were inserted into a Numpy
\footnote{Numpy is a popular Python programming library for scientific computing}
array of images, and a binary value was inserted into a seperate list at the same index.
These binary values form the labels for each image (waldo or not waldo).
Colour normalisation was performed on each so that artefacts in an image's colour profile correspond to meaningful features of the image (rather than photographic method).
\\
\par
Each original puzzle is broken down into many images, and only contains one Waldo. Although Waldo might span multiple 64$\times$64 pixel squares, this means that the non-Waldo data far outnumbers the Waldo data.
To combat the bias introduced by the skewed data, all Waldo images were artificially augmented by performing random rotations, reflections, and introducing random noise in the image to produce news images.
In this way, each original Waldo image was used to produce an additional 10 variations of the image, inserted into the image array.
This provided more variation in the true positives of the data set and assists in the development of more robust methods by exposing each technique to variations of the image during the training phase.
\\
\par
Despite the additional data, there were still over ten times as many non-Waldo images than Waldo images.
Therefore, it was necessary to cull the no-Waldo data, so that there was an even split of Waldo and non-Waldo images, improving the representation of true positives in the image data set.
\\
% Kelvin Start
\subsection{Benchmarking}\label{benchmarking}
% Kelvin Start
\subsection{Benchmarking}\label{benchmarking}
In order to benchmark the Neural Networks, the performance of these
algorithms are evaluated against other Machine Learning algorithms. We
use Support Vector Machines, K-Nearest Neighbours (\(K=5\)), Gaussian
Naive Bayes and Random Forest classifiers, as provided in Scikit-Learn.
In order to benchmark the Neural Networks, the performance of these
algorithms are evaluated against other Machine Learning algorithms. We
use Support Vector Machines, K-Nearest Neighbours (\(K=5\)), Gaussian
Naive Bayes and Random Forest classifiers, as provided in Scikit-Learn.
\subsection{Performance Metrics}\label{performance-metrics}
\subsection{Performance Metrics}\label{performance-metrics}
To evaluate the performance of the models, we record the time taken by
each model to train, based on the training data and statistics about the
predictions the models make on the test data. These prediction
statistics include:
To evaluate the performance of the models, we record the time taken by
each model to train, based on the training data and statistics about the
predictions the models make on the test data. These prediction
statistics include:
\begin{itemize}
\item
\textbf{Accuracy:}
\[a = \dfrac{|correct\ predictions|}{|predictions|} = \dfrac{tp + tn}{tp + tn + fp + fn}\]
\item
\textbf{Precision:}
\[p = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|predicted\ as\ Waldo|} = \dfrac{tp}{tp + fp}\]
\item
\textbf{Recall:}
\[r = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|actually\ Waldo|} = \dfrac{tp}{tp + fn}\]
\item
\textbf{F1 Measure:} \[f1 = \dfrac{2pr}{p + r}\] where \(tp\) is the
number of true positives, \(tn\) is the number of true negatives,
\(fp\) is the number of false positives, and \(tp\) is the number of
false negatives.
\end{itemize}
\begin{itemize}
\item
\textbf{Accuracy:}
\[a = \dfrac{|correct\ predictions|}{|predictions|} = \dfrac{tp + tn}{tp + tn + fp + fn}\]
\item
\textbf{Precision:}
\[p = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|predicted\ as\ Waldo|} = \dfrac{tp}{tp + fp}\]
\item
\textbf{Recall:}
\[r = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|actually\ Waldo|} = \dfrac{tp}{tp + fn}\]
\item
\textbf{F1 Measure:} \[f1 = \dfrac{2pr}{p + r}\] where \(tp\) is the
number of true positives, \(tn\) is the number of true negatives,
\(fp\) is the number of false positives, and \(tp\) is the number of
false negatives.
\end{itemize}
Accuracy is a common performance metric used in Machine Learning,
however in classification problems where the training data is heavily
biased toward one category, sometimes a model will learn to optimize its
accuracy by classifying all instances as one category. I.e. the
classifier will classify all images that do not contain Waldo as not
containing Waldo, but will also classify all images containing Waldo as
not containing Waldo. Thus we use, other metrics to measure performance
as well.
Accuracy is a common performance metric used in Machine Learning,
however in classification problems where the training data is heavily
biased toward one category, sometimes a model will learn to optimize its
accuracy by classifying all instances as one category. I.e. the
classifier will classify all images that do not contain Waldo as not
containing Waldo, but will also classify all images containing Waldo as
not containing Waldo. Thus we use, other metrics to measure performance
as well.
\emph{Precision} returns the percentage of classifications of Waldo that
are actually Waldo. \emph{Recall} returns the percentage of Waldos that
were actually predicted as Waldo. In the case of a classifier that
classifies all things as Waldo, the recall would be 0. \emph{F1-Measure}
returns a combination of precision and recall that heavily penalises
classifiers that perform poorly in either precision or recall.
% Kelvin End
\emph{Precision} returns the percentage of classifications of Waldo that
are actually Waldo. \emph{Recall} returns the percentage of Waldos that
were actually predicted as Waldo. In the case of a classifier that
classifies all things as Waldo, the recall would be 0. \emph{F1-Measure}
returns a combination of precision and recall that heavily penalises
classifiers that perform poorly in either precision or recall.
% Kelvin End
\section{Results} \label{sec:results}
\section{Conclusion} \label{sec:conclusion}
\clearpage % Ensures that the references are on a seperate page
\pagebreak
% References
\section{References}
\renewcommand{\refname}{}
\bibliographystyle{alpha}
\bibliography{references}
\end{document}

77
mini_proj/waldo_model.py
View File

@ -25,7 +25,7 @@ from keras.utils import to_categorical
'''
Model definition define the network structure
'''
def FCN():
def CNN():
## List of model layers
inputs = Input((3, 64, 64))
@ -33,7 +33,6 @@ def FCN():
m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool1)
#drop1 = Dropout(0.2)(conv2) # Drop some portion of features to prevent overfitting
m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool2)
@ -47,13 +46,81 @@ def FCN():
drop3 = Dropout(0.2)(dense)
classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify
## Define the model structure
## Define the model start and end
model = Model(inputs=inputs, outputs=classif)
# Optimizer recommended Adadelta values (lr=0.01)
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1])
return model
'''
Model definition for a fully convolutional (no dense layers) network structure
'''
def FCN():
## List of model layers
inputs = Input((3, 64, 64))
conv1 = Conv2D(16, (3, 3), activation='relu', padding='same', input_shape=(64, 64, 3))(inputs)
m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool1)
m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool2)
drop2 = Dropout(0.2)(conv3) # Drop some portion of features to prevent overfitting
m_pool2 = MaxPooling2D(pool_size=(2, 2))(drop2)
conv4 = Conv2D(64, (2, 2), activation='relu', padding='same')(m_pool2)
flat = Flatten()(conv4) # Makes data 1D
drop3 = Dropout(0.2)(flat)
classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify
## Define the model start and end
model = Model(inputs=inputs, outputs=classif)
# Optimizer recommended Adadelta values (lr=0.01)
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1])
return model
'''
Model definition for the network structure of LeNet
Note: LeNet was designed to classify into 10 classes, but we are only performing binary classification
'''
def LeNet():
## List of model layers
inputs = Input((3, 64, 64))
conv1 = Conv2D(6, (5, 5), activation='relu', padding='valid', input_shape=(64, 64, 3))(inputs)
m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(16, (5, 5), activation='relu', padding='valid')(m_pool1)
m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
flat = Flatten()(m_pool2) # Makes data 1D
dense1 = Dense(120, activation='relu')(flat) # Fully connected layer
dense2 = Dense(84, activation='relu')(dense1) # Fully connected layer
drop3 = Dropout(0.2)(dense2)
classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify
## Define the model start and end
model = Model(inputs=inputs, outputs=classif)
model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1])
return model
'''
AlexNet architecture
'''
def AlexNet():
inputs = Input(shape=(3, 64, 64))
return model
def f1(y_true, y_pred):
def recall(y_true, y_pred):
"""Recall metric.
@ -110,14 +177,16 @@ lbl_train = to_categorical(lbl_train) # One hot encoding the labels
lbl_test = to_categorical(lbl_test)
## Define model
#model = CNN()
model = FCN()
#model = LeNet()
# svm_iclf = ImageClassifier(svm.SVC)
# tree_iclf = ImageClassifier(tree.DecisionTreeClassifier)
# naive_bayes_iclf = ImageClassifier(naive_bayes.GaussianNBd)
# ensemble_iclf = ImageClassifier(ensemble.RandomForestClassifier)
## Define training parameters
epochs = 10 # an epoch is one forward pass and back propogation of all training data
epochs = 25 # an epoch is one forward pass and back propogation of all training data
batch_size = 150 # batch size - number of training example used in one forward/backward pass
# (higher batch size uses more memory, smaller batch size takes more time)
#lrate = 0.01 # Learning rate of the model - controls magnitude of weight changes in training the NN