Clockwork RNN
-This project is an implementation of the Clockwork RNN
-(see paper).
-People : Paul Mustière, Pandav2 aka David Panou
-Organization : UPMC - Master Data Science
-Clockwork-RNN
-The model can be found under models/clockwork_rnn.py.
-The current main.py replicates the sequence generation task
-described in the paper, but the implementation should be able
-to handle other tasks.
-To monitor the training, you can use TensorBoard:
-
tensorboard --reload_interval 2 --logdir log
-
-Results
-The following picture show the obtain MSE loss obtain on the generation task
-
\ No newline at end of file
diff --git a/README.md b/README.md
index d85b0ff..0d4344b 100644
--- a/README.md
+++ b/README.md
@@ -3,6 +3,8 @@
This project is an implementation of the Clockwork RNN
(see [paper](https://arxiv.org/abs/1402.3511)).
+See [this section](#dependencies) for dependencies.
+
### People : Paul Mustière, Pandav2 aka David Panou
### Organization : UPMC - Master Data Science
@@ -27,17 +29,33 @@ with similar numbers of parameters.
The following table summarizes the results:
| Number of parameters | Clockwork RNN (MSE) | LSTM (MSE) |
-|:--------------------:|:-------------------:|------------|
-| ~70 | 4.3e-2 | 1.5e-1 |
-| ~480 | 3.4e-3 | 1.0e-1 |
-| ~800 | 1.8e-3 | 9.3e-2 |
+|:--------------------:|:-------------------:|:----------:|
+| ~70 | 5.0e-2 | 1.4e-1 |
+| ~480 | 2.8e-3 | 8.2e-2 |
+| ~800 | 1.7e-3 | 8.8-2 |
They were obtained with a learning rate of 0.01 for Clockwork RNN,
-and 0.001 for LSTM (which was more unstable).
+and 0.001 for LSTM (which was more unstable). Please beware due to hardward
+constraints, those results are not average over several runs.
+
+This table shows the dimension of the hidden state used and the resulting exact
+number of parameters:
+
+| Number of parameters | Clockwork RNN (Hidden dimension / Total parameters) | LSTM (Hidden dimension / Total parameters) |
+|:--------------------:|:---------------------------------------------------:|:------------------------------------------:|
+| ~70 | 9 / 73 | 3 / 64 |
+| ~480 | 27 / 469 | 10 / 491 |
+| ~800 | 36 / 802 | 13 / 794 |
+
+The following graph shows the MSE loss during training, scaled by epochs:
+
+
-The following graph shows the MSE loss during training:
+The next graph shows the same MSE loss, without the scaling by epochs.
+It shows that Clockwork RNN are indeed faster to train since they require
+less computations.
-
+
The following plot shows the generated signal (and the target)
for the best performing Clockwork RNN:
@@ -45,3 +63,13 @@ for the best performing Clockwork RNN:
Generated signals for other models can be found under `results/`
+
+## Dependencies
+
+- NumPy
+- TensorFlow (tested with 0.12.1)
+- Matplotlib
+
+For generating sequences from WAV files:
+
+- SciPy
diff --git a/article/.DS_Store b/article/.DS_Store
deleted file mode 100644
index 5008ddf..0000000
Binary files a/article/.DS_Store and /dev/null differ
diff --git a/article/bib.aux b/article/bib.aux
deleted file mode 100644
index 0b9e033..0000000
--- a/article/bib.aux
+++ /dev/null
@@ -1,38 +0,0 @@
-\relax
-\citation{srn}
-\citation{vanishing}
-\select@language{french}
-\@writefile{toc}{\select@language{french}}
-\@writefile{lof}{\select@language{french}}
-\@writefile{lot}{\select@language{french}}
-\@writefile{toc}{\contentsline {section}{\numberline {1}Abstract}{1}}
-\@writefile{toc}{\contentsline {section}{\numberline {2}Context}{1}}
-\@writefile{toc}{\contentsline {subsection}{\numberline {2.1}Recurrent Neural Networks}{1}}
-\@writefile{lomycapequ}{\contentsline {mycapequ}{\numberline {1}{\ignorespaces Hidden unit update in SRN\relax }}{1}}
-\citation{lstm}
-\citation{susts}
-\citation{grn}
-\citation{empir}
-\citation{hf_opti}
-\@writefile{toc}{\contentsline {subsection}{\numberline {2.2}LSTM}{2}}
-\@writefile{toc}{\contentsline {section}{\numberline {3}Clockwork RNN}{2}}
-\citation{bptt-1}
-\citation{bptt-2}
-\citation{rnn-mikolov}
-\citation{cache_models}
-\citation{gym}
-\bibcite{vanishing}{1}
-\@writefile{toc}{\contentsline {section}{\numberline {4}CW-RNN training}{3}}
-\@writefile{toc}{\contentsline {section}{\numberline {5}Project roadmap}{3}}
-\@writefile{toc}{\contentsline {subsection}{\numberline {5.1}Implementation}{3}}
-\@writefile{toc}{\contentsline {subsection}{\numberline {5.2}Reinforcement Learning}{3}}
-\bibcite{lstm}{2}
-\bibcite{srn}{3}
-\bibcite{suts}{4}
-\bibcite{grn}{5}
-\bibcite{empir}{6}
-\bibcite{hf_opti}{7}
-\bibcite{cache_models}{8}
-\bibcite{bptt-1}{9}
-\bibcite{bptt-2}{10}
-\bibcite{gym}{11}
diff --git a/article/bib.log b/article/bib.log
deleted file mode 100644
index edeeeb7..0000000
--- a/article/bib.log
+++ /dev/null
@@ -1,332 +0,0 @@
-This is XeTeX, Version 3.14159265-2.6-0.99992 (TeX Live 2015) (preloaded format=xelatex 2016.3.23) 10 DEC 2016 13:06
-entering extended mode
- restricted \write18 enabled.
- file:line:error style messages enabled.
- %&-line parsing enabled.
-**bib.tex
-(./bib.tex
-LaTeX2e <2015/01/01>
-Babel <3.9l> and hyphenation patterns for 79 languages loaded.
-(/usr/local/texlive/2015/texmf-dist/tex/latex/base/article.cls
-Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
-(/usr/local/texlive/2015/texmf-dist/tex/latex/base/size10.clo
-File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
-)
-\c@part=\count79
-\c@section=\count80
-\c@subsection=\count81
-\c@subsubsection=\count82
-\c@paragraph=\count83
-\c@subparagraph=\count84
-\c@figure=\count85
-\c@table=\count86
-\abovecaptionskip=\skip41
-\belowcaptionskip=\skip42
-\bibindent=\dimen102
-)
-(/usr/local/texlive/2015/texmf-dist/tex/generic/babel/babel.sty
-Package: babel 2014/09/25 3.9l The Babel package
-
-(/usr/local/texlive/2015/texmf-dist/tex/generic/babel-french/frenchb.ldf
-Language: frenchb 2015/04/10 v3.1e French support from the babel system
-
-(/usr/local/texlive/2015/texmf-dist/tex/generic/babel/babel.def
-File: babel.def 2014/09/25 3.9l Babel common definitions
-\babel@savecnt=\count87
-\U@D=\dimen103
-
-(/usr/local/texlive/2015/texmf-dist/tex/generic/babel/xebabel.def))
-\l@acadian = a dialect from \language\l@french
-\l@canadien = a dialect from \language\l@french
-\FBcolonskip=\skip43
-\FBthinskip=\skip44
-\FB@interchartokenstateORI=\count88
-Package frenchb.ldf Info: No need for active punctuation characters
-(frenchb.ldf) with this version of XeTeX!
-(frenchb.ldf) reported on input line 264.
-\FB@punctthick=\XeTeXcharclass4
-\FB@punctthin=\XeTeXcharclass5
-\FB@punctnul=\XeTeXcharclass6
-\FB@guilo=\XeTeXcharclass7
-\FB@guilf=\XeTeXcharclass8
-\FB@guilnul=\XeTeXcharclass9
-\FBguillskip=\skip45
-\FBguill@level=\count89
-\FB@Mht=\dimen104
-LaTeX Info: Redefining \degres on input line 740.
-\std@mcc=\count90
-\dec@mcc=\count91
-\listindentFB=\skip46
-\labelwidthFB=\skip47
-\leftmarginFB=\skip48
-\parindentFFN=\dimen105
-))
-(/usr/local/texlive/2015/texmf-dist/tex/latex/carlisle/scalefnt.sty)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/keyval.sty
-Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
-\KV@toks@=\toks14
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/base/inputenc.sty
-Package: inputenc 2015/03/17 v1.2c Input encoding file
-\inpenc@prehook=\toks15
-\inpenc@posthook=\toks16
-
-
-Package inputenc Warning: inputenc package ignored with utf8 based engines.
-
-) (/usr/local/texlive/2015/texmf-dist/tex/latex/base/fontenc.sty
-Package: fontenc 2005/09/27 v1.99g Standard LaTeX package
-
-(/usr/local/texlive/2015/texmf-dist/tex/latex/base/t1enc.def
-File: t1enc.def 2005/09/27 v1.99g Standard LaTeX file
-LaTeX Font Info: Redeclaring font encoding T1 on input line 48.
-))
-(/usr/local/texlive/2015/texmf-dist/tex/latex/geometry/geometry.sty
-Package: geometry 2010/09/12 v5.6 Page Geometry
-
-(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ifpdf.sty
-Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO)
-Package ifpdf Info: pdfTeX in PDF mode is not detected.
-)
-(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ifvtex.sty
-Package: ifvtex 2010/03/01 v1.5 Detect VTeX and its facilities (HO)
-Package ifvtex Info: VTeX not detected.
-)
-(/usr/local/texlive/2015/texmf-dist/tex/generic/ifxetex/ifxetex.sty
-Package: ifxetex 2010/09/12 v0.6 Provides ifxetex conditional
-)
-\Gm@cnth=\count92
-\Gm@cntv=\count93
-\c@Gm@tempcnt=\count94
-\Gm@bindingoffset=\dimen106
-\Gm@wd@mp=\dimen107
-\Gm@odd@mp=\dimen108
-\Gm@even@mp=\dimen109
-\Gm@layoutwidth=\dimen110
-\Gm@layoutheight=\dimen111
-\Gm@layouthoffset=\dimen112
-\Gm@layoutvoffset=\dimen113
-\Gm@dimlist=\toks17
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/verbatim.sty
-Package: verbatim 2014/10/28 v1.5q LaTeX2e package for verbatim enhancements
-\every@verbatim=\toks18
-\verbatim@line=\toks19
-\verbatim@in@stream=\read1
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphicx.sty
-Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)
-
-(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphics.sty
-Package: graphics 2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR)
-
-(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/trig.sty
-Package: trig 1999/03/16 v1.09 sin cos tan (DPC)
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/graphics.cfg
-File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
-)
-Package graphics Info: Driver file: xetex.def on input line 94.
-
-(/usr/local/texlive/2015/texmf-dist/tex/xelatex/xetex-def/xetex.def
-File: xetex.def 2015/03/25 v4.04 LaTeX color/graphics driver for XeTeX (TeX Liv
-e/RRM/JK)
-))
-\Gin@req@height=\dimen114
-\Gin@req@width=\dimen115
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsmath.sty
-Package: amsmath 2013/01/14 v2.14 AMS math features
-\@mathmargin=\skip49
-
-For additional information on amsmath, use the `?' option.
-(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amstext.sty
-Package: amstext 2000/06/29 v2.01
-
-(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsgen.sty
-File: amsgen.sty 1999/11/30 v2.0
-\@emptytoks=\toks20
-\ex@=\dimen116
-))
-(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsbsy.sty
-Package: amsbsy 1999/11/29 v1.2d
-\pmbraise@=\dimen117
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsopn.sty
-Package: amsopn 1999/12/14 v2.01 operator names
-)
-\inf@bad=\count95
-LaTeX Info: Redefining \frac on input line 210.
-\uproot@=\count96
-\leftroot@=\count97
-LaTeX Info: Redefining \overline on input line 306.
-\classnum@=\count98
-\DOTSCASE@=\count99
-LaTeX Info: Redefining \ldots on input line 378.
-LaTeX Info: Redefining \dots on input line 381.
-LaTeX Info: Redefining \cdots on input line 466.
-\Mathstrutbox@=\box26
-\strutbox@=\box27
-\big@size=\dimen118
-LaTeX Font Info: Redeclaring font encoding OML on input line 566.
-LaTeX Font Info: Redeclaring font encoding OMS on input line 567.
-\macc@depth=\count100
-\c@MaxMatrixCols=\count101
-\dotsspace@=\muskip10
-\c@parentequation=\count102
-\dspbrk@lvl=\count103
-\tag@help=\toks21
-\row@=\count104
-\column@=\count105
-\maxfields@=\count106
-\andhelp@=\toks22
-\eqnshift@=\dimen119
-\alignsep@=\dimen120
-\tagshift@=\dimen121
-\tagwidth@=\dimen122
-\totwidth@=\dimen123
-\lineht@=\dimen124
-\@envbody=\toks23
-\multlinegap=\skip50
-\multlinetaggap=\skip51
-\mathdisplay@stack=\toks24
-LaTeX Info: Redefining \[ on input line 2665.
-LaTeX Info: Redefining \] on input line 2666.
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/caption/caption.sty
-Package: caption 2013/05/02 v3.3-89 Customizing captions (AR)
-
-(/usr/local/texlive/2015/texmf-dist/tex/latex/caption/caption3.sty
-Package: caption3 2013/05/02 v1.6-88 caption3 kernel (AR)
-Package caption3 Info: TeX engine: e-TeX on input line 57.
-\captionmargin=\dimen125
-\captionmargin@=\dimen126
-\captionwidth=\dimen127
-\caption@tempdima=\dimen128
-\caption@indent=\dimen129
-\caption@parindent=\dimen130
-\caption@hangindent=\dimen131
-)
-\c@ContinuedFloat=\count107
-)
-(/usr/local/texlive/2015/texmf-dist/tex/latex/caption/newfloat.sty
-Package: newfloat 2013/05/11 v1.1-91 Defining new floating environments (AR)
-)
-Package newfloat Info: New float `mycapequ' with options `' on input line 13.
-\c@mycapequ=\count108
-\newfloat@ftype=\count109
-Package newfloat Info: float type `mycapequ'=8 on input line 13.
- (./bib.aux)
-\openout1 = `bib.aux'.
-
-LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 19.
-LaTeX Font Info: ... okay on input line 19.
-LaTeX Info: Redefining \dots on input line 19.
-LaTeX Info: Redefining \up on input line 19.
-
-Package frenchb.ldf Warning: Add \usepackage{fontspec} to the
-(frenchb.ldf) preamble of your document, on input line 19.
-
-*geometry* driver: auto-detecting
-*geometry* detected driver: xetex
-*geometry* verbose mode - [ preamble ] result:
-* driver: xetex
-* paper: a4paper
-* layout: