dlnd_language_translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Language Translation\n",
    "In this project, you’re going to take a peek into the realm of neural network machine translation.  You’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French.\n",
    "## Get the Data\n",
    "Since translating the whole language of English to French will take lots of time to train, we have provided you with a small portion of the English corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "source_path = 'data/small_vocab_en'\n",
    "target_path = 'data/small_vocab_fr'\n",
    "source_text = helper.load_data(source_path)\n",
    "target_text = helper.load_data(target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data\n",
    "Play around with view_sentence_range to view different parts of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Stats\n",
      "Roughly the number of unique words: 227\n",
      "Number of sentences: 137861\n",
      "Average number of words in a sentence: 13.225277634719028\n",
      "\n",
      "English sentences 51 to 60:\n",
      "she disliked a rusty yellow car .\n",
      "france is usually quiet during november , but it is sometimes warm in february .\n",
      "new jersey is never wet during november , and it is mild in august .\n",
      "we like peaches , pears , and strawberries .\n",
      "the orange is her least liked fruit , but the grapefruit is their least liked .\n",
      "china is never rainy during november , and it is quiet in january .\n",
      "china is relaxing during march , but it is sometimes snowy in september .\n",
      "paris is wonderful during march , but it is usually pleasant in june .\n",
      "new jersey is chilly during autumn , and it is sometimes pleasant in spring .\n",
      "\n",
      "French sentences 51 to 60:\n",
      "elle n'aimait pas une voiture jaune rouillée .\n",
      "la france est généralement calme en novembre , mais il est parfois chaud en février .\n",
      "new jersey est jamais humide au mois de novembre , et il est doux au mois d' août .\n",
      "nous aimons les pêches , les poires et les fraises .\n",
      "l'orange est la moins aimé des fruits , mais le pamplemousse est leur moins aimé .\n",
      "chine est jamais pluvieux en novembre , et il est calme en janvier .\n",
      "chine est relaxant au mois de mars , mais il est parfois enneigée en septembre .\n",
      "paris est merveilleux au mois de mars , mais il est généralement agréable en juin .\n",
      "new jersey est froid au cours de l' automne , et il est parfois agréable au printemps .\n"
     ]
    }
   ],
   "source": [
    "view_sentence_range = (51, 60)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))\n",
    "\n",
    "sentences = source_text.split('\\n')\n",
    "word_counts = [len(sentence.split()) for sentence in sentences]\n",
    "print('Number of sentences: {}'.format(len(sentences)))\n",
    "print('Average number of words in a sentence: {}'.format(np.average(word_counts)))\n",
    "\n",
    "print()\n",
    "print('English sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(source_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))\n",
    "print()\n",
    "print('French sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(target_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Preprocessing Function\n",
    "### Text to Word Ids\n",
    "As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function `text_to_ids()`, you'll turn `source_text` and `target_text` from words to ids.  However, you need to add the `<EOS>` word id at the end of `target_text`.  This will help the neural network predict when the sentence should end.\n",
    "\n",
    "You can get the `<EOS>` word id by doing:\n",
    "```python\n",
    "target_vocab_to_int['<EOS>']\n",
    "```\n",
    "You can get other word ids using `source_vocab_to_int` and `target_vocab_to_int`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "    Convert source and target text to proper word ids\n",
    "    :param source_text: String that contains all the source text.\n",
    "    :param target_text: String that contains all the target text.\n",
    "    :param source_vocab_to_int: Dictionary to go from the source words to an id\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :return: A tuple of lists (source_id_text, target_id_text)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    eos = source_vocab_to_int['<EOS>']\n",
    "    source_text = [[source_vocab_to_int[word] for word in sentence.split()] for sentence in source_text.split('\\n')]\n",
    "    target_text = [[target_vocab_to_int[word] for word in sentence.split()] + [eos] for sentence in target_text.split('\\n')]\n",
    "    return(source_text, target_text)\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_text_to_ids(text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess all the data and save it\n",
    "Running the code cell below will preprocess all the data and save it to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "helper.preprocess_and_save_data(source_path, target_path, text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check Point\n",
    "This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "import helper\n",
    "\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Version of TensorFlow and Access to GPU\n",
    "This will check to make sure you have the correct version of TensorFlow and access to a GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0\n",
      "Default GPU Device: /gpu:0\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.layers.core import Dense\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n",
    "You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:\n",
    "- `model_inputs`\n",
    "- `process_decoder_input`\n",
    "- `encoding_layer`\n",
    "- `decoding_layer_train`\n",
    "- `decoding_layer_infer`\n",
    "- `decoding_layer`\n",
    "- `seq2seq_model`\n",
    "\n",
    "### Input\n",
    "Implement the `model_inputs()` function to create TF Placeholders for the Neural Network. It should create the following placeholders:\n",
    "\n",
    "- Input text placeholder named \"input\" using the TF Placeholder name parameter with rank 2.\n",
    "- Targets placeholder with rank 2.\n",
    "- Learning rate placeholder with rank 0.\n",
    "- Keep probability placeholder named \"keep_prob\" using the TF Placeholder name parameter with rank 0.\n",
    "- Target sequence length placeholder named \"target_sequence_length\" with rank 1\n",
    "- Max target sequence length tensor named \"max_target_len\" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.\n",
    "- Source sequence length placeholder named \"source_sequence_length\" with rank 1\n",
    "\n",
    "Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def model_inputs():\n",
    "    \"\"\"\n",
    "    Create TF Placeholders for input, targets, learning rate, and lengths of source and target sequences.\n",
    "    :return: Tuple (input, targets, learning rate, keep probability, target sequence length,\n",
    "    max target sequence length, source sequence length)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    \n",
    "    inputs = tf.placeholder(tf.int32, shape=[None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, shape=[None, None], name='targets')\n",
    "    learning_rate = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_probability = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    target_sequence_length = tf.placeholder(tf.int32, (None,), name='target_sequence_length')\n",
    "    max_target_sequence_length = tf.reduce_max(target_sequence_length, name='max_target_sequence_length')\n",
    "    source_sequence_length = tf.placeholder(tf.int32, (None,), name='source_sequence_length')\n",
    "    \n",
    "    return(inputs, targets, learning_rate, keep_probability, target_sequence_length,\n",
    "    max_target_sequence_length, source_sequence_length)\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_model_inputs(model_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process Decoder Input\n",
    "Implement `process_decoder_input` by removing the last word id from each batch in `target_data` and concat the GO ID to the begining of each batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def process_decoder_input(target_data, target_vocab_to_int, batch_size):\n",
    "    \"\"\"\n",
    "    Preprocess target data for encoding\n",
    "    :param target_data: Target Placehoder\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :param batch_size: Batch Size\n",
    "    :return: Preprocessed target data\n",
    "    \"\"\"\n",
    "    \n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    # TODO: Implement Function\n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return(dec_input)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_process_encoding_input(process_decoder_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding\n",
    "Implement `encoding_layer()` to create a Encoder RNN layer:\n",
    " * Embed the encoder input using [`tf.contrib.layers.embed_sequence`](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence)\n",
    " * Construct a [stacked](https://github.com/tensorflow/tensorflow/blob/6947f65a374ebf29e74bb71e36fd82760056d82c/tensorflow/docs_src/tutorials/recurrent.md#stacking-multiple-lstms) [`tf.contrib.rnn.LSTMCell`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMCell) wrapped in a [`tf.contrib.rnn.DropoutWrapper`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/DropoutWrapper)\n",
    " * Pass cell and embedded input to [`tf.nn.dynamic_rnn()`](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "from imp import reload\n",
    "reload(tests)\n",
    "\n",
    "def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, \n",
    "                   source_sequence_length, source_vocab_size, \n",
    "                   encoding_embedding_size):\n",
    "    \"\"\"\n",
    "    Create encoding layer\n",
    "    :param rnn_inputs: Inputs for the RNN\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :param source_sequence_length: a list of the lengths of each sequence in the batch\n",
    "    :param source_vocab_size: vocabulary size of source data\n",
    "    :param encoding_embedding_size: embedding size of source data\n",
    "    :return: tuple (RNN output, RNN state)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    enc_embed_input = tf.contrib.layers.embed_sequence(rnn_inputs, rnn_size, encoding_embedding_size)\n",
    "    enc_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size) for _ in range(num_layers)])\n",
    "    enc_cell = tf.contrib.rnn.DropoutWrapper(enc_cell, keep_prob)\n",
    "    return tf.nn.dynamic_rnn(enc_cell, enc_embed_input, dtype=tf.float32)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_encoding_layer(encoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Training\n",
    "Create a training decoding layer:\n",
    "* Create a [`tf.contrib.seq2seq.TrainingHelper`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/TrainingHelper) \n",
    "* Create a [`tf.contrib.seq2seq.BasicDecoder`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/BasicDecoder)\n",
    "* Obtain the decoder outputs from [`tf.contrib.seq2seq.dynamic_decode`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_decode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, \n",
    "                         target_sequence_length, max_summary_length, \n",
    "                         output_layer, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a decoding layer for training\n",
    "    :param encoder_state: Encoder State\n",
    "    :param dec_cell: Decoder RNN Cell\n",
    "    :param dec_embed_input: Decoder embedded input\n",
    "    :param target_sequence_length: The lengths of each sequence in the target batch\n",
    "    :param max_summary_length: The length of the longest sequence in the batch\n",
    "    :param output_layer: Function to apply the output layer\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: BasicDecoderOutput containing training logits and sample_id\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(\n",
    "        inputs=dec_embed_input, sequence_length=target_sequence_length, time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell, \n",
    "                                                       training_helper,\n",
    "                                                       encoder_state, output_layer)\n",
    "\n",
    "    training_decoder_output, _ = tf.contrib.seq2seq.dynamic_decode(\n",
    "        training_decoder, impute_finished=True,\n",
    "        maximum_iterations=max_summary_length)\n",
    "\n",
    "    logits = training_decoder_output.rnn_output\n",
    "    sample_id = training_decoder_output.sample_id\n",
    "    logits = tf.nn.dropout(logits, keep_prob)\n",
    "    \n",
    "    return tf.contrib.seq2seq.BasicDecoderOutput(logits, sample_id)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer_train(decoding_layer_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Inference\n",
    "Create inference decoder:\n",
    "* Create a [`tf.contrib.seq2seq.GreedyEmbeddingHelper`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/GreedyEmbeddingHelper)\n",
    "* Create a [`tf.contrib.seq2seq.BasicDecoder`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/BasicDecoder)\n",
    "* Obtain the decoder outputs from [`tf.contrib.seq2seq.dynamic_decode`](https://www.tensorflow.org/api_docs/python/tf/contrib/seq2seq/dynamic_decode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,\n",
    "                         end_of_sequence_id, max_target_sequence_length,\n",
    "                         vocab_size, output_layer, batch_size, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a decoding layer for inference\n",
    "    :param encoder_state: Encoder state\n",
    "    :param dec_cell: Decoder RNN Cell\n",
    "    :param dec_embeddings: Decoder embeddings\n",
    "    :param start_of_sequence_id: GO ID\n",
    "    :param end_of_sequence_id: EOS Id\n",
    "    :param max_target_sequence_length: Maximum length of target sequences\n",
    "    :param vocab_size: Size of decoder/target vocabulary\n",
    "    :param decoding_scope: TenorFlow Variable Scope for decoding\n",
    "    :param output_layer: Function to apply the output layer\n",
    "    :param batch_size: Batch size\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :return: BasicDecoderOutput containing inference logits and sample_id\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    start_tokens = tf.tile(\n",
    "        tf.constant([start_of_sequence_id], dtype=tf.int32), [batch_size], \n",
    "        name='start_tokens')\n",
    "\n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(\n",
    "        dec_embeddings, start_tokens, end_of_sequence_id)\n",
    "\n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        encoder_state,\n",
    "                                                        output_layer)\n",
    "        \n",
    "    inference_decoder_output, _ = tf.contrib.seq2seq.dynamic_decode(\n",
    "        inference_decoder, impute_finished=True,\n",
    "        maximum_iterations=max_target_sequence_length)\n",
    "\n",
    "    logits = inference_decoder_output.rnn_output\n",
    "    sample_id = inference_decoder_output.sample_id\n",
    "    return tf.contrib.seq2seq.BasicDecoderOutput(logits, sample_id)\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer_infer(decoding_layer_infer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Decoding Layer\n",
    "Implement `decoding_layer()` to create a Decoder RNN layer.\n",
    "\n",
    "* Embed the target sequences\n",
    "* Construct the decoder LSTM cell (just like you constructed the encoder cell above)\n",
    "* Create an output layer to map the outputs of the decoder to the elements of our vocabulary\n",
    "* Use the your `decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob)` function to get the training logits.\n",
    "* Use your `decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob)` function to get the inference logits.\n",
    "\n",
    "Note: You'll need to use [tf.variable_scope](https://www.tensorflow.org/api_docs/python/tf/variable_scope) to share variables between training and inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer(dec_input, encoder_state,\n",
    "                   target_sequence_length, max_target_sequence_length,\n",
    "                   rnn_size,\n",
    "                   num_layers, target_vocab_to_int, target_vocab_size,\n",
    "                   batch_size, keep_prob, decoding_embedding_size):\n",
    "    \"\"\"\n",
    "    Create decoding layer\n",
    "    :param dec_input: Decoder input\n",
    "    :param encoder_state: Encoder state\n",
    "    :param target_sequence_length: The lengths of each sequence in the target batch\n",
    "    :param max_target_sequence_length: Maximum length of target sequences\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :param target_vocab_size: Size of target vocabulary\n",
    "    :param batch_size: The size of the batch\n",
    "    :param keep_prob: Dropout keep probability\n",
    "    :param decoding_embedding_size: Decoding embedding size\n",
    "    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    # TODO: Implement Function\n",
    "    # Embedding the target input sequences\n",
    "    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size,\n",
    "                                                    decoding_embedding_size]))\n",
    "    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)\n",
    "\n",
    "    # Making the decoder cell\n",
    "    def make_cell(rnn_size):\n",
    "        lstm = tf.contrib.rnn.LSTMCell(\n",
    "            rnn_size, initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "        return lstm\n",
    "    dec_cell = tf.contrib.rnn.MultiRNNCell([make_cell(rnn_size) for _ in range(num_layers)])\n",
    "\n",
    "    # Creating the output layer \n",
    "    output_layer = Dense(target_vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0,\n",
    "                                                                              stddev=0.1))\n",
    "    \n",
    "    # Now we use the decoding_layer_train function\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        train_logits = decoding_layer_train(encoder_state, \n",
    "                                            dec_cell, \n",
    "                                            dec_embed_input, \n",
    "                                            target_sequence_length,\n",
    "                                            max_target_sequence_length, \n",
    "                                            output_layer, \n",
    "                                            keep_prob) \n",
    "    \n",
    "    # Using the decoding_layer_infer function\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        infer_logits = decoding_layer_infer(encoder_state, \n",
    "                                            dec_cell, \n",
    "                                            dec_embeddings, \n",
    "                                            target_vocab_to_int['<GO>'], \n",
    "                                            target_vocab_to_int['<EOS>'],\n",
    "                                            max_target_sequence_length, \n",
    "                                            target_vocab_size,\n",
    "                                            output_layer, \n",
    "                                            batch_size,\n",
    "                                            keep_prob)\n",
    "\n",
    "    return(train_logits, infer_logits)\n",
    "\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_decoding_layer(decoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Neural Network\n",
    "Apply the functions you implemented above to:\n",
    "\n",
    "- Encode the input using your `encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,  source_sequence_length, source_vocab_size, encoding_embedding_size)`.\n",
    "- Process target data using your `process_decoder_input(target_data, target_vocab_to_int, batch_size)` function.\n",
    "- Decode the encoded input using your `decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size)` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, batch_size,\n",
    "                  source_sequence_length, target_sequence_length,\n",
    "                  max_target_sentence_length,\n",
    "                  source_vocab_size, target_vocab_size,\n",
    "                  enc_embedding_size, dec_embedding_size,\n",
    "                  rnn_size, num_layers, target_vocab_to_int):\n",
    "    \"\"\"\n",
    "    Build the Sequence-to-Sequence part of the neural network\n",
    "    :param input_data: Input placeholder\n",
    "    :param target_data: Target placeholder\n",
    "    :param keep_prob: Dropout keep probability placeholder\n",
    "    :param batch_size: Batch Size\n",
    "    :param source_sequence_length: Sequence Lengths of source sequences in the batch\n",
    "    :param target_sequence_length: Sequence Lengths of target sequences in the batch\n",
    "    :param source_vocab_size: Source vocabulary size\n",
    "    :param target_vocab_size: Target vocabulary size\n",
    "    :param enc_embedding_size: Decoder embedding size\n",
    "    :param dec_embedding_size: Encoder embedding size\n",
    "    :param rnn_size: RNN Size\n",
    "    :param num_layers: Number of layers\n",
    "    :param target_vocab_to_int: Dictionary to go from the target words to an id\n",
    "    :return: Tuple of (Training BasicDecoderOutput, Inference BasicDecoderOutput)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    _, enc_state = encoding_layer(input_data, \n",
    "                               rnn_size, \n",
    "                               num_layers,\n",
    "                               keep_prob,\n",
    "                               source_sequence_length,\n",
    "                               source_vocab_size, \n",
    "                               enc_embedding_size)\n",
    "    \n",
    "    \n",
    "    # Preparing target sequences to feed to the decoder in the training mode\n",
    "    dec_input = process_decoder_input(target_data, target_vocab_to_int, batch_size)\n",
    "    \n",
    "    # Passing the encoder state and decoder inputs to decoders\n",
    "    training_decoder_output, inference_decoder_output = decoding_layer(\n",
    "        dec_input, \n",
    "        enc_state, \n",
    "        target_sequence_length,\n",
    "        max_target_sentence_length, \n",
    "        rnn_size, \n",
    "        num_layers,\n",
    "        target_vocab_to_int,\n",
    "        target_vocab_size, \n",
    "        batch_size,\n",
    "        keep_prob, \n",
    "        dec_embedding_size)\n",
    "    \n",
    "    return training_decoder_output, inference_decoder_output\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_seq2seq_model(seq2seq_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Training\n",
    "### Hyperparameters\n",
    "Tune the following parameters:\n",
    "\n",
    "- Set `epochs` to the number of epochs.\n",
    "- Set `batch_size` to the batch size.\n",
    "- Set `rnn_size` to the size of the RNNs.\n",
    "- Set `num_layers` to the number of layers.\n",
    "- Set `encoding_embedding_size` to the size of the embedding for the encoder.\n",
    "- Set `decoding_embedding_size` to the size of the embedding for the decoder.\n",
    "- Set `learning_rate` to the learning rate.\n",
    "- Set `keep_probability` to the Dropout keep probability\n",
    "- Set `display_step` to state how many steps between each debug output statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "# Keep probability around 0.4 / 0.5 was not succesful\n",
    "# Using 10 and 5 epochs was to much, the training / validation scores stopped to increase after epoch 3\n",
    "\n",
    "epochs = 5\n",
    "# Batch Size\n",
    "batch_size = 256\n",
    "# RNN Size\n",
    "rnn_size = 512\n",
    "# Number of Layers\n",
    "num_layers = 2\n",
    "# Embedding Size\n",
    "encoding_embedding_size = 128\n",
    "decoding_embedding_size = 128\n",
    "# Learning Rate\n",
    "learning_rate = 0.001\n",
    "# Dropout Keep Probability\n",
    "keep_probability = 0.9\n",
    "display_step = 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Graph\n",
    "Build the graph using the neural network you implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "save_path = 'checkpoints/dev'\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()\n",
    "max_target_sentence_length = max([len(sentence) for sentence in source_int_text])\n",
    "\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()\n",
    "\n",
    "    #sequence_length = tf.placeholder_with_default(max_target_sentence_length, None, name='sequence_length')\n",
    "    input_shape = tf.shape(input_data)\n",
    "\n",
    "    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                   targets,\n",
    "                                                   keep_prob,\n",
    "                                                   batch_size,\n",
    "                                                   source_sequence_length,\n",
    "                                                   target_sequence_length,\n",
    "                                                   max_target_sequence_length,\n",
    "                                                   len(source_vocab_to_int),\n",
    "                                                   len(target_vocab_to_int),\n",
    "                                                   encoding_embedding_size,\n",
    "                                                   decoding_embedding_size,\n",
    "                                                   rnn_size,\n",
    "                                                   num_layers,\n",
    "                                                   target_vocab_to_int)\n",
    "\n",
    "\n",
    "    training_logits = tf.identity(train_logits.rnn_output, name='logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "\n",
    "    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(lr)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch and pad the source and target sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "def pad_sentence_batch(sentence_batch, pad_int):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]\n",
    "\n",
    "\n",
    "def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):\n",
    "    \"\"\"Batch targets, sources, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(sources)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "\n",
    "        # Slice the right amount for the batch\n",
    "        sources_batch = sources[start_i:start_i + batch_size]\n",
    "        targets_batch = targets[start_i:start_i + batch_size]\n",
    "\n",
    "        # Pad\n",
    "        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))\n",
    "        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))\n",
    "\n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_targets_lengths = []\n",
    "        for target in pad_targets_batch:\n",
    "            pad_targets_lengths.append(len(target))\n",
    "\n",
    "        pad_source_lengths = []\n",
    "        for source in pad_sources_batch:\n",
    "            pad_source_lengths.append(len(source))\n",
    "\n",
    "        yield pad_sources_batch, pad_targets_batch, pad_source_lengths, pad_targets_lengths\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train\n",
    "Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 Batch   25/538 - Train Accuracy: 0.4174, Validation Accuracy: 0.4766, Loss: 2.8289\n",
      "Epoch   0 Batch   50/538 - Train Accuracy: 0.4762, Validation Accuracy: 0.5172, Loss: 2.2409\n",
      "Epoch   0 Batch   75/538 - Train Accuracy: 0.5082, Validation Accuracy: 0.5231, Loss: 1.8408\n",
      "Epoch   0 Batch  100/538 - Train Accuracy: 0.5037, Validation Accuracy: 0.5320, Loss: 1.6070\n",
      "Epoch   0 Batch  125/538 - Train Accuracy: 0.5394, Validation Accuracy: 0.5359, Loss: 1.4392\n",
      "Epoch   0 Batch  150/538 - Train Accuracy: 0.5555, Validation Accuracy: 0.5605, Loss: 1.3097\n",
      "Epoch   0 Batch  175/538 - Train Accuracy: 0.5211, Validation Accuracy: 0.5305, Loss: 1.2296\n",
      "Epoch   0 Batch  200/538 - Train Accuracy: 0.5801, Validation Accuracy: 0.5710, Loss: 1.1368\n",
      "Epoch   0 Batch  225/538 - Train Accuracy: 0.5917, Validation Accuracy: 0.5932, Loss: 1.0883\n",
      "Epoch   0 Batch  250/538 - Train Accuracy: 0.5908, Validation Accuracy: 0.6078, Loss: 1.0412\n",
      "Epoch   0 Batch  275/538 - Train Accuracy: 0.5959, Validation Accuracy: 0.6220, Loss: 1.0735\n",
      "Epoch   0 Batch  300/538 - Train Accuracy: 0.6269, Validation Accuracy: 0.6199, Loss: 0.9389\n",
      "Epoch   0 Batch  325/538 - Train Accuracy: 0.6083, Validation Accuracy: 0.5842, Loss: 0.9606\n",
      "Epoch   0 Batch  350/538 - Train Accuracy: 0.6241, Validation Accuracy: 0.6271, Loss: 0.9357\n",
      "Epoch   0 Batch  375/538 - Train Accuracy: 0.6507, Validation Accuracy: 0.6472, Loss: 0.8642\n",
      "Epoch   0 Batch  400/538 - Train Accuracy: 0.6384, Validation Accuracy: 0.6419, Loss: 0.8772\n",
      "Epoch   0 Batch  425/538 - Train Accuracy: 0.6654, Validation Accuracy: 0.6717, Loss: 0.8593\n",
      "Epoch   0 Batch  450/538 - Train Accuracy: 0.6994, Validation Accuracy: 0.6726, Loss: 0.8089\n",
      "Epoch   0 Batch  475/538 - Train Accuracy: 0.7206, Validation Accuracy: 0.7156, Loss: 0.7984\n",
      "Epoch   0 Batch  500/538 - Train Accuracy: 0.7377, Validation Accuracy: 0.7257, Loss: 0.7326\n",
      "Epoch   0 Batch  525/538 - Train Accuracy: 0.7714, Validation Accuracy: 0.7587, Loss: 0.7201\n",
      "Epoch   1 Batch   25/538 - Train Accuracy: 0.8180, Validation Accuracy: 0.7834, Loss: 0.6862\n",
      "Epoch   1 Batch   50/538 - Train Accuracy: 0.8133, Validation Accuracy: 0.8074, Loss: 0.6649\n",
      "Epoch   1 Batch   75/538 - Train Accuracy: 0.8106, Validation Accuracy: 0.7805, Loss: 0.6284\n",
      "Epoch   1 Batch  100/538 - Train Accuracy: 0.8633, Validation Accuracy: 0.8281, Loss: 0.5719\n",
      "Epoch   1 Batch  125/538 - Train Accuracy: 0.8746, Validation Accuracy: 0.8466, Loss: 0.5723\n",
      "Epoch   1 Batch  150/538 - Train Accuracy: 0.8848, Validation Accuracy: 0.8624, Loss: 0.5445\n",
      "Epoch   1 Batch  175/538 - Train Accuracy: 0.8752, Validation Accuracy: 0.8580, Loss: 0.5171\n",
      "Epoch   1 Batch  200/538 - Train Accuracy: 0.8881, Validation Accuracy: 0.8690, Loss: 0.5002\n",
      "Epoch   1 Batch  225/538 - Train Accuracy: 0.9111, Validation Accuracy: 0.8521, Loss: 0.4793\n",
      "Epoch   1 Batch  250/538 - Train Accuracy: 0.9102, Validation Accuracy: 0.9096, Loss: 0.4755\n",
      "Epoch   1 Batch  275/538 - Train Accuracy: 0.8869, Validation Accuracy: 0.8874, Loss: 0.4738\n",
      "Epoch   1 Batch  300/538 - Train Accuracy: 0.9107, Validation Accuracy: 0.9190, Loss: 0.4614\n",
      "Epoch   1 Batch  325/538 - Train Accuracy: 0.9107, Validation Accuracy: 0.9112, Loss: 0.4253\n",
      "Epoch   1 Batch  350/538 - Train Accuracy: 0.9183, Validation Accuracy: 0.9155, Loss: 0.4187\n",
      "Epoch   1 Batch  375/538 - Train Accuracy: 0.9226, Validation Accuracy: 0.9181, Loss: 0.4292\n",
      "Epoch   1 Batch  400/538 - Train Accuracy: 0.9213, Validation Accuracy: 0.9254, Loss: 0.4484\n",
      "Epoch   1 Batch  425/538 - Train Accuracy: 0.8990, Validation Accuracy: 0.9387, Loss: 0.4616\n",
      "Epoch   1 Batch  450/538 - Train Accuracy: 0.9315, Validation Accuracy: 0.9217, Loss: 0.4513\n",
      "Epoch   1 Batch  475/538 - Train Accuracy: 0.9431, Validation Accuracy: 0.9189, Loss: 0.4294\n",
      "Epoch   1 Batch  500/538 - Train Accuracy: 0.9494, Validation Accuracy: 0.9338, Loss: 0.4115\n",
      "Epoch   1 Batch  525/538 - Train Accuracy: 0.9347, Validation Accuracy: 0.9334, Loss: 0.4585\n",
      "Epoch   2 Batch   25/538 - Train Accuracy: 0.9207, Validation Accuracy: 0.9496, Loss: 0.4578\n",
      "Epoch   2 Batch   50/538 - Train Accuracy: 0.9336, Validation Accuracy: 0.9244, Loss: 0.3973\n",
      "Epoch   2 Batch   75/538 - Train Accuracy: 0.9325, Validation Accuracy: 0.9444, Loss: 0.4296\n",
      "Epoch   2 Batch  100/538 - Train Accuracy: 0.9582, Validation Accuracy: 0.9535, Loss: 0.4382\n",
      "Epoch   2 Batch  125/538 - Train Accuracy: 0.9390, Validation Accuracy: 0.9604, Loss: 0.4251\n",
      "Epoch   2 Batch  150/538 - Train Accuracy: 0.9385, Validation Accuracy: 0.9254, Loss: 0.4192\n",
      "Epoch   2 Batch  175/538 - Train Accuracy: 0.9555, Validation Accuracy: 0.9249, Loss: 0.4242\n",
      "Epoch   2 Batch  200/538 - Train Accuracy: 0.9588, Validation Accuracy: 0.9467, Loss: 0.4112\n",
      "Epoch   2 Batch  225/538 - Train Accuracy: 0.9628, Validation Accuracy: 0.9485, Loss: 0.4312\n",
      "Epoch   2 Batch  250/538 - Train Accuracy: 0.9502, Validation Accuracy: 0.9542, Loss: 0.3867\n",
      "Epoch   2 Batch  275/538 - Train Accuracy: 0.9537, Validation Accuracy: 0.9503, Loss: 0.4392\n",
      "Epoch   2 Batch  300/538 - Train Accuracy: 0.9496, Validation Accuracy: 0.9487, Loss: 0.4183\n",
      "Epoch   2 Batch  325/538 - Train Accuracy: 0.9639, Validation Accuracy: 0.9473, Loss: 0.4174\n",
      "Epoch   2 Batch  350/538 - Train Accuracy: 0.9608, Validation Accuracy: 0.9476, Loss: 0.4372\n",
      "Epoch   2 Batch  375/538 - Train Accuracy: 0.9583, Validation Accuracy: 0.9407, Loss: 0.3833\n",
      "Epoch   2 Batch  400/538 - Train Accuracy: 0.9665, Validation Accuracy: 0.9567, Loss: 0.3978\n",
      "Epoch   2 Batch  425/538 - Train Accuracy: 0.9436, Validation Accuracy: 0.9615, Loss: 0.3995\n",
      "Epoch   2 Batch  450/538 - Train Accuracy: 0.9328, Validation Accuracy: 0.9435, Loss: 0.3934\n",
      "Epoch   2 Batch  475/538 - Train Accuracy: 0.9598, Validation Accuracy: 0.9407, Loss: 0.4050\n",
      "Epoch   2 Batch  500/538 - Train Accuracy: 0.9632, Validation Accuracy: 0.9421, Loss: 0.3839\n",
      "Epoch   2 Batch  525/538 - Train Accuracy: 0.9546, Validation Accuracy: 0.9439, Loss: 0.3759\n",
      "Epoch   3 Batch   25/538 - Train Accuracy: 0.9510, Validation Accuracy: 0.9652, Loss: 0.3944\n",
      "Epoch   3 Batch   50/538 - Train Accuracy: 0.9613, Validation Accuracy: 0.9600, Loss: 0.3850\n",
      "Epoch   3 Batch   75/538 - Train Accuracy: 0.9621, Validation Accuracy: 0.9588, Loss: 0.4176\n",
      "Epoch   3 Batch  100/538 - Train Accuracy: 0.9779, Validation Accuracy: 0.9691, Loss: 0.3713\n",
      "Epoch   3 Batch  125/538 - Train Accuracy: 0.9635, Validation Accuracy: 0.9664, Loss: 0.4460\n",
      "Epoch   3 Batch  150/538 - Train Accuracy: 0.9764, Validation Accuracy: 0.9597, Loss: 0.4200\n",
      "Epoch   3 Batch  175/538 - Train Accuracy: 0.9676, Validation Accuracy: 0.9370, Loss: 0.3932\n",
      "Epoch   3 Batch  200/538 - Train Accuracy: 0.9750, Validation Accuracy: 0.9593, Loss: 0.4107\n",
      "Epoch   3 Batch  225/538 - Train Accuracy: 0.9708, Validation Accuracy: 0.9611, Loss: 0.3777\n",
      "Epoch   3 Batch  250/538 - Train Accuracy: 0.9672, Validation Accuracy: 0.9512, Loss: 0.3888\n",
      "Epoch   3 Batch  275/538 - Train Accuracy: 0.9586, Validation Accuracy: 0.9567, Loss: 0.3868\n",
      "Epoch   3 Batch  300/538 - Train Accuracy: 0.9693, Validation Accuracy: 0.9567, Loss: 0.3868\n",
      "Epoch   3 Batch  325/538 - Train Accuracy: 0.9773, Validation Accuracy: 0.9670, Loss: 0.3721\n",
      "Epoch   3 Batch  350/538 - Train Accuracy: 0.9712, Validation Accuracy: 0.9602, Loss: 0.4098\n",
      "Epoch   3 Batch  375/538 - Train Accuracy: 0.9794, Validation Accuracy: 0.9606, Loss: 0.3844\n",
      "Epoch   3 Batch  400/538 - Train Accuracy: 0.9743, Validation Accuracy: 0.9663, Loss: 0.3815\n",
      "Epoch   3 Batch  425/538 - Train Accuracy: 0.9567, Validation Accuracy: 0.9700, Loss: 0.4136\n",
      "Epoch   3 Batch  450/538 - Train Accuracy: 0.9472, Validation Accuracy: 0.9592, Loss: 0.3851\n",
      "Epoch   3 Batch  475/538 - Train Accuracy: 0.9771, Validation Accuracy: 0.9666, Loss: 0.3963\n",
      "Epoch   3 Batch  500/538 - Train Accuracy: 0.9798, Validation Accuracy: 0.9688, Loss: 0.3699\n",
      "Epoch   3 Batch  525/538 - Train Accuracy: 0.9667, Validation Accuracy: 0.9728, Loss: 0.3881\n",
      "Epoch   4 Batch   25/538 - Train Accuracy: 0.9619, Validation Accuracy: 0.9727, Loss: 0.3842\n",
      "Epoch   4 Batch   50/538 - Train Accuracy: 0.9762, Validation Accuracy: 0.9746, Loss: 0.4022\n",
      "Epoch   4 Batch   75/538 - Train Accuracy: 0.9686, Validation Accuracy: 0.9666, Loss: 0.3799\n",
      "Epoch   4 Batch  100/538 - Train Accuracy: 0.9844, Validation Accuracy: 0.9648, Loss: 0.3865\n",
      "Epoch   4 Batch  125/538 - Train Accuracy: 0.9747, Validation Accuracy: 0.9794, Loss: 0.3701\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   4 Batch  150/538 - Train Accuracy: 0.9783, Validation Accuracy: 0.9616, Loss: 0.3728\n",
      "Epoch   4 Batch  175/538 - Train Accuracy: 0.9785, Validation Accuracy: 0.9590, Loss: 0.3652\n",
      "Epoch   4 Batch  200/538 - Train Accuracy: 0.9816, Validation Accuracy: 0.9719, Loss: 0.3695\n",
      "Epoch   4 Batch  225/538 - Train Accuracy: 0.9795, Validation Accuracy: 0.9693, Loss: 0.3953\n",
      "Epoch   4 Batch  250/538 - Train Accuracy: 0.9859, Validation Accuracy: 0.9739, Loss: 0.3745\n",
      "Epoch   4 Batch  275/538 - Train Accuracy: 0.9773, Validation Accuracy: 0.9647, Loss: 0.3974\n",
      "Epoch   4 Batch  300/538 - Train Accuracy: 0.9691, Validation Accuracy: 0.9739, Loss: 0.3960\n",
      "Epoch   4 Batch  325/538 - Train Accuracy: 0.9803, Validation Accuracy: 0.9574, Loss: 0.3965\n",
      "Epoch   4 Batch  350/538 - Train Accuracy: 0.9810, Validation Accuracy: 0.9652, Loss: 0.3969\n",
      "Epoch   4 Batch  375/538 - Train Accuracy: 0.9825, Validation Accuracy: 0.9624, Loss: 0.3841\n",
      "Epoch   4 Batch  400/538 - Train Accuracy: 0.9801, Validation Accuracy: 0.9737, Loss: 0.3782\n",
      "Epoch   4 Batch  425/538 - Train Accuracy: 0.9617, Validation Accuracy: 0.9632, Loss: 0.3742\n",
      "Epoch   4 Batch  450/538 - Train Accuracy: 0.9576, Validation Accuracy: 0.9716, Loss: 0.3717\n",
      "Epoch   4 Batch  475/538 - Train Accuracy: 0.9794, Validation Accuracy: 0.9751, Loss: 0.3673\n",
      "Epoch   4 Batch  500/538 - Train Accuracy: 0.9895, Validation Accuracy: 0.9721, Loss: 0.3506\n",
      "Epoch   4 Batch  525/538 - Train Accuracy: 0.9751, Validation Accuracy: 0.9627, Loss: 0.3933\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "def get_accuracy(target, logits):\n",
    "    \"\"\"\n",
    "    Calculate accuracy\n",
    "    \"\"\"\n",
    "    max_seq = max(target.shape[1], logits.shape[1])\n",
    "    if max_seq - target.shape[1]:\n",
    "        target = np.pad(\n",
    "            target,\n",
    "            [(0,0),(0,max_seq - target.shape[1])],\n",
    "            'constant')\n",
    "    if max_seq - logits.shape[1]:\n",
    "        logits = np.pad(\n",
    "            logits,\n",
    "            [(0,0),(0,max_seq - logits.shape[1])],\n",
    "            'constant')\n",
    "\n",
    "    return np.mean(np.equal(target, logits))\n",
    "\n",
    "# Split data to training and validation sets\n",
    "train_source = source_int_text[batch_size:]\n",
    "train_target = target_int_text[batch_size:]\n",
    "valid_source = source_int_text[:batch_size]\n",
    "valid_target = target_int_text[:batch_size]\n",
    "(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source,\n",
    "                                                                                                             valid_target,\n",
    "                                                                                                             batch_size,\n",
    "                                                                                                             source_vocab_to_int['<PAD>'],\n",
    "                                                                                                             target_vocab_to_int['<PAD>']))                                                                                                  \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    for epoch_i in range(epochs):\n",
    "        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(\n",
    "                get_batches(train_source, train_target, batch_size,\n",
    "                            source_vocab_to_int['<PAD>'],\n",
    "                            target_vocab_to_int['<PAD>'])):\n",
    "\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: source_batch,\n",
    "                 targets: target_batch,\n",
    "                 lr: learning_rate,\n",
    "                 target_sequence_length: targets_lengths,\n",
    "                 source_sequence_length: sources_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "\n",
    "\n",
    "                batch_train_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: source_batch,\n",
    "                     source_sequence_length: sources_lengths,\n",
    "                     target_sequence_length: targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "\n",
    "\n",
    "                batch_valid_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: valid_sources_batch,\n",
    "                     source_sequence_length: valid_sources_lengths,\n",
    "                     target_sequence_length: valid_targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "\n",
    "                train_acc = get_accuracy(target_batch, batch_train_logits)\n",
    "\n",
    "                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)\n",
    "\n",
    "                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'\n",
    "                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, save_path)\n",
    "    print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Parameters\n",
    "Save the `batch_size` and `save_path` parameters for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# Save parameters for checkpoint\n",
    "helper.save_params(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()\n",
    "load_path = helper.load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence to Sequence\n",
    "To feed a sentence into the model for translation, you first need to preprocess it.  Implement the function `sentence_to_seq()` to preprocess new sentences.\n",
    "\n",
    "- Convert the sentence to lowercase\n",
    "- Convert words into ids using `vocab_to_int`\n",
    " - Convert words not in the vocabulary, to the `<UNK>` word id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def sentence_to_seq(sentence, vocab_to_int):\n",
    "    \"\"\"\n",
    "    Convert a sentence to a sequence of ids\n",
    "    :param sentence: String\n",
    "    :param vocab_to_int: Dictionary to go from the words to an id\n",
    "    :return: List of word ids\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    unk = vocab_to_int['<UNK>']\n",
    "    sentence = sentence.lower()\n",
    "    return [vocab_to_int.get(w, unk) for w in sentence.split()]\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_sentence_to_seq(sentence_to_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate\n",
    "This will translate `translate_sentence` from English to French."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/dev\n",
      "Input\n",
      "  Word Ids:      [146, 196, 149, 45, 161, 2]\n",
      "  English Words: ['he', 'saw', 'a', 'old', 'yellow', '<UNK>']\n",
      "\n",
      "Prediction\n",
      "  Word Ids:      [58, 323, 255, 352, 269, 345, 329, 1]\n",
      "  French Words: a vu une vieille voiture jaune . <EOS>\n"
     ]
    }
   ],
   "source": [
    "translate_sentence = 'he saw a old yellow truck.'\n",
    "\n",
    "# The French output = est le gel en février = It is freezing in februari :)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    \n",
    "    # Loading the saved model\n",
    "    loader = tf.train.import_meta_graph(load_path + '.meta')\n",
    "    loader.restore(sess, load_path)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')\n",
    "    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "\n",
    "    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,\n",
    "                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,\n",
    "                                         source_sequence_length: [len(translate_sentence)]*batch_size,\n",
    "                                         keep_prob: 1.0})[0]\n",
    "\n",
    "print('Input')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_sentence]))\n",
    "print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))\n",
    "\n",
    "print('\\nPrediction')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_logits]))\n",
    "print('  French Words: {}'.format(\" \".join([target_int_to_vocab[i] for i in translate_logits])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imperfect Translation\n",
    "You might notice that some sentences translate better than others.  Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words.  For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.\n",
    "\n",
    "You can train on the [WMT10 French-English corpus](http://www.statmt.org/wmt10/training-giga-fren.tar).  This dataset has more vocabulary and richer in topics discussed.  However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided.  Just make sure you play with the WMT10 corpus after you've submitted this project.\n",
    "## Submitting This Project\n",
    "When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as \"dlnd_language_translation.ipynb\" and save it as a HTML file under \"File\" -> \"Download as\". Include the \"helper.py\" and \"problem_unittests.py\" files in your submission."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}