neural_network_basics/02-TensorFlow Basic.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:06:52.305994Z",
     "start_time": "2021-03-16T14:06:52.290037Z"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow介绍\n",
    "\n",
    "总体构成：\n",
    "\n",
    "* 高级深度学习API\n",
    "    * [tf.keras](https://www.tensorflow.org/api_docs/python/tf/keras)\n",
    "    * [tf.estimator](https://www.tensorflow.org/api_docs/python/tf/estimator)\n",
    "* 底层深度学习API\n",
    "    * [tf.nn](https://www.tensorflow.org/api_docs/python/tf/nn)\n",
    "    * [tf.losses](https://www.tensorflow.org/api_docs/python/tf/keras/losses)\n",
    "    * [tf.metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)\n",
    "    * [tf.optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)\n",
    "    * [tf.train](https://www.tensorflow.org/api_docs/python/tf/train)\n",
    "    * [tf.initializers](https://www.tensorflow.org/api_docs/python/tf/keras/initializers)  \n",
    "* 自动微分\n",
    "    * [tf.GradientTape]()\n",
    "    * [tf.gradients]()\n",
    "* I/O和预处理\n",
    "    * [tf.data](https://www.tensorflow.org/api_docs/python/tf/data)\n",
    "    * [tf.feature_column](https://www.tensorflow.org/api_docs/python/tf/feature_column)\n",
    "    * [tf.audio](https://www.tensorflow.org/api_docs/python/tf/audio)\n",
    "    * [tf.image](https://www.tensorflow.org/api_docs/python/tf/image)\n",
    "    * [tf.io](https://www.tensorflow.org/api_docs/python/tf/io)\n",
    "    * [tf.queue](https://www.tensorflow.org/api_docs/python/tf/queue)\n",
    "* TensorBoard可视化\n",
    "    * [tf.summary](https://www.tensorflow.org/api_docs/python/tf/summary)\n",
    "* 部署和优化\n",
    "    * [tf.distribute](https://www.tensorflow.org/api_docs/python/tf/distribute)\n",
    "    * [tf.saved_model](https://www.tensorflow.org/api_docs/python/tf/saved_model)\n",
    "    * [tf.autograph](https://www.tensorflow.org/api_docs/python/tf/autograph)\n",
    "    * [tf.graph_util](https://www.tensorflow.org/api_docs/python/tf/graph_util)\n",
    "    * [tf.lite](https://www.tensorflow.org/api_docs/python/tf/lite)\n",
    "    * [tf.quantization](https://www.tensorflow.org/api_docs/python/tf/quantization)\n",
    "    * [tf.tpu](https://www.tensorflow.org/api_docs/python/tf/tpu)\n",
    "    * [tf.xla](https://www.tensorflow.org/api_docs/python/tf/xla)\n",
    "* 特殊数据结构\n",
    "    * [tf.lookup](https://www.tensorflow.org/api_docs/python/tf/lookup)\n",
    "    * [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest)\n",
    "    * [tf.ragged](https://www.tensorflow.org/api_docs/python/tf/ragged)\n",
    "    * [tf.sets](https://www.tensorflow.org/api_docs/python/tf/sets)\n",
    "    * [tf.sparse](https://www.tensorflow.org/api_docs/python/tf/sparse)\n",
    "    * [tf.strings](https://www.tensorflow.org/api_docs/python/tf/strings)\n",
    "* 数学，包括线性代数和信号处理\n",
    "    * [tf.math](https://www.tensorflow.org/api_docs/python/tf/math)\n",
    "    * [tf.linalg](https://www.tensorflow.org/api_docs/python/tf/linalg)\n",
    "    * [tf.signal](https://www.tensorflow.org/api_docs/python/tf/signal)\n",
    "    * [tf.random](https://www.tensorflow.org/api_docs/python/tf/random)\n",
    "    * [tf.bitwise](https://www.tensorflow.org/api_docs/python/tf/bitwise)\n",
    "* 其他\n",
    "    * [tf.compat](https://www.tensorflow.org/api_docs/python/tf/compat)\n",
    "    * [tf.config](https://www.tensorflow.org/api_docs/python/tf/config)\n",
    "\n",
    "\n",
    "![image-20210316211612001](images/image-20210316211612001.png)\n",
    "\n",
    "开源TensorFlow项目区：https://github.com/jtoy/awesome-tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensor操作\n",
    "\n",
    "最佳实践：如果编写的代码需要移植到其他keras后端运行即不一定是TensorFlow后端，那么数学操作请使用keras(虽然keras相比TensorFlow只是部分）除此之外，请一律使用TensorFlow的操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 基础操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:34:07.352621Z",
     "start_time": "2021-03-16T13:34:07.342648Z"
    }
   },
   "outputs": [],
   "source": [
    "# 创建immutable张量, 与之相对应的是tf.Variable\n",
    "t = tf.constant([[1, 2, 3], [4, 5, 6]], dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:37:56.599131Z",
     "start_time": "2021-03-16T13:37:56.584171Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 索引取值\n",
    "t[:, 1, tf.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:35:38.468497Z",
     "start_time": "2021-03-16T13:35:38.438573Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 省略号...，要求所有未指定的轴都完全包括在内\n",
    "t[..., 1, tf.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:37:24.312349Z",
     "start_time": "2021-03-16T13:37:24.303374Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([2., 5.], dtype=float32)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 注意观察和上面的区别\n",
    "t[..., 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:38:25.794146Z",
     "start_time": "2021-03-16T13:38:25.782178Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([2, 3]), tf.float32)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看形状和dtype,和numpy一致\n",
    "t.shape, t.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:46:24.102956Z",
     "start_time": "2021-03-16T13:46:24.072003Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "add:\n",
      " [[11. 12. 13.]\n",
      " [14. 15. 16.]]\n",
      "multiply:\n",
      " [[10. 20. 30.]\n",
      " [40. 50. 60.]]\n",
      "square:\n",
      " [[ 1.  4.  9.]\n",
      " [16. 25. 36.]]\n",
      "exp:\n",
      " [[  2.7182817   7.389056   20.085537 ]\n",
      " [ 54.59815   148.41316   403.4288   ]]\n",
      "sqrt:\n",
      " [[1.        1.4142135 1.7320508]\n",
      " [2.        2.236068  2.4494898]]\n",
      "tf.Tensor(\n",
      "[[14. 32.]\n",
      " [32. 77.]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# add， multiply\n",
    "print(f\"add:\\n {t + 10}\") # or tf.add(t, 10)\n",
    "print(f\"multiply:\\n {t * 10}\") # or tf.multiply(t, 10)\n",
    "# 基础数学操作\n",
    "print(f\"square:\\n {tf.square(t)}\")\n",
    "print(f\"exp:\\n {tf.exp(t)}\")\n",
    "print(f\"sqrt:\\n {tf.sqrt(t)}\")\n",
    "# 矩阵乘法\n",
    "print(t @ tf.transpose(t)) # or tf.matmul()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:47:39.681096Z",
     "start_time": "2021-03-16T13:47:39.660152Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(6, 1), dtype=float32, numpy=\n",
       "array([[1.],\n",
       "       [2.],\n",
       "       [3.],\n",
       "       [4.],\n",
       "       [5.],\n",
       "       [6.]], dtype=float32)>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reshape\n",
    "tf.reshape(t, (-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:57:18.303137Z",
     "start_time": "2021-03-16T13:57:18.289175Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(6,), dtype=int32, numpy=array([1, 2, 1, 3, 1, 1])>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 该函数返回一个张量，这个张量是将原始input中所有维度为1的那些维都删掉的结果, 默认删除所有为1的维度\n",
    "# axis可以用来指定要删掉的为1的维度，此处要注意指定的维度必须确保其是1，否则会报错\n",
    "tf.squeeze(tf.constant([[1, 2, 1, 3, 1, 1]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T13:58:25.870136Z",
     "start_time": "2021-03-16T13:58:25.857198Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(6,), dtype=int32, numpy=array([1, 2, 1, 3, 1, 1])>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# axis可以用来指定要删掉的为1的维度，此处要注意指定的维度必须确保其是1，否则会报错\n",
    "tf.squeeze(tf.constant([[1, 2, 1, 3, 1, 1]]), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:04:46.766847Z",
     "start_time": "2021-03-16T14:04:46.740916Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "构造前：\n",
      "[[1. 2. 3.]\n",
      " [4. 5. 6.]]\n",
      "需要构造的size：\n",
      "[1 2]\n",
      "构造后：\n",
      "[[1. 2. 3. 1. 2. 3.]\n",
      " [4. 5. 6. 4. 5. 6.]]\n"
     ]
    }
   ],
   "source": [
    "# tile操作会根据给定的第二个张量的大小当做size,然后将第一个张量当做一个整体，构造一个新的张量\n",
    "print(f\"构造前：\\n{t}\")\n",
    "a=tf.constant([1, 2])\n",
    "print(f\"需要构造的size：\\n{a}\") \n",
    "print(f\"构造后：\\n{tf.tile(t, a)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:10:13.693843Z",
     "start_time": "2021-03-16T14:10:13.668937Z"
    }
   },
   "outputs": [],
   "source": [
    "# 与numpy意思相同但是名称不同的一些操作\n",
    "tn = np.array([[1., 2., 3.], [4., 5., 6.]])\n",
    "assert tf.reduce_mean(t).numpy() == np.mean(tn)\n",
    "assert tf.reduce_sum(t).numpy() == np.sum(tn)\n",
    "assert tf.reduce_max(t).numpy() == np.max(tn)\n",
    "assert tf.math.log(t).numpy().all() == np.log(tn).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:21:03.878006Z",
     "start_time": "2021-03-16T14:21:03.861051Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 4., 16., 25.])>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TensorFlow操作作用于numpy数组，但是注意作用之后会把numpy数组转变成Tensor\n",
    "a = np.array([2., 4., 5.])\n",
    "tf.square(a) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:17:29.480118Z",
     "start_time": "2021-03-16T14:17:29.462441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4, 16, 25], dtype=int32)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numpy 操作作用于Tensor， 但是注意作用之后会把Tensor转变成numpy数组\n",
    "t = tf.constant([2, 4, 5])\n",
    "np.square(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:22:30.173495Z",
     "start_time": "2021-03-16T14:22:30.162524Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([2., 4., 5.], dtype=float32)>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 从numpy数组创建tensor\n",
    "# 注意：numpy数组为64位精度，tensor是32位精度, 通常32位精度tensor对网络足以，因此从numpy创建tensor时，指定dtype=tf.float32\n",
    "t_a = tf.constant(a, dtype=tf.float32)\n",
    "t_a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:22:45.782191Z",
     "start_time": "2021-03-16T14:22:45.768228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2., 4., 5.], dtype=float32)"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tensor转numpy数组\n",
    "a_t = t_a.numpy()\n",
    "a_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:31:27.748781Z",
     "start_time": "2021-03-16T14:31:27.737811Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2]\n"
     ]
    }
   ],
   "source": [
    "# 类型转换\n",
    "# 不能把浮点张量和整数张量相加等操作\n",
    "# 不能把不同大小的浮点张量相加等操作\n",
    "try:\n",
    "    tf.constant(2.) + tf.constant(40)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:28:30.695790Z",
     "start_time": "2021-03-16T14:28:30.670857Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=42.0>"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用tf.cast进行类型转换\n",
    "t2 = tf.constant(40., dtype=tf.float64)\n",
    "tf.constant(2.) + tf.cast(t2, tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:32:51.705582Z",
     "start_time": "2021-03-16T14:32:51.691620Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建可变张量用于网络调节权重，梯度等需要改变的值\n",
    "v = tf.Variable([[1., 2., 3.,],[4.,5.,6.]])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:55:42.639590Z",
     "start_time": "2021-03-16T14:55:42.608595Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[4., 5., 6.],\n",
       "       [1., 2., 3.]], dtype=float32)>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用assign修改值\n",
    "v.assign(2 * v) # => [[2., 4., 6.], [8., 10., 12.]]\n",
    "# 使用切片和assign修改值\n",
    "v[0, 1].assign(42) # => [[2., 42., 6.], [8., 10., 12.]]\n",
    "\n",
    "# 使用scatter_update修改值\n",
    "v[:, 2].assign([0., 1.]) # => [[2., 42., 0.], [8., 10., 1.]]\n",
    "\n",
    "\n",
    "# 使用scatter_nd_update(通过切片的方式修改值)\n",
    "v.scatter_nd_update(indices=[[0, 0], [1, 2]], updates=[100., 200.])# => [[100., 42., 0.], [8., 10., 200.]]\n",
    "\n",
    "# 先创建对应索引即对应要修改的值，然后使用scatter_update\n",
    "sparse_delta = tf.IndexedSlices(values=[[1., 2., 3.], [4., 5., 6.]],\n",
    "                                indices=[1, 0])\n",
    "v.scatter_update(sparse_delta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T14:54:33.418611Z",
     "start_time": "2021-03-16T14:54:33.407641Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'ResourceVariable' object does not support item assignment\n"
     ]
    }
   ],
   "source": [
    "# 如果按照Python的方式修改变量，会报错\n",
    "try:\n",
    "    v[1] = [7., 8., 9.]\n",
    "except TypeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：实际上几乎不需要手动创建变量，因为keras提供了add_weight()方法，其次模型参数通常由优化器直接更新，因此几乎不需要手动更新变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 字符串张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:01:06.626177Z",
     "start_time": "2021-03-16T15:01:06.616204Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'hello world'>"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tf.string类型的constant张量使用字节字符串，而不是Unicode字符串，因此不存在形状\n",
    "tf.constant(b\"hello world\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:01:54.612565Z",
     "start_time": "2021-03-16T15:01:54.595586Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'caf\\xc3\\xa9'>"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 如果使用Unicode字符串创建张量，会被自动编码成UTF-8， 因此不存在形状\n",
    "tf.constant(\"café\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:01:55.306470Z",
     "start_time": "2021-03-16T15:01:55.297498Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233])>"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unicode张量通过如下方式创建，会显示形状\n",
    "u = tf.constant([ord(c) for c in \"café\"])\n",
    "u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:04:40.615184Z",
     "start_time": "2021-03-16T15:04:40.597232Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'caf\\xc3\\xa9'>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unicode 张量 encode\n",
    "b = tf.strings.unicode_encode(u, \"UTF-8\")\n",
    "tf.strings.length(b, unit=\"UTF8_CHAR\")\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:04:48.150651Z",
     "start_time": "2021-03-16T15:04:48.132717Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233])>"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unicode 张量 decode\n",
    "tf.strings.unicode_decode(b, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 字符创数组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:05:34.790020Z",
     "start_time": "2021-03-16T15:05:34.775061Z"
    }
   },
   "outputs": [],
   "source": [
    "p = tf.constant([\"Café\", \"Coffee\", \"caffè\", \"咖啡\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:05:44.151271Z",
     "start_time": "2021-03-16T15:05:44.136333Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([4, 6, 5, 2])>"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.length(p, unit=\"UTF8_CHAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:06:12.983233Z",
     "start_time": "2021-03-16T15:06:12.968257Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857]]>"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = tf.strings.unicode_decode(p, \"UTF8\")\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:06:14.709883Z",
     "start_time": "2021-03-16T15:06:14.698912Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857]]>\n"
     ]
    }
   ],
   "source": [
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 不规则张量(Ragged tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:06:51.402067Z",
     "start_time": "2021-03-16T15:06:51.373654Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 67 111 102 102 101 101], shape=(6,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "print(r[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:07:01.182110Z",
     "start_time": "2021-03-16T15:07:01.167151Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232]]>\n"
     ]
    }
   ],
   "source": [
    "print(r[1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:07:18.735910Z",
     "start_time": "2021-03-16T15:07:18.689057Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857], [65, 66], [], [67]]>\n"
     ]
    }
   ],
   "source": [
    "r2 = tf.ragged.constant([[65, 66], [], [67]])\n",
    "print(tf.concat([r, r2], axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:09:02.299916Z",
     "start_time": "2021-03-16T15:09:02.275980Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233, 68, 69, 70], [67, 111, 102, 102, 101, 101, 71], [99, 97, 102, 102, 232], [21654, 21857, 72, 73]]>\n"
     ]
    }
   ],
   "source": [
    "r3 = tf.ragged.constant([[68, 69, 70], [71], [], [72, 73]])\n",
    "print(tf.concat([r, r3], axis=1)) # 相当于按照两个张量的逐个每一行的元素拼接到一起"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:10:15.613912Z",
     "start_time": "2021-03-16T15:10:15.595934Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=string, numpy=array([b'DEF', b'G', b'', b'HI'], dtype=object)>"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.unicode_encode(r3, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:09:40.158184Z",
     "start_time": "2021-03-16T15:09:40.144218Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 6), dtype=int32, numpy=\n",
       "array([[   67,    97,   102,   233,     0,     0],\n",
       "       [   67,   111,   102,   102,   101,   101],\n",
       "       [   99,    97,   102,   102,   232,     0],\n",
       "       [21654, 21857,     0,     0,     0,     0]])>"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 不规则张量转变成规则张量\n",
    "r.to_tensor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 稀疏张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:12:45.724443Z",
     "start_time": "2021-03-16T15:12:45.709510Z"
    }
   },
   "outputs": [],
   "source": [
    "# 只显示该张量非零值的构成，但不是实际的矩阵\n",
    "s = tf.SparseTensor(indices=[[0, 1], [1, 0], [2, 3]],\n",
    "                    values=[1., 2., 3.],\n",
    "                    dense_shape=[3, 4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:12:46.868104Z",
     "start_time": "2021-03-16T15:12:46.852046Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 1]\n",
      " [1 0]\n",
      " [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:12:59.530469Z",
     "start_time": "2021-03-16T15:12:59.515537Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 1., 0., 0.],\n",
       "       [2., 0., 0., 0.],\n",
       "       [0., 0., 0., 3.]], dtype=float32)>"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 转变成常规tensor\n",
    "tf.sparse.to_dense(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:13:03.233044Z",
     "start_time": "2021-03-16T15:13:03.219055Z"
    }
   },
   "outputs": [],
   "source": [
    "s2 = s * 2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:16:55.435805Z",
     "start_time": "2021-03-16T15:16:55.431815Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unsupported operand type(s) for +: 'SparseTensor' and 'float'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    s3 = s + 1.\n",
    "except TypeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:18:34.585654Z",
     "start_time": "2021-03-16T15:18:34.562715Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[ 30.,  40.],\n",
       "       [ 20.,  40.],\n",
       "       [210., 240.]], dtype=float32)>"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 稀疏张量和普通张量做点积\n",
    "s4 = tf.constant([[10., 20.], [30., 40.], [50., 60.], [70., 80.]])\n",
    "tf.sparse.sparse_dense_matmul(s, s4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:19:38.463729Z",
     "start_time": "2021-03-16T15:19:38.453756Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 2]\n",
      " [0 1]], shape=(2, 2), dtype=int64), values=tf.Tensor([1. 2.], shape=(2,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "s5 = tf.SparseTensor(indices=[[0, 2], [0, 1]],\n",
    "                     values=[1., 2.],\n",
    "                     dense_shape=[3, 4])\n",
    "print(s5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:19:41.556790Z",
     "start_time": "2021-03-16T15:19:41.525900Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "indices[1] = [0,1] is out of order. Many sparse ops require sorted indices.\n",
      "    Use `tf.sparse.reorder` to create a correctly ordered copy.\n",
      "\n",
      " [Op:SparseToDense]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.sparse.to_dense(s5)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 2., 1., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s6 = tf.sparse.reorder(s5)\n",
    "tf.sparse.to_dense(s6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:20:51.709274Z",
     "start_time": "2021-03-16T15:20:51.699301Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 6), dtype=int32, numpy=\n",
       "array([[ 2,  3,  4,  5,  6,  7],\n",
       "       [ 0,  7,  9, 10,  0,  0]])>"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 求并集\n",
    "set1 = tf.constant([[2, 3, 5, 7], [7, 9, 0, 0]])\n",
    "set2 = tf.constant([[4, 5, 6], [9, 10, 0]])\n",
    "tf.sparse.to_dense(tf.sets.union(set1, set2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:21:01.529201Z",
     "start_time": "2021-03-16T15:21:01.514242Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=int32, numpy=\n",
       "array([[2, 3, 7],\n",
       "       [7, 0, 0]])>"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 求差集\n",
    "tf.sparse.to_dense(tf.sets.difference(set1, set2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:21:24.790225Z",
     "start_time": "2021-03-16T15:21:24.776262Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[5, 0],\n",
       "       [0, 9]])>"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 求交集\n",
    "tf.sparse.to_dense(tf.sets.intersection(set1, set2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 张量数组"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:23:39.322264Z",
     "start_time": "2021-03-16T15:23:39.312291Z"
    }
   },
   "outputs": [],
   "source": [
    "array = tf.TensorArray(dtype=tf.float32, size=3)\n",
    "array = array.write(0, tf.constant([1., 2.]))\n",
    "array = array.write(1, tf.constant([3., 10.]))\n",
    "array = array.write(2, tf.constant([5., 7.]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:23:50.054205Z",
     "start_time": "2021-03-16T15:23:50.046226Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([5., 7.], dtype=float32)>"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array.read(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:24:07.055603Z",
     "start_time": "2021-03-16T15:24:07.049634Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[1., 2.],\n",
       "       [0., 0.],\n",
       "       [0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# array 转 tensor\n",
    "array.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:24:21.062676Z",
     "start_time": "2021-03-16T15:24:21.047717Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.33333334, 0.6666667 ], dtype=float32)>"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean, variance = tf.nn.moments(array.stack(), axes=0)\n",
    "mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T15:24:54.719058Z",
     "start_time": "2021-03-16T15:24:54.704098Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.22222221, 0.88888884], dtype=float32)>"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自定义组件\n",
    "\n",
    "注意：\n",
    "- **损失函数，层，激活函数，模型，实现call方法**,\n",
    "- **正则化，初始化，和约束，实现__call__()方法**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T16:56:17.632153Z",
     "start_time": "2021-03-16T16:56:17.617188Z"
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T16:56:38.907249Z",
     "start_time": "2021-03-16T16:56:38.885308Z"
    }
   },
   "outputs": [],
   "source": [
    "housing = fetch_california_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T16:59:58.838066Z",
     "start_time": "2021-03-16T16:59:58.815325Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing['data'], housing['target'].reshape(-1, 1), random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:02:15.173396Z",
     "start_time": "2021-03-16T17:02:15.167412Z"
    }
   },
   "outputs": [],
   "source": [
    "standscaler = StandardScaler()\n",
    "X_train_scaled = standscaler.fit_transform(X_train)\n",
    "X_test_scaled = standscaler.transform(X_test)\n",
    "X_valid_scaled = standscaler.transform(X_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义损失函数\n",
    "\n",
    "假设此处要定义huber损失函数，虽然在keras.losses中可以调用，但是此处以他为例实现如果自定义\n",
    "\n",
    "Huber Loss 是对MSE和MAE的综合，包含了一个超参数 δ。δ 值的大小决定了 Huber Loss 对 MSE 和 MAE 的侧重性，当 $|y−f(x)| ≤ δ$ 时，变为 MSE；当 $|y−f(x)| > δ$ 时，则变成类似于 MAE\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "&L_{\\delta}(\\theta)=\\left\\{\\begin{array}{l}\n",
    "\\frac{1}{2}(y_i-{f(x_i)})^{2}, \\text { if }|y_i-f(x_i)| \\leq \\delta. \\\\\n",
    "\\delta|y_i-f(x_i)|-\\frac{1}{2} \\delta^{2}, \\quad \\text { otherwise }\n",
    "\\end{array}\\right.\\\\\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:10:27.516394Z",
     "start_time": "2021-03-16T17:10:27.506419Z"
    }
   },
   "outputs": [],
   "source": [
    "def huber_fn(y_true, y_pred):\n",
    "    error = y_true - y_pred\n",
    "    is_small_error = tf.abs(error) < 1\n",
    "    square_loss = tf.square(error) / 2\n",
    "    linear_loss = tf.abs(error) - 0.5\n",
    "    return tf.where(is_small_error, square_loss, linear_loss)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:33:20.903863Z",
     "start_time": "2021-03-16T17:33:20.867042Z"
    }
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Input(shape=[8]),\n",
    "    keras.layers.Dense(30, activation='elu'),\n",
    "    keras.layers.Dense(30, activation='elu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=huber_fn,\n",
    "              optimizer=keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:33:45.141162Z",
     "start_time": "2021-03-16T17:33:21.478957Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.7549 - val_loss: 0.2207\n",
      "Epoch 2/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2141 - val_loss: 0.1903\n",
      "Epoch 3/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1997 - val_loss: 0.1898\n",
      "Epoch 4/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1974 - val_loss: 0.1883\n",
      "Epoch 5/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1864 - val_loss: 0.1864\n",
      "Epoch 6/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1827 - val_loss: 0.1909\n",
      "Epoch 7/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1871 - val_loss: 0.1884\n",
      "Epoch 8/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1774 - val_loss: 0.1859\n",
      "Epoch 9/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1816 - val_loss: 0.1883\n",
      "Epoch 10/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1777 - val_loss: 0.1812\n",
      "Epoch 11/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1755 - val_loss: 0.1839\n",
      "Epoch 12/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1748 - val_loss: 0.1850\n",
      "Epoch 13/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1791 - val_loss: 0.1807\n",
      "Epoch 14/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1717 - val_loss: 0.1780\n",
      "Epoch 15/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1711 - val_loss: 0.1788\n",
      "Epoch 16/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1667 - val_loss: 0.1702\n",
      "Epoch 17/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1671 - val_loss: 0.1634\n",
      "Epoch 18/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1675 - val_loss: 0.1612\n",
      "Epoch 19/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1644 - val_loss: 0.1584\n",
      "Epoch 20/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1685 - val_loss: 0.1575\n",
      "Epoch 21/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1650 - val_loss: 0.1521\n",
      "Epoch 22/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1657 - val_loss: 0.1651\n",
      "Epoch 23/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1571 - val_loss: 0.1530\n",
      "Epoch 24/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1654 - val_loss: 0.1548\n",
      "Epoch 25/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1593 - val_loss: 0.1707\n",
      "Epoch 26/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1624 - val_loss: 0.1544\n",
      "Epoch 27/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1610 - val_loss: 0.1611\n",
      "Epoch 28/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1608 - val_loss: 0.1744\n",
      "Epoch 29/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1575 - val_loss: 0.1532\n",
      "Epoch 30/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1587 - val_loss: 0.1474\n"
     ]
    }
   ],
   "source": [
    "\n",
    "history = model.fit(X_train_scaled, y_train,\n",
    "                    epochs=30,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 保存和加载包含自定义组件的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:34:15.469004Z",
     "start_time": "2021-03-16T17:34:15.399924Z"
    }
   },
   "outputs": [],
   "source": [
    "# 保存\n",
    "model.save(\"models/my_model_with_a_custom_loss.h5\")\n",
    "\n",
    "# 加载\n",
    "model = keras.models.load_model(\"models/my_model_with_a_custom_loss.h5\",\n",
    "                                custom_objects={\"huber_fn\": huber_fn})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:34:34.647463Z",
     "start_time": "2021-03-16T17:34:32.911789Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1573 - val_loss: 0.1508\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1571 - val_loss: 0.1507\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14cab5b8fd0>"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:28:37.601135Z",
     "start_time": "2021-03-16T17:28:37.586175Z"
    }
   },
   "outputs": [],
   "source": [
    "# 上面的huber损失函数中对阀值进行了硬编码，但是需要要灵活配置，可以进行适当的wrap\n",
    "def create_huber(threahold=1.0):\n",
    "    def huber_fn(y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < threahold\n",
    "        square_loss = tf.square(error) / 2\n",
    "        linear_loss = threahold*tf.abs(error) - threahold**2 / 2\n",
    "        return tf.where(is_small_error, square_loss, linear_loss)\n",
    "    return huber_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:35:53.140705Z",
     "start_time": "2021-03-16T17:35:48.964158Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.1777 - mae: 0.4246 - val_loss: 0.2105 - val_mae: 0.4291\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.1748 - mae: 0.4209 - val_loss: 0.2704 - val_mae: 0.4645\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14ca9281d60>"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train,\n",
    "          epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不幸的是，对于上面的create_huber函数，保存模型时，阀值不会被保存， 因此加载模型时，必须指定阀值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:37:33.538830Z",
     "start_time": "2021-03-16T17:37:29.559334Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.1791 - mae: 0.4230 - val_loss: 0.2030 - val_mae: 0.4266\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.1756 - mae: 0.4193 - val_loss: 0.2357 - val_mae: 0.4512\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14cab80c640>"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "model.save(\"models/my_model_with_a_custom_loss_threshold_2.h5\")\n",
    "\n",
    "# \n",
    "model = keras.models.load_model(\"models/my_model_with_a_custom_loss_threshold_2.h5\",\n",
    "                                custom_objects={\"huber_fn\": create_huber(2.0)})\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果不想通过上方的形式去传入参数，可以通过继承keras.losses.Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:43:27.408448Z",
     "start_time": "2021-03-16T17:43:27.393488Z"
    }
   },
   "outputs": [],
   "source": [
    "class HuberLoss(keras.losses.Loss):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        self.threshold = threshold\n",
    "        # **kwargs需要传递给父类构造函数\n",
    "        super().__init__(**kwargs)\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < self.threshold\n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss  = self.threshold * tf.abs(error) - self.threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    def get_config(self):\n",
    "        # 返回一个字典，将每个超参数名称映射到其值\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:44:04.645201Z",
     "start_time": "2021-03-16T17:44:01.062960Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 1.6312 - mae: 1.4982 - val_loss: 0.5516 - val_mae: 0.6597\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.2563 - mae: 0.5257 - val_loss: 0.4740 - val_mae: 0.6163\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14caf577430>"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=[8]),\n",
    "    keras.layers.Dense(1),\n",
    "])\n",
    "\n",
    "model.compile(loss=HuberLoss(2.), optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:47:13.745474Z",
     "start_time": "2021-03-16T17:47:13.683144Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 当保存模型时，keras会调用损失实例的get_config(),并将配置以JSON格式保存到HDF5文件中\n",
    "model.save(\"models/my_model_with_a_custom_loss_class.h5\")\n",
    "\n",
    "# 加载模型时，他在huberloss类上调用from_config()类方法\n",
    "model = keras.models.load_model(\"models/my_model_with_a_custom_loss_class.h5\",\n",
    "                                custom_objects={\"HuberLoss\": HuberLoss})\n",
    "\n",
    "model.loss.threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义激活函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:54:15.809651Z",
     "start_time": "2021-03-16T17:54:15.798684Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_softplus(z):\n",
    "    return tf.math.log(1.0 + tf.exp(z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义初始化方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:57:10.296951Z",
     "start_time": "2021-03-16T17:57:10.292961Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_glorot_initializer(shape, dtype=tf.float32):\n",
    "    stddev = tf.sqrt(2.0 / (shape[0] + shape[1]))\n",
    "    return tf.random.normal(shape, mean=0., stddev=stddev, dtype=dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义正则化方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T17:58:15.602074Z",
     "start_time": "2021-03-16T17:58:15.588606Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_l1_regularizer(weights):\n",
    "    return tf.reduce_sum(tf.abs(0.01*weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义约束"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:04:00.406358Z",
     "start_time": "2021-03-16T18:04:00.391370Z"
    }
   },
   "outputs": [],
   "source": [
    "def my_positive_weights(weights):\n",
    "    return tf.where(weights < 0, tf.zeros_like(weights), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:04:00.782085Z",
     "start_time": "2021-03-16T18:04:00.745152Z"
    }
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=[8]),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=my_l1_regularizer,\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:04:05.335908Z",
     "start_time": "2021-03-16T18:04:01.729542Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 2.6346 - mae: 1.1389 - val_loss: inf - val_mae: inf\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.7021 - mae: 0.5307 - val_loss: inf - val_mae: inf\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14cb1464280>"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:06:09.350341Z",
     "start_time": "2021-03-16T18:06:09.280305Z"
    }
   },
   "outputs": [],
   "source": [
    "model.save(\"models/my_model_with_many_custom_parts.h5\")\n",
    "\n",
    "model = keras.models.load_model(\"models/my_model_with_many_custom_parts.h5\",\n",
    "           custom_objects={\"my_l1_regularizer\": my_l1_regularizer,\n",
    "                           \"my_positive_weights\": my_positive_weights,\n",
    "                           \"my_glorot_initializer\": my_glorot_initializer,\n",
    "                           \"my_softplus\": my_softplus,\n",
    "                        })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:08:57.577667Z",
     "start_time": "2021-03-16T18:08:57.569689Z"
    }
   },
   "outputs": [],
   "source": [
    "class MyL1Regularizer(keras.regularizers.Regularizer):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "    \n",
    "    def __call__(self, weights):\n",
    "        return tf.reduce_sum(tf.abs(self.factor * weights))\n",
    "    \n",
    "    def get_config(self):\n",
    "        return {\"factor\": self.factor}\n",
    "\n",
    "# 注意上面的子类中没有调用父类构造函数或父类的get_config方法，是因为factor不是父类定义的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上，如果其他的函数也要保存超参数的话，需要适当的继承类\n",
    "\n",
    "- 自定义正则化+超参数：继承keras.regularizers.Regularizer\n",
    "- 自定义约束+超参数：继承keras.constraints.Constraint\n",
    "- 自定义初始化+超参数：继承keras.initializers.Initializer\n",
    "- 自定义层+超参数：继承keras.layers.Layer\n",
    "\n",
    "注意：\n",
    "- **损失函数，层，激活函数，模型，实现call方法**,\n",
    "- **正则化，初始化，和约束，实现__call__()方法**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:09:38.649477Z",
     "start_time": "2021-03-16T18:09:34.938681Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 2.3829 - mae: 1.1635 - val_loss: 1.4155 - val_mae: 0.5607\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.6299 - mae: 0.5410 - val_loss: 1.4397 - val_mae: 0.5137\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x14cafd3f190>"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=[8]),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=MyL1Regularizer(0.01),\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-16T18:10:02.431891Z",
     "start_time": "2021-03-16T18:10:02.359126Z"
    }
   },
   "outputs": [],
   "source": [
    "model.save(\"models/my_model_with_many_custom_parts.h5\")\n",
    "\n",
    "model = keras.models.load_model(\n",
    "    \"models/my_model_with_many_custom_parts.h5\",\n",
    "    custom_objects={\n",
    "       \"MyL1Regularizer\": MyL1Regularizer,\n",
    "       \"my_positive_weights\": my_positive_weights,\n",
    "       \"my_glorot_initializer\": my_glorot_initializer,\n",
    "       \"my_softplus\": my_softplus,\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义指标\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义模型内部损失"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义训练过程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自动微分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 计算图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python函数与TF函数转换"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 转换dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### repeat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### apply"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### take"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### shuffle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### list_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### interleave"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预处理dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预处理层与Data API结合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 内置通用dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TFRecord"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "239.184px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}