{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"cellView": "form",
"id": "tuOe1ymfHZPu"
},
"source": [
"````{admonition} Copyright 2020 The TensorFlow Authors.\n",
"```\n",
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License.\n",
"```\n",
"````"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qFdPvlXBOdUN"
},
"source": [
"# 梯度和自动微分简介"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MfBg1C5NB3X0"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r6P32iYYV27b"
},
"source": [
"## 自动微分和梯度\n",
"\n",
"[自动微分](https://en.wikipedia.org/wiki/Automatic_differentiation)对于实现机器学习算法（例如，用于训练神经网络的[反向传播](https://en.wikipedia.org/wiki/Backpropagation)）非常有用。\n",
"\n",
"在本指南中，您将探索使用 TensorFlow 计算梯度的方法，尤其是在 [Eager Execution](eager.ipynb) 中。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MUXex9ctTuDB"
},
"source": [
"## 设置"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from set_env import temp_dir"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "IqR2PQG4ZaZ0"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xHxb-dlhMIzW"
},
"source": [
"## 计算梯度\n",
"\n",
"要实现自动微分，TensorFlow 需要记住在*前向*传递过程中哪些运算以何种顺序发生。随后，在*后向传递*期间，TensorFlow 以相反的顺序遍历此运算列表来计算梯度。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1CLWJl0QliB0"
},
"source": [
"## 梯度带\n",
"\n",
"TensorFlow 为自动微分提供了 `tf.GradientTape` API；即计算某个计算相对于某些输入（通常是 `tf.Variable`）的梯度。TensorFlow 会将在 `tf.GradientTape` 上下文内执行的相关运算“记录”到“条带”上。TensorFlow 随后会该使用条带通过[反向模式微分](https://en.wikipedia.org/wiki/Automatic_differentiation)计算“记录的”计算的梯度。\n",
"\n",
"例如："
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "Xq9GgTCP7a4A"
},
"outputs": [],
"source": [
"x = tf.Variable(3.0)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" y = x**2"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CR9tFAP_7cra"
},
"source": [
"记录一些运算后，使用 `GradientTape.gradient(target, sources)` 计算某个目标（通常是损失）相对于某个源（通常是模型变量）的梯度。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "LsvrwF6bHroC"
},
"outputs": [
{
"data": {
"text/plain": [
"6.0"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# dy = 2x * dx\n",
"dy_dx = tape.gradient(y, x)\n",
"dy_dx.numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q2_aqsO25Vx1"
},
"source": [
"上方示例使用标量，但是 `tf.GradientTape` 在任何张量上都可以轻松运行："
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "vacZ3-Ws5VdV"
},
"outputs": [],
"source": [
"w = tf.Variable(tf.random.normal((3, 2)), name='w')\n",
"b = tf.Variable(tf.zeros(2, dtype=tf.float32), name='b')\n",
"x = [[1., 2., 3.]]\n",
"\n",
"with tf.GradientTape(persistent=True) as tape:\n",
" y = x @ w + b\n",
" loss = tf.reduce_mean(y**2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i4eXOkrQ-9Pb"
},
"source": [
"要获得 `loss` 相对于两个变量的梯度，可以将这两个变量同时作为 `gradient` 方法的源传递。梯度带在关于源的传递方式上非常灵活，可以接受列表或字典的任何嵌套组合，并以相同的方式返回梯度结构（请参阅 `tf.nest`）。"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "luOtK1Da_BR0"
},
"outputs": [],
"source": [
"[dl_dw, dl_db] = tape.gradient(loss, [w, b])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ei4iVXi6qgM7"
},
"source": [
"相对于每个源的梯度具有源的形状："
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "aYbWRFPZqk4U"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(3, 2)\n",
"(3, 2)\n"
]
}
],
"source": [
"print(w.shape)\n",
"print(dl_dw.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dI_SzxHsvao1"
},
"source": [
"此处也为梯度计算，这一次传递了一个变量字典："
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "d73cY6NOuaMd"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_vars = {\n",
" 'w': w,\n",
" 'b': b\n",
"}\n",
"\n",
"grad = tape.gradient(loss, my_vars)\n",
"grad['b']"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HZ2LvHifEMgO"
},
"source": [
"## 相对于模型的梯度\n",
"\n",
"通常将 `tf.Variables` 收集到 `tf.Module` 或其子类之一（`layers.Layer`、`keras.Model`）中，用于[设置检查点](checkpoint.ipynb)和[导出](saved_model.ipynb)。\n",
"\n",
"在大多数情况下，需要计算相对于模型的可训练变量的梯度。 由于 `tf.Module` 的所有子类都在 `Module.trainable_variables` 属性中聚合其变量，您可以用几行代码计算这些梯度： "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "JvesHtbQESc-"
},
"outputs": [],
"source": [
"layer = tf.keras.layers.Dense(2, activation='relu')\n",
"x = tf.constant([[1., 2., 3.]])\n",
"\n",
"with tf.GradientTape() as tape:\n",
" # Forward pass\n",
" y = layer(x)\n",
" loss = tf.reduce_mean(y**2)\n",
"\n",
"# Calculate gradients with respect to every trainable variable\n",
"grad = tape.gradient(loss, layer.trainable_variables)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "PR_ezr6UFrpI"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"kernel, shape: (3, 2)\n",
"bias, shape: (2,)\n"
]
}
],
"source": [
"for var, g in zip(layer.trainable_variables, grad):\n",
" print(f'{var.name}, shape: {g.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6Gx6LS714zR"
},
"source": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(x, y, label='y')\n",
"plt.plot(x, dy_dx, label='dy/dx')\n",
"plt.legend()\n",
"_ = plt.xlabel('x')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6kADybtQzYj4"
},
"source": [
"## 控制流\n",
"\n",
"在执行运算时，由于梯度带会记录这些运算，因此会自然地处理 Python 控制流（例如 `if` 和 `while` 语句）。\n",
"\n",
"此处，`if` 的每个分支上使用不同变量。梯度仅连接到使用的变量："
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "ciFLizhrrjy7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor(1.0, shape=(), dtype=float32)\n",
"None\n"
]
}
],
"source": [
"x = tf.constant(1.0)\n",
"\n",
"v0 = tf.Variable(2.0)\n",
"v1 = tf.Variable(2.0)\n",
"\n",
"with tf.GradientTape(persistent=True) as tape:\n",
" tape.watch(x)\n",
" if x > 0.0:\n",
" result = v0\n",
" else:\n",
" result = v1**2 \n",
"\n",
"dv0, dv1 = tape.gradient(result, [v0, v1])\n",
"\n",
"print(dv0)\n",
"print(dv1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HKnLaiapsjeP"
},
"source": [
"注意，控制语句本身不可微分，因此对基于梯度的优化器不可见。\n",
"\n",
"根据上面示例中 `x` 的值，梯度带将记录 `result = v0` 或 `result = v1**2`。 相对于 `x` 的梯度始终为 `None`。"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "8k05WmuAwPm7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"dx = tape.gradient(result, x)\n",
"\n",
"print(dx)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "egypBxISAHhx"
},
"source": [
"## `gradient` 返回 `None` 的情况\n",
"\n",
"当目标未连接到源时，`gradient` 将返回 `None`。\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"id": "CU185WDM81Ut"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"x = tf.Variable(2.)\n",
"y = tf.Variable(3.)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" z = y * y\n",
"print(tape.gradient(z, x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sZbKpHfBRJym"
},
"source": [
"此处 `z` 显然未连接到 `x`，但可以通过几种不太明显的方式将梯度断开。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eHDzDOiQ8xmw"
},
"source": [
"### 1. 使用张量替换变量\n",
"\n",
"在[控制梯度带监视内容](#watches)部分中，梯度带会自动监视 `tf.Variable`，但不会监视 `tf.Tensor`。\n",
"\n",
"一个常见错误是无意中将 `tf.Variable` 替换为 `tf.Tensor`，而不使用 `Variable.assign` 更新 `tf.Variable`。见下例："
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"id": "QPKY4Tn9zX7_"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ResourceVariable : tf.Tensor(1.0, shape=(), dtype=float32)\n",
"EagerTensor : None\n"
]
}
],
"source": [
"x = tf.Variable(2.0)\n",
"\n",
"for epoch in range(2):\n",
" with tf.GradientTape() as tape:\n",
" y = x+1\n",
"\n",
" print(type(x).__name__, \":\", tape.gradient(y, x))\n",
" x = x + 1 # This should be `x.assign_add(1)`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3gwZKxgA97an"
},
"source": [
"### 2.在 TensorFlow 之外进行了计算\n",
"\n",
"如果计算退出 TensorFlow，梯度带将无法记录梯度路径。例如："
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"id": "jmoLCDJb_yw1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"x = tf.Variable([[1.0, 2.0],\n",
" [3.0, 4.0]], dtype=tf.float32)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" x2 = x**2\n",
"\n",
" # This step is calculated with NumPy\n",
" y = np.mean(x2, axis=0)\n",
"\n",
" # Like most ops, reduce_mean will cast the NumPy array to a constant tensor\n",
" # using `tf.convert_to_tensor`.\n",
" y = tf.reduce_mean(y, axis=0)\n",
"\n",
"print(tape.gradient(y, x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p3YVfP3R-tp7"
},
"source": [
"### 3.通过整数或字符串获取梯度\n",
"\n",
"整数和字符串不可微分。如果计算路径使用这些数据类型，则不会出现梯度。\n",
"\n",
"谁也不会期望字符串是可微分的，但是如果不指定 `dtype`，很容易意外创建一个 `int` 常量或变量。"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"id": "9jlHXHqfASU3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"x = tf.constant(10)\n",
"\n",
"with tf.GradientTape() as g:\n",
" g.watch(x)\n",
" y = x * x\n",
"\n",
"print(g.gradient(y, x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RsdP_mTHX9L1"
},
"source": [
"TensorFlow 不会在类型之间自动进行转换，因此，在实践中，您经常会遇到类型错误而不是缺少梯度。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WyAZ7C8qCEs6"
},
"source": [
"### 4. 通过有状态对象获取梯度\n",
"\n",
"状态会停止梯度。从有状态对象读取时，梯度带只能观察当前状态，而不能观察导致该状态的历史记录。\n",
"\n",
"`tf.Tensor` 不可变。张量创建后就不能更改。它有一个*值*，但没有*状态*。目前讨论的所有运算也都无状态：`tf.matmul` 的输出只取决于它的输入。\n",
"\n",
"`tf.Variable` 具有内部状态，即它的值。使用变量时，会读取状态。计算相对于变量的梯度是正常操作，但是变量的状态会阻止梯度计算进一步向后移动。 例如：\n"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"id": "C1tLeeRFE479"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"None\n"
]
}
],
"source": [
"x0 = tf.Variable(3.0)\n",
"x1 = tf.Variable(0.0)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" # Update x1 = x1 + x0.\n",
" x1.assign_add(x0)\n",
" # The tape starts recording from x1.\n",
" y = x1**2 # y = (x1 + x0)**2\n",
"\n",
"# This doesn't work.\n",
"print(tape.gradient(y, x0)) #dy/dx0 = 2*(x1 + x0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xKA92-dqF2r-"
},
"source": [
"类似地，`tf.data.Dataset` 迭代器和 `tf.queue` 也有状态，会停止经过它们的张量上的所有梯度。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HHvcDGIbOj2I"
},
"source": [
"## 未注册梯度"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aoc-A6AxVqry"
},
"source": [
"某些 `tf.Operation` 被**注册为不可微分**，将返回 `None`。还有一些则**未注册梯度**。\n",
"\n",
"`tf.raw_ops` 页面显示了哪些低级运算已经注册梯度。\n",
"\n",
"如果您试图通过一个没有注册梯度的浮点运算获取梯度，梯度带将抛出错误，而不是直接返回 `None`。这样一来，您可以了解某个环节出现问题。\n",
"\n",
"例如，`tf.image.adjust_contrast` 函数封装了 `raw_ops.AdjustContrastv2`，此运算可能具有梯度，但未实现该梯度：\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"id": "HSb20FXc_V0U"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LookupError: gradient registry has no entry for: AdjustContrastv2\n"
]
}
],
"source": [
"image = tf.Variable([[[0.5, 0.0, 0.0]]])\n",
"delta = tf.Variable(0.1)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" new_image = tf.image.adjust_contrast(image, delta)\n",
"\n",
"try:\n",
" print(tape.gradient(new_image, [image, delta]))\n",
" assert False # This should not happen.\n",
"except LookupError as e:\n",
" print(f'{type(e).__name__}: {e}')\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pDoutjzATiEm"
},
"source": [
"如果需要通过此运算进行微分，则需要实现梯度并注册该梯度（使用 `tf.RegisterGradient`），或者使用其他运算重新实现该函数。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GCTwc_dQXp2W"
},
"source": [
"## 零而不是 None"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TYDrVogA89eA"
},
"source": [
"在某些情况下，对于未连接的梯度，得到 0 而不是 `None` 会比较方便。您可以使用 `unconnected_gradients` 参数来决定具有未连接的梯度时返回的内容："
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"id": "U6zxk1sf9Ixx"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor([0. 0.], shape=(2,), dtype=float32)\n"
]
}
],
"source": [
"x = tf.Variable([2., 2.])\n",
"y = tf.Variable(3.)\n",
"\n",
"with tf.GradientTape() as tape:\n",
" z = y**2\n",
"print(tape.gradient(z, x, unconnected_gradients=tf.UnconnectedGradients.ZERO))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"Tce3stUlHN0L"
],
"name": "autodiff.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "xxx",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
![](\"https://tensorflow.google.cn/images/tf_logo_32px.png\"){kind=logo}
在 TensorFlow.org 上查看 ![](\"https://tensorflow.google.cn/images/colab_logo_32px.png\"){kind=logo}
在 Google Colab 中运行 ![](\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\")
在 GitHub 上查看源代码 ![](\"https://tensorflow.google.cn/images/download_logo_32px.png\"){kind=logo}
下载笔记本