{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "outputs": [],
      "source": [
        "##### Copyright 2022 The TensorFlow Compression Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# 可扩展的模型压缩"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://tensorflow.google.cn/tutorials/optimization/compression\"><img src=\"https://tensorflow.google.cn/images/tf_logo_32px.png\">在 TensorFlow.org 上查看</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/optimization/compression.ipynb\">     <img src=\"https://tensorflow.google.cn/images/colab_logo_32px.png\">     在 Google Colab 中运行</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/zh-cn/tutorials/optimization/compression.ipynb\"><img src=\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\">在 Github 上查看源代码</a>\n",
        "</td>\n",
        "  <td>     <a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/zh-cn/tutorials/optimization/compression.ipynb\"><img src=\"https://tensorflow.google.cn/images/download_logo_32px.png\">下载笔记本</a>\n",
        "</td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## 概述\n",
        "\n",
        "本笔记本展示了如何使用 [TensorFlow Compression](https://github.com/tensorflow/compression) 压缩模型。\n",
        "\n",
        "在下面的示例中，我们将 MNIST 分类器的权重压缩到比其浮点表示小得多的大小，同时保持分类准确率。这是通过基于论文 [Scalable Model Compression by Entropy Penalized Reparameterization](https://arxiv.org/abs/1906.06624) 的两步过程完成的：\n",
        "\n",
        "- 在训练期间使用显式**熵惩罚**来训练“可压缩”模型，这鼓励了模型参数的可压缩性。此惩罚的权重 $\\lambda$，能够持续控制压缩模型大小和其准确率之间的权衡。\n",
        "\n",
        "- 使用与惩罚相匹配的编码方案将可压缩模型编码为压缩模型，这意味着惩罚是对模型大小的良好预测指标。这确保了该方法不需要多次迭代训练、压缩和重新训练模型以进行微调。\n",
        "\n",
        "这种方法会严格考虑压缩模型的大小，而不是计算复杂度。它可以与模型剪枝等技术相结合，以减少大小和复杂度。\n",
        "\n",
        "各种模型的压缩结果示例：\n",
        "\n",
        "模型（数据集） | 模型大小 | 压缩率 | Top-1 错误压缩（解压缩）\n",
        "--- | --- | --- | ---\n",
        "LeNet300-100 (MNIST) | 8.56 KB | 124x | 1.9%  (1.6%)\n",
        "LeNet5-Caffe (MNIST) | 2.84 KB | 606x | 1.0%  (0.7%)\n",
        "VGG-16 (CIFAR-10) | 101 KB | 590x | 10.0%  (6.6%)\n",
        "ResNet-20-4 (CIFAR-10) | 128 KB | 134x | 8.8%  (5.0%)\n",
        "ResNet-18 (ImageNet) | 1.97 MB | 24x | 30.0% (30.0%)\n",
        "ResNet-50 (ImageNet) | 5.49 MB | 19x | 26.0% (25.0%)\n",
        "\n",
        "应用包括：\n",
        "\n",
        "- 大规模部署/广播模型到边缘设备，节省传输带宽。\n",
        "- 在联合学习中向客户端传达全局模型状态。模型架构（隐藏单元的数量等）相较于初始模型没有变化，客户端可以在解压缩的模型上继续学习。\n",
        "- 在内存极其有限的客户端上执行推断。在推断过程中，可以按顺序解压缩每一层的权重，并在计算激活后立即丢弃。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## 设置\n",
        "\n",
        "通过 `pip` 安装 TensorFlow Compression。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K489KsEgxuLI",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "# Installs the latest version of TFC compatible with the installed TF version.\n",
        "\n",
        "read MAJOR MINOR <<< \"$(pip show tensorflow | perl -p -0777 -e 's/.*Version: (\\d+)\\.(\\d+).*/\\1 \\2/sg')\"\n",
        "pip install \"tensorflow-compression<$MAJOR.$(($MINOR+1))\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfVAmHCVxpTS"
      },
      "source": [
        "导入库依赖项。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IqR2PQG4ZaZ0",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "import tensorflow_compression as tfc\n",
        "import tensorflow_datasets as tfds\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wsncKT2iymgQ"
      },
      "source": [
        "## 定义和训练一个基本的 MNIST 分类器\n",
        "\n",
        "为了高效压缩密集层和卷积层，我们需要定义自定义层类。这些类似于 `tf.keras.layers` 下的层，但我们稍后将对它们进行子类化以高效实现熵惩罚重参数化 (EPR)。为此，我们还添加了一个复制构造函数。\n",
        "\n",
        "首先，我们定义一个标准的密集层："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n_7ZRqiaO1WQ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CustomDense(tf.keras.layers.Layer):\n",
        "\n",
        "  def __init__(self, filters, name=\"dense\"):\n",
        "    super().__init__(name=name)\n",
        "    self.filters = filters\n",
        "\n",
        "  @classmethod\n",
        "  def copy(cls, other, **kwargs):\n",
        "    \"\"\"Returns an instantiated and built layer, initialized from `other`.\"\"\"\n",
        "    self = cls(filters=other.filters, name=other.name, **kwargs)\n",
        "    self.build(None, other=other)\n",
        "    return self\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
        "    if other is None:\n",
        "      kernel_shape = (input_shape[-1], self.filters)\n",
        "      kernel = tf.keras.initializers.GlorotUniform()(shape=kernel_shape)\n",
        "      bias = tf.keras.initializers.Zeros()(shape=(self.filters,))\n",
        "    else:\n",
        "      kernel, bias = other.kernel, other.bias\n",
        "    self.kernel = tf.Variable(\n",
        "        tf.cast(kernel, self.variable_dtype), name=\"kernel\")\n",
        "    self.bias = tf.Variable(\n",
        "        tf.cast(bias, self.variable_dtype), name=\"bias\")\n",
        "    self.built = True\n",
        "\n",
        "  def call(self, inputs):\n",
        "    outputs = tf.linalg.matvec(self.kernel, inputs, transpose_a=True)\n",
        "    outputs = tf.nn.bias_add(outputs, self.bias)\n",
        "    return tf.nn.leaky_relu(outputs)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RUZkcXegc0yR"
      },
      "source": [
        "类似地，定义一个 2D 卷积层："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RDibtb8EWCSj",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CustomConv2D(tf.keras.layers.Layer):\n",
        "\n",
        "  def __init__(self, filters, kernel_size,\n",
        "               strides=1, padding=\"SAME\", name=\"conv2d\"):\n",
        "    super().__init__(name=name)\n",
        "    self.filters = filters\n",
        "    self.kernel_size = kernel_size\n",
        "    self.strides = strides\n",
        "    self.padding = padding\n",
        "\n",
        "  @classmethod\n",
        "  def copy(cls, other, **kwargs):\n",
        "    \"\"\"Returns an instantiated and built layer, initialized from `other`.\"\"\"\n",
        "    self = cls(filters=other.filters, kernel_size=other.kernel_size,\n",
        "               strides=other.strides, padding=other.padding, name=other.name,\n",
        "               **kwargs)\n",
        "    self.build(None, other=other)\n",
        "    return self\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
        "    if other is None:\n",
        "      kernel_shape = 2 * (self.kernel_size,) + (input_shape[-1], self.filters)\n",
        "      kernel = tf.keras.initializers.GlorotUniform()(shape=kernel_shape)\n",
        "      bias = tf.keras.initializers.Zeros()(shape=(self.filters,))\n",
        "    else:\n",
        "      kernel, bias = other.kernel, other.bias\n",
        "    self.kernel = tf.Variable(\n",
        "        tf.cast(kernel, self.variable_dtype), name=\"kernel\")\n",
        "    self.bias = tf.Variable(\n",
        "        tf.cast(bias, self.variable_dtype), name=\"bias\")\n",
        "    self.built = True\n",
        "\n",
        "  def call(self, inputs):\n",
        "    outputs = tf.nn.convolution(\n",
        "        inputs, self.kernel, strides=self.strides, padding=self.padding)\n",
        "    outputs = tf.nn.bias_add(outputs, self.bias)\n",
        "    return tf.nn.leaky_relu(outputs)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xWa1hHMdCpG"
      },
      "source": [
        "在继续模型压缩之前，我们来检查一下是否可以成功地训练一个常规分类器。\n",
        "\n",
        "定义模型架构："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8yZESLgW-vp1",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "classifier = tf.keras.Sequential([\n",
        "    CustomConv2D(20, 5, strides=2, name=\"conv_1\"),\n",
        "    CustomConv2D(50, 5, strides=2, name=\"conv_2\"),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    CustomDense(500, name=\"fc_1\"),\n",
        "    CustomDense(10, name=\"fc_2\"),\n",
        "], name=\"classifier\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9iRSvt_CdUuY"
      },
      "source": [
        "加载训练数据："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L4bsA3HFF2k0",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def normalize_img(image, label):\n",
        "  \"\"\"Normalizes images: `uint8` -> `float32`.\"\"\"\n",
        "  return tf.cast(image, tf.float32) / 255., label\n",
        "\n",
        "training_dataset, validation_dataset = tfds.load(\n",
        "    \"mnist\",\n",
        "    split=[\"train\", \"test\"],\n",
        "    shuffle_files=True,\n",
        "    as_supervised=True,\n",
        "    with_info=False,\n",
        ")\n",
        "training_dataset = training_dataset.map(normalize_img)\n",
        "validation_dataset = validation_dataset.map(normalize_img)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rR9WYjt_daRG"
      },
      "source": [
        "最后，训练模型："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ROn2DbzsBirI",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def train_model(model, training_data, validation_data, **kwargs):\n",
        "  model.compile(\n",
        "      optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),\n",
        "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n",
        "      # Uncomment this to ease debugging:\n",
        "      # run_eagerly=True,\n",
        "  )\n",
        "  kwargs.setdefault(\"epochs\", 5)\n",
        "  kwargs.setdefault(\"verbose\", 1)\n",
        "  log = model.fit(\n",
        "      training_data.batch(128).prefetch(8),\n",
        "      validation_data=validation_data.batch(128).cache(),\n",
        "      validation_freq=1,\n",
        "      **kwargs,\n",
        "  )\n",
        "  return log.history[\"val_sparse_categorical_accuracy\"][-1]\n",
        "\n",
        "classifier_accuracy = train_model(\n",
        "    classifier, training_dataset, validation_dataset)\n",
        "\n",
        "print(f\"Accuracy: {classifier_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QupWKZ91di-y"
      },
      "source": [
        "成功！该模型训练良好，在 5 个周期内的验证集上的准确率达到了 98% 以上。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRqZFwb5dqQm"
      },
      "source": [
        "## 训练可压缩分类器\n",
        "\n",
        "熵惩罚重参数化（EPR）有两个主要组成部分：\n",
        "\n",
        "- 在训练期间对模型权重施加**惩罚**，该惩罚对应于概率模型下的熵，并与权重的编码方案相匹配。下面，我们定义一个实现此惩罚的 Keras `Regularizer`。\n",
        "\n",
        "- **重新参数化**权重，即将它们带入更具可压缩性的潜在表示中（在可压缩性和模型性能之间达成更好的权衡）。对于卷积核，[已经证明](https://arxiv.org/abs/1906.06624)傅里叶域是一个很好的表示。对于其他参数，以下示例仅使用具有不同量化步长的标量量化（舍入）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e4jmnqEmO6eB"
      },
      "source": [
        "首先，定义惩罚。\n",
        "\n",
        "下面的示例使用在 `tfc.PowerLawEntropyModel` 类中实现的代码/概率模型，灵感来自论文 [Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory](https://arxiv.org/abs/2201.02664)。惩罚定义为：$$ \\log \\Bigl(\\frac {|x| + \\alpha} \\alpha\\Bigr)，$$ 其中 $x$ 是模型参数或其潜在表示的一个元素，$\\alpha$ 是一个数值稳定性在 0 附近小常量。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hh57nxjuwocc",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "_ = tf.linspace(-5., 5., 501)\n",
        "plt.plot(_, tfc.PowerLawEntropyModel(0).penalty(_));\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gr3-6vLrwo-H"
      },
      "source": [
        "这种惩罚实际上是一种正则化损失（有时称为“权重损失”）。它是凹形的，顶点为零，这一事实鼓励权重稀疏。用于压缩权重的编码方案是 [Elias gamma 码](https://en.wikipedia.org/wiki/Elias_gamma_coding)，它为元素大小产生长度为 $ 1 + \\lfloor \\log_2 |x| \\rfloor $ 比特的编码。也就是说，它与惩罚相匹配，并应用惩罚从而最小化预期的代码长度。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H1Yt6e1ub6pU",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class PowerLawRegularizer(tf.keras.regularizers.Regularizer):\n",
        "\n",
        "  def __init__(self, lmbda):\n",
        "    super().__init__()\n",
        "    self.lmbda = lmbda\n",
        "\n",
        "  def __call__(self, variable):\n",
        "    em = tfc.PowerLawEntropyModel(coding_rank=variable.shape.rank)\n",
        "    return self.lmbda * em.penalty(variable)\n",
        "\n",
        "# Normalizing the weight of the penalty by the number of model parameters is a\n",
        "# good rule of thumb to produce comparable results across models.\n",
        "regularizer = PowerLawRegularizer(lmbda=2./classifier.count_params())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kyQc35QTf8Aq"
      },
      "source": [
        "其次，定义 `CustomDense` 和 `CustomConv2D` 的子类，它们具有以下附加功能：\n",
        "\n",
        "- 它们接受上述 Regularizer 的一个实例，并将其应用于训练期间的内核和偏差。\n",
        "- 它们将内核和偏差定义为 `@property`，每当访问变量时，它们都会使用直通梯度执行量化。这准确地反映了稍后在压缩模型中执行的计算。\n",
        "- 它们定义了额外的 `log_step` 变量，代表量化步长的对数。量化越粗，模型越小，但准确率越低。每个模型参数的量化步长都可训练，因此对惩罚损失函数执行优化将确定最佳量化步长。\n",
        "\n",
        "量化步长定义如下："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "60fMt3avgSFw",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def quantize(latent, log_step):\n",
        "  step = tf.exp(log_step)\n",
        "  return tfc.round_st(latent / step) * step\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stKrchp7mB0b"
      },
      "source": [
        "有了它，我们可以定义密集层："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ciz1F1WsXre_",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CompressibleDense(CustomDense):\n",
        "\n",
        "  def __init__(self, regularizer, *args, **kwargs):\n",
        "    super().__init__(*args, **kwargs)\n",
        "    self.regularizer = regularizer\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
        "    super().build(input_shape, other=other)\n",
        "    if other is not None and hasattr(other, \"kernel_log_step\"):\n",
        "      kernel_log_step = other.kernel_log_step\n",
        "      bias_log_step = other.bias_log_step\n",
        "    else:\n",
        "      kernel_log_step = bias_log_step = -4.\n",
        "    self.kernel_log_step = tf.Variable(\n",
        "        tf.cast(kernel_log_step, self.variable_dtype), name=\"kernel_log_step\")\n",
        "    self.bias_log_step = tf.Variable(\n",
        "        tf.cast(bias_log_step, self.variable_dtype), name=\"bias_log_step\")\n",
        "    self.add_loss(lambda: self.regularizer(\n",
        "        self.kernel_latent / tf.exp(self.kernel_log_step)))\n",
        "    self.add_loss(lambda: self.regularizer(\n",
        "        self.bias_latent / tf.exp(self.bias_log_step)))\n",
        "\n",
        "  @property\n",
        "  def kernel(self):\n",
        "    return quantize(self.kernel_latent, self.kernel_log_step)\n",
        "\n",
        "  @kernel.setter\n",
        "  def kernel(self, kernel):\n",
        "    self.kernel_latent = tf.Variable(kernel, name=\"kernel_latent\")\n",
        "\n",
        "  @property\n",
        "  def bias(self):\n",
        "    return quantize(self.bias_latent, self.bias_log_step)\n",
        "\n",
        "  @bias.setter\n",
        "  def bias(self, bias):\n",
        "    self.bias_latent = tf.Variable(bias, name=\"bias_latent\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CsykbQO0hxzW"
      },
      "source": [
        "卷积层类似。此外，只要设置了卷积核，就会将卷积核作为其实值离散傅里叶变换 (RDFT) 存储，并且每当使用该核时，变换都会被反转。由于内核的不同频率分量往往或多或少是可压缩的，因此其中的每个分量都被分配了自己的量化步长。\n",
        "\n",
        "按如下方式定义傅里叶变换及其逆变换："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rUFMKGHDguJS",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def to_rdft(kernel, kernel_size):\n",
        "  # The kernel has shape (H, W, I, O) -> transpose to take DFT over last two\n",
        "  # dimensions.\n",
        "  kernel = tf.transpose(kernel, (2, 3, 0, 1))\n",
        "  # The RDFT has type complex64 and shape (I, O, FH, FW).\n",
        "  kernel_rdft = tf.signal.rfft2d(kernel)\n",
        "  # Map real and imaginary parts into regular floats. The result is float32\n",
        "  # and has shape (I, O, FH, FW, 2).\n",
        "  kernel_rdft = tf.stack(\n",
        "      [tf.math.real(kernel_rdft), tf.math.imag(kernel_rdft)], axis=-1)\n",
        "  # Divide by kernel size to make the DFT orthonormal (length-preserving).\n",
        "  return kernel_rdft / kernel_size\n",
        "\n",
        "def from_rdft(kernel_rdft, kernel_size):\n",
        "  # Undoes the transformations in to_rdft.\n",
        "  kernel_rdft *= kernel_size\n",
        "  kernel_rdft = tf.dtypes.complex(*tf.unstack(kernel_rdft, axis=-1))\n",
        "  kernel = tf.signal.irfft2d(kernel_rdft, fft_length=2 * (kernel_size,))\n",
        "  return tf.transpose(kernel, (2, 3, 0, 1))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "esZZrJ5ImVDY"
      },
      "source": [
        "这样，将卷积层定义为："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YKzXBNCO7bjB",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CompressibleConv2D(CustomConv2D):\n",
        "\n",
        "  def __init__(self, regularizer, *args, **kwargs):\n",
        "    super().__init__(*args, **kwargs)\n",
        "    self.regularizer = regularizer\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    \"\"\"Instantiates weights, optionally initializing them from `other`.\"\"\"\n",
        "    super().build(input_shape, other=other)\n",
        "    if other is not None and hasattr(other, \"kernel_log_step\"):\n",
        "      kernel_log_step = other.kernel_log_step\n",
        "      bias_log_step = other.bias_log_step\n",
        "    else:\n",
        "      kernel_log_step = tf.fill(self.kernel_latent.shape[2:], -4.)\n",
        "      bias_log_step = -4.\n",
        "    self.kernel_log_step = tf.Variable(\n",
        "        tf.cast(kernel_log_step, self.variable_dtype), name=\"kernel_log_step\")\n",
        "    self.bias_log_step = tf.Variable(\n",
        "        tf.cast(bias_log_step, self.variable_dtype), name=\"bias_log_step\")\n",
        "    self.add_loss(lambda: self.regularizer(\n",
        "        self.kernel_latent / tf.exp(self.kernel_log_step)))\n",
        "    self.add_loss(lambda: self.regularizer(\n",
        "        self.bias_latent / tf.exp(self.bias_log_step)))\n",
        "\n",
        "  @property\n",
        "  def kernel(self):\n",
        "    kernel_rdft = quantize(self.kernel_latent, self.kernel_log_step)\n",
        "    return from_rdft(kernel_rdft, self.kernel_size)\n",
        "\n",
        "  @kernel.setter\n",
        "  def kernel(self, kernel):\n",
        "    kernel_rdft = to_rdft(kernel, self.kernel_size)\n",
        "    self.kernel_latent = tf.Variable(kernel_rdft, name=\"kernel_latent\")\n",
        "\n",
        "  @property\n",
        "  def bias(self):\n",
        "    return quantize(self.bias_latent, self.bias_log_step)\n",
        "\n",
        "  @bias.setter\n",
        "  def bias(self, bias):\n",
        "    self.bias_latent = tf.Variable(bias, name=\"bias_latent\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-ekDDQ9jidI"
      },
      "source": [
        "使用与上面相同的架构定义分类器模型，但使用以下修改后的层："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TQgp84L7qalw",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def make_mnist_classifier(regularizer):\n",
        "  return tf.keras.Sequential([\n",
        "      CompressibleConv2D(regularizer, 20, 5, strides=2, name=\"conv_1\"),\n",
        "      CompressibleConv2D(regularizer, 50, 5, strides=2, name=\"conv_2\"),\n",
        "      tf.keras.layers.Flatten(),\n",
        "      CompressibleDense(regularizer, 500, name=\"fc_1\"),\n",
        "      CompressibleDense(regularizer, 10, name=\"fc_2\"),\n",
        "  ], name=\"classifier\")\n",
        "\n",
        "compressible_classifier = make_mnist_classifier(regularizer)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJ-TMHE1kNFc"
      },
      "source": [
        "并训练模型："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6L5ZJAX4EiXW",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "penalized_accuracy = train_model(\n",
        "    compressible_classifier, training_dataset, validation_dataset)\n",
        "\n",
        "print(f\"Accuracy: {penalized_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZuE4NeY_kTDz"
      },
      "source": [
        "可压缩模型已达到与普通分类器相似的准确率。\n",
        "\n",
        "但是，该模型实际上还没有被压缩。为此，我们定义了另一组子类，它们以压缩形式存储内核和偏差（作为位序列）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AZhj8A2gnBkD"
      },
      "source": [
        "## 压缩分类器\n",
        "\n",
        "下面定义的 `CustomDense` 和 `CustomConv2D` 的子类将可压缩密集层的权重转换为二进制字符串。此外，它们以半精度存储量化步长的对数以节省空间。每当通过 `@property` 访问内核或偏差时，它们就会从其字符串表示中解压缩并去量化。\n",
        "\n",
        "首先，定义函数来压缩和解压缩模型参数："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xS19FhDajeto",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def compress_latent(latent, log_step, name):\n",
        "  em = tfc.PowerLawEntropyModel(latent.shape.rank)\n",
        "  compressed = em.compress(latent / tf.exp(log_step))\n",
        "  compressed = tf.Variable(compressed, name=f\"{name}_compressed\")\n",
        "  log_step = tf.cast(log_step, tf.float16)\n",
        "  log_step = tf.Variable(log_step, name=f\"{name}_log_step\")\n",
        "  return compressed, log_step\n",
        "\n",
        "def decompress_latent(compressed, shape, log_step):\n",
        "  latent = tfc.PowerLawEntropyModel(len(shape)).decompress(compressed, shape)\n",
        "  step = tf.exp(tf.cast(log_step, latent.dtype))\n",
        "  return latent * step\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bPPABE9fjqHJ"
      },
      "source": [
        "有了这些，我们可以定义 `CompressedDense`："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CnaiNzhgaZ7s",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CompressedDense(CustomDense):\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    assert isinstance(other, CompressibleDense)\n",
        "    self.input_channels = other.kernel.shape[0]\n",
        "    self.kernel_compressed, self.kernel_log_step = compress_latent(\n",
        "        other.kernel_latent, other.kernel_log_step, \"kernel\")\n",
        "    self.bias_compressed, self.bias_log_step = compress_latent(\n",
        "        other.bias_latent, other.bias_log_step, \"bias\")\n",
        "    self.built = True\n",
        "\n",
        "  @property\n",
        "  def kernel(self):\n",
        "    kernel_shape = (self.input_channels, self.filters)\n",
        "    return decompress_latent(\n",
        "        self.kernel_compressed, kernel_shape, self.kernel_log_step)\n",
        "\n",
        "  @property\n",
        "  def bias(self):\n",
        "    bias_shape = (self.filters,)\n",
        "    return decompress_latent(\n",
        "        self.bias_compressed, bias_shape, self.bias_log_step)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tzvMCM0El2iW"
      },
      "source": [
        "卷积层类与上面类似。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hS-2ADA6iWeQ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "class CompressedConv2D(CustomConv2D):\n",
        "\n",
        "  def build(self, input_shape, other=None):\n",
        "    assert isinstance(other, CompressibleConv2D)\n",
        "    self.input_channels = other.kernel.shape[2]\n",
        "    self.kernel_compressed, self.kernel_log_step = compress_latent(\n",
        "        other.kernel_latent, other.kernel_log_step, \"kernel\")\n",
        "    self.bias_compressed, self.bias_log_step = compress_latent(\n",
        "        other.bias_latent, other.bias_log_step, \"bias\")\n",
        "    self.built = True\n",
        "\n",
        "  @property\n",
        "  def kernel(self):\n",
        "    rdft_shape = (self.input_channels, self.filters,\n",
        "                  self.kernel_size, self.kernel_size // 2 + 1, 2)\n",
        "    kernel_rdft = decompress_latent(\n",
        "        self.kernel_compressed, rdft_shape, self.kernel_log_step)\n",
        "    return from_rdft(kernel_rdft, self.kernel_size)\n",
        "\n",
        "  @property\n",
        "  def bias(self):\n",
        "    bias_shape = (self.filters,)\n",
        "    return decompress_latent(\n",
        "        self.bias_compressed, bias_shape, self.bias_log_step)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJLCPoe3l8jG"
      },
      "source": [
        "要将可压缩模型转换为压缩模型，我们可以方便地使用 `clone_model` 函数。`compress_layer` 可以将任何可压缩层转换为压缩层，并简单地传递给任何其他类型的层（例如 `Flatten` 等）。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WEHroUyhG56m",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def compress_layer(layer):\n",
        "  if isinstance(layer, CompressibleDense):\n",
        "    return CompressedDense.copy(layer)\n",
        "  if isinstance(layer, CompressibleConv2D):\n",
        "    return CompressedConv2D.copy(layer)\n",
        "  return type(layer).from_config(layer.get_config())\n",
        "\n",
        "compressed_classifier = tf.keras.models.clone_model(\n",
        "    compressible_classifier, clone_function=compress_layer)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3wbN1XQmkDg"
      },
      "source": [
        "现在，我们来验证压缩模型是否仍按预期执行："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R95kuURITpa9",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "compressed_classifier.compile(metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n",
        "_, compressed_accuracy = compressed_classifier.evaluate(validation_dataset.batch(128))\n",
        "\n",
        "print(f\"Accuracy of the compressible classifier: {penalized_accuracy:0.4f}\")\n",
        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KtFhpXh6uaIY"
      },
      "source": [
        "压缩模型的分类准确率与训练期间达到的分类准确率相同！\n",
        "\n",
        "此外，压缩后的模型权重的大小远小于原始模型的大小："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qp-ecfuYufbs",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def get_weight_size_in_bytes(weight):\n",
        "  if weight.dtype == tf.string:\n",
        "    return tf.reduce_sum(tf.strings.length(weight, unit=\"BYTE\"))\n",
        "  else:\n",
        "    return tf.size(weight) * weight.dtype.size\n",
        "\n",
        "original_size = sum(map(get_weight_size_in_bytes, classifier.weights))\n",
        "compressed_size = sum(map(get_weight_size_in_bytes, compressed_classifier.weights))\n",
        "\n",
        "print(f\"Size of original model weights: {original_size} bytes\")\n",
        "print(f\"Size of compressed model weights: {compressed_size} bytes\")\n",
        "print(f\"Compression ratio: {(original_size/compressed_size):0.0f}x\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8A8v0df6TR2"
      },
      "source": [
        "将模型存储在磁盘上需要一些开销来存储模型架构、函数图等。\n",
        "\n",
        "ZIP 等无损压缩方法擅长压缩此类数据，但不擅长压缩权重本身。这就是为什么在应用了 ZIP 压缩之后，当计算模型大小（包括开销）时，EPR 仍然具有显著优势："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4hunDYxH1zqb",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import shutil\n",
        "\n",
        "def get_disk_size(model, path):\n",
        "  model.save(path)\n",
        "  zip_path = shutil.make_archive(path, \"zip\", path)\n",
        "  return os.path.getsize(zip_path)\n",
        "\n",
        "original_zip_size = get_disk_size(classifier, \"/tmp/classifier\")\n",
        "compressed_zip_size = get_disk_size(\n",
        "    compressed_classifier, \"/tmp/compressed_classifier\")\n",
        "\n",
        "print(f\"Original on-disk size (ZIP compressed): {original_zip_size} bytes\")\n",
        "print(f\"Compressed on-disk size (ZIP compressed): {compressed_zip_size} bytes\")\n",
        "print(f\"Compression ratio: {(original_zip_size/compressed_zip_size):0.0f}x\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSITvJrlAhZs"
      },
      "source": [
        "## 正则化效果和大小-准确度权衡\n",
        "\n",
        "上面，$\\lambda$ 超参数被设置为 2（通过模型中的参数数量进行标准化）。随着我们增加 $\\lambda$，模型权重的可压缩性受到越来越严重的惩罚。\n",
        "\n",
        "对于较低的值，惩罚可以起到权重调节器的作用。它实际上对分类器的泛化性能有有益的影响，并且可以在验证数据集上产生略高的准确率：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "4rhmKu98FdPJ",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "\n",
        "print(f\"Accuracy of the vanilla classifier: {classifier_accuracy:0.4f}\")\n",
        "print(f\"Accuracy of the penalized classifier: {penalized_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UCfC4LQFdjL"
      },
      "source": [
        "对于更高的值，我们看到模型大小越来越小，但准确率也在逐渐降低。为了看到这一点，我们来训练几个模型，并绘制它们的大小与准确率之间的关系图：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "diApPKHbAIqa",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def compress_and_evaluate_model(lmbda):\n",
        "  print(f\"lambda={lmbda:0.0f}: training...\", flush=True)\n",
        "  regularizer = PowerLawRegularizer(lmbda=lmbda/classifier.count_params())\n",
        "  compressible_classifier = make_mnist_classifier(regularizer)\n",
        "  train_model(\n",
        "      compressible_classifier, training_dataset, validation_dataset, verbose=0)\n",
        "  print(\"compressing...\", flush=True)\n",
        "  compressed_classifier = tf.keras.models.clone_model(\n",
        "      compressible_classifier, clone_function=compress_layer)\n",
        "  compressed_size = sum(map(\n",
        "      get_weight_size_in_bytes, compressed_classifier.weights))\n",
        "  compressed_zip_size = float(get_disk_size(\n",
        "      compressed_classifier, \"/tmp/compressed_classifier\"))\n",
        "  print(\"evaluating...\", flush=True)\n",
        "  compressed_classifier = tf.keras.models.load_model(\n",
        "      \"/tmp/compressed_classifier\")\n",
        "  compressed_classifier.compile(\n",
        "      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n",
        "  _, compressed_accuracy = compressed_classifier.evaluate(\n",
        "      validation_dataset.batch(128), verbose=0)\n",
        "  print()\n",
        "  return compressed_size, compressed_zip_size, compressed_accuracy\n",
        "\n",
        "lambdas = (2., 5., 10., 20., 50.)\n",
        "metrics = [compress_and_evaluate_model(l) for l in lambdas]\n",
        "metrics = tf.convert_to_tensor(metrics, tf.float32)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "bhAi85KzGqTz",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "\n",
        "def plot_broken_xaxis(ax, compressed_sizes, original_size, original_accuracy):\n",
        "  xticks = list(range(\n",
        "      int(tf.math.floor(min(compressed_sizes) / 5) * 5),\n",
        "      int(tf.math.ceil(max(compressed_sizes) / 5) * 5) + 1,\n",
        "      5))\n",
        "  xticks.append(xticks[-1] + 10)\n",
        "  ax.set_xlim(xticks[0], xticks[-1] + 2)\n",
        "  ax.set_xticks(xticks[1:])\n",
        "  ax.set_xticklabels(xticks[1:-1] + [f\"{original_size:0.2f}\"])\n",
        "  ax.plot(xticks[-1], original_accuracy, \"o\", label=\"float32\")\n",
        "\n",
        "sizes, zip_sizes, accuracies = tf.transpose(metrics)\n",
        "sizes /= 1024\n",
        "zip_sizes /= 1024\n",
        "\n",
        "fig, (axl, axr) = plt.subplots(1, 2, sharey=True, figsize=(10, 4))\n",
        "axl.plot(sizes, accuracies, \"o-\", label=\"EPR compressed\")\n",
        "axr.plot(zip_sizes, accuracies, \"o-\", label=\"EPR compressed\")\n",
        "plot_broken_xaxis(axl, sizes, original_size/1024, classifier_accuracy)\n",
        "plot_broken_xaxis(axr, zip_sizes, original_zip_size/1024, classifier_accuracy)\n",
        "\n",
        "axl.set_xlabel(\"size of model weights [kbytes]\")\n",
        "axr.set_xlabel(\"ZIP compressed on-disk model size [kbytes]\")\n",
        "axl.set_ylabel(\"accuracy\")\n",
        "axl.legend(loc=\"lower right\")\n",
        "axr.legend(loc=\"lower right\")\n",
        "axl.grid()\n",
        "axr.grid()\n",
        "for i in range(len(lambdas)):\n",
        "  axl.annotate(f\"$\\lambda = {lambdas[i]:0.0f}$\", (sizes[i], accuracies[i]),\n",
        "               xytext=(10, -5), xycoords=\"data\", textcoords=\"offset points\")\n",
        "  axr.annotate(f\"$\\lambda = {lambdas[i]:0.0f}$\", (zip_sizes[i], accuracies[i]),\n",
        "               xytext=(10, -5), xycoords=\"data\", textcoords=\"offset points\")\n",
        "plt.tight_layout()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ajrHaFTAaLd2"
      },
      "source": [
        "理想情况下，该图应显示肘形大小-准确率权衡，但准确率指标有些噪声也正常。根据初始化的不同，曲线可能会出现一些曲折。\n",
        "\n",
        "由于正则化效应，对于较小的 $\\lambda$ 值，EPR 压缩模型在测试集上比原始模型更准确。即使我们比较附加 ZIP 压缩后的大小，EPR 压缩模型也要小很多倍。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RBhdXZTzoWw"
      },
      "source": [
        "## 解压缩分类器\n",
        "\n",
        "`CompressedDense` 和 `CompressedConv2D` 在每次前向传递时会解压缩它们的权重。这使得它们非常适合内存有限的设备，但解压缩的计算成本可能很高，尤其是对于小批次。\n",
        "\n",
        "要将模型解压缩一次，并将其用于进一步的训练或推断，我们可以使用常规层或可压缩层将其转换回模型。这在模型部署或联合学习场景中很有用。\n",
        "\n",
        "首先，转换回普通模型，我们可以进行推断，和/或继续进行常规训练，而不会有压缩惩罚："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QBB2-X5XzvwB",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def decompress_layer(layer):\n",
        "  if isinstance(layer, CompressedDense):\n",
        "    return CustomDense.copy(layer)\n",
        "  if isinstance(layer, CompressedConv2D):\n",
        "    return CustomConv2D.copy(layer)\n",
        "  return type(layer).from_config(layer.get_config())\n",
        "\n",
        "decompressed_classifier = tf.keras.models.clone_model(\n",
        "    compressed_classifier, clone_function=decompress_layer)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ehE2ov8U0p0G",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "decompressed_accuracy = train_model(\n",
        "    decompressed_classifier, training_dataset, validation_dataset, epochs=1)\n",
        "\n",
        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n",
        "print(f\"Accuracy of the decompressed classifier after one more epoch of training: {decompressed_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jiSCvemQ04o8"
      },
      "source": [
        "请注意，在训练额外的周期后验证准确率会下降，因为训练是在没有正则化的情况下完成的。\n",
        "\n",
        "或者，我们可以将模型转换回“可压缩”模型，以进行推断和/或进一步训练，并带有压缩惩罚："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JDppVUdx1BvY",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def decompress_layer_with_penalty(layer):\n",
        "  if isinstance(layer, CompressedDense):\n",
        "    return CompressibleDense.copy(layer, regularizer=regularizer)\n",
        "  if isinstance(layer, CompressedConv2D):\n",
        "    return CompressibleConv2D.copy(layer, regularizer=regularizer)\n",
        "  return type(layer).from_config(layer.get_config())\n",
        "\n",
        "decompressed_classifier = tf.keras.models.clone_model(\n",
        "    compressed_classifier, clone_function=decompress_layer_with_penalty)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AJcnyOFW2IcK",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "decompressed_accuracy = train_model(\n",
        "    decompressed_classifier, training_dataset, validation_dataset, epochs=1)\n",
        "\n",
        "print(f\"Accuracy of the compressed classifier: {compressed_accuracy:0.4f}\")\n",
        "print(f\"Accuracy of the decompressed classifier after one more epoch of training: {decompressed_accuracy:0.4f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ciol315T_TwQ"
      },
      "source": [
        "在这里，在训练一个额外的周期后，准确率会提高。"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "Tce3stUlHN0L",
        "xHxb-dlhMIzW"
      ],
      "name": "compression.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}