{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PTQ 与 QAT 实践\n",
    "\n",
    "本文主要介绍如何使用 PyTorch 将浮点模型转换为 PTQ 或者 QAT 模型。\n",
    "\n",
    "## 背景\n",
    "\n",
    "{guilabel}`目标`：快速将浮点模型转换为 PTQ 或者 QAT 模型。\n",
    "\n",
    "### 读者\n",
    "\n",
    "本教程适用于会使用 PyTorch 编写 CNN 等模块的的算法工程师。\n",
    "\n",
    "### 环境配置\n",
    "\n",
    "本文使用 Python 3.10.0 （其他版本请自测），暂时仅 Linux 平台被测试。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看 `torch` 和 `torchvision` 的版本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch: 1.11.0 \n",
      "torchvision: 0.12.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "\n",
    "print(f'torch: {torch.__version__} \\n'\n",
    "      f'torchvision: {torchvision.__version__}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置一些警告配置："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置 warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\n",
    "    action='ignore',\n",
    "    category=DeprecationWarning,\n",
    "    module='.*'\n",
    ")\n",
    "warnings.filterwarnings(\n",
    "    action='ignore',\n",
    "    module='torch.ao.quantization'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述：PQT 与 QAT\n",
    "\n",
    "参考：[量化](https://pytorch.org/docs/master/quantization.html)\n",
    "\n",
    "`训练后量化`\n",
    ":   简称 PTQ（Post Training Quantization）：权重量化，激活量化，需要借助数据在训练后进行校准。\n",
    "\n",
    "`静态量化感知训练`\n",
    ":   简称 QAT（static quantization aware training）：权重量化，激活量化，在训练过程中的量化数值进行建模。\n",
    "\n",
    "`浮点模型`\n",
    ":   模型的 **权重** 和 **激活** 均为浮点类型（如 {data}`torch.float32`, {data}`torch.float64`）。\n",
    "\n",
    "`量化模型`\n",
    ":   模型的 **权重** 和 **激活** 均为量化类型（如 {data}`torch.qint32`, {data}`torch.qint8`, {data}`torch.quint8`, {data}`torch.quint2x4`, {data}`torch.quint4x2`）。\n",
    "\n",
    "\n",
    "下面举例说明如何将浮点模型转换为量化模型。\n",
    "\n",
    "为了方便说明定义如下模块：\n",
    "\n",
    "```{rubric} 定义简单的浮点模块\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn, Tensor\n",
    "\n",
    "\n",
    "class M(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv = torch.nn.Conv2d(1, 1, 1)\n",
    "        self.relu = torch.nn.ReLU()\n",
    "\n",
    "    def _forward_impl(self, x: Tensor) -> Tensor:\n",
    "        '''提供便捷函数'''\n",
    "        x = self.conv(x)\n",
    "        x = self.relu(x)\n",
    "        return x\n",
    "\n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        x= self._forward_impl(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 定义可量化模块\n",
    "```\n",
    "\n",
    "将浮点模块 `M` 转换为可量化模块 `QM`（量化流程的最关键的一步）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.ao.quantization import QuantStub, DeQuantStub\n",
    "\n",
    "\n",
    "class QM(M):\n",
    "    '''\n",
    "    Args:\n",
    "        is_print: 为了测试需求，打印一些信息\n",
    "    '''\n",
    "    def __init__(self, is_print: bool=False):\n",
    "        super().__init__()\n",
    "        self.is_print = is_print\n",
    "        self.quant = QuantStub() # 将张量从浮点转换为量化\n",
    "        self.dequant = DeQuantStub() # 将张量从量化转换为浮点\n",
    "\n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        # 手动指定张量将在量化模型中从浮点模块转换为量化模块的位置\n",
    "        x = self.quant(x)\n",
    "        if self.is_print:\n",
    "            print('量化前的类型：', x.dtype)\n",
    "        x = self._forward_impl(x)\n",
    "        if self.is_print:\n",
    "            print('量化中的类型：',x.dtype)\n",
    "        # 在量化模型中手动指定张量从量化到浮点的转换位置\n",
    "        x = self.dequant(x)\n",
    "        if self.is_print:\n",
    "            print('量化后的类型：', x.dtype)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "简单测试前向过程的激活数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.float32\n",
      "量化中的类型： torch.float32\n",
      "量化后的类型： torch.float32\n"
     ]
    }
   ],
   "source": [
    "input_fp32 = torch.randn(4, 1, 4, 4) # 输入的数据\n",
    "\n",
    "m = QM(is_print=True)\n",
    "x = m(input_fp32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看权重的数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.conv.weight.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出，此时模块 `m` 是浮点模块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PTQ 简介\n",
    "\n",
    "当内存带宽和计算空间都很重要时，通常会使用训练后量化，而 CNN 就是其典型的用例。训练后量化对模型的 **权重** 和 **激活** 进行量化。它在可能的情况下将 **激活** 融合到前面的层中。它需要用具有代表性的数据集进行 **校准**，以确定激活的最佳量化参数。\n",
    "\n",
    "```{rubric} 示意图\n",
    "```\n",
    "\n",
    "```\n",
    "# 原始模型\n",
    "# 全部的张量和计算均在浮点上进行\n",
    "previous_layer_fp32 -- linear_fp32 -- activation_fp32 -- next_layer_fp32\n",
    "                    /\n",
    "    linear_weight_fp32\n",
    "\n",
    "# 静态量化模型\n",
    "# weights 和 activations 在 int8 上\n",
    "previous_layer_int8 -- linear_with_activation_int8 -- next_layer_int8\n",
    "                    /\n",
    "  linear_weight_int8\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "直接创建浮点模块的实例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建浮点模型实例\n",
    "model_fp32 = QM(is_print=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要使 PTQ 生效，必须将模型设置为 `eval` 模式：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QM(\n",
       "  (conv): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n",
       "  (relu): ReLU()\n",
       "  (quant): QuantStub()\n",
       "  (dequant): DeQuantStub()\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_fp32.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看此时的数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.float32\n",
      "量化中的类型： torch.float32\n",
      "量化后的类型： torch.float32\n",
      "激活和权重的数据类型分别为：torch.float32, torch.float32\n"
     ]
    }
   ],
   "source": [
    "input_fp32 = torch.randn(4, 1, 4, 4)\n",
    "\n",
    "x = model_fp32(input_fp32)\n",
    "print('激活和权重的数据类型分别为：'\n",
    "      f'{x.dtype}, {model_fp32.conv.weight.dtype}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 定义观测器\n",
    "```\n",
    "\n",
    "赋值实例变量 `qconfig`，其中包含关于要附加哪种观测器的信息：\n",
    "\n",
    "- 使用 [`'fbgemm'`](https://github.com/pytorch/FBGEMM) 用于带 AVX2 的 x86（没有AVX2，一些运算的实现效率很低）；使用 [`'qnnpack'`](https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native/quantized/cpu/qnnpack) 用于 ARM CPU（通常出现在移动/嵌入式设备中）。\n",
    "- 其他量化配置，如选择对称或非对称量化和 `MinMax` 或 `L2Norm` 校准技术，可以在这里指定。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32.qconfig = torch.ao.quantization.get_default_qconfig('fbgemm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看此时的数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.float32\n",
      "量化中的类型： torch.float32\n",
      "量化后的类型： torch.float32\n",
      "激活和权重的数据类型分别为：torch.float32, torch.float32\n"
     ]
    }
   ],
   "source": [
    "input_fp32 = torch.randn(4, 1, 4, 4)\n",
    "\n",
    "x = model_fp32(input_fp32)\n",
    "print('激活和权重的数据类型分别为：'\n",
    "      f'{x.dtype}, {model_fp32.conv.weight.dtype}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 融合激活层\n",
    "```\n",
    "\n",
    "在适用的地方，融合 activation 到前面的层（这需要根据模型架构手动完成）。常见的融合包括 `conv + relu` 和 `conv + batchnorm + relu`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QM(\n",
       "  (conv): ConvReLU2d(\n",
       "    (0): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n",
       "    (1): ReLU()\n",
       "  )\n",
       "  (relu): Identity()\n",
       "  (quant): QuantStub()\n",
       "  (dequant): DeQuantStub()\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_fp32_fused = torch.ao.quantization.fuse_modules(model_fp32,\n",
    "                                                      [['conv', 'relu']])\n",
    "                                                    \n",
    "model_fp32_fused"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到 `model_fp32_fused` 中 `ConvReLU2d` 融合 `model_fp32` 的两个层 `conv` 和 `relu`。\n",
    "\n",
    "查看此时的数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.float32\n",
      "量化中的类型： torch.float32\n",
      "量化后的类型： torch.float32\n",
      "激活和权重的数据类型分别为：torch.float32, torch.float32\n"
     ]
    }
   ],
   "source": [
    "input_fp32 = torch.randn(4, 1, 4, 4)\n",
    "\n",
    "x = model_fp32_fused(input_fp32)\n",
    "print('激活和权重的数据类型分别为：'\n",
    "      f'{x.dtype}, {model_fp32.conv.weight.dtype}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 启用观测器\n",
    "```\n",
    "\n",
    "在融合后的模块中启用观测器，用于在校准期间观测激活（activation）张量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32_prepared = torch.quantization.prepare(model_fp32_fused)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 校准准备好的模型\n",
    "```\n",
    "校准准备好的模型，以确定量化参数的激活在现实世界的设置，校准具有代表性的数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.float32\n",
      "量化中的类型： torch.float32\n",
      "量化后的类型： torch.float32\n",
      "激活和权重的数据类型分别为：torch.float32, torch.float32\n"
     ]
    }
   ],
   "source": [
    "input_fp32 = torch.randn(4, 1, 4, 4)\n",
    "\n",
    "x = model_fp32_prepared(input_fp32)\n",
    "print('激活和权重的数据类型分别为：'\n",
    "      f'{x.dtype}, {model_fp32.conv.weight.dtype}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 模型转换\n",
    "```\n",
    "\n",
    "```{note}\n",
    "量化权重，计算和存储每个激活张量要使用的尺度（scale）和偏差（bias）值，并用量化实现替换关键算子。\n",
    "```\n",
    "\n",
    "转换已校准好的模型为量化模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QM(\n",
       "  (conv): QuantizedConvReLU2d(1, 1, kernel_size=(1, 1), stride=(1, 1), scale=0.010650944896042347, zero_point=0)\n",
       "  (relu): Identity()\n",
       "  (quant): Quantize(scale=tensor([0.0351]), zero_point=tensor([74]), dtype=torch.quint8)\n",
       "  (dequant): DeQuantize()\n",
       ")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_int8 = torch.quantization.convert(model_fp32_prepared)\n",
    "model_int8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看权重的数据类型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.qint8"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_int8.conv.weight().dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出此时权重的元素大小为 1 字节，而不是 FP32 的 4 字节："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_int8.conv.weight().element_size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行模型，相关的计算将在 {data}`torch.qint8` 中发生。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "量化前的类型： torch.quint8\n",
      "量化中的类型： torch.quint8\n",
      "量化后的类型： torch.float32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = model_int8(input_fp32)\n",
    "res.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要了解更多关于量化意识训练的信息，请参阅 [QAT 教程](https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QAT 概述\n",
    "\n",
    "与其他量化方法相比，QAT 在 **训练过程中** 模拟量化的效果，可以获得更高的 accuracy。在训练过程中，所有的计算都是在浮点上进行的，使用 fake_quant 模块通过夹紧和舍入的方式对量化效果进行建模，模拟 INT8 的效果。模型转换后，权值和激活被量化，激活在可能的情况下被融合到前一层。它通常与 CNN 一起使用，与 PTQ 相比具有更高的 accuracy。\n",
    "\n",
    "\n",
    "```{rubric} 示意图\n",
    "```\n",
    "\n",
    "```\n",
    "# 原始模型\n",
    "# 全部张量和计算均在浮点上\n",
    "previous_layer_fp32 -- linear_fp32 -- activation_fp32 -- next_layer_fp32\n",
    "                      /\n",
    "    linear_weight_fp32\n",
    "\n",
    "# 在训练过程中使用 fake_quants 建模量化数值\n",
    "previous_layer_fp32 -- fq -- linear_fp32 -- activation_fp32 -- fq -- next_layer_fp32\n",
    "                           /\n",
    "   linear_weight_fp32 -- fq\n",
    "\n",
    "# 量化模型\n",
    "# weights 和 activations 在 int8 上\n",
    "previous_layer_int8 -- linear_with_activation_int8 -- next_layer_int8\n",
    "                     /\n",
    "   linear_weight_int8\n",
    "```\n",
    "\n",
    "定义比 `M` 稍微复杂一点的浮点模块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class M2(M):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.bn = torch.nn.BatchNorm2d(1)\n",
    "\n",
    "    def _forward_impl(self, x: Tensor) -> Tensor:\n",
    "        '''提供便捷函数'''\n",
    "        x = self.conv(x)\n",
    "        x = self.bn(x)\n",
    "        x = self.relu(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同样需要定义可量化模块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QM2(M2, QM):\n",
    "    def __init__(self):\n",
    "        super().__init__()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建浮点模型实例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QM2(\n",
       "  (conv): Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1))\n",
       "  (relu): ReLU()\n",
       "  (quant): QuantStub()\n",
       "  (dequant): DeQuantStub()\n",
       "  (bn): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       ")"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建模型实例\n",
    "model_fp32 = QM2()\n",
    "model_fp32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型必须设置为训练模式，以便 QAT 可用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32.train();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "添加量化配置（与 PTQ 相同相似）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32.qconfig = torch.ao.quantization.get_default_qat_qconfig('fbgemm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 融合 QAT 模块\n",
    "```\n",
    "\n",
    "QAT 的模块融合与 PTQ 相同相似："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.ao.quantization import fuse_modules_qat\n",
    "\n",
    "model_fp32_fused = fuse_modules_qat(model_fp32,\n",
    "                                    [['conv', 'bn', 'relu']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 准备 QAT 模型\n",
    "```\n",
    "\n",
    "这将在模型中插入观测者和伪量化模块，它们将在校准期间观测权重和激活的张量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32_prepared = torch.quantization.prepare_qat(model_fp32_fused)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{rubric} 训练 QAT 模型\n",
    "```\n",
    "\n",
    "```python\n",
    "# 下文会编写实际的例子，此处没有显示\n",
    "training_loop(model_fp32_prepared)\n",
    "```\n",
    "\n",
    "将观测到的模型转换为量化模型。需要：\n",
    "\n",
    "- 量化权重，计算和存储用于每个激活张量的尺度（scale）和偏差（bias）值，\n",
    "- 在适当的地方融合模块，并用量化实现替换关键算子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_fp32_prepared.eval()\n",
    "model_int8 = torch.quantization.convert(model_fp32_prepared)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行模型，相关的计算将在 {data}`torch.qint8` 中发生。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = model_int8(input_fp32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要了解更多关于量化意识训练的信息，请参阅 [QAT 教程](https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html)。\n",
    "\n",
    "### PTQ/QAT 统一的量化流程\n",
    "\n",
    "PTQ 和 QAT 的量化流程十分相似，为了统一接口，可以使用 `torchvision` 提供的函数 {func}`~torchvision.models.quantization.utils._fuse_modules`。\n",
    "\n",
    "下面利用函数 {func}`~torchvision.models.quantization.utils._fuse_modules` 可量化模块 `QM2`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any\n",
    "from torch.ao.quantization import fuse_modules, fuse_modules_qat\n",
    "from torch.ao.quantization import get_default_qconfig, get_default_qat_qconfig\n",
    "from torch.ao.quantization import quantize, quantize_qat\n",
    "\n",
    "def _fuse_modules(\n",
    "    model: nn.Module, modules_to_fuse: list[str] | list[list[str]], is_qat: bool | None, **kwargs: Any\n",
    "):\n",
    "    if is_qat is None:\n",
    "        is_qat = model.training\n",
    "    method = fuse_modules_qat if is_qat else fuse_modules\n",
    "    return method(model, modules_to_fuse, **kwargs)\n",
    "\n",
    "\n",
    "class QM3(QM2):\n",
    "    '''可量化模型\n",
    "    Args:\n",
    "        is_qat: 是否使用 QAT 模式\n",
    "    '''\n",
    "    def __init__(self, is_qat: bool | None = None, backend='fbgemm'):\n",
    "        super().__init__()\n",
    "        self.is_qat = is_qat\n",
    "        # 定义观测器\n",
    "        if is_qat:\n",
    "            self.train()\n",
    "            self.qconfig = get_default_qat_qconfig(backend)\n",
    "        else:\n",
    "            self.eval()\n",
    "            self.qconfig = get_default_qconfig(backend)\n",
    "\n",
    "    def fuse_model(self) -> None:\n",
    "        '''模块融合'''\n",
    "        if self.is_qat:\n",
    "            modules_to_fuse = ['bn', 'relu']\n",
    "        else:\n",
    "            modules_to_fuse = ['conv', 'bn', 'relu']\n",
    "        return _fuse_modules(self,\n",
    "                      modules_to_fuse,\n",
    "                      self.is_qat,\n",
    "                      inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了可量化模块 `QM3`，可以十分便利的切换 PTQ/QAT了。\n",
    "\n",
    "比如，PTQ，可以这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_fn(model, num_epochs):\n",
    "    for _ in range(num_epochs):\n",
    "        input_fp32 = torch.randn(4, 1, 4, 4)\n",
    "        model(input_fp32)\n",
    "\n",
    "num_epochs = 10\n",
    "ptq_model = QM3(is_qat=False)\n",
    "model_fused = ptq_model.fuse_model()\n",
    "quanted_model = quantize(model_fused, run_fn, [num_epochs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "QAT 可以这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 10\n",
    "qat_model = QM3(is_qat=True)\n",
    "model_fused = qat_model.fuse_model()\n",
    "quanted_model = quantize_qat(model_fused, run_fn, [num_epochs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PTQ/QAT 量化策略\n",
    "\n",
    "对于通用量化技术，需要了解：\n",
    "\n",
    "1. 将任何需要输出再量化请求的运算（因此有额外的参数）从函数形式转换为模块形式（例如，使用 {class}`torch.nn.ReLU` 而不是 {func}`torch.nn.functional.relu`）。\n",
    "1. 通过在子模块上指定 `.qconfig` 属性或指定 `qconfig_dict` 来指定模型的哪些部分需要量化。例如，设置 `model.conv1.qconfig = None` 表示 `model.conv1` 层不量化，设置 `model.linear1.qconfig = custom_qconfig` 表示 `model.linear1` 将使用 `custom_qconfig` 而不是全局 `qconfig`。\n",
    "\n",
    "对于量化激活的静态量化技术（即对模型的权重和激活均进行量化，包括 PTQ 和 QAT），用户还需要做以下工作：\n",
    "\n",
    "1. 指定量化和反量化激活的位置。这是使用 {class}`~torch.ao.quantization.stubs.QuantStub` 和 {class}`~torch.ao.quantization.stubs.DeQuantStub` 模块完成的。\n",
    "1. 使用 {class}`~torch.nn.quantized.FloatFunctional` 将需要对量化进行特殊处理的张量运算封装到模块中。例如像 {func}`add` 和 {func}`cat` 这样需要特殊处理来确定输出量化参数的运算。\n",
    "1. 融合模块：将运算/模块组合成单个模块，获得更高的 accuracy 和性能。这是使用 {func}`~torch.ao.quantization.fuse_modules.fuse_modules` API 完成的，该 API 接受要融合的模块列表。目前支持以下融合：`[Conv, Relu]`、 `[Conv, BatchNorm]`、 `[Conv, BatchNorm, Relu]` 和 `[Linear, Relu]`。\n",
    "\n",
    "示例：\n",
    "\n",
    "```{figure} images/resnet.png\n",
    ":align: center\n",
    ":class: w3-border\n",
    "\n",
    "倒置残差块的转换前后对比\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PTQ 和 QAT 实战\n",
    "\n",
    "\n",
    "```{rubric} 模型对比\n",
    "```\n",
    "\n",
    "类型|大小（MB）|accuracy（$\\%$）\n",
    ":-|:-|:-\n",
    "浮点|9.188|94.91\n",
    "浮点融合|8.924|94.91\n",
    "QAT|2.657|94.41\n",
    "\n",
    "```{rubric} 不同 QConfig 的静态 PTQ 模型\n",
    "```\n",
    "\n",
    "accuracy（$\\%$）|激活|权重|\n",
    ":-|:-|:-\n",
    "|51.11|{data}`~torch.ao.quantization.observer.MinMaxObserver`.`with_args(quant_min=0, quant_max=127)`|{data}`~torch.ao.quantization.observer.MinMaxObserver`.`with_args(dtype=torch.qint8, qscheme=torch.per_tensor_symmetric)`\n",
    "80.42|{data}`~torch.ao.quantization.observer.HistogramObserver`.`with_args(quant_min=0, quant_max=127)`|{data}`~torch.ao.quantization.observer.PerChannelMinMaxObserver`.`with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric)`\n",
    "\n",
    "为了提供一致的量化工具接口，我们使用 Python 包 `torchq`。\n",
    "\n",
    "本地载入临时 `torchq` 包："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mod import torchq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{tip}\n",
    "本文使用 `torchq` 的 `'0.0.1-alpha'` 版本。\n",
    "```\n",
    "\n",
    "更方便的是：使用 `pip` 安装：\n",
    "\n",
    "```shell\n",
    "pip install torchq==0.0.1-alpha\n",
    "```\n",
    "\n",
    "接着，便可以直接导入：\n",
    "\n",
    "```python\n",
    "import torchq\n",
    "```\n",
    "\n",
    "```{tip}\n",
    "本文使用 `torchq` 的 `'0.0.1-alpha'` 版本。\n",
    "```\n",
    "\n",
    "可以看出 PTQ 和 QAT 需要用户自定义的内容主要集中在： **模块融合** 和 **算子替换**。\n",
    "\n",
    "{func}`~torchvision.models.quantization.utils._fuse_modules` 提供了 {func}`~torch.ao.quantization.fuse_modules.fuse_modules` 和 {func}`~torch.ao.quantization.fuse_modules.fuse_modules_qat` 的统一接口。下面以 MobileNetV2 为例，简述如何使用 {func}`~torchvision.models.quantization.utils._fuse_modules` 函数和 {class}`~torch.nn.quantized.FloatFunctional` 类定制可量化的模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''参考 torchvision/models/quantization/mobilenetv2.py\n",
    "'''\n",
    "from typing import Any\n",
    "from torch import Tensor\n",
    "from torch import nn\n",
    "\n",
    "from torchvision._internally_replaced_utils import load_state_dict_from_url\n",
    "from torchvision.ops.misc import ConvNormActivation\n",
    "from torchvision.models.quantization.utils import _fuse_modules, _replace_relu, quantize_model\n",
    "from torch.ao.quantization import QuantStub, DeQuantStub\n",
    "from torchvision.models.mobilenetv2 import InvertedResidual, MobileNetV2, model_urls\n",
    "\n",
    "quant_model_urls = {\n",
    "    \"mobilenet_v2_qnnpack\": \"https://download.pytorch.org/models/quantized/mobilenet_v2_qnnpack_37f702c5.pth\"\n",
    "}\n",
    "\n",
    "\n",
    "class QuantizableInvertedResidual(InvertedResidual):\n",
    "    def __init__(self, *args: Any, **kwargs: Any) -> None:\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.skip_add = nn.quantized.FloatFunctional()\n",
    "\n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        if self.use_res_connect:\n",
    "            return self.skip_add.add(x, self.conv(x))\n",
    "        else:\n",
    "            return self.conv(x)\n",
    "\n",
    "    def fuse_model(self, is_qat: bool | None = None) -> None:\n",
    "        for idx in range(len(self.conv)):\n",
    "            if type(self.conv[idx]) is nn.Conv2d:\n",
    "                _fuse_modules(self.conv,\n",
    "                              [str(idx),\n",
    "                               str(idx + 1)],\n",
    "                              is_qat,\n",
    "                              inplace=True)\n",
    "\n",
    "\n",
    "class QuantizableMobileNetV2(MobileNetV2):\n",
    "    def __init__(self, *args: Any, **kwargs: Any) -> None:\n",
    "        \"\"\"\n",
    "        MobileNet V2 main class\n",
    "\n",
    "        Args:\n",
    "           继承自浮点 MobileNetV2 的参数\n",
    "        \"\"\"\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.quant = QuantStub()\n",
    "        self.dequant = DeQuantStub()\n",
    "\n",
    "    def forward(self, x: Tensor) -> Tensor:\n",
    "        x = self.quant(x)\n",
    "        x = self._forward_impl(x)\n",
    "        x = self.dequant(x)\n",
    "        return x\n",
    "\n",
    "    def fuse_model(self, is_qat: bool | None=None) -> None:\n",
    "        for m in self.modules():\n",
    "            if type(m) is ConvNormActivation:\n",
    "                _fuse_modules(m, [\"0\", \"1\", \"2\"], is_qat, inplace=True)\n",
    "            if type(m) is QuantizableInvertedResidual:\n",
    "                m.fuse_model(is_qat)\n",
    "\n",
    "\n",
    "def mobilenet_v2(\n",
    "    pretrained: bool = False,\n",
    "    progress: bool = True,\n",
    "    quantize: bool = False,\n",
    "    **kwargs: Any,\n",
    ") -> QuantizableMobileNetV2:\n",
    "    \"\"\"\n",
    "    从 `MobileNetV2：反向残差和线性瓶颈 <https://arxiv.org/abs/1801.04381>`_ 构建 MobileNetV2 架构。\n",
    "\n",
    "    注意，quantize = True 返回具有 8 bit 权值的量化模型。量化模型只支持推理并在 CPU 上运行。\n",
    "    目前还不支持 GPU 推理\n",
    "\n",
    "    Args:\n",
    "        pretrained (bool): 如果为 True，返回在 ImageNet 上训练过的模型。\n",
    "        progress (bool): 如果为 True，则显示下载到标准错误的进度条\n",
    "        quantize(bool): 如果为 True，则返回量化模型，否则返回浮点模型\n",
    "    \"\"\"\n",
    "    model = QuantizableMobileNetV2(block=QuantizableInvertedResidual, **kwargs)\n",
    "    _replace_relu(model)\n",
    "\n",
    "    if quantize:\n",
    "        # TODO use pretrained as a string to specify the backend\n",
    "        backend = \"qnnpack\"\n",
    "        quantize_model(model, backend)\n",
    "    else:\n",
    "        assert pretrained in [True, False]\n",
    "\n",
    "    if pretrained:\n",
    "        if quantize:\n",
    "            model_url = quant_model_urls[\"mobilenet_v2_\" + backend]\n",
    "        else:\n",
    "            model_url = model_urls[\"mobilenet_v2\"]\n",
    "\n",
    "        state_dict = load_state_dict_from_url(model_url, progress=progress)\n",
    "        model.load_state_dict(state_dict)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一些准备工作\n",
    "\n",
    "下面以 Cifar10 为了来说明 PTQ/QAT 的量化流程。\n",
    "\n",
    "定义几个[辅助函数](https://github.com/pytorch/examples/blob/master/imagenet/main.py)来帮助评估模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchq.helper import evaluate, print_size_of_model, load_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置超参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "saved_model_dir = 'models/'\n",
    "float_model_file = 'mobilenet_pretrained_float.pth'\n",
    "scripted_float_model_file = 'mobilenet_float_scripted.pth'\n",
    "scripted_ptq_model_file = 'mobilenet_ptq_scripted.pth'\n",
    "scripted_quantized_model_file = 'mobilenet_quantization_scripted_quantized.pth'\n",
    "scripted_qat_model_file = 'mobilenet_qat_scripted_quantized.pth'\n",
    "\n",
    "learning_rate = 5e-5\n",
    "num_epochs = 30\n",
    "batch_size = 16\n",
    "num_classes = 10\n",
    "\n",
    "# 设置评估策略\n",
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义数据集和数据加载器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "from torchq.xinet import CV\n",
    "\n",
    "# 为了 cifar10 匹配 ImageNet，需要将其 resize 到 224\n",
    "train_iter, test_iter = CV.load_data_cifar10(batch_size=batch_size,\n",
    "                                             resize=224)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看数据集的 batch 次数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练、测试批次分别为： 3125 625\n"
     ]
    }
   ],
   "source": [
    "print('训练、测试批次分别为：',\n",
    "      len(train_iter), len(test_iter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获取训练和测试数据集的大小："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 10000)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_train = sum(len(ys) for _, ys in train_iter)\n",
    "num_eval = sum(len(ys) for _, ys in test_iter)\n",
    "num_train, num_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 微调浮点模型\n",
    "\n",
    "配置浮点模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from torchvision.models.quantization import mobilenet_v2\n",
    "\n",
    "# 定义模型\n",
    "def create_model(quantize=False,\n",
    "                 num_classes=10,\n",
    "                 pretrained=False):\n",
    "    float_model = mobilenet_v2(pretrained=pretrained,\n",
    "                               quantize=quantize)\n",
    "    # 匹配 ``num_classes``\n",
    "    float_model.classifier[1] = nn.Linear(float_model.last_channel,\n",
    "                                          num_classes)\n",
    "    return float_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "float_model = create_model(pretrained=True,\n",
    "                           quantize=False,\n",
    "                           num_classes=num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义微调的函数 {class}`torchq.xinet.CV`.{func}`train_fine_tuning` 用于模型。\n",
    "\n",
    "微调浮点模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.012, train acc 0.996, test acc 0.949\n",
      "276.9 examples/sec on cuda:2\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg height=\"184.455469pt\" version=\"1.1\" viewBox=\"0 0 238.965625 184.455469\" width=\"238.965625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <metadata>\n  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2022-03-29T17:34:51.353391</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.4.0, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 184.455469 \nL 238.965625 184.455469 \nL 238.965625 -0 \nL 0 -0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 30.103125 146.899219 \nL 225.403125 146.899219 \nL 225.403125 10.999219 \nL 30.103125 10.999219 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 57.041056 146.899219 \nL 57.041056 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m3de9c93945\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"57.041056\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 5 -->\n      <g transform=\"translate(53.859806 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 90.71347 146.899219 \nL 90.71347 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"90.71347\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 10 -->\n      <g transform=\"translate(84.35097 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n        <path d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 124.385884 146.899219 \nL 124.385884 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"124.385884\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 15 -->\n      <g transform=\"translate(118.023384 161.497656)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 158.058297 146.899219 \nL 158.058297 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"158.058297\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 20 -->\n      <g transform=\"translate(151.695797 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 191.730711 146.899219 \nL 191.730711 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"191.730711\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 25 -->\n      <g transform=\"translate(185.368211 161.497656)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 225.403125 146.899219 \nL 225.403125 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"225.403125\" xlink:href=\"#m3de9c93945\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 30 -->\n      <g transform=\"translate(219.040625 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-33\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(112.525 175.175781)scale(0.1 -0.1)\">\n      <defs>\n       <path d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_13\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 146.899219 \nL 225.403125 146.899219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m55cfeea55f\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.0 -->\n      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_15\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 119.719219 \nL 225.403125 119.719219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"119.719219\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.2 -->\n      <g transform=\"translate(7.2 123.518437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 92.539219 \nL 225.403125 92.539219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"92.539219\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.4 -->\n      <g transform=\"translate(7.2 96.338437)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 65.359219 \nL 225.403125 65.359219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"65.359219\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.6 -->\n      <g transform=\"translate(7.2 69.158437)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 38.179219 \nL 225.403125 38.179219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"38.179219\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 0.8 -->\n      <g transform=\"translate(7.2 41.978437)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_23\">\n      <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 10.999219 \nL 225.403125 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#m55cfeea55f\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 1.0 -->\n      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_25\">\n    <path clip-path=\"url(#p639c8b1aac)\" d=\"M 24.715539 8.560968 \nL 26.062435 40.549836 \nL 27.409332 55.505064 \nL 28.756228 64.51953 \nL 30.103125 71.166239 \nL 31.450022 102.779484 \nL 32.796918 103.106751 \nL 35.490711 105.43524 \nL 36.837608 106.03371 \nL 38.184504 113.986373 \nL 39.531401 113.538475 \nL 40.878297 113.73424 \nL 42.225194 114.186225 \nL 43.572091 114.260246 \nL 44.918987 121.028374 \nL 46.265884 120.515439 \nL 47.61278 120.617613 \nL 48.959677 120.930524 \nL 50.306573 120.884085 \nL 51.65347 124.439494 \nL 53.000366 125.191764 \nL 54.347263 124.831315 \nL 55.694159 125.074212 \nL 57.041056 124.91375 \nL 58.387953 128.058153 \nL 59.734849 128.134854 \nL 61.081746 127.917934 \nL 62.428642 128.038899 \nL 63.775539 127.831466 \nL 65.122435 131.769974 \nL 67.816228 130.960006 \nL 69.163125 131.11094 \nL 70.510022 130.87434 \nL 71.856918 133.028536 \nL 73.203815 133.359903 \nL 74.550711 133.451506 \nL 77.244504 133.34691 \nL 78.591401 135.179561 \nL 79.938297 135.774051 \nL 82.632091 136.027742 \nL 83.978987 135.786159 \nL 85.325884 136.631538 \nL 86.67278 136.591038 \nL 88.019677 136.855287 \nL 89.366573 136.791906 \nL 90.71347 136.899408 \nL 92.060366 138.294513 \nL 93.407263 138.557221 \nL 97.447953 137.995133 \nL 98.794849 138.911982 \nL 101.488642 139.338203 \nL 104.182435 139.240028 \nL 105.529332 140.475049 \nL 106.876228 140.285133 \nL 108.223125 140.350958 \nL 110.916918 140.232266 \nL 112.263815 141.421973 \nL 113.610711 141.081506 \nL 114.957608 141.115737 \nL 116.304504 140.940856 \nL 117.651401 140.924676 \nL 118.998297 141.497791 \nL 121.692091 141.598784 \nL 123.038987 141.80475 \nL 124.385884 141.689543 \nL 125.73278 141.878605 \nL 127.079677 142.306362 \nL 131.120366 142.36258 \nL 132.467263 143.090326 \nL 133.814159 143.151743 \nL 135.161056 143.015163 \nL 137.854849 142.97891 \nL 139.201746 143.554058 \nL 145.936228 143.178564 \nL 151.323815 143.536351 \nL 152.670711 143.568133 \nL 154.017608 143.718964 \nL 158.058297 143.617599 \nL 160.752091 143.840993 \nL 162.098987 143.687949 \nL 164.79278 143.741102 \nL 166.139677 144.007999 \nL 167.486573 144.151764 \nL 170.180366 144.148034 \nL 171.527263 144.140437 \nL 172.874159 144.555539 \nL 176.914849 144.459345 \nL 178.261746 144.423621 \nL 179.608642 144.796955 \nL 180.955539 144.775386 \nL 182.302435 144.61729 \nL 190.383815 144.861704 \nL 194.424504 144.776871 \nL 198.465194 144.745803 \nL 199.812091 144.949761 \nL 201.158987 145.0354 \nL 202.505884 144.902945 \nL 205.199677 144.861942 \nL 206.546573 145.496579 \nL 207.89347 145.27427 \nL 211.934159 145.164227 \nL 213.281056 145.566602 \nL 214.627953 145.551557 \nL 215.974849 145.341209 \nL 218.668642 145.311088 \nL 220.015539 144.973421 \nL 221.362435 145.258658 \nL 225.403125 145.25123 \nL 225.403125 145.25123 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_26\">\n    <path clip-path=\"url(#p639c8b1aac)\" d=\"M 24.715539 56.865469 \nL 26.062435 46.543864 \nL 27.409332 41.703559 \nL 28.756228 38.862116 \nL 30.103125 36.638113 \nL 31.450022 25.961809 \nL 32.796918 26.050144 \nL 35.490711 25.278911 \nL 36.837608 25.007791 \nL 38.184504 22.428409 \nL 40.878297 22.451059 \nL 43.572091 22.287073 \nL 44.918987 19.778359 \nL 46.265884 19.968619 \nL 50.306573 19.873489 \nL 51.65347 18.840649 \nL 53.000366 18.432949 \nL 54.347263 18.586969 \nL 57.041056 18.454693 \nL 58.387953 17.495239 \nL 59.734849 17.400109 \nL 63.775539 17.712679 \nL 65.122435 16.204189 \nL 67.816228 16.616419 \nL 69.163125 16.516759 \nL 70.510022 16.560247 \nL 71.856918 15.782899 \nL 74.550711 15.624349 \nL 77.244504 15.671461 \nL 78.591401 14.926729 \nL 79.938297 14.654929 \nL 82.632091 14.641339 \nL 83.978987 14.779957 \nL 85.325884 14.355949 \nL 86.67278 14.451079 \nL 90.71347 14.391283 \nL 92.060366 14.016199 \nL 94.754159 13.912009 \nL 97.447953 14.122201 \nL 98.794849 13.717219 \nL 102.835539 13.635679 \nL 104.182435 13.673731 \nL 105.529332 12.996949 \nL 106.876228 13.126054 \nL 110.916918 13.157311 \nL 112.263815 12.643609 \nL 113.610711 12.840664 \nL 120.345194 12.813484 \nL 123.038987 12.691174 \nL 124.385884 12.752329 \nL 129.77347 12.507709 \nL 131.120366 12.491401 \nL 132.467263 12.167959 \nL 137.854849 12.214165 \nL 139.201746 11.896159 \nL 145.936228 12.235909 \nL 149.976918 12.100009 \nL 151.323815 12.110881 \nL 152.670711 11.964109 \nL 156.711401 11.960711 \nL 164.79278 12.059239 \nL 166.139677 11.855389 \nL 170.180366 11.913146 \nL 171.527263 11.901595 \nL 172.874159 11.678719 \nL 178.261746 11.776567 \nL 179.608642 11.515639 \nL 193.077608 11.569999 \nL 195.771401 11.724019 \nL 198.465194 11.700463 \nL 199.812091 11.569999 \nL 201.158987 11.556409 \nL 202.505884 11.660599 \nL 205.199677 11.665129 \nL 206.546573 11.338969 \nL 209.240366 11.488459 \nL 211.934159 11.526511 \nL 214.627953 11.366149 \nL 217.321746 11.434099 \nL 218.668642 11.466715 \nL 220.015539 11.610769 \nL 222.709332 11.474869 \nL 225.403125 11.504767 \nL 225.403125 11.504767 \n\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_27\">\n    <path clip-path=\"url(#p639c8b1aac)\" d=\"M 30.103125 24.833839 \nL 36.837608 21.368389 \nL 43.572091 20.430679 \nL 50.306573 19.574509 \nL 57.041056 19.126039 \nL 63.775539 19.153219 \nL 70.510022 18.691159 \nL 77.244504 18.473719 \nL 83.978987 18.201919 \nL 90.71347 18.405769 \nL 97.447953 18.297049 \nL 104.182435 17.821399 \nL 110.916918 17.902939 \nL 117.651401 18.256279 \nL 124.385884 17.794219 \nL 131.120366 18.038839 \nL 137.854849 17.807809 \nL 144.589332 17.685499 \nL 151.323815 18.052429 \nL 158.058297 17.998069 \nL 164.79278 17.807809 \nL 171.527263 17.726269 \nL 178.261746 18.052429 \nL 184.996228 17.685499 \nL 191.730711 17.848579 \nL 198.465194 17.685499 \nL 205.199677 17.902939 \nL 211.934159 17.671909 \nL 218.668642 17.767039 \nL 225.403125 17.916529 \n\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 30.103125 146.899219 \nL 30.103125 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 225.403125 146.899219 \nL 225.403125 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 30.103125 146.899219 \nL 225.403125 146.899219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 30.103125 10.999219 \nL 225.403125 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 140.634375 141.899219 \nL 218.403125 141.899219 \nQ 220.403125 141.899219 220.403125 139.899219 \nL 220.403125 96.864844 \nQ 220.403125 94.864844 218.403125 94.864844 \nL 140.634375 94.864844 \nQ 138.634375 94.864844 138.634375 96.864844 \nL 138.634375 139.899219 \nQ 138.634375 141.899219 140.634375 141.899219 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_28\">\n     <path d=\"M 142.634375 102.963281 \nL 162.634375 102.963281 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_29\"/>\n    <g id=\"text_14\">\n     <!-- train loss -->\n     <g transform=\"translate(170.634375 106.463281)scale(0.1 -0.1)\">\n      <defs>\n       <path d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\n      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-6c\"/>\n      <use x=\"292.333984\" xlink:href=\"#DejaVuSans-6f\"/>\n      <use x=\"353.515625\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"405.615234\" xlink:href=\"#DejaVuSans-73\"/>\n     </g>\n    </g>\n    <g id=\"line2d_30\">\n     <path d=\"M 142.634375 117.641406 \nL 162.634375 117.641406 \n\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_31\"/>\n    <g id=\"text_15\">\n     <!-- train acc -->\n     <g transform=\"translate(170.634375 121.141406)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\n      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"325.830078\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"380.810547\" xlink:href=\"#DejaVuSans-63\"/>\n     </g>\n    </g>\n    <g id=\"line2d_32\">\n     <path d=\"M 142.634375 132.319531 \nL 162.634375 132.319531 \n\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_33\"/>\n    <g id=\"text_16\">\n     <!-- test acc -->\n     <g transform=\"translate(170.634375 135.819531)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"192.041016\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"223.828125\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"285.107422\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"340.087891\" xlink:href=\"#DejaVuSans-63\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p639c8b1aac\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"30.103125\" y=\"10.999219\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "CV.train_fine_tuning(float_model, train_iter, test_iter,\n",
    "                     learning_rate=learning_rate,\n",
    "                     num_epochs=num_epochs,\n",
    "                     device='cuda:2',\n",
    "                     param_group=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "保存模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(float_model.state_dict(), saved_model_dir + float_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 配置可量化模型\n",
    "\n",
    "加载浮点模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "float_model = create_model(quantize=False,\n",
    "                           num_classes=num_classes)\n",
    "float_model = load_model(float_model, saved_model_dir + float_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看浮点模型的信息："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_info(model,\n",
    "               model_type='浮点模型',\n",
    "               test_iter=test_iter,\n",
    "               criterion=criterion, num_eval=num_eval):\n",
    "    '''打印信息'''\n",
    "    print_size_of_model(model)\n",
    "    top1, top5 = evaluate(model, criterion, test_iter)\n",
    "    print(f'\\n{model_type}：\\n\\t'\n",
    "          f'在 {num_eval} 张图片上评估 accuracy 为: {top1.avg:2.5f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型大小：9.187789 MB\n",
      "\n",
      "浮点模型：\n",
      "\t在 10000 张图片上评估 accuracy 为: 94.91000\n"
     ]
    }
   ],
   "source": [
    "print_info(float_model, model_type='浮点模型')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以先查看融合前的 inverted residual 块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): ConvNormActivation(\n",
       "    (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)\n",
       "    (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU()\n",
       "  )\n",
       "  (1): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "  (2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       ")"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "float_model.features[1].conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "融合模块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "float_model.fuse_model(is_qat=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看融合后的 inverted residual 块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): ConvNormActivation(\n",
       "    (0): ConvReLU2d(\n",
       "      (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)\n",
       "      (1): ReLU()\n",
       "    )\n",
       "    (1): Identity()\n",
       "    (2): Identity()\n",
       "  )\n",
       "  (1): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1))\n",
       "  (2): Identity()\n",
       ")"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "float_model.features[1].conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了得到“基线”精度，看看融合模块的非量化模型的精度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline 模型大小\n",
      "模型大小：8.923757 MB\n"
     ]
    }
   ],
   "source": [
    "model_type = '融合后的浮点模型'\n",
    "print(\"baseline 模型大小\")\n",
    "print_size_of_model(float_model)\n",
    "\n",
    "top1, top5 = evaluate(float_model, criterion, test_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "融合后的浮点模型：\n",
      "\t在 10000 张图片上评估 accuracy 为: 94.91\n"
     ]
    }
   ],
   "source": [
    "from torch import jit\n",
    "print(f'\\n{model_type}：\\n\\t在 {num_eval} 张图片上评估 accuracy 为: {top1.avg:2.2f}')\n",
    "# 保存\n",
    "jit.save(jit.script(float_model), saved_model_dir + scripted_float_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这将是我们进行比较的基准。接下来，尝试不同的量化方法。\n",
    "\n",
    "### PTQ 实战"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载模型\n",
    "myModel = create_model(pretrained=False,\n",
    "                       quantize=False,\n",
    "                       num_classes=num_classes)\n",
    "float_model = load_model(myModel,\n",
    "                         saved_model_dir + float_model_file)\n",
    "myModel.eval()\n",
    "\n",
    "# 融合\n",
    "myModel.fuse_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "指定量化配置（从简单的最小/最大范围估计和加权的逐张量量化开始）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QConfig(activation=functools.partial(<class 'torch.ao.quantization.observer.MinMaxObserver'>, quant_min=0, quant_max=127){}, weight=functools.partial(<class 'torch.ao.quantization.observer.MinMaxObserver'>, dtype=torch.qint8, qscheme=torch.per_tensor_symmetric){})"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch.ao.quantization.qconfig import default_qconfig\n",
    "\n",
    "myModel.qconfig = default_qconfig\n",
    "myModel.qconfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "开始校准准备："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PTQ 准备：插入观测者\n",
      "\n",
      " 查看观测者插入后的 inverted residual \n",
      "\n",
      " Sequential(\n",
      "  (0): ConvNormActivation(\n",
      "    (0): ConvReLU2d(\n",
      "      (0): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)\n",
      "      (1): ReLU()\n",
      "      (activation_post_process): MinMaxObserver(min_val=inf, max_val=-inf)\n",
      "    )\n",
      "    (1): Identity()\n",
      "    (2): Identity()\n",
      "  )\n",
      "  (1): Conv2d(\n",
      "    32, 16, kernel_size=(1, 1), stride=(1, 1)\n",
      "    (activation_post_process): MinMaxObserver(min_val=inf, max_val=-inf)\n",
      "  )\n",
      "  (2): Identity()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from torch.ao.quantization.quantize import prepare\n",
    "\n",
    "print('PTQ 准备：插入观测者')\n",
    "prepare(myModel, inplace=True)\n",
    "print('\\n 查看观测者插入后的 inverted residual \\n\\n',\n",
    "      myModel.features[1].conv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用数据集校准："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PTQ：校准完成！\n"
     ]
    }
   ],
   "source": [
    "num_calibration_batches = 200 # 取部分训练集做校准\n",
    "evaluate(myModel, criterion, train_iter, neval_batches=num_calibration_batches)\n",
    "print('\\nPTQ：校准完成！')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "转换为量化模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PTQ：转换完成！\n"
     ]
    }
   ],
   "source": [
    "from torch.ao.quantization.quantize import convert\n",
    "\n",
    "convert(myModel, inplace=True)\n",
    "print('PTQ：转换完成！')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "融合并量化后，查看融合模块的 Inverted Residual 块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): ConvNormActivation(\n",
       "    (0): QuantizedConvReLU2d(32, 32, kernel_size=(3, 3), stride=(1, 1), scale=0.1370350867509842, zero_point=0, padding=(1, 1), groups=32)\n",
       "    (1): Identity()\n",
       "    (2): Identity()\n",
       "  )\n",
       "  (1): QuantizedConv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), scale=0.20581312477588654, zero_point=69)\n",
       "  (2): Identity()\n",
       ")"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myModel.features[1].conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "量化后的模型大小："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型大小：2.356113 MB\n"
     ]
    }
   ],
   "source": [
    "print_size_of_model(myModel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PTQ 模型：\n",
      "\t在 10000 张图片上评估 accuracy 为: 51.11\n"
     ]
    }
   ],
   "source": [
    "model_type = 'PTQ 模型'\n",
    "top1, top5 = evaluate(myModel, criterion, test_iter)\n",
    "print(f'\\n{model_type}：\\n\\t在 {num_eval} 张图片上评估 accuracy 为: {top1.avg:2.2f}')\n",
    "# jit.save(jit.script(myModel), saved_model_dir + scripted_ptq_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用了简单的 min/max 观测器来确定量化参数，将模型的大小减少到了 2.36 MB 以下，几乎减少了 4 倍。\n",
    "\n",
    "此外，通过使用不同的量化配置来显著提高精度（对于量化 ARM 架构的推荐配置重复同样的练习）。该配置的操作如下：\n",
    "\n",
    "- 在 per-channel 基础上量化权重\n",
    "- 使用直方图观测器，收集激活的直方图，然后以最佳方式选择量化参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QConfig(activation=functools.partial(<class 'torch.ao.quantization.observer.HistogramObserver'>, reduce_range=True){}, weight=functools.partial(<class 'torch.ao.quantization.observer.PerChannelMinMaxObserver'>, dtype=torch.qint8, qscheme=torch.per_channel_symmetric){})"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_channel_quantized_model = create_model(quantize=False,\n",
    "                                           num_classes=num_classes)\n",
    "per_channel_quantized_model = load_model(per_channel_quantized_model,\n",
    "                                         saved_model_dir + float_model_file)\n",
    "per_channel_quantized_model.eval()\n",
    "per_channel_quantized_model.fuse_model()\n",
    "per_channel_quantized_model.qconfig = get_default_qconfig('fbgemm')\n",
    "per_channel_quantized_model.qconfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_calibration_batches = 200 # 仅仅取 200 个批次\n",
    "prepare(per_channel_quantized_model, inplace=True)\n",
    "evaluate(per_channel_quantized_model, criterion,\n",
    "         train_iter, num_calibration_batches)\n",
    "\n",
    "model_type = 'PTQ 模型（直方图观测器）'\n",
    "convert(per_channel_quantized_model, inplace=True)\n",
    "top1, top5 = evaluate(per_channel_quantized_model, criterion, test_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PTQ 模型（直方图观测器）：\n",
      "\t在 10000 张图片上评估 accuracy 为: 80.42\n"
     ]
    }
   ],
   "source": [
    "print(f'\\n{model_type}：\\n\\t在 {num_eval} 张图片上评估 accuracy 为: {top1.avg:2.2f}')\n",
    "jit.save(jit.script(per_channel_quantized_model),\n",
    "         saved_model_dir + scripted_quantized_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "仅仅改变这种量化配置方法，就可以将准确度提高到 $80.42\\%$ 以上！尽管如此，这还是比 $95\\%$ 的基线水平低了 $15\\%$。\n",
    "\n",
    "### QAT 实战\n",
    "\n",
    "使用 QAT，所有的权值和激活都在前向和后向训练过程中被“伪量化”：也就是说，浮点值被舍入以模拟 int8 值，但所有的计算仍然使用浮点数完成。因此，训练过程中的所有权重调整都是在“感知到”模型最终将被量化的情况下进行的；因此，在量化之后，这种方法通常比动态量化或训练后的静态量化产生更高的精度。\n",
    "\n",
    "实际执行 QAT 的总体工作流程与之前非常相似：\n",
    "\n",
    "- 可以使用与以前相同的模型：不需要为量化感知训练做额外的准备。\n",
    "- 需要使用 `qconfig` 来指定在权重和激活之后插入何种类型的伪量化，而不是指定观测者。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_qat_model(num_classes,\n",
    "                     model_path,\n",
    "                     quantize=False,\n",
    "                     backend='fbgemm'):\n",
    "    qat_model = create_model(quantize=quantize,\n",
    "                             num_classes=num_classes)\n",
    "    qat_model = load_model(qat_model, model_path)\n",
    "    qat_model.fuse_model()\n",
    "    qat_model.qconfig = get_default_qat_qconfig(backend=backend)\n",
    "    return qat_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，`prepare_qat` 执行“伪量化”，为量化感知训练准备模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.ao.quantization.quantize import prepare_qat\n",
    "\n",
    "model_path = saved_model_dir + float_model_file\n",
    "qat_model = create_qat_model(num_classes, model_path)\n",
    "qat_model = prepare_qat(qat_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inverted Residual Block：准备好 QAT 后，注意伪量化模块："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): ConvNormActivation(\n",
       "    (0): ConvBnReLU2d(\n",
       "      32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False\n",
       "      (bn): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (weight_fake_quant): FusedMovingAvgObsFakeQuantize(\n",
       "        fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.qint8, quant_min=-128, quant_max=127, qscheme=torch.per_channel_symmetric, reduce_range=False\n",
       "        (activation_post_process): MovingAveragePerChannelMinMaxObserver(min_val=tensor([]), max_val=tensor([]))\n",
       "      )\n",
       "      (activation_post_process): FusedMovingAvgObsFakeQuantize(\n",
       "        fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True\n",
       "        (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)\n",
       "      )\n",
       "    )\n",
       "    (1): Identity()\n",
       "    (2): Identity()\n",
       "  )\n",
       "  (1): ConvBn2d(\n",
       "    32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False\n",
       "    (bn): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (weight_fake_quant): FusedMovingAvgObsFakeQuantize(\n",
       "      fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.qint8, quant_min=-128, quant_max=127, qscheme=torch.per_channel_symmetric, reduce_range=False\n",
       "      (activation_post_process): MovingAveragePerChannelMinMaxObserver(min_val=tensor([]), max_val=tensor([]))\n",
       "    )\n",
       "    (activation_post_process): FusedMovingAvgObsFakeQuantize(\n",
       "      fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True\n",
       "      (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)\n",
       "    )\n",
       "  )\n",
       "  (2): Identity()\n",
       ")"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qat_model.features[1].conv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练具有高精确度的量化模型要求在推理时对数值进行精确的建模。因此，对于量化感知训练，我们对训练循环进行如下修改：\n",
    "\n",
    "- 将批处理范数转换为训练结束时的运行均值和方差，以更好地匹配推理数值。\n",
    "- 冻结量化器参数（尺度和零点）并微调权重。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.013, train acc 0.996, test acc 0.948\n",
      "55.3 examples/sec on cuda:2\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg height=\"184.455469pt\" version=\"1.1\" viewBox=\"0 0 245.328125 184.455469\" width=\"245.328125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <metadata>\n  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2022-03-30T04:56:14.107754</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.4.0, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 184.455469 \nL 245.328125 184.455469 \nL 245.328125 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 36.465625 146.899219 \nL 231.765625 146.899219 \nL 231.765625 10.999219 \nL 36.465625 10.999219 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 63.403556 146.899219 \nL 63.403556 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"md03c04327e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"63.403556\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 5 -->\n      <g transform=\"translate(60.222306 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 691 4666 \nL 3169 4666 \nL 3169 4134 \nL 1269 4134 \nL 1269 2991 \nQ 1406 3038 1543 3061 \nQ 1681 3084 1819 3084 \nQ 2600 3084 3056 2656 \nQ 3513 2228 3513 1497 \nQ 3513 744 3044 326 \nQ 2575 -91 1722 -91 \nQ 1428 -91 1123 -41 \nQ 819 9 494 109 \nL 494 744 \nQ 775 591 1075 516 \nQ 1375 441 1709 441 \nQ 2250 441 2565 725 \nQ 2881 1009 2881 1497 \nQ 2881 1984 2565 2268 \nQ 2250 2553 1709 2553 \nQ 1456 2553 1204 2497 \nQ 953 2441 691 2322 \nL 691 4666 \nz\n\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 97.07597 146.899219 \nL 97.07597 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"97.07597\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 10 -->\n      <g transform=\"translate(90.71347 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 794 531 \nL 1825 531 \nL 1825 4091 \nL 703 3866 \nL 703 4441 \nL 1819 4666 \nL 2450 4666 \nL 2450 531 \nL 3481 531 \nL 3481 0 \nL 794 0 \nL 794 531 \nz\n\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n        <path d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 130.748384 146.899219 \nL 130.748384 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.748384\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 15 -->\n      <g transform=\"translate(124.385884 161.497656)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 164.420797 146.899219 \nL 164.420797 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"164.420797\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 20 -->\n      <g transform=\"translate(158.058297 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 198.093211 146.899219 \nL 198.093211 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"198.093211\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 25 -->\n      <g transform=\"translate(191.730711 161.497656)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 231.765625 146.899219 \nL 231.765625 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"231.765625\" xlink:href=\"#md03c04327e\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 30 -->\n      <g transform=\"translate(225.403125 161.497656)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 2597 2516 \nQ 3050 2419 3304 2112 \nQ 3559 1806 3559 1356 \nQ 3559 666 3084 287 \nQ 2609 -91 1734 -91 \nQ 1441 -91 1130 -33 \nQ 819 25 488 141 \nL 488 750 \nQ 750 597 1062 519 \nQ 1375 441 1716 441 \nQ 2309 441 2620 675 \nQ 2931 909 2931 1356 \nQ 2931 1769 2642 2001 \nQ 2353 2234 1838 2234 \nL 1294 2234 \nL 1294 2753 \nL 1863 2753 \nQ 2328 2753 2575 2939 \nQ 2822 3125 2822 3475 \nQ 2822 3834 2567 4026 \nQ 2313 4219 1838 4219 \nQ 1578 4219 1281 4162 \nQ 984 4106 628 3988 \nL 628 4550 \nQ 988 4650 1302 4700 \nQ 1616 4750 1894 4750 \nQ 2613 4750 3031 4423 \nQ 3450 4097 3450 3541 \nQ 3450 3153 3228 2886 \nQ 3006 2619 2597 2516 \nz\n\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-33\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epoch -->\n     <g transform=\"translate(118.8875 175.175781)scale(0.1 -0.1)\">\n      <defs>\n       <path d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 1159 525 \nL 1159 -1331 \nL 581 -1331 \nL 581 3500 \nL 1159 3500 \nL 1159 2969 \nQ 1341 3281 1617 3432 \nQ 1894 3584 2278 3584 \nQ 2916 3584 3314 3078 \nQ 3713 2572 3713 1747 \nQ 3713 922 3314 415 \nQ 2916 -91 2278 -91 \nQ 1894 -91 1617 61 \nQ 1341 213 1159 525 \nz\nM 3116 1747 \nQ 3116 2381 2855 2742 \nQ 2594 3103 2138 3103 \nQ 1681 3103 1420 2742 \nQ 1159 2381 1159 1747 \nQ 1159 1113 1420 752 \nQ 1681 391 2138 391 \nQ 2594 391 2855 752 \nQ 3116 1113 3116 1747 \nz\n\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 1959 3097 \nQ 1497 3097 1228 2736 \nQ 959 2375 959 1747 \nQ 959 1119 1226 758 \nQ 1494 397 1959 397 \nQ 2419 397 2687 759 \nQ 2956 1122 2956 1747 \nQ 2956 2369 2687 2733 \nQ 2419 3097 1959 3097 \nz\nM 1959 3584 \nQ 2709 3584 3137 3096 \nQ 3566 2609 3566 1747 \nQ 3566 888 3137 398 \nQ 2709 -91 1959 -91 \nQ 1206 -91 779 398 \nQ 353 888 353 1747 \nQ 353 2609 779 3096 \nQ 1206 3584 1959 3584 \nz\n\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3122 3366 \nL 3122 2828 \nQ 2878 2963 2633 3030 \nQ 2388 3097 2138 3097 \nQ 1578 3097 1268 2742 \nQ 959 2388 959 1747 \nQ 959 1106 1268 751 \nQ 1578 397 2138 397 \nQ 2388 397 2633 464 \nQ 2878 531 3122 666 \nL 3122 134 \nQ 2881 22 2623 -34 \nQ 2366 -91 2075 -91 \nQ 1284 -91 818 406 \nQ 353 903 353 1747 \nQ 353 2603 823 3093 \nQ 1294 3584 2113 3584 \nQ 2378 3584 2631 3529 \nQ 2884 3475 3122 3366 \nz\n\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 4863 \nL 1159 4863 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-65\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_13\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 146.899219 \nL 231.765625 146.899219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m0fb2e4dc96\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m0fb2e4dc96\" y=\"146.899219\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 0.80 -->\n      <g transform=\"translate(7.2 150.698437)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 684 794 \nL 1344 794 \nL 1344 0 \nL 684 0 \nL 684 794 \nz\n\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n        <path d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_15\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 112.924219 \nL 231.765625 112.924219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m0fb2e4dc96\" y=\"112.924219\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 0.85 -->\n      <g transform=\"translate(7.2 116.723437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_17\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 78.949219 \nL 231.765625 78.949219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m0fb2e4dc96\" y=\"78.949219\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- 0.90 -->\n      <g transform=\"translate(7.2 82.748438)scale(0.1 -0.1)\">\n       <defs>\n        <path d=\"M 703 97 \nL 703 672 \nQ 941 559 1184 500 \nQ 1428 441 1663 441 \nQ 2288 441 2617 861 \nQ 2947 1281 2994 2138 \nQ 2813 1869 2534 1725 \nQ 2256 1581 1919 1581 \nQ 1219 1581 811 2004 \nQ 403 2428 403 3163 \nQ 403 3881 828 4315 \nQ 1253 4750 1959 4750 \nQ 2769 4750 3195 4129 \nQ 3622 3509 3622 2328 \nQ 3622 1225 3098 567 \nQ 2575 -91 1691 -91 \nQ 1453 -91 1209 -44 \nQ 966 3 703 97 \nz\nM 1959 2075 \nQ 2384 2075 2632 2365 \nQ 2881 2656 2881 3163 \nQ 2881 3666 2632 3958 \nQ 2384 4250 1959 4250 \nQ 1534 4250 1286 3958 \nQ 1038 3666 1038 3163 \nQ 1038 2656 1286 2365 \nQ 1534 2075 1959 2075 \nz\n\" id=\"DejaVuSans-39\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_19\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 44.974219 \nL 231.765625 44.974219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m0fb2e4dc96\" y=\"44.974219\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0.95 -->\n      <g transform=\"translate(7.2 48.773437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_21\">\n      <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 10.999219 \nL 231.765625 10.999219 \n\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m0fb2e4dc96\" y=\"10.999219\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 1.00 -->\n      <g transform=\"translate(7.2 14.798437)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-31\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_23\">\n    <path clip-path=\"url(#p91ee78b89a)\" d=\"M 31.078039 70.615523 \nL 32.424935 72.149463 \nL 33.771832 73.974414 \nL 35.118728 74.335338 \nL 36.465625 76.015284 \nL 37.812522 86.703997 \nL 39.159418 88.161496 \nL 40.506315 90.029004 \nL 43.200108 91.898526 \nL 44.547004 102.847997 \nL 45.893901 102.299912 \nL 47.240797 104.532696 \nL 48.587694 105.06605 \nL 49.934591 105.835775 \nL 51.281487 110.27355 \nL 52.628384 109.25242 \nL 55.322177 108.929008 \nL 56.669073 108.247929 \nL 59.362866 111.802944 \nL 60.709763 111.747194 \nL 62.056659 112.15115 \nL 63.403556 112.412372 \nL 64.750453 113.430062 \nL 66.097349 113.952392 \nL 67.444246 115.794983 \nL 68.791142 115.991017 \nL 70.138039 115.697004 \nL 71.484935 119.376502 \nL 74.178728 120.07993 \nL 75.525625 119.361033 \nL 76.872522 117.860135 \nL 78.219418 116.713112 \nL 79.566315 120.993078 \nL 80.913211 120.474337 \nL 82.260108 120.106967 \nL 83.607004 120.699894 \nL 84.953901 120.879738 \nL 86.300797 122.010243 \nL 87.647694 122.528537 \nL 88.994591 123.839023 \nL 90.341487 123.906835 \nL 91.688384 123.472706 \nL 93.03528 124.954689 \nL 94.382177 125.470114 \nL 97.07597 125.263453 \nL 98.422866 126.248128 \nL 99.769763 123.978746 \nL 101.116659 125.097742 \nL 102.463556 125.402897 \nL 103.810453 126.075004 \nL 105.157349 128.304803 \nL 106.504246 127.89696 \nL 107.851142 128.510186 \nL 109.198039 128.603277 \nL 110.544935 128.170605 \nL 111.891832 129.364174 \nL 113.238728 129.330643 \nL 114.585625 128.807286 \nL 115.932522 129.266167 \nL 117.279418 129.55765 \nL 118.626315 129.512733 \nL 119.973211 130.269941 \nL 121.320108 130.69975 \nL 124.013901 130.830016 \nL 125.360797 132.735197 \nL 126.707694 131.267384 \nL 128.054591 131.465208 \nL 129.401487 131.2067 \nL 132.09528 131.16353 \nL 133.442177 131.432938 \nL 134.789073 131.477423 \nL 137.482866 131.703942 \nL 140.176659 133.522378 \nL 141.523556 132.255562 \nL 142.870453 132.235303 \nL 144.217349 132.697386 \nL 145.564246 133.468391 \nL 146.911142 134.39028 \nL 148.258039 134.246047 \nL 149.604935 133.534188 \nL 150.951832 133.530884 \nL 152.298728 136.660609 \nL 153.645625 135.545191 \nL 154.992522 134.798966 \nL 156.339418 134.581701 \nL 157.686315 134.035415 \nL 159.033211 132.962147 \nL 160.380108 132.070981 \nL 161.727004 132.53534 \nL 163.073901 131.864772 \nL 164.420797 132.214483 \nL 165.767694 136.174205 \nL 167.114591 135.255114 \nL 169.808384 135.121916 \nL 171.15528 135.501301 \nL 172.502177 132.683101 \nL 173.849073 133.467715 \nL 175.19597 133.547182 \nL 176.542866 134.036376 \nL 177.889763 134.065951 \nL 179.236659 135.14947 \nL 180.583556 135.445245 \nL 181.930453 136.53371 \nL 183.277349 135.944888 \nL 184.624246 135.703902 \nL 185.971142 136.362435 \nL 187.318039 136.025007 \nL 188.664935 135.902756 \nL 190.011832 135.500679 \nL 191.358728 135.839849 \nL 192.705625 134.820861 \nL 194.052522 134.206323 \nL 195.399418 134.250105 \nL 196.746315 134.689309 \nL 198.093211 134.968594 \nL 199.440108 138.658525 \nL 200.787004 137.133369 \nL 202.133901 136.530263 \nL 203.480797 137.035581 \nL 204.827694 137.383782 \nL 206.174591 138.294842 \nL 207.521487 136.645425 \nL 208.868384 136.868177 \nL 210.21528 137.515696 \nL 211.562177 137.357385 \nL 212.909073 141.152067 \nL 214.25597 139.317938 \nL 215.602866 138.803798 \nL 216.949763 138.737913 \nL 218.296659 138.45686 \nL 219.643556 137.810523 \nL 220.990453 138.008781 \nL 222.337349 137.6813 \nL 223.684246 137.602418 \nL 225.031142 137.69467 \nL 226.378039 138.114574 \nL 227.724935 138.045231 \nL 230.418728 138.301875 \nL 231.765625 138.134319 \nL 231.765625 138.134319 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_24\">\n    <path clip-path=\"url(#p91ee78b89a)\" d=\"M 31.078039 36.412519 \nL 32.424935 36.242644 \nL 33.771832 35.665069 \nL 35.118728 35.444231 \nL 36.465625 34.795309 \nL 37.812522 31.859869 \nL 39.159418 31.214344 \nL 40.506315 30.387619 \nL 41.853211 30.246056 \nL 43.200108 29.984449 \nL 44.547004 26.220019 \nL 45.893901 26.559769 \nL 47.240797 25.676419 \nL 48.587694 25.336669 \nL 49.934591 25.268719 \nL 51.281487 24.385369 \nL 52.628384 24.827044 \nL 53.97528 24.996919 \nL 55.322177 24.894994 \nL 56.669073 24.915379 \nL 59.362866 23.230219 \nL 60.709763 23.207569 \nL 62.056659 22.941431 \nL 63.403556 23.026369 \nL 66.097349 22.788544 \nL 67.444246 22.165669 \nL 68.791142 22.075069 \nL 70.138039 22.143019 \nL 71.484935 20.987869 \nL 72.831832 20.682094 \nL 75.525625 20.580169 \nL 76.872522 21.205309 \nL 78.219418 21.123769 \nL 79.566315 19.492969 \nL 80.913211 19.923319 \nL 82.260108 20.138494 \nL 83.607004 19.859899 \nL 84.953901 20.444269 \nL 86.300797 19.832719 \nL 87.647694 19.560919 \nL 88.994591 19.153219 \nL 90.341487 19.302709 \nL 91.688384 18.609619 \nL 93.03528 18.575644 \nL 94.382177 18.269869 \nL 95.729073 18.371794 \nL 97.07597 18.337819 \nL 98.422866 17.998069 \nL 99.769763 18.779494 \nL 101.116659 18.473719 \nL 102.463556 18.405769 \nL 107.851142 17.341219 \nL 109.198039 17.403506 \nL 110.544935 17.658319 \nL 111.891832 17.114719 \nL 114.585625 17.160019 \nL 115.932522 16.808944 \nL 117.279418 16.747789 \nL 118.626315 17.250619 \nL 119.973211 16.639069 \nL 124.013901 16.353679 \nL 125.360797 15.891619 \nL 126.707694 16.027519 \nL 129.401487 16.163419 \nL 132.09528 16.435219 \nL 133.442177 16.129444 \nL 136.13597 16.061494 \nL 137.482866 15.905209 \nL 140.176659 15.348019 \nL 141.523556 15.778369 \nL 144.217349 15.579049 \nL 146.911142 14.702494 \nL 148.258039 14.872369 \nL 149.604935 15.246094 \nL 150.951832 15.293659 \nL 152.298728 14.532619 \nL 153.645625 14.838394 \nL 154.992522 15.302719 \nL 157.686315 15.443149 \nL 160.380108 15.993544 \nL 161.727004 15.710419 \nL 163.073901 15.959569 \nL 164.420797 15.973159 \nL 165.767694 14.532619 \nL 167.114591 14.940319 \nL 168.461487 15.076219 \nL 169.808384 15.093206 \nL 171.15528 14.994679 \nL 172.502177 16.027519 \nL 173.849073 15.585844 \nL 175.19597 15.619819 \nL 176.542866 15.449944 \nL 177.889763 15.443149 \nL 179.236659 14.940319 \nL 180.583556 14.872369 \nL 181.930453 14.419369 \nL 184.624246 14.641339 \nL 185.971142 14.328769 \nL 187.318039 14.498644 \nL 188.664935 14.532619 \nL 190.011832 14.753456 \nL 191.358728 14.627749 \nL 194.052522 15.314044 \nL 195.399418 15.415969 \nL 198.093211 15.212119 \nL 199.440108 13.649269 \nL 200.787004 14.498644 \nL 202.133901 14.781769 \nL 206.174591 13.853119 \nL 207.521487 14.430694 \nL 208.868384 14.306119 \nL 210.21528 14.039981 \nL 211.562177 14.097739 \nL 212.909073 12.630019 \nL 214.25597 13.343494 \nL 215.602866 13.490719 \nL 216.949763 13.496381 \nL 218.296659 13.635679 \nL 219.643556 14.464669 \nL 220.990453 14.090944 \nL 223.684246 14.056969 \nL 226.378039 13.989019 \nL 227.724935 14.226844 \nL 231.765625 14.056969 \nL 231.765625 14.056969 \n\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_25\">\n    <path clip-path=\"url(#p91ee78b89a)\" d=\"M 36.465625 54.351319 \nL 43.200108 65.970769 \nL 49.934591 67.329769 \nL 56.669073 48.779419 \nL 63.403556 59.583469 \nL 70.138039 47.420419 \nL 76.872522 53.671819 \nL 83.607004 49.662769 \nL 90.341487 49.458919 \nL 97.07597 47.352469 \nL 103.810453 47.012719 \nL 110.544935 46.129369 \nL 117.279418 46.740919 \nL 124.013901 47.760169 \nL 130.748384 45.246019 \nL 137.482866 45.993469 \nL 144.217349 47.284519 \nL 150.951832 45.313969 \nL 157.686315 46.129369 \nL 164.420797 46.876819 \nL 171.15528 48.711469 \nL 177.889763 45.449869 \nL 184.624246 46.401169 \nL 191.358728 46.605019 \nL 198.093211 46.740919 \nL 204.827694 46.605019 \nL 211.562177 47.148619 \nL 218.296659 44.702419 \nL 225.031142 43.139569 \nL 231.765625 46.537069 \n\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 36.465625 146.899219 \nL 36.465625 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 231.765625 146.899219 \nL 231.765625 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 36.465625 146.899219 \nL 231.765625 146.899219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 36.465625 10.999219 \nL 231.765625 10.999219 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 122.714063 102.466406 \nL 224.765625 102.466406 \nQ 226.765625 102.466406 226.765625 100.466406 \nL 226.765625 57.432031 \nQ 226.765625 55.432031 224.765625 55.432031 \nL 122.714063 55.432031 \nQ 120.714063 55.432031 120.714063 57.432031 \nL 120.714063 100.466406 \nQ 120.714063 102.466406 122.714063 102.466406 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_26\">\n     <path d=\"M 124.714063 63.530469 \nL 144.714063 63.530469 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_27\"/>\n    <g id=\"text_13\">\n     <!-- 0.8+train loss -->\n     <g transform=\"translate(152.714063 67.030469)scale(0.1 -0.1)\">\n      <defs>\n       <path d=\"M 2944 4013 \nL 2944 2272 \nL 4684 2272 \nL 4684 1741 \nL 2944 1741 \nL 2944 0 \nL 2419 0 \nL 2419 1741 \nL 678 1741 \nL 678 2272 \nL 2419 2272 \nL 2419 4013 \nL 2944 4013 \nz\n\" id=\"DejaVuSans-2b\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 1172 4494 \nL 1172 3500 \nL 2356 3500 \nL 2356 3053 \nL 1172 3053 \nL 1172 1153 \nQ 1172 725 1289 603 \nQ 1406 481 1766 481 \nL 2356 481 \nL 2356 0 \nL 1766 0 \nQ 1100 0 847 248 \nQ 594 497 594 1153 \nL 594 3053 \nL 172 3053 \nL 172 3500 \nL 594 3500 \nL 594 4494 \nL 1172 4494 \nz\n\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2194 1759 \nQ 1497 1759 1228 1600 \nQ 959 1441 959 1056 \nQ 959 750 1161 570 \nQ 1363 391 1709 391 \nQ 2188 391 2477 730 \nQ 2766 1069 2766 1631 \nL 2766 1759 \nL 2194 1759 \nz\nM 3341 1997 \nL 3341 0 \nL 2766 0 \nL 2766 531 \nQ 2569 213 2275 61 \nQ 1981 -91 1556 -91 \nQ 1019 -91 701 211 \nQ 384 513 384 1019 \nQ 384 1609 779 1909 \nQ 1175 2209 1959 2209 \nL 2766 2209 \nL 2766 2266 \nQ 2766 2663 2505 2880 \nQ 2244 3097 1772 3097 \nQ 1472 3097 1187 3025 \nQ 903 2953 641 2809 \nL 641 3341 \nQ 956 3463 1253 3523 \nQ 1550 3584 1831 3584 \nQ 2591 3584 2966 3190 \nQ 3341 2797 3341 1997 \nz\n\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 603 3500 \nL 1178 3500 \nL 1178 0 \nL 603 0 \nL 603 3500 \nz\nM 603 4863 \nL 1178 4863 \nL 1178 4134 \nL 603 4134 \nL 603 4863 \nz\n\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 3513 2113 \nL 3513 0 \nL 2938 0 \nL 2938 2094 \nQ 2938 2591 2744 2837 \nQ 2550 3084 2163 3084 \nQ 1697 3084 1428 2787 \nQ 1159 2491 1159 1978 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1366 3272 1645 3428 \nQ 1925 3584 2291 3584 \nQ 2894 3584 3203 3211 \nQ 3513 2838 3513 2113 \nz\n\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n       <path d=\"M 2834 3397 \nL 2834 2853 \nQ 2591 2978 2328 3040 \nQ 2066 3103 1784 3103 \nQ 1356 3103 1142 2972 \nQ 928 2841 928 2578 \nQ 928 2378 1081 2264 \nQ 1234 2150 1697 2047 \nL 1894 2003 \nQ 2506 1872 2764 1633 \nQ 3022 1394 3022 966 \nQ 3022 478 2636 193 \nQ 2250 -91 1575 -91 \nQ 1294 -91 989 -36 \nQ 684 19 347 128 \nL 347 722 \nQ 666 556 975 473 \nQ 1284 391 1588 391 \nQ 1994 391 2212 530 \nQ 2431 669 2431 922 \nQ 2431 1156 2273 1281 \nQ 2116 1406 1581 1522 \nL 1381 1569 \nQ 847 1681 609 1914 \nQ 372 2147 372 2553 \nQ 372 3047 722 3315 \nQ 1072 3584 1716 3584 \nQ 2034 3584 2315 3537 \nQ 2597 3491 2834 3397 \nz\n\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-30\"/>\n      <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n      <use x=\"95.410156\" xlink:href=\"#DejaVuSans-38\"/>\n      <use x=\"159.033203\" xlink:href=\"#DejaVuSans-2b\"/>\n      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"282.03125\" xlink:href=\"#DejaVuSans-72\"/>\n      <use x=\"323.144531\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"384.423828\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"412.207031\" xlink:href=\"#DejaVuSans-6e\"/>\n      <use x=\"475.585938\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"507.373047\" xlink:href=\"#DejaVuSans-6c\"/>\n      <use x=\"535.15625\" xlink:href=\"#DejaVuSans-6f\"/>\n      <use x=\"596.337891\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"648.4375\" xlink:href=\"#DejaVuSans-73\"/>\n     </g>\n    </g>\n    <g id=\"line2d_28\">\n     <path d=\"M 124.714063 78.208594 \nL 144.714063 78.208594 \n\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_29\"/>\n    <g id=\"text_14\">\n     <!-- train acc -->\n     <g transform=\"translate(152.714063 81.708594)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\n      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"325.830078\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"380.810547\" xlink:href=\"#DejaVuSans-63\"/>\n     </g>\n    </g>\n    <g id=\"line2d_30\">\n     <path d=\"M 124.714063 92.886719 \nL 144.714063 92.886719 \n\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_31\"/>\n    <g id=\"text_15\">\n     <!-- test acc -->\n     <g transform=\"translate(152.714063 96.386719)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-65\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-73\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-74\"/>\n      <use x=\"192.041016\" xlink:href=\"#DejaVuSans-20\"/>\n      <use x=\"223.828125\" xlink:href=\"#DejaVuSans-61\"/>\n      <use x=\"285.107422\" xlink:href=\"#DejaVuSans-63\"/>\n      <use x=\"340.087891\" xlink:href=\"#DejaVuSans-63\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"p91ee78b89a\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"36.465625\" y=\"10.999219\"/>\n  </clipPath>\n </defs>\n</svg>\n",
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "CV.train_fine_tuning(qat_model,\n",
    "                     train_iter,\n",
    "                     test_iter,\n",
    "                     learning_rate=learning_rate,\n",
    "                     num_epochs=30,\n",
    "                     device='cuda:2',\n",
    "                     param_group=True,\n",
    "                     is_freeze=False,\n",
    "                     is_quantized_acc=False,\n",
    "                     need_qconfig=False,\n",
    "                     ylim=[0.8, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "这里的损失函数向上平移了 0.8 以提供更好的视觉效果。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于量化模型暂仅支持 CPU，故而需要先将模型转换为 CPU 版本，则转为量化版本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert(qat_model.cpu().eval(), inplace=True)\n",
    "qat_model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型大小：2.656573 MB\n",
      "\n",
      "QAT 模型：\n",
      "\t在 10000 张图片上评估 accuracy 为: 94.41000\n"
     ]
    }
   ],
   "source": [
    "print_info(qat_model,'QAT 模型')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "量化感知训练在整个数据集上的准确率超过 $94.4\\%$，接近浮点精度 $95\\%$。\n",
    "\n",
    "更多关于 QAT 的内容：\n",
    "\n",
    "- QAT 是后训练量化技术的超集，允许更多的调试。例如，我们可以分析模型的准确性是否受到权重或激活量化的限制。\n",
    "- 也可以在浮点上模拟量化模型的准确性，因为使用伪量化来模拟实际量化算法的数值。\n",
    "- 也可以很容易地模拟训练后量化。\n",
    "\n",
    "保存 QAT 模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "jit.save(jit.script(qat_model), saved_model_dir + scripted_qat_model_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 小结\n",
    "\n",
    "同样可以使用 {func}`~torch.ao.quantization.quantize.quantize` 和 {func}`~torch.ao.quantization.quantize.quantize_qat` 简化流程。\n",
    "\n",
    "比如，QAT 流程可以这样：\n",
    "\n",
    "```python\n",
    "model_path = saved_model_dir + float_model_file\n",
    "qat_model = create_qat_model(num_classes, model_path)\n",
    "num_epochs = 30\n",
    "ylim = [0.8, 1]\n",
    "device = 'cuda:2'\n",
    "is_freeze = False\n",
    "is_quantized_acc = False\n",
    "need_qconfig = True # 做一些 QAT 的量化配置工作\n",
    "param_group = True\n",
    "\n",
    "# 提供位置参数\n",
    "args = [train_iter,\n",
    "        test_iter,\n",
    "        learning_rate,\n",
    "        num_epochs,\n",
    "        device,\n",
    "        is_freeze,\n",
    "        is_quantized_acc,\n",
    "        need_qconfig,\n",
    "        param_group,\n",
    "        ylim]\n",
    "\n",
    "quantized_model = quantize_qat(qat_model, CV.train_fine_tuning, args)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "简而言之，不管是 PTQ 还是 QAT，我们只需要自定义融合模块函数和量化校准函数（比如 QAT 的训练中校准，PTQ 的训练后校准）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ccd751c8c176f1a7084878738c6c59984a17d1189ffe2fae146e3d74e2010826"
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 (conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}