{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 量化计算机视觉分类\n", "\n", "{guilabel}`来源`：[quantized_transfer_learning](https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html)\n", " \n", "```{admonition} 任务\n", "使用量化模型提取特征。\n", "```\n", "\n", "## 准备\n", "\n", "在进入迁移学习之前，让我们回顾一下“先决条件”，例如安装和数据加载/可视化。\n", "\n", "详细内容见：{doc}`图像分类的迁移学习 `。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from matplotlib import pyplot as plt\n", "import torch\n", "from torchvision.models.quantization import resnet18\n", "\n", "from mod import load_mod\n", "\n", "plt.ion()\n", "# 载入自定义模块\n", "load_mod()\n", "\n", "from pytorch_book.datasets.examples import Hymenoptera\n", "from xinet import ModuleTool, CV" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "loader = Hymenoptera()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1 基于量化特征提取器训练自定义分类器\n", "\n", "在本节中，将使用“冻结”的量化特征提取器，并在此基础上训练自定义分类器头部。与浮点模型不同，您不需要为量化模型设置 `requires_grad=False`，因为它没有可训练的参数。请参阅 [文档](https://pytorch.org/docs/stable/quantization.html) 了解更多细节。\n", "\n", "加载预训练模型：在这个练习中，使用 [ResNet-18](https://pytorch.org/hub/pytorch_vision_resnet/)。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/pc/xinet/anaconda3/envs/ai/lib/python3.9/site-packages/torch/ao/quantization/observer.py:172: UserWarning: Please use quant_min and quant_max to specify the range for observers. reduce_range will be deprecated in a future release of PyTorch.\n", " warnings.warn(\n" ] } ], "source": [ "# 我们需要 `fc` 中过滤器的数量，以便将来使用。\n", "# 这里每个输出样本的大小设置为 2。\n", "# 或者，它可以推广到 nn.Linear(num_ftrs, len(class_names))。\n", "model_fe = resnet18(pretrained=True, progress=True, quantize=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "此时，您需要修改预训练的模型。该模型在开始和结束都有量化/反量化块。但是，因为你只会使用特征提取器，所以反量化层必须在线性层（头部）之前移动。最简单的方法是将模型封装在 ``nn.Sequential`` 模块中。第一步是分离 ResNet 模型中的特征提取器。虽然在这个例子中，你的任务是使用除 ``fc`` 以外的所有层作为特征提取器，但实际上，你可以根据需要取尽可能多的部分。如果您也想替换一些卷积层，这将是有用的。\n", "\n", "```{note}\n", "当将特征提取器与量化模型的其余部分分离时，您必须手动将 quantizer/dequantized 放置在希望保持量化的部分的开头和结尾。\n", "```\n", "\n", "下面的函数创建一个带有自定义头部的模型。" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from torch import nn\n", "\n", "\n", "def create_combined_model(model_fe):\n", " num_ftrs = model_fe.fc.in_features\n", " # 步骤1：分离特征提取器\n", " model_fe_features = nn.Sequential(\n", " model_fe.quant, # 量化 input\n", " model_fe.conv1,\n", " model_fe.bn1,\n", " model_fe.relu,\n", " model_fe.maxpool,\n", " model_fe.layer1,\n", " model_fe.layer2,\n", " model_fe.layer3,\n", " model_fe.layer4,\n", " model_fe.avgpool,\n", " model_fe.dequant, # 反量化 output\n", " )\n", "\n", " # 步骤2：创建一个新的“头”\n", " new_head = nn.Sequential(\n", " nn.Dropout(p=0.5),\n", " nn.Linear(num_ftrs, 2),\n", " )\n", "\n", " # 步骤3：合并，不要忘记量化 stubs\n", " new_model = nn.Sequential(\n", " model_fe_features,\n", " nn.Flatten(1),\n", " new_head,\n", " )\n", " return new_model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{warning}\n", "目前，量化模型只能在 CPU 上运行。然而，可以将模型的反量化部分发送给 GPU。\n", "```" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import torch.optim as optim\n", "new_model = create_combined_model(model_fe)\n", "\n", "device = 'cpu'\n", "train_iter = loader.dataloaders['train']\n", "test_iter = loader.dataloaders['val']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{rubric} 训练和评估\n", "```\n", "\n", "该步骤在 CPU 上耗时约 15-25 分钟。由于量化模型只能在 CPU 上运行，所以不能在 GPU 上运行训练。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "loss 0.132, train acc 0.951, test acc 0.967\n", "164.5 examples/sec on cpu\n" ] }, { "data": { "image/svg+xml": "\n\n\n \n \n \n \n 2022-03-21T16:21:18.890574\n image/svg+xml\n \n \n Matplotlib v3.4.0, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "CV.train_fine_tuning(new_model, train_iter, test_iter,\n", " learning_rate=1e-3,\n", " num_epochs=25,\n", " param_group=False,\n", " device=device)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": "\n\n\n \n \n \n \n 2022-03-21T16:21:54.103149\n image/svg+xml\n \n \n Matplotlib v3.4.0, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "for xs, _ in test_iter:\n", " break\n", "\n", "MT = ModuleTool(xs)\n", "MT.imshow(new_model, loader.class_names, device,\n", " num_rows=1, num_cols=4, scale=2)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2 微调量化模型\n", "\n", "在这一部分中，对用于迁移学习的特征提取器进行了微调，并对特征提取器进行量化。注意，在第 1 部分和第 2 部分中，特征提取器都是量化的。不同之处在于，在第 1 部分中，使用了预训练的量化模型。在这一部分中，我们在对感兴趣的数据集进行微调后，创建了量化的特征提取器，因此这是一种利用迁移学习获得更好准确性的方法，同时具有量化的好处。请注意，在我们的具体示例中，训练集非常小（120 张图片），因此对整个模型进行微调的好处并不明显。然而，这里显示的过程将提高迁移学习的准确性于更大的数据集。\n", "\n", "预训练的特征提取器必须是可量化的。要确保它是可量化的，请执行以下步骤：\n", "\n", "1. 使用 {func}`torch.quantization.fuse_modules` 融合 ``(Conv, BN, ReLU)``, ``(Conv, BN)`` 和 ``(Conv, ReLU)``。\n", "2. 连接特征提取器与自定义头。这需要反量化特征提取器的输出。\n", "3. 在特征提取器中适当位置插入伪量化模块，训练时模拟量化。\n", "\n", "对于步骤（1），我们使用来自 {mod}`torchvision.models.quantization` 的模型 `resnet18`，它们有成员方法 ``fuse_model``。这个函数融合了 ``conv``，``bn`` 和 ``relu`` 模块。对于自定义模型，这将需要调用 {func}`torch.quantization.fuse_modules` API 中包含了需要手动融合的模块列表。\n", "\n", "步骤（2）由前一节中使用的 ``create_combined_model`` 函数执行。\n", "\n", "步骤（3）通过使用 {func}`torch.quantization.prepare_qat`，插入 fake-quantization 模块。\n", "\n", "作为步骤（4），您可以开始“微调”模型，然后将其转换为完全量化的版本（步骤5）。\n", "\n", "要将微调模型转换为量化模型，可以调用 {func}`torch.quantization.convert` 函数（在我们的例子中，只有特征提取器被量化）。\n", "\n", "```{note}\n", "由于随机初始化，您的结果可能与本教程中显示的结果不同。\n", "```" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# 注意 `quantize=False`\n", "model = resnet18(pretrained=True, progress=True, quantize=False)\n", "\n", "# Step 1\n", "model.train()\n", "model.fuse_model()\n", "# Step 2\n", "model_ft = create_combined_model(model)\n", "model_ft[0].qconfig = torch.quantization.default_qat_qconfig # 使用默认 QAT 配置\n", "# Step 3\n", "model_ft = torch.quantization.prepare_qat(model_ft, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{rubric} Step 4. 微调模型\n", "```\n", "\n", "在本教程中，对整个模型进行了微调。一般来说，这将导致更高的准确性。但是，由于这里使用的训练集较小，我们最终会对训练集过拟合。" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "for param in model_ft.parameters():\n", " param.requires_grad = True\n", "\n", "# 如果可用的话，我们可以对 GPU 进行微调\n", "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "loss 0.417, train acc 0.832, test acc 0.928\n", "60.5 examples/sec on cuda:0\n" ] }, { "data": { "image/svg+xml": "\n\n\n \n \n \n \n 2022-03-21T16:25:09.974072\n image/svg+xml\n \n \n Matplotlib v3.4.0, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "CV.train_fine_tuning(model_ft, train_iter, test_iter,\n", " learning_rate=1e-3,\n", " num_epochs=25,\n", " param_group=False,\n", " device=device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{rubric} Step 5. 转换已量化的模型\n", "```" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "from torch.quantization import convert\n", "model_ft.cpu()\n", "\n", "model_quantized_and_trained = convert(model_ft, inplace=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "让我们看看量化模型在一些图像上的表现：" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": "\n\n\n \n \n \n \n 2022-03-21T16:25:19.538247\n image/svg+xml\n \n \n Matplotlib v3.4.0, https://matplotlib.org/\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "for xs, _ in test_iter:\n", " break\n", "\n", "MT = ModuleTool(xs)\n", "MT.imshow(model_quantized_and_trained,\n", " loader.class_names,\n", " 'cpu',\n", " num_rows=1, num_cols=4, scale=2)\n", "plt.tight_layout()\n", "plt.ioff()\n", "plt.tight_layout()\n", "plt.show()" ] } ], "metadata": { "interpreter": { "hash": "61447c3ddb95e77cf825d46d6f70227c36ed08aec1baab023b2c78b109ed3829" }, "kernelspec": { "display_name": "Python 3.9.7 ('ai')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" }, "orig_nbformat": 4 }, "nbformat": 4, "nbformat_minor": 2 }