{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "Ic4_occAAiAT"
},
"outputs": [],
"source": [
"import os\n",
"os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # 设置日志级别为ERROR，以减少警告信息\n",
"# 禁用 Gemini 的底层库（gRPC 和 Abseil）在初始化日志警告\n",
"os.environ[\"GRPC_VERBOSITY\"] = \"ERROR\"\n",
"os.environ[\"GLOG_minloglevel\"] = \"3\" # 0: INFO, 1: WARNING, 2: ERROR, 3: FATAL\n",
"os.environ[\"GLOG_minloglevel\"] = \"true\"\n",
"import logging\n",
"import tensorflow as tf\n",
"tf.get_logger().setLevel(logging.ERROR)\n",
"tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\n",
"!export TF_FORCE_GPU_ALLOW_GROWTH=true"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"cellView": "form",
"id": "ioaprt5q5US7"
},
"outputs": [],
"source": [
"# Copyright 2019 The TensorFlow Authors.\n",
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"cellView": "form",
"id": "yCl0eTNH5RS3"
},
"outputs": [],
"source": [
"#@title MIT License\n",
"#\n",
"# Copyright (c) 2017 François Chollet\n",
"#\n",
"# Permission is hereby granted, free of charge, to any person obtaining a\n",
"# copy of this software and associated documentation files (the \"Software\"),\n",
"# to deal in the Software without restriction, including without limitation\n",
"# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
"# and/or sell copies of the Software, and to permit persons to whom the\n",
"# Software is furnished to do so, subject to the following conditions:\n",
"#\n",
"# The above copyright notice and this permission notice shall be included in\n",
"# all copies or substantial portions of the Software.\n",
"#\n",
"# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
"# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
"# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
"# DEALINGS IN THE SOFTWARE."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ItXfxkxvosLH"
},
"source": [
"# 电影评论文本分类"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hKY4XMc9o8iB"
},
"source": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Eg62Pmz3o83v"
},
"source": [
"本教程演示了从存储在磁盘上的纯文本文件开始的文本分类。您将训练一个二元分类器对 IMDB 数据集执行情感分析。在笔记本的最后，有一个练习供您尝试，您将在其中训练一个多类分类器来预测 Stack Overflow 上编程问题的标签。\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "8RZOuS9LWQvv"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import os\n",
"import re\n",
"import shutil\n",
"import string\n",
"import tensorflow as tf\n",
"\n",
"from tensorflow.keras import layers\n",
"from tensorflow.keras import losses\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "6-tTFS04dChr"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2.17.0\n"
]
}
],
"source": [
"print(tf.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NBTI1bi8qdFV"
},
"source": [
"## 情感分析\n",
"\n",
"此笔记本训练了一个情感分析模型，利用评论文本将电影评论分类为*正面*或*负面*评价。这是一个*二元*（或二类）分类示例，也是一个重要且应用广泛的机器学习问题。\n",
"\n",
"您将使用 [Large Movie Review Dataset](https://ai.stanford.edu/~amaas/data/sentiment/)，其中包含 [Internet Movie Database](https://www.imdb.com/) 中的 50,000 条电影评论文本 。我们将这些评论分为两组，其中 25,000 条用于训练，另外 25,000 条用于测试。训练集和测试集是*均衡的*，也就是说其中包含相等数量的正面评价和负面评价。\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iAsKG535pHep"
},
"source": [
"### 下载并探索 IMDB 数据集\n",
"\n",
"我们下载并提取数据集，然后浏览一下目录结构。"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"temp_dir = Path(\".temp\")\n",
"temp_dir.mkdir(parents=True, exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "k7ZYnuajVlFN"
},
"outputs": [],
"source": [
"url = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n",
"\n",
"dataset = tf.keras.utils.get_file(\"aclImdb_v1\", url,\n",
" untar=True, cache_dir=temp_dir,\n",
" cache_subdir='')\n",
"\n",
"# dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')\n",
"dataset_dir = Path(dataset).parent/'aclImdb'"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['labeledBow.feat',\n",
" 'neg',\n",
" 'pos',\n",
" 'unsupBow.feat',\n",
" 'urls_neg.txt',\n",
" 'urls_pos.txt',\n",
" 'urls_unsup.txt',\n",
" 'unsup']"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_dir = dataset_dir/'train'\n",
"[p.name for p in train_dir.iterdir()]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ysMNMI1CWDFD"
},
"source": [
"`aclImdb/train/pos` 和 `aclImdb/train/neg` 目录包含许多文本文件，每个文件都是一条电影评论。我们来看看其中的一条评论。"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "R7g8hFvzWLIZ"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Rachel Griffiths writes and directs this award winning short film. A heartwarming story about coping with grief and cherishing the memory of those we've loved and lost. Although, only 15 minutes long, Griffiths manages to capture so much emotion and truth onto film in the short space of time. Bud Tingwell gives a touching performance as Will, a widower struggling to cope with his wife's death. Will is confronted by the harsh reality of loneliness and helplessness as he proceeds to take care of Ruth's pet cow, Tulip. The film displays the grief and responsibility one feels for those they have loved and lost. Good cinematography, great direction, and superbly acted. It will bring tears to all those who have lost a loved one, and survived.\n"
]
}
],
"source": [
"sample_file = train_dir/'pos/1181_9.txt'\n",
"with open(sample_file) as f:\n",
" print(f.read())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mk20TEm6ZRFP"
},
"source": [
"### 加载数据集\n",
"\n",
"接下来，您将从磁盘加载数据并将其准备为适合训练的格式。为此，您将使用有用的 [text_dataset_from_directory](https://tensorflow.google.cn/api_docs/python/tf/keras/preprocessing/text_dataset_from_directory) 实用工具，它期望的目录结构如下所示。\n",
"\n",
"```\n",
"main_directory/\n",
"...class_a/\n",
"......a_text_1.txt\n",
"......a_text_2.txt\n",
"...class_b/\n",
"......b_text_1.txt\n",
"......b_text_2.txt\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nQauv38Lnok3"
},
"source": [
"要准备用于二元分类的数据集，磁盘上需要有两个文件夹，分别对应于 `class_a` 和 `class_b`。这些将是正面和负面的电影评论，可以在 `aclImdb/train/pos` 和 `aclImdb/train/neg` 中找到。由于 IMDB 数据集包含其他文件夹，因此您需要在使用此实用工具之前将其移除。"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "VhejsClzaWfl"
},
"outputs": [],
"source": [
"remove_dir = train_dir/'unsup'\n",
"shutil.rmtree(remove_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "95kkUdRoaeMw"
},
"source": [
"接下来，您将使用 `text_dataset_from_directory` 实用工具创建带标签的 `tf.data.Dataset`。[tf.data](https://tensorflow.google.cn/guide/data) 是一组强大的数据处理工具。\n",
"\n",
"运行机器学习实验时，最佳做法是将数据集拆成三份：[训练](https://developers.google.com/machine-learning/glossary#training_set)、[验证](https://developers.google.com/machine-learning/glossary#validation_set) 和 [测试](https://developers.google.com/machine-learning/glossary#test-set)。\n",
"\n",
"IMDB 数据集已经分成训练集和测试集，但缺少验证集。我们来通过下面的 `validation_split` 参数，使用 80:20 拆分训练数据来创建验证集。"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "nOrK-MTYaw3C"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 25000 files belonging to 2 classes.\n",
"Using 20000 files for training.\n"
]
}
],
"source": [
"batch_size = 32\n",
"seed = 42\n",
"\n",
"raw_train_ds = tf.keras.utils.text_dataset_from_directory(\n",
" train_dir, \n",
" batch_size=batch_size, \n",
" validation_split=0.2, \n",
" subset='training', \n",
" seed=seed)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5Y33oxOUpYkh"
},
"source": [
"如上所示，训练文件夹中有 25,000 个样本，您将使用其中的 80%（或 20,000 个）进行训练。稍后您将看到，您可以通过将数据集直接传递给 `model.fit` 来训练模型。如果您不熟悉 `tf.data`，还可以遍历数据集并打印出一些样本，如下所示。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "51wNaPPApk1K"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Review b'\"Pandemonium\" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. \"Airplane\", \"The Naked Gun\" trilogy, \"Blazing Saddles\", \"High Anxiety\", and \"Spaceballs\" are some of my favorite comedies that spoof a particular genre. \"Pandemonium\" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\\'s all this film has going for it. Geez, \"Scream\" had more laughs than this film and that was more of a horror film. How bizarre is that?
*1/2 (out of four)'\n",
"Label 0\n",
"Review b\"David Mamet is a very interesting and a very un-equal director. His first movie 'House of Games' was the one I liked best, and it set a series of films with characters whose perspective of life changes as they get into complicated situations, and so does the perspective of the viewer.
So is 'Homicide' which from the title tries to set the mind of the viewer to the usual crime drama. The principal characters are two cops, one Jewish and one Irish who deal with a racially charged area. The murder of an old Jewish shop owner who proves to be an ancient veteran of the Israeli Independence war triggers the Jewish identity in the mind and heart of the Jewish detective.
This is were the flaws of the film are the more obvious. The process of awakening is theatrical and hard to believe, the group of Jewish militants is operatic, and the way the detective eventually walks to the final violent confrontation is pathetic. The end of the film itself is Mamet-like smart, but disappoints from a human emotional perspective.
Joe Mantegna and William Macy give strong performances, but the flaws of the story are too evident to be easily compensated.\"\n",
"Label 0\n",
"Review b'Great documentary about the lives of NY firefighters during the worst terrorist attack of all time.. That reason alone is why this should be a must see collectors item.. What shocked me was not only the attacks, but the\"High Fat Diet\" and physical appearance of some of these firefighters. I think a lot of Doctors would agree with me that,in the physical shape they were in, some of these firefighters would NOT of made it to the 79th floor carrying over 60 lbs of gear. Having said that i now have a greater respect for firefighters and i realize becoming a firefighter is a life altering job. The French have a history of making great documentary\\'s and that is what this is, a Great Documentary.....'\n",
"Label 1\n"
]
}
],
"source": [
"for text_batch, label_batch in raw_train_ds.take(1):\n",
" for i in range(3):\n",
" print(\"Review\", text_batch.numpy()[i])\n",
" print(\"Label\", label_batch.numpy()[i])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JWq1SUIrp1a-"
},
"source": [
"请注意，评论包含原始文本（带有标点符号和偶尔出现的 HTML 代码，如 ` `）。我们将在以下部分展示如何处理这些问题。\n",
"\n",
"标签为 0 或 1。要查看它们与正面和负面电影评论的对应关系，可以查看数据集上的 `class_names` 属性。\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "MlICTG8spyO2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Label 0 corresponds to neg\n",
"Label 1 corresponds to pos\n"
]
}
],
"source": [
"print(\"Label 0 corresponds to\", raw_train_ds.class_names[0])\n",
"print(\"Label 1 corresponds to\", raw_train_ds.class_names[1])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pbdO39vYqdJr"
},
"source": [
"接下来，您将创建验证数据集和测试数据集。您将使用训练集中剩余的 5,000 条评论进行验证。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SzxazN8Hq1pF"
},
"source": [
"注：使用 `validation_split` 和 `subset` 参数时，请确保要么指定随机种子，要么传递 `shuffle=False`，这样验证拆分和训练拆分就不会重叠。"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "JsMwwhOoqjKF"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 25000 files belonging to 2 classes.\n",
"Using 5000 files for validation.\n"
]
}
],
"source": [
"raw_val_ds = tf.keras.utils.text_dataset_from_directory(\n",
" train_dir, \n",
" batch_size=batch_size, \n",
" validation_split=0.2, \n",
" subset='validation', \n",
" seed=seed)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "rdSr0Nt3q_ns"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 25000 files belonging to 2 classes.\n"
]
}
],
"source": [
"raw_test_ds = tf.keras.utils.text_dataset_from_directory(\n",
" dataset_dir/'test', \n",
" batch_size=batch_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qJmTiO0IYAjm"
},
"source": [
"### 准备用于训练的数据集\n",
"\n",
"接下来，您将使用有用的 `tf.keras.layers.TextVectorization` 层对数据进行标准化、词例化和向量化。\n",
"\n",
"标准化是指对文本进行预处理，通常是移除标点符号或 HTML 元素以简化数据集。词例化是指将字符串分割成词例（例如，通过空格将句子分割成单个单词）。向量化是指将词例转换为数字，以便将它们输入神经网络。所有这些任务都可以通过这个层完成。\n",
"\n",
"正如您在上面看到的，评论包含各种 HTML 代码，例如 ` `。`TextVectorization` 层（默认情况下会将文本转换为小写并去除标点符号，但不会去除 HTML）中的默认标准化程序不会移除这些代码。您将编写一个自定义标准化函数来移除 HTML。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZVcHl-SLrH-u"
},
"source": [
"注：为了防止[训练-测试偏差](https://developers.google.com/machine-learning/guides/rules-of-ml#training-serving_skew)（也称为训练-应用偏差），在训练和测试时间对数据进行相同的预处理非常重要。为此，可以将 `TextVectorization` 层直接包含在模型中，如本教程后面所示。"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "SDRI_s_tX1Hk"
},
"outputs": [],
"source": [
"def custom_standardization(input_data):\n",
" lowercase = tf.strings.lower(input_data)\n",
" stripped_html = tf.strings.regex_replace(lowercase, ' ', ' ')\n",
" return tf.strings.regex_replace(stripped_html,\n",
" '[%s]' % re.escape(string.punctuation),\n",
" '')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d2d3Aw8dsUux"
},
"source": [
" 接下来，您将创建一个 `TextVectorization` 层。您将使用该层对我们的数据进行标准化、词例化和向量化。您将 `output_mode` 设置为 `int` 以便为每个词例创建唯一的整数索引。\n",
"\n",
"请注意，您使用的是默认拆分函数，以及您在上面定义的自定义标准化函数。您还将为模型定义一些常量，例如显式的最大 `sequence_length`，这会使层将序列填充或截断为精确的 `sequence_length` 值。"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "-c76RvSzsMnX"
},
"outputs": [],
"source": [
"max_features = 10000\n",
"sequence_length = 250\n",
"\n",
"vectorize_layer = layers.TextVectorization(\n",
" standardize=custom_standardization,\n",
" max_tokens=max_features,\n",
" output_mode='int',\n",
" output_sequence_length=sequence_length)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vlFOpfF6scT6"
},
"source": [
"接下来，您将调用 `adapt` 以使预处理层的状态适合数据集。这会使模型构建字符串到整数的索引。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lAhdjK7AtroA"
},
"source": [
"注：在调用时请务必仅使用您的训练数据（使用测试集会泄漏信息）。"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"id": "GH4_2ZGJsa_X"
},
"outputs": [],
"source": [
"# Make a text-only dataset (without labels), then call adapt\n",
"train_text = raw_train_ds.map(lambda x, y: x)\n",
"vectorize_layer.adapt(train_text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SHQVEFzNt-K_"
},
"source": [
"我们来创建一个函数来查看使用该层预处理一些数据的结果。"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "SCIg_T50wOCU"
},
"outputs": [],
"source": [
"def vectorize_text(text, label):\n",
" text = tf.expand_dims(text, -1)\n",
" return vectorize_layer(text), label"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"id": "XULcm6B3xQIO"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Review tf.Tensor(b'Silent Night, Deadly Night 5 is the very last of the series, and like part 4, it\\'s unrelated to the first three except by title and the fact that it\\'s a Christmas-themed horror flick.
Except to the oblivious, there\\'s some obvious things going on here...Mickey Rooney plays a toymaker named Joe Petto and his creepy son\\'s name is Pino. Ring a bell, anyone? Now, a little boy named Derek heard a knock at the door one evening, and opened it to find a present on the doorstep for him. Even though it said \"don\\'t open till Christmas\", he begins to open it anyway but is stopped by his dad, who scolds him and sends him to bed, and opens the gift himself. Inside is a little red ball that sprouts Santa arms and a head, and proceeds to kill dad. Oops, maybe he should have left well-enough alone. Of course Derek is then traumatized by the incident since he watched it from the stairs, but he doesn\\'t grow up to be some killer Santa, he just stops talking.
There\\'s a mysterious stranger lurking around, who seems very interested in the toys that Joe Petto makes. We even see him buying a bunch when Derek\\'s mom takes him to the store to find a gift for him to bring him out of his trauma. And what exactly is this guy doing? Well, we\\'re not sure but he does seem to be taking these toys apart to see what makes them tick. He does keep his landlord from evicting him by promising him to pay him in cash the next day and presents him with a \"Larry the Larvae\" toy for his kid, but of course \"Larry\" is not a good toy and gets out of the box in the car and of course, well, things aren\\'t pretty.
Anyway, eventually what\\'s going on with Joe Petto and Pino is of course revealed, and as with the old story, Pino is not a \"real boy\". Pino is probably even more agitated and naughty because he suffers from \"Kenitalia\" (a smooth plastic crotch) so that could account for his evil ways. And the identity of the lurking stranger is revealed too, and there\\'s even kind of a happy ending of sorts. Whee.
A step up from part 4, but not much of one. Again, Brian Yuzna is involved, and Screaming Mad George, so some decent special effects, but not enough to make this great. A few leftovers from part 4 are hanging around too, like Clint Howard and Neith Hunter, but that doesn\\'t really make any difference. Anyway, I now have seeing the whole series out of my system. Now if I could get some of it out of my brain. 4 out of 5.', shape=(), dtype=string)\n",
"Label neg\n",
"Vectorized review (, )\n"
]
}
],
"source": [
"# retrieve a batch (of 32 reviews and labels) from the dataset\n",
"text_batch, label_batch = next(iter(raw_train_ds))\n",
"first_review, first_label = text_batch[0], label_batch[0]\n",
"print(\"Review\", first_review)\n",
"print(\"Label\", raw_train_ds.class_names[first_label])\n",
"print(\"Vectorized review\", vectorize_text(first_review, first_label))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6u5EX0hxyNZT"
},
"source": [
"正如您在上面看到的，每个词例都被一个整数替换了。您可以通过在该层上调用 `.get_vocabulary()` 来查找每个整数对应的词例（字符串）。"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "kRq9hTQzhVhW"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1287 ---> silent\n",
" 313 ---> night\n",
"Vocabulary size: 10000\n"
]
}
],
"source": [
"print(\"1287 ---> \",vectorize_layer.get_vocabulary()[1287])\n",
"print(\" 313 ---> \",vectorize_layer.get_vocabulary()[313])\n",
"print('Vocabulary size: {}'.format(len(vectorize_layer.get_vocabulary())))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XD2H6utRydGv"
},
"source": [
"你几乎已经准备好训练你的模型了。作为最后的预处理步骤，你将在训练、验证和测试数据集上应用之前创建的TextVectorization层。"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"id": "2zhmpeViI1iG"
},
"outputs": [],
"source": [
"train_ds = raw_train_ds.map(vectorize_text)\n",
"val_ds = raw_val_ds.map(vectorize_text)\n",
"test_ds = raw_test_ds.map(vectorize_text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YsVQyPMizjuO"
},
"source": [
"### 配置数据集以提高性能\n",
"\n",
"以下是加载数据时应该使用的两种重要方法，以确保 I/O 不会阻塞。\n",
"\n",
"从磁盘加载后，`.cache()` 会将数据保存在内存中。这将确保数据集在训练模型时不会成为瓶颈。如果您的数据集太大而无法放入内存，也可以使用此方法创建高性能的磁盘缓存，这比许多小文件的读取效率更高。\n",
"\n",
"`prefetch()` 会在训练时将数据预处理和模型执行重叠。\n",
"\n",
"您可以在[数据性能指南](https://tensorflow.google.cn/guide/data_performance)中深入了解这两种方法，以及如何将数据缓存到磁盘。"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"id": "wMcs_H7izm5m"
},
"outputs": [],
"source": [
"AUTOTUNE = tf.data.AUTOTUNE\n",
"\n",
"train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)\n",
"val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)\n",
"test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LLC02j2g-llC"
},
"source": [
"### 创建模型\n",
"\n",
"是时候创建您的神经网络了："
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "dkQP6in8yUBR"
},
"outputs": [],
"source": [
"embedding_dim = 16"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "xpKOoWgu-llD"
},
"outputs": [
{
"data": {
"text/html": [
"
![](\"https://tensorflow.google.cn/images/tf_logo_32px.png\"){kind=logo}
在 TensorFlow.org 上查看 ![](\"https://tensorflow.google.cn/images/colab_logo_32px.png\"){kind=logo}
在 Google Colab 中运行 ![](\"https://tensorflow.google.cn/images/GitHub-Mark-32px.png\")
在 GitHub 上查看源代码 ![](\"https://tensorflow.google.cn/images/download_logo_32px.png\"){kind=logo}
下载笔记本