Part1.文本预处理
文本预处理主要分为：
- 读入文本
- 分词
- 建立字典
- 将文本的词序列转为索引序列
读入文本
|
1 2 3 4 5 6 7 8 9 10 |
import collections import re def read_time_machine(): with open('/home/kesci/input/timemachine7163/timemachine.txt', 'r') as f: lines = [re.sub('[^a-z]+', ' ', line.strip().lower()) for line in f] return lines lines = read_time_machine() print('# sentences %d' % len(lines)) |
分词
我们对每个句子进行分词，也就是将一个句子划分成若干个词（token），转换为一个词的序列。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
def tokenize(sentences, token='word'): #Split sentences into word or char tokens if token == 'word': return [sentence.split(' ') for sentence in sentences] elif token == 'char': return [list(sentence) for sentence in sentences] else: print('ERROR: unkown token type '+token) tokens = tokenize(lines) tokens[0:2] #[['the', 'time', 'machine', 'by', 'h', 'g', 'wells', ''], ['']] |
我们也可以通过现有的分词工具进行分词，例如spaCy和NLTK。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
text = "Mr. Chen doesn't agree with my suggestion." ####spaCY import spacy nlp = spacy.load('en_core_web_sm') doc = nlp(text) print([token.text for token in doc]) #['Mr.', 'Chen', 'does', "n't", 'agree', 'with', 'my', 'suggestion', '.'] ####NLTK from nltk.tokenize import word_tokenize from nltk import data data.path.append('/home/kesci/input/nltk_data3784/nltk_data') print(word_tokenize(text)) #['Mr.', 'Chen', 'does', "n't", 'agree', 'with', 'my', 'suggestion', '.'] |
Part2.语言模型
本部分主要包括
- 语言模型
- n元语法
- 时序数据采样
- 随机采样
- 相邻采样
语言模型
给定一个长度为T的词的序列\(w_1, w_2, \ldots, w_T\)，语言模型的目标就是评估该序列是否合理，即计算该序列的概率：$$P(w_1, w_2, \ldots, w_T).$$假设序列\(w_1, w_2, \ldots, w_T\)中的每个词是依次生成的，那么通过条件概率我们可以把概率公式改写为：$$P(w_1, w_2, \ldots, w_T)
= \prod_{t=1}^T P(w_t \mid w_1, \ldots, w_{t-1})\\
= P(w_1)P(w_2 \mid w_1) \cdots P(w_T \mid w_1w_2\cdots w_{T-1})
$$其中单个词的概率可以通过语料库的词频进行计算，联合概率可以利用条件概率进行计算。
n元语法
n元语法通过马尔可夫假设简化模型，马尔科夫假设是指一个词的出现只与前面n个词相关，即n阶马尔可夫链。基于n-1阶马尔可夫链，我们可以将语言模型改写为：$$P(w_1, w_2, \ldots, w_T) = \prod_{t=1}^T P(w_t \mid w_{t-(n-1)}, \ldots, w_{t-1}) .$$以上也叫n元语法（n-grams），它是基于n-1阶马尔可夫链的概率语言模型。
时序数据采样
时序数据的一个样本通常包含连续的字符。假设时间步数为5，样本序列为5个字符，即“想”“要”“有”“直”“升”。该样本的标签序列为这些字符分别在训练集中的下一个字符，即“要”“有”“直”“升”“机”，即X=“想要有直升”，Y=“要有直升机”。
现在我们考虑序列“想要有直升机，想要和你飞到宇宙去”，如果时间步数为5，有以下可能的样本和标签：
X：“想要有直升”，Y：“要有直升机”
X：“要有直升机”，Y：“有直升机，”
X：“有直升机，”，Y：“直升机，想”
...
X：“要和你飞到”，Y：“和你飞到宇”
X：“和你飞到宇”，Y：“你飞到宇宙”
X：“你飞到宇宙”，Y：“飞到宇宙去”
可以看到，如果序列的长度为T，时间步数为n，那么一共有T-n个合法的样本，但是这些样本有大量的重合，我们通常采用更加高效的采样方式。我们有两种方式对时序数据进行采样，分别是随机采样和相邻采样。
随机采样
在随机采样中，每个样本是原始序列上任意截取的一段序列，相邻的两个随机小批量在原始序列上的位置不一定相毗邻。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import torch import random def data_iter_random(corpus_indices, batch_size, num_steps, device=None): # 减1是因为对于长度为n的序列，X最多只有包含其中的前n - 1个字符 num_examples = (len(corpus_indices) - 1) // num_steps # 下取整，得到不重叠情况下的样本个数 example_indices = [i * num_steps for i in range(num_examples)] # 每个样本的第一个字符在corpus_indices中的下标 random.shuffle(example_indices) def _data(i): # 返回从i开始的长为num_steps的序列 return corpus_indices[i: i + num_steps] if device is None: device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') for i in range(0, num_examples, batch_size): # 每次选出batch_size个随机样本 batch_indices = example_indices[i: i + batch_size] # 当前batch的各个样本的首字符的下标 X = [_data(j) for j in batch_indices] Y = [_data(j + 1) for j in batch_indices] yield torch.tensor(X, device=device), torch.tensor(Y, device=device) |
相邻采样
在相邻采样中，相邻的两个随机小批量在原始序列上的位置相毗邻。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
def data_iter_consecutive(corpus_indices, batch_size, num_steps, device=None): if device is None: device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') corpus_len = len(corpus_indices) // batch_size * batch_size # 保留下来的序列的长度 corpus_indices = corpus_indices[: corpus_len] # 仅保留前corpus_len个字符 indices = torch.tensor(corpus_indices, device=device) indices = indices.view(batch_size, -1) # resize成(batch_size, ) batch_num = (indices.shape[1] - 1) // num_steps for i in range(batch_num): i = i * num_steps X = indices[:, i: i + num_steps] Y = indices[:, i + 1: i + num_steps + 1] yield X, Y |
Part3.RNN基础
该部分主要包含
- RNN简介
- 手动实现RNN
- 基于pytorch实现RNN
RNN（循环神经网络）
循环神经网络引入一个隐藏变量H，用\(H_t\)表示在时间步t的值。\(H_t\)的计算基于\(X_t\)和\(H_{t-1}\)，可以认为\(H_t\)记录了到当前字符为止的序列信息，利用\(H_t\)对序列的下一个字符进行预测。
具体地，假设\(\boldsymbol{X}_t \in \mathbb{R}^{n \times d}\)是时间步t的输入，\(\boldsymbol{H}_t \in \mathbb{R}^{n \times t}\)是时间步t的隐藏变量，那么：$$\boldsymbol{H}_t = \phi(\boldsymbol{X}_t \boldsymbol{W}_{xh} + \boldsymbol{H}_{t-1} \boldsymbol{W}_{hh} + \boldsymbol{b}_h).$$在时间步t的输出为：$$\boldsymbol{O}_t = \boldsymbol{H}_t \boldsymbol{W}_{hq} + \boldsymbol{b}_q.$$
手动实现RNN
将以构件一个RNN语言模型为例子展示
准备数据
|
1 2 3 4 5 6 7 8 9 |
import torch import torch.nn as nn import time import math import sys sys.path.append('/home/kesci/input') import d2l_jay9460 as d2l (corpus_indices, char_to_idx, idx_to_char, vocab_size) = d2l.load_data_jay_lyrics() device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') |
one-hot向量
我们需要将字符表示成向量，这里采用one-hot向量。假设词典大小是N，每次字符对应一个从0到N-1的唯一的索引，则该字符的向量是一个长度为N的向量，若字符的索引是i，则该向量的第i个位置为1，其他位置为0。
|
1 2 3 4 5 6 |
def one_hot(x, n_class, dtype=torch.float32): result = torch.zeros(x.shape[0], n_class, dtype=dtype, device=x.device) # shape: (n, n_class) result.scatter_(1, x.long().view(-1, 1), 1) # result[i, x[i, 0]] = 1 return result def to_onehot(X, n_class): return [one_hot(X[:, i], n_class) for i in range(X.shape[1])] |
初始化模型参数
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
num_inputs, num_hiddens, num_outputs = vocab_size, 256, vocab_size # num_inputs: d # num_hiddens: h, 隐藏单元的个数是超参数 # num_outputs: q def get_params(): def _one(shape): param = torch.zeros(shape, device=device, dtype=torch.float32) nn.init.normal_(param, 0, 0.01) return torch.nn.Parameter(param) # 隐藏层参数 W_xh = _one((num_inputs, num_hiddens)) W_hh = _one((num_hiddens, num_hiddens)) b_h = torch.nn.Parameter(torch.zeros(num_hiddens, device=device)) # 输出层参数 W_hq = _one((num_hiddens, num_outputs)) b_q = torch.nn.Parameter(torch.zeros(num_outputs, device=device)) return (W_xh, W_hh, b_h, W_hq, b_q) |
定义模型
|
1 2 3 4 5 6 7 8 9 10 11 12 |
def rnn(inputs, state, params): # inputs和outputs皆为num_steps个形状为(batch_size, vocab_size)的矩阵 W_xh, W_hh, b_h, W_hq, b_q = params H, = state outputs = [] for X in inputs: H = torch.tanh(torch.matmul(X, W_xh) + torch.matmul(H, W_hh) + b_h) Y = torch.matmul(H, W_hq) + b_q outputs.append(Y) return outputs, (H,) def init_rnn_state(batch_size, num_hiddens, device): return (torch.zeros((batch_size, num_hiddens), device=device), ) |
裁剪梯度
循环神经网络中较容易出现梯度衰减或梯度爆炸，这会导致网络几乎无法训练。裁剪梯度（clip gradient）是一种应对梯度爆炸的方法。假设我们把所有模型参数的梯度拼接成一个向量\(\boldsymbol{g}\) ，并设裁剪的阈值是\(\theta\)。裁剪后的梯度:$$\min\left(\frac{\theta}{|\boldsymbol{g}|}, 1\right)\boldsymbol{g}$
|
1 2 3 4 5 6 7 8 |
def grad_clipping(params, theta, device): norm = torch.tensor([0.0], device=device) for param in params: norm += (param.grad.data ** 2).sum() norm = norm.sqrt().item() if norm > theta: for param in params: param.grad.data *= (theta / norm) |
预测函数
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
def predict_rnn(prefix, num_chars, rnn, params, init_rnn_state, num_hiddens, vocab_size, device, idx_to_char, char_to_idx): state = init_rnn_state(1, num_hiddens, device) output = [char_to_idx[prefix[0]]] # output记录prefix加上预测的num_chars个字符 for t in range(num_chars + len(prefix) - 1): # 将上一时间步的输出作为当前时间步的输入 X = to_onehot(torch.tensor([[output[-1]]], device=device), vocab_size) # 计算输出和更新隐藏状态 (Y, state) = rnn(X, state, params) # 下一个时间步的输入是prefix里的字符或者当前的最佳预测字符 if t < len(prefix) - 1: output.append(char_to_idx[prefix[t + 1]]) else: output.append(Y[0].argmax(dim=1).item()) return "".join([idx_to_char[i] for i in output]) |
训练函数
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
def train_and_predict_rnn(rnn, get_params, init_rnn_state, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, is_random_iter, num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes): if is_random_iter: data_iter_fn = d2l.data_iter_random else: data_iter_fn = d2l.data_iter_consecutive params = get_params() loss = nn.CrossEntropyLoss() for epoch in range(num_epochs): if not is_random_iter: # 如使用相邻采样，在epoch开始时初始化隐藏状态 state = init_rnn_state(batch_size, num_hiddens, device) l_sum, n, start = 0.0, 0, time.time() data_iter = data_iter_fn(corpus_indices, batch_size, num_steps, device) for X, Y in data_iter: if is_random_iter: # 如使用随机采样，在每个小批量更新前初始化隐藏状态 state = init_rnn_state(batch_size, num_hiddens, device) else: # 否则需要使用detach函数从计算图分离隐藏状态 for s in state: s.detach_() # inputs是num_steps个形状为(batch_size, vocab_size)的矩阵 inputs = to_onehot(X, vocab_size) # outputs有num_steps个形状为(batch_size, vocab_size)的矩阵 (outputs, state) = rnn(inputs, state, params) # 拼接之后形状为(num_steps * batch_size, vocab_size) outputs = torch.cat(outputs, dim=0) # Y的形状是(batch_size, num_steps)，转置后再变成形状为 # (num_steps * batch_size,)的向量，这样跟输出的行一一对应 y = torch.flatten(Y.T) # 使用交叉熵损失计算平均分类误差 l = loss(outputs, y.long()) # 梯度清0 if params[0].grad is not None: for param in params: param.grad.data.zero_() l.backward() grad_clipping(params, clipping_theta, device) # 裁剪梯度 d2l.sgd(params, lr, 1) # 因为误差已经取过均值，梯度不用再做平均 l_sum += l.item() * y.shape[0] n += y.shape[0] if (epoch + 1) % pred_period == 0: print('epoch %d, perplexity %f, time %.2f sec' % ( epoch + 1, math.exp(l_sum / n), time.time() - start)) for prefix in prefixes: print(' -', predict_rnn(prefix, pred_len, rnn, params, init_rnn_state, num_hiddens, vocab_size, device, idx_to_char, char_to_idx)) |
基于pytorch的实现
定义模型
我们使用Pytorch中的nn.RNN来构造循环神经网络。在本节中，我们主要关注nn.RNN的以下几个构造函数参数：
- input_size - The number of expected features in the input x
- hidden_size – The number of features in the hidden state h
- nonlinearity – The non-linearity to use. Can be either 'tanh' or 'relu'. Default: 'tanh'
- batch_first – If True, then the input and output tensors are provided as (batch_size, num_steps, input_size). Default: False
这里的batch_first决定了输入的形状，我们使用默认的参数False，对应的输入形状是 (num_steps, batch_size, input_size)。
forward函数的参数为：
- input of shape (num_steps, batch_size, input_size): tensor containing the features of the input sequence.
- h_0 of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the RNN is bidirectional, num_directions should be 2, else it should be 1.
forward函数的返回值是： - output of shape (num_steps, batch_size, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t.
- h_n of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the hidden state for t = num_steps.
现在我们构造一个nn.RNN实例。
1234567891011121314151617class RNNModel(nn.Module):def __init__(self, rnn_layer, vocab_size):super(RNNModel, self).__init__()self.rnn = rnn_layerself.hidden_size = rnn_layer.hidden_size * (2 if rnn_layer.bidirectional else 1)self.vocab_size = vocab_sizeself.dense = nn.Linear(self.hidden_size, vocab_size)def forward(self, inputs, state):# inputs.shape: (batch_size, num_steps)X = to_onehot(inputs, vocab_size)X = torch.stack(X) # X.shape: (num_steps, batch_size, vocab_size)hiddens, state = self.rnn(X, state)hiddens = hiddens.view(-1, hiddens.shape[-1]) # hiddens.shape: (num_steps * batch_size, hidden_size)output = self.dense(hiddens)return output, state
def predict_rnn_pytorch(prefix, num_chars, model, vocab_size, device, idx_to_char,
char_to_idx):
state = None
output = [char_to_idx[prefix[0]]] # output记录prefix加上预测的num_chars个字符
for t in range(num_chars + len(prefix) - 1):
X = torch.tensor([output[-1]], device=device).view(1, 1)
(Y, state) = model(X, state) # 前向计算不需要传入模型参数
if t < len(prefix) - 1:
output.append(char_to_idx[prefix[t + 1]])
else:
output.append(Y.argmax(dim=1).item())
return ''.join([idx_to_char[i] for i in output])
训练函数
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
def train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device, corpus_indices, idx_to_char, char_to_idx, num_epochs, num_steps, lr, clipping_theta, batch_size, pred_period, pred_len, prefixes): loss = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=lr) model.to(device) for epoch in range(num_epochs): l_sum, n, start = 0.0, 0, time.time() data_iter = d2l.data_iter_consecutive(corpus_indices, batch_size, num_steps, device) # 相邻采样 state = None for X, Y in data_iter: if state is not None: # 使用detach函数从计算图分离隐藏状态 if isinstance (state, tuple): # LSTM, state:(h, c) state[0].detach_() state[1].detach_() else: state.detach_() (output, state) = model(X, state) # output.shape: (num_steps * batch_size, vocab_size) y = torch.flatten(Y.T) l = loss(output, y.long()) optimizer.zero_grad() l.backward() grad_clipping(model.parameters(), clipping_theta, device) optimizer.step() l_sum += l.item() * y.shape[0] n += y.shape[0] if (epoch + 1) % pred_period == 0: print('epoch %d, perplexity %f, time %.2f sec' % ( epoch + 1, math.exp(l_sum / n), time.time() - start)) for prefix in prefixes: print(' -', predict_rnn_pytorch( prefix, pred_len, model, vocab_size, device, idx_to_char, char_to_idx)) |
