Using Transformers

Setup

设置要注意几件事情：

1. 使用jupyter notebook时想导入transformers发现会报错，至今还不清楚具体解决方法，感觉是版本兼容的问题。
2. 按照官方的教程，下面的做法是在Google Collab上做的，发现确实可行。

!pip install transformers[sentencepiece]

介绍

• Ease of use: Downloading, loading, and using a state-of-the-art NLP model for inference can be done in just two lines of code.
• Flexibility: At their core, all models are simple PyTorch nn.Module or TensorFlow tf.keras.Model classes and can be handled like any other models in their respective machine learning (ML) frameworks.
• Simplicity: Hardly any abstractions are made across the library. The “All in one file” is a core concept: a model’s forward pass is entirely defined in a single file, so that the code itself is understandable and hackable.The models are not built on modules that are shared across files; instead, each model has its own layers. In addition to making the models more approachable and understandable, this allows you to easily experiment on one model without affecting others.

Behind the pipeline

Processing with a tokenizer

tokenizer的作用：

• Splitting the input into words, subwords, or symbols (like punctuation) that are called tokens
• Mapping each token to an integer
• Adding additional inputs that may be useful to the model

tokenizer的处理细节必须和预训练模型匹配，所以我们需要先指定具体的预训练模型配套使用。

下面的代码在第一次运行时会下载预训练模型，之后的运行就会直接加载缓存：

from transformers import AutoTokenizer

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

HBox(children=(FloatProgress(value=0.0, description='Downloading', max=48.0, style=ProgressStyle(description_w…






HBox(children=(FloatProgress(value=0.0, description='Downloading', max=629.0, style=ProgressStyle(description_…






HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…

raw_inputs = [
    "I've been waiting for a HuggingFace course my whole life.", 
    "I hate this so much!",
]
# return_tensors指定返回的tensor类型，pt是指pytorch，如果没有指定类型，会得到a list of lists
inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="pt")
print(inputs)

{'input_ids': tensor([[  101,  1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,
          2607,  2026,  2878,  2166,  1012,   102],
        [  101,  1045,  5223,  2023,  2061,  2172,   999,   102,     0,     0,
             0,     0,     0,     0,     0,     0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])}

返回的是一个字典，包含两个key：

• input_ids
• attention_mask

Going through the model

与tokenizer类似，我们可以定义相应的预训练模型：

from transformers import AutoModel

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModel.from_pretrained(checkpoint)

HBox(children=(FloatProgress(value=0.0, description='Downloading', max=267844284.0, style=ProgressStyle(descri…





Some weights of the model checkpoint at distilbert-base-uncased-finetuned-sst-2-english were not used when initializing DistilBertModel: ['classifier.weight', 'pre_classifier.weight', 'classifier.bias', 'pre_classifier.bias']
- This IS expected if you are initializing DistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

outputs = model(**inputs)
print(outputs.last_hidden_state.shape)

torch.Size([2, 16, 768])

输入预训练模型，得到的是hidden states，这些hidden states还需要作为输入到head才能得到最后的结果。

得到的输出有三维：

1. Batch size: The number of sequences processed at a time (2 in our example).
2. Sequence length: The length of the numerical representation of the sequence (16 in our example).
3. Hidden size: The vector dimension of each model input. (768 in our example).

可以通过outputs.last_hidden_state或者outputs["last_hidden_state"]或者outputs[0]得到结果。

Model heads: Making sense out of numbers:

head通常由一层或几层线性层组合而成。

下面以sequence classification(输出只有两类：positive, negative)举例：

from transformers import AutoModelForSequenceClassification

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
outputs = model(**inputs)

print(outputs.logits.shape)

torch.Size([2, 2])

因为输出只有两类，输入有两句话，所以得到的参数只有四个（二维矩阵）。

Postprocessing the output

print(outputs.logits)

tensor([[-1.5607,  1.6123],
        [ 4.1692, -3.3464]], grad_fn=<AddmmBackward>)

为了将输出转换为概率，所以需要一层softmax：

import torch

predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
print(predictions)

tensor([[4.0195e-02, 9.5980e-01],
        [9.9946e-01, 5.4418e-04]], grad_fn=<SoftmaxBackward>)

可以看出模型的预测结果为：对第一句话：[0.0402, 0.9598],对第二句话：[0.9995, 0.0005]。

为了将真实的类别和序号对应，可以调用：

model.config.id2label

{0: 'NEGATIVE', 1: 'POSITIVE'}

最终得到：

• First sentence: NEGATIVE: 0.0402, POSITIVE: 0.9598
• Second sentence: NEGATIVE: 0.9995, POSITIVE: 0.0005