03_optimize_inc_quantize.ipynb 7.6 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quantize models with Optimum Intel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh \n",
    "pip -q uninstall torch -y\n",
    "pip -q install torch==1.11.0+cpu --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
EC2 Default User's avatar
EC2 Default User committed
28
    "pip -q install transformers datasets evaluate scipy --upgrade\n",
EC2 Default User's avatar
EC2 Default User committed
29
    "pip -q install optimum[intel] --upgrade"
30
31
32
33
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
34
   "execution_count": null,
35
36
37
38
39
40
41
42
43
44
   "metadata": {},
   "outputs": [],
   "source": [
    "import evaluate\n",
    "from datasets import load_dataset, load_from_disk\n",
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
45
   "execution_count": null,
46
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
47
   "outputs": [],
48
   "source": [
EC2 Default User's avatar
EC2 Default User committed
49
50
    "# If you need to build or resize the dataset. If not, skip this cell\n",
    "\n",
Julien Simon's avatar
Julien Simon committed
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
    "dataset = load_dataset(\"amazon_us_reviews\", \"Shoes_v1_00\", split=\"train[:1%]\")\n",
    "\n",
    "dataset = dataset.remove_columns(\n",
    "    [\n",
    "        \"marketplace\",\n",
    "        \"customer_id\",\n",
    "        \"review_id\",\n",
    "        \"product_id\",\n",
    "        \"product_parent\",\n",
    "        \"product_title\",\n",
    "        \"product_category\",\n",
    "        \"helpful_votes\",\n",
    "        \"total_votes\",\n",
    "        \"vine\",\n",
    "        \"verified_purchase\",\n",
    "        \"review_headline\",\n",
    "        \"review_date\",\n",
    "    ]\n",
    ")\n",
    "dataset = dataset.rename_column(\"star_rating\", \"labels\")\n",
    "dataset = dataset.rename_column(\"review_body\", \"text\")\n",
72
73
74
    "\n",
    "\n",
    "def decrement_stars(row):\n",
Julien Simon's avatar
Julien Simon committed
75
76
    "    return {\"labels\": row[\"labels\"] - 1}\n",
    "\n",
77
78
79
    "\n",
    "dataset = dataset.map(decrement_stars)\n",
    "\n",
EC2 Default User's avatar
EC2 Default User committed
80
    "dataset_split = dataset.train_test_split(test_size=0.01, shuffle=True, seed=59)\n",
Julien Simon's avatar
Julien Simon committed
81
82
    "dataset_split[\"test\"].save_to_disk(\"data_quantize/test\")\n",
    "dataset_split[\"test\"]"
83
84
   ]
  },
EC2 Default User's avatar
EC2 Default User committed
85
86
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
87
   "execution_count": null,
EC2 Default User's avatar
EC2 Default User committed
88
89
90
91
92
93
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_dataset = load_from_disk(\"./data_quantize/test\")"
   ]
  },
94
95
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
96
   "execution_count": null,
97
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
98
   "outputs": [],
99
100
101
102
   "source": [
    "model_name = \"juliensimon/distilbert-amazon-shoe-reviews\"\n",
    "\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)\n",
EC2 Default User's avatar
EC2 Default User committed
103
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
104
105
106
107
108
109
110
111
112
113
114
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define an evaluation function that will be used during the optimization process."
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
115
   "execution_count": null,
116
117
118
119
120
121
122
123
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval_func(model):\n",
    "    eval = evaluate.evaluator(\"text-classification\")\n",
    "    results = eval.compute(\n",
    "        model_or_pipeline=model,\n",
    "        tokenizer=tokenizer,\n",
Julien Simon's avatar
Julien Simon committed
124
125
    "        data=eval_dataset,\n",
    "        metric=evaluate.load(\"mse\"),\n",
126
    "        label_column=\"labels\",\n",
Julien Simon's avatar
Julien Simon committed
127
    "        label_mapping=model.config.label2id,\n",
128
    "    )\n",
EC2 Default User's avatar
EC2 Default User committed
129
    "    return results[\"mse\"]"
130
131
132
133
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
134
   "execution_count": null,
135
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
136
   "outputs": [],
137
138
139
140
141
142
143
144
   "source": [
    "eval_func(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
EC2 Default User's avatar
EC2 Default User committed
145
146
147
    "Then, we set up quantization. We decide to use a bayesian optimization strategy (```tuning.strategy.name```) with a maximum loss of accuracy (```accuracy_criterion.relative```) set to 1%. We let the job run for 100 trials (```tuning.exit_policy.max_trials```).\n",
    "\n",
    "You can learn about tuning strategies in the Intel Neural Compressor [documentation](https://intel.github.io/neural-compressor/docs/tuning_strategies.html)."
148
149
150
151
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
152
   "execution_count": null,
153
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
154
   "outputs": [],
EC2 Default User's avatar
EC2 Default User committed
155
156
157
158
159
160
   "source": [
    "!pygmentize intel_neural_compressor/quantize.yml"
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
161
   "execution_count": null,
EC2 Default User's avatar
EC2 Default User committed
162
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
163
   "outputs": [],
164
165
166
167
168
   "source": [
    "from optimum.intel.neural_compressor import IncOptimizer, IncQuantizer\n",
    "from optimum.intel.neural_compressor.configuration import IncQuantizationConfig\n",
    "\n",
    "quantization_config = IncQuantizationConfig.from_pretrained(\n",
EC2 Default User's avatar
EC2 Default User committed
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
    "    config_name_or_path=\"./intel_neural_compressor\",\n",
    "    config_file_name=\"quantize.yml\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also load a configuration directly from the Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantization_config = IncQuantizationConfig.from_pretrained(\n",
    "    config_name_or_path=\"config_name_or_path=\"juliensimon/distilbert-amazon-shoe-reviews\",\",\n",
Julien Simon's avatar
Julien Simon committed
189
    "    config_file_name=\"quantize.yml\",\n",
190
191
192
193
194
195
196
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
EC2 Default User's avatar
EC2 Default User committed
197
    "We then create the ```IncQuantizer``` and ```IncOptimizer``` objects, and we launch quantization."
198
199
200
201
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
202
   "execution_count": null,
203
204
205
   "metadata": {},
   "outputs": [],
   "source": [
EC2 Default User's avatar
EC2 Default User committed
206
    "inc_quantizer = IncQuantizer(quantization_config, eval_func)"
207
208
209
210
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
211
   "execution_count": null,
212
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
213
   "outputs": [],
214
   "source": [
EC2 Default User's avatar
EC2 Default User committed
215
    "inc_optimizer = IncOptimizer(model, quantizer=inc_quantizer)\n",
216
217
218
219
    "\n",
    "inc_model = inc_optimizer.fit()"
   ]
  },
EC2 Default User's avatar
EC2 Default User committed
220
221
222
223
224
225
226
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we save the quantized model and its configuration file."
   ]
  },
227
228
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
229
   "execution_count": null,
230
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
231
   "outputs": [],
232
233
   "source": [
    "model_dir = \"./model_inc\"\n",
EC2 Default User's avatar
EC2 Default User committed
234
    "inc_optimizer.save_pretrained(model_dir)\n",
Julien Simon's avatar
Julien Simon committed
235
    "tokenizer.save_pretrained(model_dir)"
236
237
238
239
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
240
   "execution_count": null,
241
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
242
   "outputs": [],
243
244
245
   "source": [
    "from optimum.intel.neural_compressor.quantization import IncQuantizedModelForSequenceClassification\n",
    "\n",
Julien Simon's avatar
Julien Simon committed
246
    "inc_model = IncQuantizedModelForSequenceClassification.from_pretrained(model_name_or_path=model_dir)"
247
248
   ]
  },
EC2 Default User's avatar
EC2 Default User committed
249
250
251
252
253
254
255
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the ```Linear``` operator has been replaced by its quantized equivalent."
   ]
  },
256
257
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
258
   "execution_count": null,
259
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
260
   "outputs": [],
261
262
263
264
265
266
   "source": [
    "model.distilbert.transformer.layer[0]"
   ]
  },
  {
   "cell_type": "code",
EC2 Default User's avatar
EC2 Default User committed
267
   "execution_count": null,
268
   "metadata": {},
EC2 Default User's avatar
EC2 Default User committed
269
   "outputs": [],
270
   "source": [
EC2 Default User's avatar
EC2 Default User committed
271
    "inc_model.distilbert.transformer.layer[0]"
272
   ]
EC2 Default User's avatar
EC2 Default User committed
273
274
275
276
277
278
279
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
280
281
282
283
284
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
EC2 Default User's avatar
EC2 Default User committed
285
   "display_name": "conda_python3",
286
   "language": "python",
EC2 Default User's avatar
EC2 Default User committed
287
   "name": "conda_python3"
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}