NEWS 17 KB
Newer Older
Lars Wirzenius's avatar
Lars Wirzenius committed
1
2
3
NEWS for vmdb2, the Debian disk image builder
=============================================================================

Lars Wirzenius's avatar
Lars Wirzenius committed
4
5
6
7
8
Project is now in "selfish maintenance mode". I make changes when I
need them for myself, but I'm not developing new features otherwise. I
will review and merge patches, though.

Version 0.26, released 2022-04-20
9
10
-----------------------------------------------------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
11
12
13
14
15
16
* Bug fix: the Ansible plugin now correctly deletes the temporary
  files it creates.


Version 0.25, released 2022-02-19
-----------------------------------------------------------------------------
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

* Florian La Roche fixed the grub plugin to not set
  systemd.show_status multiple times, and other issues with how vmdb2
  generates the kernel command line.

* Zygmunt Krynicki changed vmdb2 to detect if it should use the
  --no-extra-removable option. Debian and Ubuntu versions of grub
  differ on whether it's needed.

* Andy Piper fixed cleaning up mapped partitions in Docker on macOS.

* Andy Piper added a "fill" option to the lvcreate plugin.

* Andy Piper added support for filling ext2/3/4 file systems with
  zeroes after they're unmounted. This happens by default, but can be
  disabled in the mount step.

* Andy Piper added the copy_dir plugin to copy directory trees into
  the build chroot.

* Andy Piper added the set_part plugin for setting flags on partition.

* Andy Piper fixed the grub plugin to run "apt update" before grub is
  installed to handle the case where no packages get installed
  previously in a vmdb2 run.

* Andy Piper fixed the grub plugin to work if the root file system is
  on a logical volume.

* Andy Pipe added the --variable option to vmdb2 for setting variables
  for jinja2 template expansion on .vmdb files.

* Andy Piper extended the ansible plugin to make it more configurable.

* Andy Piper fixed the debootstrap plugin to not run apt-get update
  unnecessarily, and to add the include and install_keyring fields,
  to allow additional packages to be installed.

* Lars Wirzenius added the cryptsetup setup.

* Sebastian Bachmann fixed bugs related partition naming, when device
  names end in a number, and redundant removal of loop devices.

* Lars Wirzenius made the debootstrap plugin require the target
  directory to not exist, or be an empty directory, with an override
  for this check.

* Lars Wirzenius changed the test suite so there is one script that
  can run it entirely. It's called check-all. The old check-all is now
  check-images.

* Lars Wirzenius changed the luks plugin to use luks2 by default.


71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
Version 0.24, released 2021-08-22
-----------------------------------------------------------------------------

* This release disables the non-Intel architectures in the
  `./check-all` scripts. The builds work when done manually, but now
  when done by the script. I'm tired of debugging this and will be
  rewriting the test suite to be more sane.

* Aaron Dewes merged `qemu-debootstrap` with plain `debootstrap`. As
  of Debian 10 (buster) plain `debootstrap` can do the same foreign
  bootstrap thing.

* Andres Salomon fixed the logic for the `apt` plugin's `recommends`
  option. The option has reversed the meaning to make it easier for
  the user to get right.

* Lars Wirzenius disabled using `realpath` to resolve the output
  device to not include symbolic links, to fix LVM2 support. This
  probably breaks something else.

* Lars Wirzenius changed the `lvcreate` step to run the `lvcreate`
  command in a way the prevents it from asking questions.

Lars Wirzenius's avatar
Lars Wirzenius committed
94
Version 0.23, released 2021-04-01
95
96
-----------------------------------------------------------------------------

Matthias Klein's avatar
Matthias Klein committed
97
98
* Matthias Klein added the optional `options` field to the `mkfs`
  step to set aditional options for mkfs.
99

Lars Wirzenius's avatar
Lars Wirzenius committed
100
101
* Gunnar Wolf added support for not installing recommended packages in
  the `apt` step in a .vmdb file.
102
103

Version 0.22, released 2021-01-16
104
105
-----------------------------------------------------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
106
107
108
109
* This release has a lot of changes from Ryutaroh Matsumoto and
  Christian Kastner to fix and clean up things, and to add support for
  building images for I385 (UEFI and BIOS), armhf+UEFI, and ppc64el.

110
111

Version 0.21, released 2020-12-18
112
113
-----------------------------------------------------------------------------

114
* The grub plugin is able to install the UEFI version of grub on arm64
Lars Wirzenius's avatar
Lars Wirzenius committed
115
116
117
  targets, patch from David Edmondson.[

* The documentation tile for the `resize_rootfs` step is now fixed.
118
119
120


Version 0.20, released 2020-10-18
121
122
-----------------------------------------------------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
123
124
* All use of the Python `cliapp` library has been dropped from vmdb2 now.

125
126
127
* New step `resize-rootfs` adds a systemd service to resize the root
  file system to fit the actual drive on boot.

128

129
Version 0.19, released 2020-09-16
130
131
-----------------------------------------------------------------------------

132
133
134
135
136
137
138
139
140
141
Change in this release are entirely internal code changes that should
not affect users of vmdb2.

* Python Code has been formatted with the Black code formatting tool.

* Changes have been made towards dropping the dependency on the
  `cliapp` library: the Python standard library module `subprocess` is
  used instead of `cliapp.runcmd`, and the `cliapp` plugin system has
  been replaced with a custom one.

142

Lars Wirzenius's avatar
Lars Wirzenius committed
143
Version 0.18, released 2020-08-13
144
145
-----------------------------------------------------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
146
147
148
149
* The way the manual was formatted could sometimes lead to different
  plugins' documentation being merged into the same section. This has
  now been fixed.

150

Lars Wirzenius's avatar
Lars Wirzenius committed
151
Version 0.17, released 2020-07-13
152
153
-----------------------------------------------------------------------------

154
155
156
157
158
* The luks plugin now runs the `cryptsetup` utility with the `-q`
  (alias for `--batch-mode`) option for batch mode, to prevent the
  utility from asking any questions of the user. Fix by Birger
  Schacht.

159

Lars Wirzenius's avatar
Lars Wirzenius committed
160
161
Version 0.16, released 2020-06-06
-----------------------------------------------------------------------------
162
163
164
165
166
167

* Progress output is now flushed after every write. This matters, when
  the output goes to a pipe, because then Python would buffer the
  output, meaning that progress output gets delayed until the buffer
  fills up, and that's useless.

168
169
170
171
* The documentation is now formatted using Subplot. Previously, the
  manual and the acceptance test suite were split into separate, but
  confusingly named files (vmdb2.md vs vmdb2.mdwn); now they are
  combined. The manual has been cleaned up a bit.
172

173
174
175
176
177
* Add a `quiet` field to the **grub step** to configure the kernel
  boot to be quiet or not. Default is now not quiet, which is a change
  from before.

* Add a `timeout` field to the **grub step** to configure grub menu
178
179
180
  timeout. Default is now zero seconds, which is a change from the
  Debian default of five seconds.

181
182
183
* Symbolic links can now be used in the **mkpart step** for device
  files, thanks to Tim Small.

184

185
Version 0.15, released 2020-05-05
Lars Wirzenius's avatar
Lars Wirzenius committed
186
-----------------------------------------------------------------------------
187

188
189
190
191
192
193
194
### New and changed features

* Document and rename the `virtual-filesystems` plugin. This allows
  installation of Debian packages that require `/proc` to be mounted
  to install, such as `ca-certificates-java`.
  
  Undo previous changes to have `/proc` automatically mounted in the
195
  chroot, when running commands in the chroot. That wasn't enough
196
197
  (e.g., it didn't cover Ansible), and `virtual-filesystems` seems
  generally like the more general and better solution.
198

199
200
  The `grub` step handles the filesystems it needs itself, but doesn't
  mind if they're already mounted.
201
202

  This change should NOT require changes to existing, working .vmdb
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
  files. If it does, please report a bug.

* The .vmdb file can now be passed in via the standard input, by
  specifying `-` as the filename. Patch from voney.

* The manual now documents that un-partitioned images are not
  supported, at least not yet.
  
* The "All went fine" message is now written at the very end of the
  vmdb2 run, if at all. Previously all the messages from teardown
  parts of steps were printed after it, hiding it effectively.
  
* The standard output and error output of the commands vmdb2 runs are
  no longer logged in real time, only after the command has finished.
  This is a side effect of the `cliapp.runcmd` helper function no
  longer being used. The Python standard library subprocess module is
  used directly instead. This change is a step to drop the dependency
  on the cliapp library.

### Bug fixes

* The `grub` step documentation now uses the correct key for
  specifying the image device (`image-dev`). Thanks: Sebastian
  Bachmann.

* The `mkpart` step now waits for the new device node to be created.
  Previously it sometimes took enough time to be created that the next
  step failed. This only affects when the output disk is a real block
  device, not a disk image file.

### Changes mainly affecting vmdb2 and plugin developers
234

235
236
237
* The `./check` script only runs the yarn integration testing tool if
  it's new enough to have the yarnutils Python library included. Yarn
  is being replaced with Subplot.
238

239
240
241
242
243
244
* Step runners must now use a new way of specifying expected and
  allowed keys for steps. The new way specifies types for mandatory
  keys, and default values for optional keys. The old
  `get_required_keys` method is gone. This change makes mistakes in
  step runner code a little less likely, and opens up the possibility
  of generating documentation about step keys automatically.
245

246
247
248
249
250
251
* Failures when un-mounting filesystems are now ignored. Such failures
  can happen for various reasons, but by far the most common reason is
  because the mount had already been unmounted and it's simpler to
  just ignore failures than keep meticulous track of what's been
  unmounted already.
  
252

253
Version 0.14.1
Lars Wirzenius's avatar
Lars Wirzenius committed
254
-----------------------------------------------------------------------------
Lars Wirzenius's avatar
Lars Wirzenius committed
255

Lars Wirzenius's avatar
Lars Wirzenius committed
256
257
258
This is a bug fix release to fix bugs in the 0.14 release that slipped
through due to insufficient testing.

259
* The shell plugin source file was misnamed. Fixed now.
Lars Wirzenius's avatar
Lars Wirzenius committed
260

Lars Wirzenius's avatar
Lars Wirzenius committed
261
262
263
* Fixes to problems found while building real images. The new
  `check-external` script builds my own images, and Debian's Raspberry
  Pi images.
Lars Wirzenius's avatar
Lars Wirzenius committed
264
265


266
Version 0.14, released 2020-03-21
Lars Wirzenius's avatar
Lars Wirzenius committed
267
-----------------------------------------------------------------------------
268

269
270
271
272
273
274
275
276
277
278
279
280
281
282
It has been two years since the previous release. These release notes
are a little sketchy.

* All plugins now provide exactly one type of step. This is a little
  simpler than having multiple steps provided by the same plugin.
  Simplicity is good.

* There is now some better automated testing.

* vmdb2 now checks at start time that all required keys are set for
  all steps and that keys have values of the correct type.

* Source code for vmdb2 is now hosted on gitlab.com and git.liw.fi,
  moved there from GitHub.
283

Lars Wirzenius's avatar
Lars Wirzenius committed
284
285
286
287
288
* Fixed smoke test timeout for starting a VM to 5 min. The previous
  timeout (30 s) was short enough that tests often failed because of
  it.

* The `debootstrap` action now finishes with `apt-get update`. For
Lars Wirzenius's avatar
Lars Wirzenius committed
289
  some reason thing had changed so that this became necessary.
Lars Wirzenius's avatar
Lars Wirzenius committed
290

Lars Wirzenius's avatar
Lars Wirzenius committed
291
292
293
* Christian Schlüter added the optional `label` field to the `mkfs`
  step to set the filesystem label upon creation.

Lars Wirzenius's avatar
Lars Wirzenius committed
294
295
296
297
298
* Changed the `grub` step to take an optional `image-dev` field to
  override the heuristics for finding the device where GRUB is to be
  installed. This was necessary to allow vmdb2 to install Debian onto
  real hardware.

Lars Wirzenius's avatar
Lars Wirzenius committed
299
300
* The smoke test now tests a UEFI image as well.

301
302
303
304
* The rootfs unpacking action now copies `/etc/resolv.conf` into the
  chroot, overwriting what the tarball has. This allows vmdb2 to work
  with a tarball generated in a different network location.

Lars Wirzenius's avatar
Lars Wirzenius committed
305
306
* Add plugins for doing full-disk encryption via cryptsetup, and LVM2.

Lars Wirzenius's avatar
Lars Wirzenius committed
307
308
* There is now some documentation.

Lars Wirzenius's avatar
Lars Wirzenius committed
309
310
* Christian Schlüter added support for f2fs labels in the `mkfs` step.

Lars Wirzenius's avatar
Lars Wirzenius committed
311
312
313
314
315
316
* Step runners may now implement the `run_even_if_skipped` method to
  have code that is run whether the step is skipped or not. The
  `debootstrap` step now uses this to run `apt-get update` always.
  This allows a rootfs tarball that is old to be used, without the
  Packages files being too old to be usable.

317
318
319
* The `cache_rootfs` step now caches all the explicitly mounted
  filesystems, not just the root filesystem.

320
Version 0.13.2, released 2018-05-06
321
322
------------------------------------

323
* New build, after CI screwed up.
324

325
Version 0.13.1, released 2018-04-30
326
327
----------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
328
* Fix build-dependency on pandoc.
329

Lars Wirzenius's avatar
Lars Wirzenius committed
330
Version 0.13.1, released 2018-04-30
331
332
333
----------------------------------


334
Version 0.12, released 2018-02-24
335
336
----------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
337
* Minor Debian packaging fixes.
338

339
Version 0.11, released 2018-02-10
340
341
----------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
342
343
* Build fixes so that .deb doesn't contain Python 2.7 stuff, which
  isn't wanted or used.
344

345
Version 0.10, released 2018-02-10
346
347
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
348
* Build tweaks to make builds work on Debian unstable.
349

350
Version 0.9, released 2017-10-11
351
352
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
353
354
355
356
357
358
359
360
* Stuart Prescott added a `variant` parameter to the `vmdebootstrap`
  plugin.

* Stuart and Lars documented some of the dependencies in README. If
  they're insufficient, report a bug please.

* Stuart added a `qemu-debootstrap` plugin to build images for a
  foreign architecture.
361

362
363
364
* Lars Wirzenius added the function `vmdb.runcm_chroot` for executing
  programs inside a chroot. It's a short wrapper around `vmdb.runcmd`,
  but is a little easier to get right than invoking **chroot**(8) via
Lars Wirzenius's avatar
Lars Wirzenius committed
365
  `runcmd`. Michael Stapelberg fixed the function.
366

Lars Wirzenius's avatar
Lars Wirzenius committed
367
368
369
370
371
372
373
* Lars Wirzenius changed it so that `.vmdb` files can have structured
  YAML values and their constituent strings are used as Jinja2
  templates. Previously vmdb2 would just crash. This opens a
  possibility to have things like "list of packages to install" for
  the `apt` step as a YAML list, instead of having the step runner
  parse a space delimited list.

Lars Wirzenius's avatar
Lars Wirzenius committed
374
375
376
* Michael Stapelberg added the `fs-type` parameter to the `mkpart`
  step so that Raspberry Pi images can be created.

Lars Wirzenius's avatar
Lars Wirzenius committed
377
378
379
380
381
* Lars Wirzenius changed the `apt` step. Previously the `apt` field
  values was the name of one package to install. Now the value MUST be
  `install` and a separate field `packages` is a YAML list of package
  names to install. These will all be installed at once.

Lars Wirzenius's avatar
Lars Wirzenius committed
382
383
384
385
* Michael Stapelberg and Lars Wirzenius added support for a
  `components` list to the `qemu-debootstrap` step, to allow giving a
  list of components to the debootstrap run by the step.

386
Version 0.8, released 2017-07-29
387
388
---------------------------------

389
390
391
392
393
* The `mount` plugin can now specify a mount point on an already
  mounted filesystem, such as /boot inside the root filesystem already
  mounted. This allows constructing a system with a separate /boot
  partition. Thanks to Stuart Prescott for reporting the lack of this
  feature.
394

395
Version 0.7, released 2017-06-18
396
397
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
398
399
400
401
402
* New plugin `virtuals` provides step `mount-virtual-filesystems` for
  mounting virtual filesystems such as `/proc` and `/dev` as well.

* The `apt` step installs `eatmydata` and runs `apt` under it, to
  speed up package installs.
403

404
Version 0.6, released 2017-06-11
405
406
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
407
408
* setup.py now installs the plugins, making the .deb package actually
  be usable.
409

410
Version 0.5, released 2017-06-04
411
412
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
413
414
415
416
417
* The "unless:" part of vmdb spec files now actually works. Previously
  it was entrely unimplmented (there was code to implement the check,
  but it was never actually called). The apt and debootstrap plugins
  have been fixed to not do "has rootfs tarball been unpacked" checks
  themselves.
418

419
Version 0.4, released 2017-06-03
420
421
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
422
423
424
425
426
427
428
429
* Add a rudimenteary smoke test yarn for vmdb2-built images.

* Add a plugin to run Ansible to configure an image at build time.
  Doing this via Ansible saves me from having to write equivalent
  functionality directly into vmdb2, which would be quite a lot of
  work. (If someone wants puppet, a plugin for that is probably easily
  doable, assuming it can run against a chroot. I don't now, I've
  never used puppet.)
430

431
Version 0.3, released 2017-05-21
432
433
---------------------------------

434
435
436
437
* Simplify progress reporting to go to stdout, plus stop logging
  progress reports ar as errors.

* Add a BIOS flavor for installing GRUB.
438

439
Version 0.2, released 2017-05-14
440
441
---------------------------------

Lars Wirzenius's avatar
Lars Wirzenius committed
442
443
444
445
446
447
448
449
450
451
452
453
* Add plugin to provide steps to cache the rootfs, and to unpack the
  rootfs from the cache, instead of running debootstrap or installing
  packages. This speeds up iteration time from about 9 minutes to 40
  seconds on my laptop. See small.vmdb for an example.

* Add a generic "unless this condition is true" functionality to
  steps. If a step has a field "unless: foo" it is skipped the
  variable foo exists and is true. The variables are set by steps, and
  currently only the rootfs unpacking step sets a variable (the
  `rootfs_unpacked` variable). This allows debootstrap to be skipped
  if the rootfs has already been created by unpacking a cached
  tarball.
454

455
Version 0.1, released 2017-05-13
Lars Wirzenius's avatar
Lars Wirzenius committed
456
457
458
459
-----------------------------------------------------------------------------

This is the first release. It can build a UEFI image for the amd64
architecture. It's not meant to really be useful for other people.