Press ?
for help.
All features are anonymous.
Michael Heironimus
Some people, especially if they're not very experienced, simply don't think in terms of automation. Others approach it in a completely linear manner.
If you don't know Vagrant, it's automation around copying and configuring VMs from images.
Packer creates Docker images without a Dockerfile by running things in the image and committing or saving.
Setup examples: shell scripts, Puppet, Chef, Salt, Ansible, etc.
Post-install examples: Vagrant box package, docker push/tag/etc.
AWS CloudFormation also has special JSON keys for annotations.
The build file is small in absolute terms, but it's a lot of JSON to manage by hand. There is also a second that is slightly larger.
{
"description": "Build Vagrant-ready CentOS images that look like the official AMIs in the AWS Marketplace.",
"builders": [
{
"name": "centos-6-virtualbox",
"type": "virtualbox-iso",
"boot_command": [
"<tab> text {{user `c6_ks_opts`}} ks=http://{{.HTTPIP}}:{{.HTTPPort}}/{{user `c6_name`}}.ks<enter><wait>"
],
"disk_size": 8192,
"export_opts": ["--manifest"],
"guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
"guest_os_type": "RedHat_64",
"hard_drive_interface": "sata",
"headless": "{{user `headless`}}",
"http_directory": "http",
"iso_checksum": "{{user `c6_iso_checksum`}}",
"iso_checksum_type": "sha512",
"iso_url": "{{user `c6_iso_path`}}",
"output_directory": "builds/{{build_name}}-{{user `stamp`}}",
"shutdown_command": "echo 'vagrant' | sudo -S /sbin/halt -h -p",
"ssh_password": "vagrant",
"ssh_username": "vagrant",
"ssh_wait_timeout": "10000s",
"vboxmanage": [
["modifyvm", "{{.Name}}", "--memory", "1280"],
["modifyvm", "{{.Name}}", "--cpus", "1"],
["modifyvm", "{{.Name}}", "--vram", "12"]
],
"vm_name": "{{user `c6_name`}}-{{user `stamp`}}"
},
{
"name": "centos-7-virtualbox",
"type": "virtualbox-iso",
"boot_command": [
"<tab> text {{user `c7_ks_opts`}} ks=http://{{.HTTPIP}}:{{.HTTPPort}}/{{user `c7_name`}}.ks<enter><wait>"
],
"disk_size": 8192,
"export_opts": ["--manifest"],
"guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
"guest_os_type": "RedHat_64",
"hard_drive_interface": "sata",
"headless": "{{user `headless`}}",
"http_directory": "http",
"iso_checksum": "{{user `c7_iso_checksum`}}",
"iso_checksum_type": "sha512",
"iso_url": "{{user `c7_iso_path`}}",
"output_directory": "builds/{{build_name}}-{{user `stamp`}}",
"shutdown_command": "echo 'vagrant' | sudo -S /sbin/halt -h -p",
"ssh_password": "vagrant",
"ssh_username": "vagrant",
"ssh_wait_timeout": "10000s",
"vboxmanage": [
["modifyvm", "{{.Name}}", "--memory", "1280"],
["modifyvm", "{{.Name}}", "--cpus", "1"],
["modifyvm", "{{.Name}}", "--vram", "12"]
],
"vm_name": "{{user `c7_name`}}-{{user `stamp`}}"
},
{
"type": "docker",
"name": "centos-6",
"commit": true,
"image": "{{user `c6_docker_image`}}",
"pull": false
},
{
"type": "docker",
"name": "centos-7",
"commit": true,
"image": "{{user `c7_docker_image`}}",
"pull": false
}
],
"post-processors": [
{
"type": "vagrant",
"only": ["centos-6-virtualbox", "centos-7-virtualbox"],
"compression_level": 9,
"keep_input_artifact": true,
"output": "builds/{{.BuildName}}-{{user `stamp`}}.box"
},
[
{
"type": "docker-tag",
"only": ["centos-6", "centos-7"],
"repository": "mkheironimus/{{build_name}}",
"tag": "{{user `stamp`}}"
},
{
"type": "docker-save",
"only": ["centos-6", "centos-7"],
"path": "builds/{{build_name}}-{{user `stamp`}}.tar"
}
]
],
"provisioners": [
{
"type": "shell",
"environment_vars": [
"HOME_DIR=/home/vagrant",
"http_proxy={{user `http_proxy`}}",
"https_proxy={{user `https_proxy`}}",
"no_proxy={{user `no_proxy`}}",
"VERBOSE={{user `verbose`}}"
],
"override": {
"centos-6-virtualbox": {
"execute_command": "env {{.Vars}} sudo -S -E bash -eu '{{.Path}}'"
},
"centos-7-virtualbox": {
"execute_command": "env {{.Vars}} sudo -S -E bash -eu '{{.Path}}'"
}
},
"execute_command": "env {{.Vars}} bash -eu '{{.Path}}'",
"scripts": [
"shell/yum.sh",
"shell/docker_prep.sh",
"shell/grub.sh",
"shell/patch.sh"
]
},
{
"type": "shell",
"pause_before": "10s",
"environment_vars": [
"HOME_DIR=/home/vagrant",
"http_proxy={{user `http_proxy`}}",
"https_proxy={{user `https_proxy`}}",
"no_proxy={{user `no_proxy`}}",
"VERBOSE={{user `verbose`}}"
],
"override": {
"centos-6-virtualbox": {
"execute_command": "env {{.Vars}} sudo -S -E bash -eu '{{.Path}}'"
},
"centos-7-virtualbox": {
"execute_command": "env {{.Vars}} sudo -S -E bash -eu '{{.Path}}'"
}
},
"execute_command": "env {{.Vars}} bash -eu '{{.Path}}'",
"scripts": [
"shell/vmtools.sh",
"shell/vagrant.sh",
"shell/vm_ami_cleanup.sh",
"shell/cleanup.sh",
"shell/whiteout.sh"
]
}
],
"variables": {
"headless": "true",
"http_proxy": "{{env `http_proxy`}}",
"https_proxy": "{{env `https_proxy`}}",
"no_proxy": "{{env `no_proxy`}}",
"stamp": "{{isotime \"20060102\"}}",
"verbose": "true",
"c6_name": "centos-6",
"c6_iso_checksum": "c694ba4903ac2eb07d996ffa06b92d1dda78ec035d1ad87fdc9681e7245e7fc363eae987ec2476a408cf0bcaed0080ab05c7f26d7a9141eec8f898993c1057b1",
"c6_iso_path": "iso/CentOS-6.7-x86_64-bin-DVD1.iso",
"c6_ks_opts": "",
"c6_docker_image": "centos:6",
"c7_name": "centos-7",
"c7_iso_checksum": "3e084199713f0a7a39942a123ad698d36e0ee0a2253774e96c44a85df1574ed4ab6f7f13f52e778318902e59d7e0eb056b8557397946e1629119dec1b3b8d442",
"c7_iso_path": "iso/CentOS-7-x86_64-DVD-1511.iso",
"c7_ks_opts": "rd.live.check=0",
"c7_docker_image": "centos:7"
}
}
Builders run in parallel, so you may benefit from putting them together.
Variables are also only usable in text fields, and not even in all of those.
Worry about a nice file first, but how do you pick a new file format?
Packer does use nested arrays, most notably for ordering postprocessors.
Remember when computers were going to save us from mundane repetitive tasks?
For simplicity, includes can't be nested.
Those global settings may go away. They don't do anything you can't do with includes.
---
builders:
centos-6-virtualbox:
config_template:
- tmpl_vbox.yml
- tmpl_vbox_iso.yml
iso_url: iso/CentOS-6.7-x86_64-bin-DVD1.iso
boot_command:
- <tab> text ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-6.ks<enter><wait>
vm_name: "centos-6-{{user `stamp`}}"
iso_checksum: c694ba4903ac2eb07d996ffa06b92d1dda78ec035d1ad87fdc9681e7245e7fc363eae987ec2476a408cf0bcaed0080ab05c7f26d7a9141eec8f898993c1057b1
centos-7-virtualbox:
config_template:
- tmpl_vbox.yml
- tmpl_vbox_iso.yml
iso_url: iso/CentOS-7-x86_64-DVD-1511.iso
boot_command:
- <tab> text rd.live.check=0 ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-7.ks<enter><wait>
vm_name: "centos-7-{{user `stamp`}}"
iso_checksum: 3e084199713f0a7a39942a123ad698d36e0ee0a2253774e96c44a85df1574ed4ab6f7f13f52e778318902e59d7e0eb056b8557397946e1629119dec1b3b8d442
centos-6:
config_template: tmpl_docker_commit.yml
image: centos:6
centos-7:
config_template: tmpl_docker_commit.yml
image: centos:7
provisioner_override:
centos-6:
execute_command: "env {{.Vars}} bash -eu '{{.Path}}'"
centos-7:
execute_command: "env {{.Vars}} bash -eu '{{.Path}}'"
provisioners:
-
config_template: tmpl_shell.yml
scripts:
- shell/yum.sh
- shell/docker_prep.sh
- shell/grub.sh
- shell/patch.sh
-
config_template: tmpl_shell.yml
scripts:
- shell/vmtools.sh
- shell/vagrant.sh
- shell/vm_ami_cleanup.sh
- shell/cleanup.sh
- shell/whiteout.sh
pause_before: 10s
post-processors:
-
config_template: tmpl_vagrant.yml
only:
- centos-6-virtualbox
- centos-7-virtualbox
-
-
config_template: tmpl_docker_tag.yml
only:
- centos-6
- centos-7
-
config_template: tmpl_docker_save.yml
only:
- centos-6
- centos-7
# defaults, to be overridden on command line or var file.
variables:
headless: "true"
http_proxy: "{{env `http_proxy`}}"
https_proxy: "{{env `https_proxy`}}"
no_proxy: "{{env `no_proxy`}}"
stamp: "{{isotime \"20060102\"}}"
verbose: "true"
centos-6-virtualbox:
config_template:
- tmpl_vbox.yml
- tmpl_vbox_iso.yml
iso_url: iso/CentOS-6.7-x86_64-bin-DVD1.iso
boot_command:
- <tab> text ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-6.ks<enter><wait>
vm_name: "centos-6-{{user `stamp`}}"
iso_checksum: c694ba4903ac2eb07d996ffa06b92d1dda78ec035d1ad87fdc9681e7245e7fc363eae987ec2476a408cf0bcaed0080ab05c7f26d7a9141eec8f898993c1057b1
Excerpt from the top-level YAML file.
output_directory: "builds/{{build_name}}-{{user `stamp`}}"
ssh_password: vagrant
ssh_wait_timeout: 10000s
headless: "{{user `headless`}}"
export_opts:
- --manifest
ssh_username: vagrant
vboxmanage:
-
- modifyvm
- "{{.Name}}"
- --memory
- "1280"
-
- modifyvm
- "{{.Name}}"
- --cpus
- "1"
-
- modifyvm
- "{{.Name}}"
- --vram
- "12"
shutdown_command: echo 'vagrant' | sudo -S /sbin/halt -h -p
tmpl_vbox.yml, common to both ISO and OVF VirtualBox builders.
# Overlay for tmpl_vbox.yml
type: virtualbox-iso
http_directory: http
hard_drive_interface: sata
guest_os_type: RedHat_64
guest_additions_path: VBoxGuestAdditions_{{.Version}}.iso
disk_size: 8192
iso_checksum_type: sha512
tmpl_vbox_iso.yml, second template in the list.
centos-6-virtualbox:
config_template:
- tmpl_vbox.yml
- tmpl_vbox_iso.yml
iso_url: iso/CentOS-6.7-x86_64-bin-DVD1.iso
boot_command:
- <tab> text ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-6.ks<enter><wait>
vm_name: "centos-6-{{user `stamp`}}"
iso_checksum: c694ba4903ac2eb07d996ffa06b92d1dda78ec035d1ad87fdc9681e7245e7fc363eae987ec2476a408cf0bcaed0080ab05c7f26d7a9141eec8f898993c1057b1
Everything but config_template is layered on top, and the name is moved in to the "name" key.
Builders run in parallel so the order doesn't matter, so they can be named in the YAML. Provisioners and postprocessors require a specific order, so they would need a sort key.
At this stage error handling is more of a "neatness counts" thing than a real need. Plus I'm still in the "how does this work/look" stage with Perl 6.
YAMLish doesn't say what's wrong, just that parsing failed.
Since the script does almost no transformation, you can use Packer's error on your YAML just as well as on the JSON.
Lessons learned.
JSON::Tiny should work, but it's untested.
You can't yet assume that everything will "just work" in Perl 6.
I filed an issue on github about quoting numeric strings and the YAMLish author said he needed to have it quote a lot more, and that output was generally less mature than input.
use JSON::Pretty;
use YAMLish;
my $f = slurp('centos.json');
my $cfg = from-json($f);
my $yaml = save-yaml($cfg);
say $yaml;
Done for the initial file, but don't forget the quoting problem.
use v6;
use JSON::Pretty;
use YAMLish;
class PackerGen {
has %!cfg = ();
has %.output = ();
# Constructor taking an optional input file, not inherited.
submethod BUILD(Str :$yaml='') {
self.load($yaml);
self.generate();
}
# Load a YAML file from stdin or the named file.
method load(Str $in) {
%!cfg = load-yaml(($in ne '') ?? slurp($in)
!! $*IN.slurp-rest);
}
# Write the generated JSON to stdout or the given filename.
multi method save(Str $out where { $out eq '' }) {
say to-json(%!output);
}
multi method save(Str $out) {
# to-json doesn't have a final newline
spurt($out, to-json(%!output) ~ "\n");
}
# If a config_template element exists, load that content and
# overlay the supplied data. Otherwise just return the data.
multi method expand_entry(%data
where { %data<config_template>:exists } --> Hash) {
my %new = %data<config_template>:delete.map({
load-yaml(slurp($^a)) });
%(%new, %data);
}
multi method expand_entry(%data --> Hash) {
%data;
}
# Process templates and add the name key.
method build(Pair (:key(:$name), :value(:$data)) --> Hash) {
%(self.expand_entry($data), 'name' => $name);
}
# Process templates and add the global override definition if
# there is no override in the provisioner.
method provision(%prov --> Hash) {
my %new = self.expand_entry(%prov);
if (%!cfg<provisioner_override>:exists &&
!(%prov<override>:exists)) {
%new<override> = %!cfg<provisioner_override>;
}
%new;
}
# Process templates for a sequence or a single postprocessor.
multi method postproc(@post --> Seq) {
@post.map({ self.expand_entry(%^a) });
}
multi method postproc(%post --> Hash) {
self.expand_entry(%post);
}
method generate(--> Hash) {
%!output<builders> = %!cfg<builders>.map({
self.build($^a) });
%!output<provisioners> = %!cfg<provisioners>.map({
self.provision($^a) })
if (%!cfg<provisioners>:exists);
%!output<post-processors> = %!cfg<post-processors>.map({
self.postproc($^a) })
if (%!cfg<post-processors>:exists);
%!output<description> = %!cfg<description>
if (%!cfg<description>:exists);
%!output<variables> = %!cfg<variables>
if (%!cfg<variables>:exists);
%!output;
}
}
# --in=input.yaml --out=output.json
sub MAIN(Str :$in='', Str :$out='') {
my $pg = PackerGen.new(yaml => $in);
$pg.save($out);
}
"Dynamic templating" means more advanced variable expansions, macros, etc.
People have written lots of similar tools for CloudFormation, but AWS uses JSON all over the place.
Some people throw around the term "DSL" really freely. Here it means a real workflow configuration language (maybe using a Grammar), not just JSON or Ruby with a library.
Code and presentation available online.
Pause for questions, comments, and tomatoes.