Creating a Metalsmith static site powered by node.js
I have been converting some of my lesser-updated sites to static, where we pre-render the HTML files and serve only HTML (and some images, perhaps) from the webserver. This post will walk you through the process of converting a simple blog site, originally based on Drupal 7, to a static site with similar features.
Step 1: Choose a static site generator (metalsmith.io in this case)
I selected metalsmith.io as my static site generator. There are lots of static site generators out there but this one is written in node.js, which I have some familiarity with, and I also liked the fact that it is not specifically for blogging, so I can potentially adapt it to other uses down the road.
My colleague ran more or less the same experiment using Jekyll which is a Ruby-based static site generator that is very popular. PHP has one called Sculpin You can find others at StaticGen.com.
Most of the static generators out there use Markdown as a formatting tool, so we will use that. Metalsmith itself has an example site which uses Handlebars.js for the layouts, so I kept things easy by choosing that as my layout engine.
Note that I am not planning to create a GUI experience for editing the content. Just go into the 'src' folder and write HTML or Markdown posts. If you want a CMS back-end, check out headless.org for some options.
Step 2: Hello, Static World!
In the second phase of my investigation of static sites was to actually run a builder. To do this, I cloned the metalsmith, and made sure I had run the node.js installer on my local machine.
To build the site you simply go into the root of the metalsmith repo:
cd examples/static-site/
make build
If you don't see any folders you should now see content in the build subdirectory.
You can create new ".md" files or ".html" files in the /src folder. Just make sure to start them with the metadata prefix like so:
---
layout: post.html
title: My lazy Saturday
date: 2017-08-05 14:45:00
---
# Neato, a heading!!
This is the body content
... and run make build to see the result. I used this Drupal module to export my content as .md files. Next time around I might try a JSON source rather than MD.
From here on we will be customizing the index.js file that lives in the root.
Step 3: Gathering dependencies, aka, "site building"
The "site building" process revolves around the index.js file that lives in the root of the site.
At the top of the file you load any dependencies. Let's take a look at what I have included for this website:
var Metalsmith = require('metalsmith');
var markdown = require('metalsmith-markdown');
var layouts = require('metalsmith-layouts');
var permalinks = require('metalsmith-permalinks');
var sitemap = require('metalsmith-mapsite');
var collections = require('metalsmith-collections');
var rss = require('metalsmith-rss');
var aliases = require('metalsmith-aliases-nginx');
var assets = require('metalsmith-assets');
var dateFormatter = require('metalsmith-date-formatter');
var discoverPartials = require('metalsmith-discover-partials');
Quite a few things! Let's review some of them...
3.1 Adding modules and managing package.json
If you are new to node.js or JavaScript in general you will only need to know one thing: when you installed node.js, you also installed the node.js pacakge manager "npm". While you are in the root of the project you can add any required dependencies by running the install command in the root of the project:
npm install --save metalsmith-assets
After you run that command, the dependency (and any subsequent dependencies) will be downloaded to the /node-modules subfolder of your project.
The --save switch adds that dependency to package.json in the root of your project.
You will want to maintain a package.json file listing all of these dependencies and ideally the version(s) you are willing to use of each. If you create a pacakge.json you can simply run "npm install" to get them all. I recommend you do this. You can use a similar procedure for doing updates.
IMPORTANT: you must now load that dependency by adding the var assets = require('metalsmith-assets'); yourself. You ALSO will need to add it to the process chain so that it can be "used" by your project.
3.2 Loading and making use of modules
Now that our modules are loaded, we need to "use" them somewhere. In metalsmith, this method is conveniently called "use".
When I added the metalsmith-assets module I went and consulted the module page on npm which is simply a rendering of the module's README.md file. It contains the information needed to install in our "app" in index.js after the "assets" variable has loaded the library:
.use(assets({
source: './assets',
destination: './'
}))
That's it! Now we have added the "markdown-assets" module to our configuration and we run it each time. This is a very simple module that simply copies anything I put into the /assets folder in my project and copies it verbatem with no changes over to the /build folder during the build process. I put things like Google Webmaster Tools authentication files here, robots.txt files, images, etc.
If you haven't already done so, create an "assets" folder in the root of the project, put some files there, and try running "make build" again. This time the metalsmith-assets module should get loaded into the variable "assets" and the "use" method in metalsmith will run it when you issue the command make build.
Step 4: Extract, Transform, Load process
Now you have an overview of running a node.js builder, loading modules using npm, and finally registering and making use of the module in your static site "application" we are building.
Let's look at the index.js file in more detail now. If you are familiar with Drupal migrations, there is something very similar happening here: index.js is effectively an "extract-tranform-load" migration process!
4.1 Setup global options, including some site metadata
In the first section, all we provide is some simple metatdata that can be re-used throughout the site. Note that we did not end this statement with a ; because we are going to chain the next settings off that:
Metalsmith(__dirname)
.metadata({
title: "verbosity.ca",
description: "web developer ramblings",
generator: "Metalsmith",
url: "http://verbosity.ca/"
})
.source('./src')
.destination('./build') // no semicolon here! the .use statements will follow!
Source and destination are both just references to folders that exist in the root of your project. Simple.
4.2 Collections module
.use(collections({
posts: {
pattern: ['**/*.md'],
sortBy: 'date',
reverse: true,
limit: 8
},
planet: {
pattern: ['**/*drupal*.md', 'php/drupal/*.md'],
sortBy: 'date',
reverse: true,
limit: 3
},
archive: {
pattern: ['**/*.md'],
sortBy: 'date',
reverse: true,
},
})) // no semicolon here! more .use statements will follow!
Now we are in the "process" chain issuing "use" calls for each step in the chain, delegating responsibility for each step to a module with some associated settings for that module.
In this case, "metalsmith-collections" is a module we are using to create a few lists: the first one grabs the 8 most recent posts in any section, and saves it to the variable "posts". The second statement, "planet", creates our Drupal Planet feed, which will get all posts in the /php/drupal directory, plus any posts anywhere on the site that have "drupal" in their title.
Note, I am specifically looking for ".md" files. I chose to make index.html files that are not processed by the markdown module to keep the indexes out of my collections. You can also use a path pattern to exclude "index.md" files if you wish.
One thing I really like about collections is that you can have a many-to-one relationship as we have done in the "planet" feed. That is super convenient.
4.4 RSS module
This module creates an RSS feed for our site's readers.
These filters should all likely explain themselves. They are just here to give us some additional nice things that we come to expect of our Drupal sites.
.use(rss({
feedOptions: {
title: "verbosity.ca",
description: "Blog posts by Ryan Weal",
site_url: "http://verbosity.ca/",
},
collection: "posts",
encoding: "utf8",
destination: "rss.xml"
}))
.use(rss({
feedOptions: {
title: "verbosity.ca drupal planet",
description: "Blog posts by Ryan Weal",
site_url: "http://verbosity.ca/",
},
collection: "planet",
encoding: "utf8",
destination: "drupal-planet/feed"
}))
This section was hard to build... I had to dig around for the right settings of the upstream RSS module to get verything right. I also had to patch one of the modules to get my rendered XML just right the way I wanted it.
4.5 Pretty URLS, pathauto, and input formats...
These methods will take care of how URLs are generated (or no generated) depending on some preferences we can set (we're not setting any). It is also where we convert any markdown files to HTML or do any other transormations like that.
.clean(true)
.use(markdown())
.use(permalinks())
The aliases module is an interesting one - I forked someone's repo on github. Their module made meta-refresh redirects but I don't like the idea of doing so much bouncing around so I implemented a version that creates an nginx file which can be imported into the webserver config.
.use(aliases())
That produces a file called nginx.conf in my root directory.
I include it in my Nginx server with the following syntax:
include /opt/static-site/nginx.conf;
4.6 Theme-related modules (see also next section)
.use(dateFormatter({
dates: 'date'
}))
.use(discoverPartials({
directory: 'partials',
pattern: /\.hbs$/
}))
.use(layouts({
engine: 'handlebars',
}))
These modules all get used for theming. Did you notice we are doing theming after all of the lists are built, and after we have rendered the XML feeds? That is on purpose. Everything runs in the order we specify and it wouldn't be very nice if our XML machine names were all formatted dates!
The "discover partials" module is very interesting. It allows you to create a folder called /hbs which can contain text files with .hbs as the extention. You can put anything in these, and inject them into your templates using the syntax {{>filename }} where filename refers to /hbs/filename.hbs. I use this in place of Drupal's "blocks" module primarily, but I also use it any time I wish to split off any string of text and have it load into the template dynamically.
4.6 Static Assets
This is very helpful if you have things like vlidation codes to place on the web server or assets like jpg and png files which should not have further processing done to them.
.use(assets({
source: './assets',
destination: './'
}))
4.7 XML sitemap
I had to customize this a bit to hide the shame of index.html in the result. Otherwise it was pretty easy.
// fyi, I added a slash to omitIndex in mapsite!
// prevents from rendering trailing /index.html in sitemap.xml
// evenually I will do this a better way without hacks.
.use(sitemap({
hostname: 'http://verbosity.ca',
omitIndex: true
}))
Finally we are done with all of our .use methods... time to trigger the build process at the end of the chained commands:
.build(function(err, files) {
if (err) { throw err; }
});
Now we're just making sure our assets are in place as described above and then taking care of the actual run process. That's it!
Step 5: Handlebars
The first templating engine I used with Metalsmith is the one used by the static-site example that comes with Metalsmith: Handlebars.
Coming from the PHP universe, it is very similar to twig. More specifically though, it borrows much of the syntax ideas from another JS project, Moustache.js.
Syntax
Moustache calls itself "logic-less templates" and I found that to be initially a bit difficult but after working with it a bit I really like it.
Handlebars lets you use any of the variables on the page as variables within the page. So for the title: parameter in your content source files will get rendered if you use {{ title }}. That will escape any odd characters you may have. If you need the raw HTML in that field, you can use the unescaped version by tripple-wrapping it. You will often see this done with the {{{content}}} variable that contains your body text.
Global variables can be used, and you can access anything off the "this" object that might be set globally. I used {{ locale }} and {{ this.altFiles.fr.path }} in one of my projects. Those variables are created by other modules... the point here is just to show you the syntax.
You can also do some basic iteration using the {{#each}} syntax. That is well documented elsewhere so we're not going to cover it here.
Debugging Handlebars
You can review what is in your "this" varaible to see what options you have available by doing the following in your template:
{{ log this }} {{ log this.title }} {{ log title }}
There is also a {{ debugger }} tag for creating breakpoints in your debug process. So far I have not had to use this as my problems have been fairly simple to figure out.
5.1 Using partials
If you have ever used "blocks" to define content in a CMS like Drupal, partials are your friend.
To use partials, we registered metalsmith-discover-partials module to the discoverPartials variable and set it to look for partials in a folder called partials.
That is all you need to do. Now, in your layout files, you can use partials as your own variables.
Here is an example. Let's say we want to make a block with some "contact us" info that will show on every page.
We go into the /partials/ folder in our repository, create a file called contact.hbs and put our HTML content in there.
Now we go to our layout file and we can use {{>contact}} to inject our content. Simple as that.
5.2 Writing and using your own partials
Many of the things we have mentioned above are actually "helpers" ... the log command, the partials we're injecting... all helpers. You can write your own, too, and you will want to at some point.
In my experience, any time you want an "if" statement in your template, or if you want to do something such as have a variable that defines a section of the site, you will probably find a need to write a small helper yourself.
It is important to note that you can't simply run JavaScript between the handlebars. If you are familiar with other JS frameworks this will probably bother you on some level, but get over it... this is very easy stuff to overcome and is done to benefit performance when you generate the site.
I made a language switcher for one of my projects. I have the {{locale}} variable already from another module, and I want to use that in my own code.
I created a folder called /helpers.
I added the package metalsmith-register-helpers in my index.js: var registerHelpers = require('metalsmith-register-helpers');.
And then I wrote my helper, named switcher.js in my helpers folder: module.exports = function (lang) { return this.altFiles[lang].path.replace(//index.html/i, '').replace(//$/i, ''); }
Now I can use it in my template like this:
{{ switcher 'en' }} // gives me a link to the English version
{{ switcher 'fr' }} // gives me a link to the French version
Nothing very fancy going on here. We could easily make it better but this gets the job done. I also filtered out the "index.html" part of the path while I was at it.
You can combine the syntax...
{{> (i18n 'menu') }}
What this is going to do is first between the parenetheses it will find a helper I created called i18n.js in my helpers folder. That is going to return a string, in this case, it will return "menu_en" (trust me on this, since I'm not showing you that code here). I happen to have a file called menu_en.hbs in my partials folder, so now the partials module will take over and inject that template.
The key take-away for me relating to handlebars is that you can't run plain JavaScript within the handlebars. Not when you're using {{ title }} which is a variable, not when you're using {{ >contact }} to import partials/contact.hbs. Even if you wrap things in parentheses you will still be running a helper, whether it be a built-in or one you write yourself. In these examples, "i18n" and "switcher" are helpers I created myself.
{{ log }} and {{ debugger }} as well as {{ #each }} are great examples of built-in helpers.
Step 6: Hide the shame (of .html) and gzip the files (in nginx)
We're using Nginx to host our files. We use the "try_files" parameter to mask the index.html if it is not provided in the URL. Currently we do not redirect users who type it in manually.
gzip on;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
try_files $uri $uri/index.html $uri/ =404;
}
// in step 4 we made a redirects file to include:
include /vagrant/www/verbosity-nginx.conf;
...we also gzip the content because text content compresses really well and saves substantially on bandwidth.
Step 7: Deploy!
Some day I will get fancy and add a webhook to my GitHub and/or Bitbucket repos that contain the files, have that hit an endpoint I define that will then check out the repo, build it, and deploy.
For now I don't need to do all of that. I simply rsync the files into place and then let my vagrant provisioning take care of the rest.
rsync -av --delete build/ /path/to/files/deployment/path/
After running this /path/to/files/deployment/path will contain everything in your project's build folder, and will also delete any posts that are no longer "live" since your last build.
Conclusion
If you have made it this far, congratulations! You have a node.js-powered static site and you can deploy it anywhere you wish!
Best of all, be have nice URLs, XMLsitemaps, RSS feeds, and all of that other typical things we might usually associate with having a CMS. If we were to take this a step further we might add a commenting service such as Disqus if we wanted to have comments on the site.
You can even do multilingual as I have done with my consulting site, http://kafei.ca. If you are interested in doing the same I recommend looking into the metalsmith-multi-language package and Belén Albeza's post about creating i18n metalsmith sites. That tutorial uses pug (formerly "jade") templates but the rest of the concepts are the same as discussed here.
Overall I really like using Metalsmith and I will continue using it for personal projects. I'm also looking into using Vue.js to generate static sites as both Vue and React now have many options for doing pre-generated as well as server-generated versions of apps built using those platforms. For most server-oriented tasks I'm planning to use Node.js to write custon JSON endpoints on an as-needed basis unless a suitable subscription service is available.
I am looking for JavaScript gigs. Contact me at http://kafei.ca about your project.
