Tonight I presented an overview of pump.io to the Montréal JavaScript meetup. My slides are attached to this post.

A good portion of the introduction is looking at the crazy stack that social networks generally use, typically something resembling LAMP, and that it is quite heavy for sending short messages. Mention to socket.io and node.js for being ideal to manage these situations. Some discussion was had about Diaspora and Frendika Frendica* which was interesting to note. Long story short, the stack has been screwing us all.

Then we dove into "what does it look like"... so I made a test post: https://comn.ca/ryanweal/note/Maamnc73RqW3kBPssERXcw and then one person asked to demo how to follow someone. *Thank you whoever you are! *That was a great question. Then we quickly went back to the slides.

After this I have two dense slides of install tips which are largely based on my own experience and interactions with other people running their own pump. We took a quick look at a typical config file. Everything a would-be admin will need to know.

To wrap up I provide a list of links to resources that people can use and three people to follow.

I blasted through these slides in just over 20 minutes! Yes, they are probably full of spelling errors, that's how I roll.

I have also attached the source document if you wish to recompile these into documentation, wiki pages, presentations to your friends, your boss, etc. Enjoy.

Slides

In pdf format

ODP source document

Step 1: Choose a static site generator (metalsmith.io in this case)

I selected metalsmith.io as my static site generator. There are lots of static site generators out there but this one is written in node.js, which I have some familiarity with, and I also liked the fact that it is not specifically for blogging, so I can potentially adapt it to other uses down the road.

My colleague ran more or less the same experiment using Jekyll which is a Ruby-based static site generator that is very popular. PHP has one called Sculpin You can find others at StaticGen.com.

Most of the static generators out there use Markdown as a formatting tool, so we will use that. Metalsmith itself has an example site which uses Handlebars.js for the layouts, so I kept things easy by choosing that as my layout engine.

Note that I am not planning to create a GUI experience for editing the content. Just go into the 'src' folder and write HTML or Markdown posts. If you want a CMS back-end, check out headless.org for some options.

Step 2: Hello, Static World!

In the second phase of my investigation of static sites was to actually run a builder. To do this, I cloned the metalsmith, and made sure I had run the node.js installer on my local machine.

To build the site you simply go into the root of the metalsmith repo:

cd examples/static-site/

make build

If you don't see any folders you should now see content in the build subdirectory.

You can create new ".md" files or ".html" files in the /src folder. Just make sure to start them with the metadata prefix like so:

---
layout: post.html
title: My lazy Saturday
date: 2017-08-05 14:45:00
---
# Neato, a heading!!

This is the body content

... and run make build to see the result. I used this Drupal module to export my content as .md files. Next time around I might try a JSON source rather than MD.

From here on we will be customizing the index.js file that lives in the root.

Step 3: Gathering dependencies, aka, "site building"

The "site building" process revolves around the index.js file that lives in the root of the site.

At the top of the file you load any dependencies. Let's take a look at what I have included for this website:

var Metalsmith  = require('metalsmith');
var markdown    = require('metalsmith-markdown');
var layouts     = require('metalsmith-layouts');
var permalinks  = require('metalsmith-permalinks');
var sitemap = require('metalsmith-mapsite');
var collections = require('metalsmith-collections');
var rss = require('metalsmith-rss');
var aliases = require('metalsmith-aliases-nginx');
var assets = require('metalsmith-assets');
var dateFormatter = require('metalsmith-date-formatter');
var discoverPartials = require('metalsmith-discover-partials');

Quite a few things! Let's review some of them...

3.1 Adding modules and managing package.json

If you are new to node.js or JavaScript in general you will only need to know one thing: when you installed node.js, you also installed the node.js pacakge manager "npm". While you are in the root of the project you can add any required dependencies by running the install command in the root of the project:

npm install --save metalsmith-assets

After you run that command, the dependency (and any subsequent dependencies) will be downloaded to the /node-modules subfolder of your project.

The --save switch adds that dependency to package.json in the root of your project.

You will want to maintain a package.json file listing all of these dependencies and ideally the version(s) you are willing to use of each. If you create a pacakge.json you can simply run "npm install" to get them all. I recommend you do this. You can use a similar procedure for doing updates.

IMPORTANT: you must now load that dependency by adding the var assets = require('metalsmith-assets'); yourself. You ALSO will need to add it to the process chain so that it can be "used" by your project.

3.2 Loading and making use of modules

Now that our modules are loaded, we need to "use" them somewhere. In metalsmith, this method is conveniently called "use".

When I added the metalsmith-assets module I went and consulted the module page on npm which is simply a rendering of the module's README.md file. It contains the information needed to install in our "app" in index.js after the "assets" variable has loaded the library:

.use(assets({
  source: './assets',
  destination: './'
}))

That's it! Now we have added the "markdown-assets" module to our configuration and we run it each time. This is a very simple module that simply copies anything I put into the /assets folder in my project and copies it verbatem with no changes over to the /build folder during the build process. I put things like Google Webmaster Tools authentication files here, robots.txt files, images, etc.

If you haven't already done so, create an "assets" folder in the root of the project, put some files there, and try running "make build" again. This time the metalsmith-assets module should get loaded into the variable "assets" and the "use" method in metalsmith will run it when you issue the command make build.

Step 4: Extract, Transform, Load process

Now you have an overview of running a node.js builder, loading modules using npm, and finally registering and making use of the module in your static site "application" we are building.

Let's look at the index.js file in more detail now. If you are familiar with Drupal migrations, there is something very similar happening here: index.js is effectively an "extract-tranform-load" migration process!

4.1 Setup global options, including some site metadata

In the first section, all we provide is some simple metatdata that can be re-used throughout the site. Note that we did not end this statement with a ; because we are going to chain the next settings off that:

Metalsmith(__dirname)
  .metadata({
    title: "verbosity.ca",
    description: "web developer ramblings",
    generator: "Metalsmith",
    url: "http://verbosity.ca/"
  })
  .source('./src')
  .destination('./build') // no semicolon here! the .use statements will follow!

Source and destination are both just references to folders that exist in the root of your project. Simple.

4.2 Collections module

  .use(collections({
    posts: {
      pattern: ['**/*.md'],
      sortBy: 'date',
      reverse: true,
      limit: 8
    },
    planet: {
      pattern: ['**/*drupal*.md', 'php/drupal/*.md'],
      sortBy: 'date',
      reverse: true,
      limit: 3
    },
    archive: {
      pattern: ['**/*.md'],
      sortBy: 'date',
      reverse: true,
    },
  }))  // no semicolon here! more .use statements will follow!

Now we are in the "process" chain issuing "use" calls for each step in the chain, delegating responsibility for each step to a module with some associated settings for that module.

In this case, "metalsmith-collections" is a module we are using to create a few lists: the first one grabs the 8 most recent posts in any section, and saves it to the variable "posts". The second statement, "planet", creates our Drupal Planet feed, which will get all posts in the /php/drupal directory, plus any posts anywhere on the site that have "drupal" in their title.

Note, I am specifically looking for ".md" files. I chose to make index.html files that are not processed by the markdown module to keep the indexes out of my collections. You can also use a path pattern to exclude "index.md" files if you wish.

One thing I really like about collections is that you can have a many-to-one relationship as we have done in the "planet" feed. That is super convenient.

4.4 RSS module

This module creates an RSS feed for our site's readers.

These filters should all likely explain themselves. They are just here to give us some additional nice things that we come to expect of our Drupal sites.

  .use(rss({
    feedOptions: {
        title: "verbosity.ca",
        description: "Blog posts by Ryan Weal",
        site_url: "http://verbosity.ca/",
      },
      collection: "posts",
      encoding: "utf8",
      destination: "rss.xml"
  }))
  .use(rss({
    feedOptions: {
        title: "verbosity.ca drupal planet",
        description: "Blog posts by Ryan Weal",
        site_url: "http://verbosity.ca/",
      },
      collection: "planet",
      encoding: "utf8",
      destination: "drupal-planet/feed"
  }))

This section was hard to build... I had to dig around for the right settings of the upstream RSS module to get verything right. I also had to patch one of the modules to get my rendered XML just right the way I wanted it.

4.5 Pretty URLS, pathauto, and input formats...

These methods will take care of how URLs are generated (or no generated) depending on some preferences we can set (we're not setting any). It is also where we convert any markdown files to HTML or do any other transormations like that.

  .clean(true)
  .use(markdown())
  .use(permalinks())

The aliases module is an interesting one - I forked someone's repo on github. Their module made meta-refresh redirects but I don't like the idea of doing so much bouncing around so I implemented a version that creates an nginx file which can be imported into the webserver config.

  .use(aliases())

That produces a file called nginx.conf in my root directory.

I include it in my Nginx server with the following syntax:

include /opt/static-site/nginx.conf;

4.6 Theme-related modules (see also next section)

  .use(dateFormatter({
    dates: 'date'
  }))
  .use(discoverPartials({
    directory: 'partials',
    pattern: /\.hbs$/
  }))
  .use(layouts({
    engine: 'handlebars',
  }))

These modules all get used for theming. Did you notice we are doing theming after all of the lists are built, and after we have rendered the XML feeds? That is on purpose. Everything runs in the order we specify and it wouldn't be very nice if our XML machine names were all formatted dates!

The "discover partials" module is very interesting. It allows you to create a folder called /hbs which can contain text files with .hbs as the extention. You can put anything in these, and inject them into your templates using the syntax {{>filename }} where filename refers to /hbs/filename.hbs. I use this in place of Drupal's "blocks" module primarily, but I also use it any time I wish to split off any string of text and have it load into the template dynamically.

4.6 Static Assets

This is very helpful if you have things like vlidation codes to place on the web server or assets like jpg and png files which should not have further processing done to them.

  .use(assets({
    source: './assets',
    destination: './'
  }))

4.7 XML sitemap

I had to customize this a bit to hide the shame of index.html in the result. Otherwise it was pretty easy.

  // fyi, I added a slash to omitIndex in mapsite!
  // prevents from rendering trailing /index.html in sitemap.xml
  // evenually I will do this a better way without hacks.
  .use(sitemap({
    hostname: 'http://verbosity.ca',
    omitIndex: true
  }))

Finally we are done with all of our .use methods... time to trigger the build process at the end of the chained commands:

  .build(function(err, files) {
    if (err) { throw err; }
  });

Now we're just making sure our assets are in place as described above and then taking care of the actual run process. That's it!

Step 5: Handlebars

The first templating engine I used with Metalsmith is the one used by the static-site example that comes with Metalsmith: Handlebars.

Coming from the PHP universe, it is very similar to twig. More specifically though, it borrows much of the syntax ideas from another JS project, Moustache.js.

Syntax

Moustache calls itself "logic-less templates" and I found that to be initially a bit difficult but after working with it a bit I really like it.

Handlebars lets you use any of the variables on the page as variables within the page. So for the title: parameter in your content source files will get rendered if you use {{ title }}. That will escape any odd characters you may have. If you need the raw HTML in that field, you can use the unescaped version by tripple-wrapping it. You will often see this done with the {{{content}}} variable that contains your body text.

Global variables can be used, and you can access anything off the "this" object that might be set globally. I used {{ locale }} and {{ this.altFiles.fr.path }} in one of my projects. Those variables are created by other modules... the point here is just to show you the syntax.

You can also do some basic iteration using the {{#each}} syntax. That is well documented elsewhere so we're not going to cover it here.

Debugging Handlebars

You can review what is in your "this" varaible to see what options you have available by doing the following in your template:

{{ log this }} {{ log this.title }} {{ log title }}

There is also a {{ debugger }} tag for creating breakpoints in your debug process. So far I have not had to use this as my problems have been fairly simple to figure out.

5.1 Using partials

If you have ever used "blocks" to define content in a CMS like Drupal, partials are your friend.

To use partials, we registered metalsmith-discover-partials module to the discoverPartials variable and set it to look for partials in a folder called partials.

That is all you need to do. Now, in your layout files, you can use partials as your own variables.

Here is an example. Let's say we want to make a block with some "contact us" info that will show on every page.

We go into the /partials/ folder in our repository, create a file called contact.hbs and put our HTML content in there.

Now we go to our layout file and we can use {{>contact}} to inject our content. Simple as that.

5.2 Writing and using your own partials

Many of the things we have mentioned above are actually "helpers" ... the log command, the partials we're injecting... all helpers. You can write your own, too, and you will want to at some point.

In my experience, any time you want an "if" statement in your template, or if you want to do something such as have a variable that defines a section of the site, you will probably find a need to write a small helper yourself.

It is important to note that you can't simply run JavaScript between the handlebars. If you are familiar with other JS frameworks this will probably bother you on some level, but get over it... this is very easy stuff to overcome and is done to benefit performance when you generate the site.

I made a language switcher for one of my projects. I have the {{locale}} variable already from another module, and I want to use that in my own code.

I created a folder called /helpers.

I added the package metalsmith-register-helpers in my index.js: var registerHelpers = require('metalsmith-register-helpers');.

And then I wrote my helper, named switcher.js in my helpers folder: module.exports = function (lang) { return this.altFiles[lang].path.replace(//index.html/i, '').replace(//$/i, ''); }

Now I can use it in my template like this:

{{ switcher 'en' }} // gives me a link to the English version
{{ switcher 'fr' }} // gives me a link to the French version

Nothing very fancy going on here. We could easily make it better but this gets the job done. I also filtered out the "index.html" part of the path while I was at it.

You can combine the syntax...

{{> (i18n 'menu') }}

What this is going to do is first between the parenetheses it will find a helper I created called i18n.js in my helpers folder. That is going to return a string, in this case, it will return "menu_en" (trust me on this, since I'm not showing you that code here). I happen to have a file called menu_en.hbs in my partials folder, so now the partials module will take over and inject that template.

The key take-away for me relating to handlebars is that you can't run plain JavaScript within the handlebars. Not when you're using {{ title }} which is a variable, not when you're using {{ >contact }} to import partials/contact.hbs. Even if you wrap things in parentheses you will still be running a helper, whether it be a built-in or one you write yourself. In these examples, "i18n" and "switcher" are helpers I created myself.

{{ log }} and {{ debugger }} as well as {{ #each }} are great examples of built-in helpers.

Step 6: Hide the shame (of .html) and gzip the files (in nginx)

We're using Nginx to host our files. We use the "try_files" parameter to mask the index.html if it is not provided in the URL. Currently we do not redirect users who type it in manually.

gzip on;
location / {
  root   /usr/share/nginx/html;
  index  index.html index.htm;
  try_files $uri $uri/index.html $uri/ =404;
}
// in step 4 we made a redirects file to include:
include /vagrant/www/verbosity-nginx.conf;

...we also gzip the content because text content compresses really well and saves substantially on bandwidth.

Step 7: Deploy!

Some day I will get fancy and add a webhook to my GitHub and/or Bitbucket repos that contain the files, have that hit an endpoint I define that will then check out the repo, build it, and deploy.

For now I don't need to do all of that. I simply rsync the files into place and then let my vagrant provisioning take care of the rest.

rsync -av --delete build/ /path/to/files/deployment/path/

After running this /path/to/files/deployment/path will contain everything in your project's build folder, and will also delete any posts that are no longer "live" since your last build.

Conclusion

If you have made it this far, congratulations! You have a node.js-powered static site and you can deploy it anywhere you wish!

Best of all, be have nice URLs, XMLsitemaps, RSS feeds, and all of that other typical things we might usually associate with having a CMS. If we were to take this a step further we might add a commenting service such as Disqus if we wanted to have comments on the site.

You can even do multilingual as I have done with my consulting site, http://kafei.ca. If you are interested in doing the same I recommend looking into the metalsmith-multi-language package and Belén Albeza's post about creating i18n metalsmith sites. That tutorial uses pug (formerly "jade") templates but the rest of the concepts are the same as discussed here.

Overall I really like using Metalsmith and I will continue using it for personal projects. I'm also looking into using Vue.js to generate static sites as both Vue and React now have many options for doing pre-generated as well as server-generated versions of apps built using those platforms. For most server-oriented tasks I'm planning to use Node.js to write custon JSON endpoints on an as-needed basis unless a suitable subscription service is available.

I am looking for JavaScript gigs. Contact me at http://kafei.ca about your project.

First, some background: I have been a developer for many years now but I was mostly doing PHP until a few years ago. I have been exposed to JavaScript for decades. I just wasn't very good at JavaScript for most of that time and for most of that time I have been calling myself a back-end programmer, but lately obviously I am working on the front-end again.

So with that being said... I dove back into JavaScript at a very complex time. The time when moment.js was in every project. When webpack was preferred for everything. When ES modules were just becoming a thing. I didn't even know how to include files beyond the classical script tag technique. I learned a lot when I was getting into Vue. Learning Svelte has been like a top-up to my knowledge of current JavaScript practices.

No window.*

In Vue.js and therefore also Nuxt, you do not always have access to the window object, or any objects outside of Vue, during parts of the Vue lifecycle.

This leads to many situations where you have to architect around the framework, or to simply defer those actions until mounted() which is when you will have access to the outside world. This can get quite complex when you are trying to statically render and have very specific requirements about when you want things to load.

In Svelte, it seems you just do the things you want. Framework doesn't care. Whether that is good or bad is up for you to decide.

There's a module for that

In the Nuxt community there seems to be a trend to solve the above issue (and other issues) by creating wrapper modules.

Wrapper modules distance you from the task at hand. What if you decided to re-implement some Vue code in Svelte? Do you know the API or just the wrapper module?

For the years I was working with Drupal I thought that modules were a good thing, and in many cases they are, but I think wrapper modules are another story. A wrapper module is when some random person (generally speaking) picks up a popular project and makes a wrapper module so it is "easier" to install or to use within your project.

I find that wrapper modules often get WAY out of date. For this reason they can be a huge security risk because it is usually not the original author doing the packaging, but some random person who may or may not care to keep up with the upstream/original maintainers.

This isn't really a Vue or Nuxt-specific problem but I see this issue creeping into many of my Nuxt projects. You can avoid this issue if you resist the urge to give preference to a community module simply because it exists (this is a very common policy when doing Drupal projects).

For similar reasons it is important not to choose a framework "because" it has so many modules. Saying that there are lots of modules does not address how many of them are actually useful for what you are trying to do. It could make things harder. I am of the belief that less is more when it comes to modules.

My experience of doing more JavaScript at work caused me to reject most wrapper modules in favor of working directly with a library. I suspect it has saved me countless hours in debugging time. I have applied the same philosophy to my legacy projects in other languages as well. Use what is fresh. Easily be able to re-implement the same concepts and code when switching frameworks without relying on quirky and/or stale wrappers. It will make your life as a developer better.

Loading things in the right order

What happens if you write code for, let's say, Google Maps. Let's say you do lots of custom code.

Your code runs fast, sometimes so fast that Google Maps hasn't loaded yet! What happens then? You could use a wrapper module for Google Maps, but why when you already have the latest version running?

I found out that in Svelte there is an on:load parameter you can use when adding any script tag, which will trigger a custom function. This function can do whatever you want! What a dream!

It got me to thinking, this problem has got to be trivial in Nuxt also, but why have I not found the solution yet?

The solution, of course, was easy: use vue-meta's callback option when adding a script tag. In Nuxt, vue-meta is implemented as the head() method. It works pretty much the same as Svelte and it does have access to the component's this object so you can use it to make changes to data(). Wonderful.

This particular thing made me realize that learning different frameworks is a really great way to increase your knowledge of the frameworks you already know. Seeing how other people do things really sheds light on how you can improve your applications with very simple adjustments.

Statically generating pages: what is going on?

Nuxt likes to abstract away a lot of the API-related business that you want to do with the site. It is a Good Thing™ but I find it confusing at times.

When I started using Svelte it all became painfully obvious to me. The generate process starts up a webserver, which serves dynamic pages, and then it literally downloads the output from that temporary website. I should have understood this when using Nuxt... and I did to a certain extent... but Sapper's bundling process made it painfully obvious as I had to write code for both of the contexts. It forced me to better acknowledge the separation of build-time vs. run-time code.

This helped me overcome a lot of weird bugs and mysteries that I never fully understood until working with Svelte.

These days Nuxt has a wonderful static generation system that allows for different API URLs for build-time and for run-time (in static mode). This is a huge help for static sites and for building within Docker, when you may not have the same API URL as in production. Svelte helped me to understand the build context better so I'm much less confused when I don't get what I expect out of Nuxt's generate process.

Running in "SSR" mode vs. Nginx webserver config

Having come from the world of content management systems I generally dislike the idea that the front-end code should be running on the server for every page request. I would much rather do a build and not have the back-end exposed to the public at all. For smaller sites it makes maintenance much easier.

Many people like to run Nuxt on the server rather than generate static pages. I find this a bit shocking honestly, but to each their own!

Nuxt handles both SSR and static cases well but some modules and/or plugins may only work in the SSR context, so that is something to be aware of.

Most of the modules that will not run in SSR mode are for doing things that you could handle on the webserver level. I have had to brush-up my skills on doing advanced Nginx configuration alongside my JavaScript adventures. I have been using Nginx for a long time and it has proven itself to be capable of some crazy configurations that I never would have ever thought possible.

Svelte proved to me that Nuxt's wide variety of configuration options is great for doing static-only sites but also having support for mixed-mode sites that try to hydrate after pre-rendering is a great option. Knowing that SSR is available as a fallback means I know I can adapt to any situation if/when things need to scale in ways I had not initially planned for.

Conclusion

Reading this you might think I am now all-in with Svelte and that Nuxt (and Vue) are therefore dead to me but that is not the case at all!

Learning Svelte has reaffirmed why I started using Nuxt and Vue in the first place. It also has me thinking about how I can better architect apps using Nuxt. I now have a better system for evaluating modules and more knowledge on how to keep things organized for long-term maintenance. Overall it was a great learning experience.

I will continue to use both Svelte and Nuxt in the future. I will decide which to use on a per-project basis and I will have some well-informed opinions on both when going into a project evaluation.

The rendered version of the site includes no JavaScript itself, and much care and attention was made to ensure I have clean URLs, a nice xml sitemap, rss feeds, and all the other typical blogging features.

Why am I moving away from a CMS?

• Since I am not in the habit of updating often it makes sense from a maintenance perspective.

• I wanted the site to be really fast, no matter where you live or how you are connected... it will still be reasonable on 2G connections in far away places. I had Drupal running fast as well for non-logged in users, but it was a struggle in some ways.

• I have been doing a lot of work on the topic of increasing site performance: from the DNS, to reverse-proxies/edge caches/CDN type of things, that extend what I was doing with Vagrant and Docker. It is amazing all the things you can do with Nginx!

• Comment spam was a maintenance nightmare, whether your strategy be proactive or reactive in nature. I'm not entirely comfortable handing over comments to a service like Mollom, nor do I want to feel obligated to reply or constantly be cleaning up spam. Lots of tools exist for managing these, and more importantly, entire sites like Reddit and HackerNews are much better discussion forums.

What about social networking?

In 2017 I'm trying to reduce my level of participation on big social networks. I still post on twitter, but my connections to pump.io and statusnet are long since gone... and I left Facebook even before that. I consider my LinkedIn account virtually read-only, so here we are. Back to blogging.

For now I will be cross-posting blog articles to twitter.

I plan to do more "in real life" social networking at JavaScript-focused conferences.

Why Metalsmith/node.js rather than $X static site builder?

I spent some time in the summer of 2016 "retooling" by doing small projects that brought me back into the JavaScript world. I have worked with Vue.js 1.x and 2.x on a couple "app-like" projects, but for this site I wanted to see how far I could get by making the final product be rendered files. This will hopefully help preserve some of the SEO juice that has built up over the years.

My first Vue.js project was my task-list tool Bulletpad which was the inspiration to my offline talk which I presented at Badcamp in Califoria and at DrupalCamp Northern Lights in Iceland. My second Vue.js project was an Electron app interactive kiosk touchscreen application. Neither of these projects resemble the idea of building "webpages" so choosing Metalsmith was a nice middle ground where I can use JavaScript and familiar tools, but to achive a different outcome (static files rather than an "app").

Since I was already spending a lot of time with Nginx on performance work I customized my configuration there to have URLs as nice as I could get (to match what Drupal does) and I also forked an NPM module to make URL aliases for my old paths - using this most performant Nginx syntax possible, to prevent a mandatory 301 when hitting directory paths. That is now on Github and NPM under the name metalsmith-aliases-nginx. It was my first contribution to NPM! The other redirection solutions involved multiple round-trips to the server. Tuning a static site on the server level is important...

Finally, the most exciging arhitectural feature: each page on this blog is fully self-contained unit! To load this page your browser probably only made one request. That is something that is near-impossible to do in Drupal (which, if you are interested, involves throwing out all of Drupal's CSS and JS, which is never fun!). Making the page a single-request should virtually flatten the loading times around the world since there is only one round trip to the server, and thus, the latency tax only happnes once (or twice, if you count DNS lookup, but DNS is cached regionally quite often so it costs less).

What next?

My plan is to get more JavaScript-related articles posted. I have some drafts in the works. One of the next posts will likely be a technical run-down of how I converted this blog over. I will likely still do the occaional post on Drupal but I suspect that based on my latest reasearch and general interest that there will be a greater focus on DevOps with things that compliment the work I do in JavaScript.

You probably won't find much more about Drupal Migrate here as I'm largely moved on (although I do still accept migration projects). For the time being I have made a "cheat sheet" that contains the most up-to-date configuration reference at kafei.fyi/migrate.

For a long time my team debated how we could structure our Vue.js applications so we could pre-render the entire site as static, but somehow with Vue.js baked in to make use of the router and other interactive features. It turns out Nuxt is this tool. It isn’t really “another framework” on top of Vue.js, but rather just a structure that is enforced by way of a script that helps you build the initial project with the most commonly used packages.

Things I love about Nuxt

What is cool about Nuxt is that we have found a lot of our Content Management System (CMS) use cases are easily met with Vue.js when we use Nuxt - whether that be a small chunk of the site or the entire site itself.

We have developed things that have nothing to do with the concept of “pages”... such as a search interface to Apache Solr (proxied via Nginx) as one example, and a donation app (landing page) that loads a 3rd party payment form for a charity we work with as another example. We didn’t have big plans to introduce Nuxt in these cases, it just made sense as we were working on it to use Nuxt when the need to use some of it’s dependencies started coming up. In one project we adopted Nuxt to gain the router. As a side effect the refactoring organized a lot of the code in a much better way. That could be more of a side-effect of having done something twice though.

If you know Vue.js, getting started with Nuxt is easy… you just run the setup command and then start putting your components into the pages or components directory, depending on whether the component is to represent a “page” (with a URL) or a chunk of data (ie, a component to use as a sub-component) respectively. Nuxt provides a hot-loading development server so every time you press save in your code editor it updates your browser with the new version of the things you are working on. When you are finished building, you simply run the build command and it packages everything up in a pre-rendered static form. Super easy. Super awesome.

When we pre-render sites we get great performance for general content pages and we can ensure the site follows good SEO practices. We sometimes go a step further and enable caching as an “offline app” (ie, a "pwa" or progressive web app) on the user’s device so they never need to visit our front-end server again! We can deploy the entire pre-rendered static site to a CDN.

The pre-rendering process also validates images are sized appropriately and also allows us to hook into webpack's pipeline with our own configuration if we need to. During the build process the code is minimized and if additional browser support is needed for IE and Safari quirks it tries to deal with it automatically. For the most part I never think about browser problems anymore and this is great.

A good place to start if you are looking into pre-rendering Nuxt data from an API is Christian Nwamba’s blog post, “Building server-rendered apps in Vue using Nuxt.js”.

Still, sometimes pre-rendering isn’t enough…

...and for those cases we sill pre-render most of the time, in some way.

For “live” data we cannot pre-render we make use of APIs to take care of any interactivity using Node.js (with Express) and Apache Cassandra. We choose these technologies because we can ensure zero downtime by having multiple replicas in different regions, and we can also we can scale by adding more servers. We can use similar hosting strategies to ensure redundancy/fail-over as we do with our static sites which is a nice benefit. Our build strategy overall is pretty close to what people have been calling the “JAMstack” with the addition of planning for offline situations and having good redundancy across the entire application stack.

I am having a tough time finding things to dislike about using Vue.js with Nuxt! Both are fun and motivating to use. It is a great time to be developing front-end web projects. If you’re interested in learning more about using Nuxt as a Vue.js developer my teammates and I are currently reviewing this Udemy course on Nuxt. It seems to cover most of the typical configurations we have encountered.

Myself and some of my teammates will be attending VueConf in Toronto in November. Hope to see you there!

This post is a summary of things I have been learning that have all turned into things I have been doing for work as a freelancer. Want to know what type of things are on my desk at the start of 2018? This is it.

"Front End" React & Vue ("web applications")

Transitioning between technologies is especially hard for a consultant who already has established business in another area but I have fumbled my way through:

• I started with a Vue.js project for an agency where my friend worked. They didn't care what the underlying techonlogy was, so long as we could build a kiosk with it. Vue.js was very fun to work with, and I will use it again. We packaged the project with Electron. Prior to this project I had made a little hello-world in Vue.js.
• Next up was a React project. I like to call this one a "pants-less drupal" project because "headless drupal" is all the rage in that community, but in this case we were making the editing - not display - side headless (don't do this, do the front-end with React too, and use Node.js or Go as the back-end). It was a complicated and big project. I learned a lot about using JS promises. I can see where React is more flexible than Vue.js, but I found them remarkably similar.

Prior to these two projects I had done an ES6 JavaScript course by Wes Bos, which was a great top-up for someone like myself who always "knew" a bit of JS but spent more time doing back-end programming in other languages.

I was introduced to Vue.js with Laracasts "Learn Vue 2: Step By Step" which is a great (not to mention, free) introduction to programming with Vue. For React I got going by watching a few videos on YouTube as many of the concepts were familiar from Vue. Eventually I paid up for another Wes Bos React course, "React for Beginners" and used that as a guide to audit our code before going live.

If you are thinking of taking on "front-end" JavaScript projects in 2018 I would focus my efforts on React (where the money is) or Vue. Angular is a good option as well. All of these make use of a "virtual dom" that makes things faster for users and more pleasant for developers.

If you just wrote-off that previous comment because you're not a "front-end" developer I have news for you: coding in React and Vue.js is more like application programming and at the end of the day you are still going to hand it over to your CSS specialist to fine-tune it. So don't avoid these technologies just because you are a so-called "back-end" dev.

There is also an entire category of "front-end" JS work that is powered by back-end node.js pacakages like webpack are great too. I am using metalsmith.js to generate this static blog site. You can find more about that in my previous blog post.

How JavaScript Has Changed

I have often mentioned that JavaScript has undergone a bit of a renaissance with the transition from "ES5" to "ES6" (otherwise known as ECMAScript6 or ECMA Script 2015). The language added many new features that make lives better for developers. It has continued to improve since then.

Since I had not been "focused" on JS for many years I struggled with a few points that look obvious now:

• import vs. require vs. script tag. This should be easy, but add in npm and bower (not to mention now we also have yarn), I found this difficult to differentiate which context and when I should use each. For now I'm using node.js' require because import is still too new when used in browsers. If using node.js to render static sites, require is used but then the final rendered output might get attached to a page using a script tag. Now that I understand it... It is obvious. Less has changed here than I had imagined.
• transpiling vs. polyfilling. One converts your beautiful new code into grotesque old-timey code with the potential for new bugs, the other loads a bunch of code in old browsers to make them capable of understanding your new code. I had not separated these ideas out in my head. They didn't really matter to me before though.
• legacy JS. Learning ES6 was great. Using ES6 is even better. Until you inherit an old project and start having to mix the old and new paradigms. If I never see another ES5 slideshow with jQuery and three other libraries it will be too soon. Plain ES6 will get you much further in less time and with less debugging.
• never click the link to "w3 schools", there are much better docs at "mdn" now.

Another worthy note about how JavaScript has changed: I feel like it is much simpler to manage code in a way that is not a memory-leaking disaster. Maybe I've just grown older and wiser in this department though.

Onward: All the Things!

As 2018 begins JavaScript has started to take over other parts of my developer life:

• I'm using JS to perform content migration work and audit those migrations. Most of these are currently in-house node.js scripts but they are starting to take shape as potential npm packages.
• I am working on a node.js API service in my spare time. Finding this to be valuable as a dojo experience, but I have plans to release it to the public at some point. It will replace some of my most complex projects when it is ready.
• End-to-end integration testing and QA work has started to hit my desk more often. This caught me by surprise... but "Headless Chrome" is fantastic and lets you code your tests in ES6/7/8 syntax. I'm using puppeteer and mocha with chai as my test running solution for now.

The neat thing about that last point... well, actually many of these points is that these tasks situate me in such a way that I can keep working "around" Drupal and PHP communities where my existing clients are as I learn new things and take on new JavaScript projects.

formapi_ajax_node.pdf

formapi_ajax_node.odp

Over the past two years I have been a big fan of StatusNet, the social software that is open source and works similarly to twitter. On this blog I have documented my experiences running the software on my own server and it has been a fun time.

Earlier this year Evan, the founder of StatusNet, announced that he had rewritten the engine of the software with JavaScript on node.js.

I believe this was a smart choice for this software as the Apache, PHP and MySQL parts of the StatusNet stack are incredibly heavy. Evan had been working on this new JavaScript rewrite for about a year before announcing it, and I have been running it in some form since January when it was announced.

In June I finally put my instance live at comn.ca. It had the unfortunate consequence of topping out the memory on my server so I decided to shut down my StatusNet instance (see graph attached).

The largest StatusNet site, identi.ca, is due to be converted to pump.io any day now. When that happens the pump.io network will overtake StatusNet. The PHP-based StatusNet will live on under the name GNU Social (new updates to come soon).

Some of my contributions to the StatusNet community were on my SN server. If anyone would like a copy of the themes I made please reach out to me.

If you have a pump.io account please follow me at my new instance: https://comn.ca/ryanweal

... want to try pump.io? This link will give you a random server that is accepting users. Don't like the domain? Just hit that link again and you'll get a different one.

For those interested, I have recreated my Chinese language flash card bot. It posts random words every 10 minutes.

PS. The attached photo shows pump.io AND StatusNet running concurrently, until I turned off SN. You can see the result... pump.io is smooooooth running.

PPS. I hope more Drupal folks end up on pump.io than were on StatusNet. I still find it hard to believe the Drupal community uses twitter rather than open source messaging platforms...

Puppeteer, for those who are unfamiliar, is "headless Chrome" (or more specifically, chromium, the open-source version of the Chrome web browser). We are going to use Mocha to run Puppeteer, and Chai to test some expectations on what we should be seeing inside puppeteer.

I have used a variety of test frameworks in the past, particularly in PHP... and I was never really very interested in it. That changed due to some of the reasons I enjoy puppeteer which you will see documented below.

Testing the happy path...

Before we get any further with this it is worth noting that end-to-end tests are typically very brittle so we are going to focus on testing the happy path which is what we think the user will be doing in the majority of cases. Following this, we will add additional tests to catch the edge cases we may not uncover until we work through some of the initial workflows of our site.

With that in mind, if you want to test a site that already exists and doesn't have any tests - you are in the perfect place. Puppeteer is perfect for making tests for projects that are already live because you are using a standard browser to run the tests.

Puppeteer: what is so good about it?

Some of the key features of Puppeteer set it apart from the other tools out there, here is a quick summary:

1. Very simple installation

• A specific chromium (open source Chrome) version gets downloaded with the test suite. You will 100% know what version is running the tests, always (especially if you hard-code the version in package.json). For this reason it is usually good to make sure your node_modules folder is in your .gitignore file to prevent clogging your repo with a big chromium binary.

  Your teammates will simply "npm i" and then they can run your tests. Zero config. Beautiful.

2. Less limiting than other libraries

• The request library is often used for automated testing, but it is limiting. Having the full browser at your disposal allows you to test for UI elements that may not render correctly, delayed, or not at all in headless environments.

  Puppeteer also has an option to turn headless mode off if you distrust "headless mode" which will allow you to watch the tests in real time.

  So unlike with many other tools, you can fully test Vue.js, React, Angular, and any other dynamic site. You just "wait" for selectors to appear at the right time and then test it like any other context (note: waiting for selector can be a bit flaky so I often wait based on time).

  You won't be installing awkward add-ons for basic functionality. You get it all by default.

3. Streamlined use of selectors

• These are the selectors you know... from JavaScript. You can query any object, and then perform actions on it (click, touch, keyup, etc). There are actions described throughout the API that will help with this part.

If you are just learning these skills in JavaScript they are all immediately transferable to other forms of work. As a business owner, I love this.

After getting a few tests written you will see that they are very
repetitive. Your workflow gets down to using dev tools
inspector to select an object, right-click the element, choose Copy, and
Copy Selector, and then dropping the result into your boilerplate
pattern. Super easy. The rest is just mastering the basics of (ES6+) JavaScript.

• It is very quick to begin writing tests. Most of the time you just need to find a selector on your rendered content that points to the thing to click/touch/etc, and a second selector (or more) to test the result of what you want to happen. You can find these very simply by opening the Chrome inspector (Control-Shift-J on most systems), clicking the arrow icon in the toolbar, and selecting the HTML element you want to get the selector for. Finally, when the selector is highlighted in code, you can right-click on that, choose "copy->selector" and then paste that into your test.

4. Async bonus

• A side effect of writing tests with puppeteer is you will become very aware of when you need to be using async and await, as well as promises, and for me, especially Promise.all(). These things will become second nature to you after awhile and the knowledge will benefit you when coding other JavaScript projects in the future.

Remembering what we are not testing...

Finally, before we begin it is important to note that Puppeteer is Chrome-focused. It shouldn't be used for browser compatibility testing... for that you should keep using tools like Selenium which automate the interaction across browsers but use an abstraction layer to do so, rather than native JavaScript.

Things you can not test:

• The output of a generated PDF... you can click a link, open that PDF (it goes into a new tab) but I haven't been bothered to try switching context to that other tab to do anything with it. PDFs are a known limitation.

• You can't test OS-level features. On MacOS, but not other operating systems, "Select All" is done by MacOS. So you can't use "select-all" behavior and expect that to work on any environment.

Getting setup (package.json, .gitignore)

Okay, enough talk. Let's get setup with some defaults so that our team can begin writing puppeteer-based tests!

Requirements:

• package.json (generate one with npm init)
  • install our 4 packages: puppeteer, dotenv, mocha, chai
  • edit the file to add the following:
    • scripts pointing to the files
    • relative path to mocha binary (so users do not need to install puppeteer globally)
• .gitignore to prevent committing the browser in node_modules to git
• tests folder with your tests

Bonus points if you make these all "dev" requirements so they do not roll out into your production scripts. In my case this package was not tied to another application so there was no distinction between dev and prod.

Example package.json file:

{
  "name": "mocha-tests",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test-courses": "node_modules/mocha/bin/mocha tests/courses.js || exit 0",
    "test-content": "node_modules/mocha/bin/mocha tests/content.js || exit 0"
  },
  "author": "Joe Somebody",
  "license": "GPL-2.0",
  "dependencies": {
    "chai": "^4.1.2",
    "dotenv": "^5.0.0",
    "mocha": "^5.0.0",
    "puppeteer": "^1.0.0"
  }
}

This package.json file does the following:

• It allows us to "npm install" to get all the dependencies for the tests, including: chai, dotenv (we use this for variable storage), mocha, and puppeteer).

If you used "npm init" to create the package.json file, you can add our dependencies like so:

   npm i puppeteer dotenv mocha chai

• You will need to manually create the two "scripts" entries that are in this case project-specific. After installation users can simply run the following to test the site's "courses" tests contained in tests/courses.js:

  npm run test-courses
  

  ... and that will run:

  node tests/courses.js, using the puppeteer browser in node_modules

  or alternatively, run the content tests from tests/content.js:

  npm run test-content
  

  ... and that will run

  node tests/content.js, using the puppeteer browser in node_modules

Example .gitignore file

The .gitignore file will keep the puppeteer browser from getting committed to your project repo, saving a ton of potentially wasted space in your repos.

node_modules/
variables.env

Creating a test template (tests/content.js, variables.env)

• Create the file tests/content.js file for some tests:

  // load your custom settings from your variables.env file
  require('dotenv').config(
    {path: __dirname.substring(__dirname.length -5,0) + 'variables.env'}
  );
  
  // dependencies
  const expect = require('chai').expect;
  const puppeteer = require('puppeteer');
  
  // pointer to our browser tab so it will persist between tests
  let page;
  const host = process.env.MOCHA_TEST_HOST;
  
  // the test suite
  describe('My test suite', function () {
    this.timeout(10000); // Useful when testing really slow Drupal sites
  
    // open a new browser tab and set the page variable to point at it
    before (async function () {
      global.expect = expect;
      global.browser = await puppeteer.launch( { headless: false } );
      page = await browser.newPage();
      page.setViewport({width: 1187, height: 1000});
    });
  
    // close the browser when the tests are finished
    after (async function () {
      await page.close();
      await browser.close();
    });
  
    // @todo tests go here!!!
  
  });
  

• to run it, go to the root of your project folder

• ensure there is a variables.env file in that folder

  • the file contains: MOCHA_TEST_HOST=http://example.com

• then run "npm run test-content" per the name you set in package.json

• the test will load the browser, pop open a window, and then close it because there are no tests yet! No failures == success!

If you cannot run the tests:

• Run "npm install" first.
• If that doesn't work, make sure you have node.js installed.

Example variables.env file:

MOCHA_TEST_HOST=http://example.com

Command to run the tests:

npm run test-content

Set "test-content" to run a specific file in your package.json.

Sample tests (add to tests/content.js)

These all go in your tests/content.js file, where we left that @todo in the example above.

PS. You can nest the describe/it blocks if you want!

1. Check the page title.

The first example we have just looks at the metadata coming back from puppeteer to check the page title:

    it('homepage loads and has correct page title', async function () {
      const [response] = await Promise.all([
        page.goto(host, {timeout:0}),
        page.waitForNavigation({timeout:0}),
      ]);
      expect(await page.title()).to.eql('Crazy page that takes a lot of time to load');
    });

This test is nice and quick because we aren't parsing any HTML.

2. User clicks the checkout button

This one is part of our previous example.

Note that click and waitForNavigation can happen in any order. We're not doing anything until both the click and the load of the next page are complete.

      it('proceeds to checkout', async function () {
        const [response] = await Promise.all([
          page.waitForNavigation({timeout:0}),
          page.click('#edit-checkout'),
        ]);
        expect(await page.title()).to.eql('Checkout | My Awesome Webstore');
      });

3. User logs into the application

This example loads the login form for a CMS, inputs the data, and submits it.

      it('can login', async function () {
        await page.goto(host + '/user', {timeout:0});
        await page.type('#edit-name', 'joesomebody');
        await page.type('#edit-pass', 'password1');
        const [response] = await Promise.all([
          page.waitForNavigation({timeout:0}),
          page.click('#edit-submit'),
        ]);
        let result = await page.$eval('.field-group-htabs-wrapper', e => e.innerHTML);
        expect(result).to.include('Add a New Class');
      });

The test passes if the user successfully logs in and sees some HTML with the class "field-group-htabs-wrapper", and if inside that class there is a string 'Add a New Class' that is present. If there are many HTML elements using this class, the first one is used for the test.

This test can be somewhat problematic because we are actually doing two page loads within the same test (first for the /user page, then for the "click submit" to come back). Keeping your tests to one request will make things smoother, and alleviate the need to explicitly set timeouts everywhere. You probably won't run into this problem outside of slow CMS testing though.

4. Just force it to waitFor a second...

This is a very brittle example but it does a few new things. Rather than waiting for a page to load, it simply waits 1 second (1000 ms):

    it('selects an item from a slow AJAX select field', async function () {
      const [response1] = await Promise.all([
        page.click('#edit-field-class-level-und'),
        page.waitFor(1000),
      ]);
      const [response2] = await Promise.all([
        page.keyboard.press('ArrowDown'),
        page.waitFor(1000),
      ]);
      await page.keyboard.press('Enter');

      // @todo we should have an expect here, to test for a result
    });

In this example we also make use of the virtual keyboard that puppeteer provides. After one second we presume that the AJAX content has loaded, press the down arrow, wait one more second, and then press enter. We don't actually test for anything in this example, so this would "pass" regardless of what happens unless we add an "expect" command in there.

Creating tests is now pretty much like this:

Now that you have seen a few examples you can simplify your workflow by using the provided examples.

To start a new project:

• Run npm init, create tests folder, update package.json scripts. Add a gitignore file.

To create a new set of tests in a new test file:

• Use the example above or copy an existing file in /tests and then update package.json to add it into the "scripts" list.
• Populate variables.env with any new re-usable settings

To create a new test within an existing test file (such as content.js above). This is mostly a copy-and-paste adventure now:

• copy 5-6 lines of code from elsewhere in the file (or from examples above)
• paste the lines of code elsewhere in the file
• update the "it" line to describe what you are testing
• update the "expect" line to reflect the result you want
• update the selector(s) you will be using in your test:
  • in chrome or chromium, go to the page you are testing manually, and use the inspector to select what you want your test to click,
  • right-click on the selected element (ie, the code in inspector), choose "copy->selector"
  • paste that over top of the first selector in the example code

In other words, to find a selector: when you are looking at the page in chrome, press control-shift-j to launch the inspector, click the arrow in the corner, then click on the element on the page you want to use for your test. After clicking, the inspector will highlight some HTML in the inspector window. Right click that HTML. Choose Copy->Copy Selector. Paste that into your test.

Occasionally the test selector is too specific. I have found if the selector is very long I can often take elements out of the middle... makes it generic enough to target the first element. Depends on what garbage HTML you are dealing with. More often than not simply copying over a selector "just works" but sometimes you will have to edit them.

Things to watch out for

• Waiting for a div to appear using waitFor didn't seem to work so we stuck to testing based on time or waitForNavigation (page load). Not ideal, perhaps it has been fixed, not sure what the current status is.

• Many requests will run in parallel unless you explicitly prevent them from doing so. This is why I provided some examples with Promise.all, and also with sequential use of Promise.all.

• Timeouts are expected to pass quickly. For this reason it is often simpler to test one page at a time, rather than loading multiple pages within a single test. It can be too granular at times to have to do that and sometimes you won't have a result (or even need one).

   // for the times when you have a few tests to run
   // in sequence, but while they are running, there
   // is not really anything to test.
   expect(true).to.eql(true);
   // or maybe we just didn't write the test yet:
   // expect(true).to.eql(false);
  

• Delays are sometimes necessary for text input. It also makes it more fun to watch if you disable headless mode. Some captcha systems test for "input too fast" so delaying it might be necessary in many cases.

• "Select All" is an operating-system specific function and will differ on different platforms. May not be testable.

• PDF rendering cannot be tested in great detail.

Refactoring ideas... Puppeteer as a migration source!

It should be pretty obvious that all of this could easily be a web scraper rather than a test suite. My team has been making use of Puppeteer in other contexts to extract data for extract-transform-load migrations. Puppeteer provides a quick way of getting a real-life rendered DOM with a consistent use of selectors. Huge time-saver when the alternative is doing dozens of database joins.

Most of my work these days is actually in Vue/Nuxt JavaScript universe but I decided to use Svelte for this project for a number of reasons:

• The default template is nearly perfect. Oriented toward good typography, out of the box.
• Using rollup.js rather than webpack by default is very fast!
• Sapper, the site generator, makes it very clear what code is running during generate vs. what code runs for the user at runtime.
• No more crazy "custom link" code, such as the nuxt-link tag. Just use anchor tags.

These reasons all amounted to Svelte being "good enough" to give it a try and it was overall a very pleasant experience. Things did take longer than I expected, as they always tend to do, but this time it was all based on foward development rather than having to "undo" presumptions that came with the framework.

That is a fancy way of saying "I did not get stuck on anything" and "I did not give up because I was confused"... everything was very clear and I only took awhile to finish it because I wanted to give it the extra love and attention to make it the perfect fit for what I wanted.

Where I made customizations

• Most of the content is imported from Markdown
• Each section of the site has it's own source folder and outputs "category" RSS feeds
• Every post has a comment form that sends me an email (no public comments, sorry, haha!)
• All posts have a donation option!

The donation option was interesting - I ported an entire app from Vue.js to make this possible. It made things take considerably longer but I have a much deeper understanding of the framework having done this.

One cool thing about the donation workflow - it was very easy to include external scripts and work with them any way I liked. So rather than have an EU Cookie Compliance banner I was able to not load any cookies at all until you click the donation link. So the donation link itself is the cookie consent. Up until that point no cookies are served so no need for the cookie banner! Yay!

Some experimental things

• I provided a link to a service that outputs the RSS feeds to your email. This means I do not have to manage any mailing lists. I don't know if the service is any good, but it is worth a try!
• Custom infrastructure... I always self-host, even though it is extra effort. This time it is fully dockerized.
• Serverless functions. The comment and donation workflows both require some server processing. I wrote these as though they were serverless functions that can run on a service like Vercel... but then I also wrote an API loader that is similar to Vercel so I can run the same functions on my local infrastructure.

These exeprimental things mostly help me manage the site as an individual without getting into too much overhead. If I need to change any of these things it will be easy to adapt them to use other platforms/technologies and that is the most exciting part of all!