We run a few blogs out of my office which has led me to have some criticisms of tagging in general. The first lesson that I have taken away is to keep the format your tags consistent. In my del.icio.us bookmarks I use all lower case words for my tagging for consistency, while on the blog I'm trying to reinforce proper capitalization of words to bring it more in line with news media. Many blogs I see have a mixture of both and it appears unprofessional to the casual viewer visiting the blog for the first time.

Another strategy that I am keen to embrace is to limit the number of tags. I hinted in a previous post but I think more needs to be said. Some bloggers will add new keywords each time they post an article but this quickly gets out of control. One particular entry in a blog I read has only one story about religion but a tag was created so this is visible on every single page. The author may not realize this, but the post challenges the blog's credibility in other ways when the general topic of the blog has nothing to do with religion. To avoid these mishaps I think the best strategy is to limit yourself to a set number of tags. If something doesn't fit, cut the article as it will deviate from the defined scope of the blog (or tag it with something more meaningful).

Since we're now in the business of defining our tags I think a further step is needed in blog functionality that has been missing for some time. Tags as landing pages. Yes, that's right, when you click on a tag you should be taken to some kind of customized section. Perhaps a definition of the tag, what it means to you, it's history and why you use it on the blog. This makes sense when you think about the tags as "categories" of information. Users will rejoice that your tags will become research topics for them and the SEO (search engine optimization) benefits will be great too: no longer would search engines ignore your tags but rather see them as individual pages worthy of indexing. The thought of that sounds enticing.

Why ecartis as opposed to mailman? It was a tough choice, but it basically came down to the configuration steps required on Debian. Mailman needed to be recompiled or something like that or I would have had to redesign a bunch of webpages to get it going. Ecartis just works once you know the system.

Let's update your mailing list

You need to interact with ecartis by sending it email. You can be either a manager or a user, it doesn't matter. Your commands will be different depending on who you want to be.

Letting users take care of things themselves:

To add or remove yourself from a list you must email ecartis with the word "subscribe" or "unsbuscribe' from the account you are using. The email address to send to is often -request@

Subscribing

• Draft an email to ecartis
• Put "subscribe" in the subject line
• Send the message

Unsubscribing

• Draft an email to ecartis
• Put "unsubscribe" in the subject line
• Send the message

For administrators:

Your commands will be the following, email them to ecartis. It will take these values and send you a ticket you must approve before the changes are made.

# always start with admin2 and the
        # list name

        admin2 testlist

        # then manage your old users,
        # changing all their settings around

        setfor user@site.xyz MODERATOR
        setfor user@site.xyz CCERRORS
        setfor user@site.xyz REPORTS
        setfor user@site.xyz ADMIN

        # a command structured as these unset
        # commands will assume the last email
        # from above will be the same here

        unset CCERRORS
        unset REPORTS

        # we can also add people to the list
        # or remove them.  bye joe!

        unsubscribe joe@example.com

        # the rest of the commands for
        # managing users, use them as the
        # commands above are used

        NOPOST 
        DIGEST
        DIGEST2 (digest & the rest)
        VACATION
        ECHOPOST (self copy)
        HIDDEN

        # say goodbye to ecartis!
        adminend2
        

Within the evening I had unintentionally upgraded my computer to Debian 4.0, the latest version which until now had escaped my notice. For the Windows and Mac users out there, that's like a major upgrade (like from XP to Vista but without the scary hardware considerations).

A few things broke but I was able to fix them all really quickly. Though I had to change some configurations I was well aware of which areas of the system that needed the changes and I'm already working away and enjoying the new stability.

Another bonus to this new release is that my Wacom tablet finally works on Linux for PowerPC. Previously I had only been able to use it with my Mac desktop (which is extremely slow) or with a Windows computer (which I only use at work). Now I can have my cake and eat it too.

This laptop is long for this world, the display and the hard disk each routinely fail. I doubt it will survive until the next Debian upgrade. To those who contributed to the release, thank you. Once again your efforts have gone far and beyond my expectations. My next PC will be running an x86 chipset but for now my PowerPC is still going strong. Nice.

Justification for buying two computers is easy in my world. I'm always telling people about some fun things you can do with servers but it's hard to demonstrate when all your important files are just sitting there waiting to be corrupted by a demonstration. A second computer would be great for experimentation with new software and if I get bored with it I can always find a home for it, after all, it's so little. Even as a simple music player it would serve a very fitting purpose.

So when the crashes started I was a little taken aback. I had plans for this system and suddenly I wasn't so sure. Memories of computing past came back to haunt me. That time I bought a multimedia kit for my 486 computer which never worked at all, the second one I quickly killed by crossing some wires, and so on. It happened more recently with my Blackberry which promises "instant mail" but takes up to 15 minutes to arrive (if you're lucky and the servers aren't overwhelmed like usual). Promises not delivered for one reason or another and unfortunately all to common in the technology arena.

Once again my high hopes were quickly shot down.

Then I thought about the purpose of this adventure. It's to help me get my business off the ground. By September I was hoping to be in the late planning stages. So far, still on target. This computer is a serious threat to my productivity though. With this in mind I looked across the room and suddenly that spare computer I bought never looked so sexy.

So this is round two. I'm now up and running on the second computer with my fingers crossed, hoping for the sake of my timelines that this one will do the trick. Sure, I won't have that spare tire lying around anymore but isn't this exactly what a spare tire is for? In this case, it certainly is.

Key to the success of any list is a general understanding of how and when your audience uses computers. For those of us who work day jobs this is relatively simple. Catch us during working hours, Monday through Friday, and you've got our attention.

When to send

• Monday morning review. This is the best time to have a great impact with your readership
• Tuesday individuals. Contact people individually to get the buzz going
• Wednesday business. Get whatever it is you need to complete done today Why not?
• Thursday event. This is a very reliable day to host any event, it beats NBC sitcoms
• Friday wrap up. The thank-you letters must go out now, or they won't go out at all

Distribution methods

• Mailing list. This will make managing the list simple. Web forms are ok too.
• Blog. The common man'z way of learnin stuff.

This computer will replace my server which I built in 2004 out of spare parts from around Vancouver. Back then I just needed something that I could churn out resumes with while my laptop was in repair. It's still going strong and now manages my entire media collection and all of my communication related archives.

The new unit will sit on the book shelf as opposed to on the floor and should be much quieter than it's predecessor. I will also be purchasing an LCD display so I can retire my laptop.

The other neat feature is this post, which was originally an email from my wireless device. What fun!

I was slightly smug that day as the night before I had emailed the trainees and encouraged them to download the materials in advance, and most had done so! Those who did not were able to use Pantheon's hosted environment as their DNS provider seemed to be unaffected where we were.

We carried on. The training went smoothly.

Throughout the day I kept thinking to myself... I run a DNS. I should probably do a full review of the aftermath of this incident. So I did...

What should we be doing for DNS

Later on in reading about the aftermath I read a comment about DNS servers. Most people have their websites hosted by two DNS servers. I did the same... no problems here, right?

However, there were problems... I noticed them first when travelling to South America a couple years prior. The first request to my website on any network seemed to timeout, and on second try the DNS would resolve. At the time my servers were in Canada. So I moved them to New York. Problem solved. Or so I thought...

A couple years later I was in India. Guess what happened? More or less the same thing as in South America, but it was possible on some networks to get a reply on first hit. Whenever I got a request it was slow to arrive.

Back to 2016 in California I was surprised to learn that a lot of people "in the know" on these matters recommend you have at least THREE (3) authoratiative name servers for your domains!

There was also a suggestion by some commentators during that huge DNS outage in 2016 that having diversity within those three servers would be a good idea as well. Both geographic and also TLD-diversity.

Knowing all of this, I knew I could be doing better on DNS. I created a modest goal: improve the reachability of the site; to increase resiliance to network failures.

Time for some tests

Simply testing connection speeds from where I live in Montréal to New York wasn't going to do the trick. Ping times are typically really fast. What I needed was a way to measure global latency. There are a few providers out there but I settled on Maplatency.com as it seemed to do what I needed and the DNS tests were free.

The tests confirmed my experiences travelling internationally. For people close to North America thigns are good, but once you start crossing oceans - and multiple oceans - the picture changes quite a lot.

Improvement 1: a third DNS server

The next step was to add a third DNS server. My existing servers were in NYC and San Francisco so I added one to Europe (Frankfurt) as that is my next most visited place in real life.

Huge improvement already. Wow. 62 users are getting the response within 50ms (up from 53 users before), and now we have 88 people getting a response (up from 71 before).

Note that I'm not doing anything fancy here - no load balancing, no geo IP targeting, just simply adding an IP address to the pool and letting the magic of the Internet work the way it was designed.

Improvement 2: a fourth DNS server, on a different TLD

Following the advice of commentators I thought I would setup a new domain (on the ".global" tld in this case) to "diversify" my DNS in case the top-level domain that I normally use goes down. Simple enough to do... but I forgot about how to add a custom DNS server to my registrar to "authorize" a DNS running on that domain. Minor detail, but I have been running DNS for so long I had forgotten about it.

Once I figured that out we were in business. I used MXtoolbox to check the results and made changes until everything synced up nicely.

Time for more tests:

Wow! That is an amazing improvement. Now roughly the same 62 people are getting the request within 50ms, but there are so many more users now, 99 people are getting replies (the progression was 71->88->99).

I compared this result to one of my Cloudflare-powered sites and although Cloudflare preforms slightly better at this task the chart is looking very much the same. Not bad for a first attempt!

Huge win for very little effort. I won't have any timeouts on the first request when I travel anymore.

If you're still not willing to host your own DNS, use a service like Cloudflare or Fastly to speed things up. They will likely make you publish only 2 nameservers but they have other magic to achieve similar results.

When you're ready to go to the next level I recommend reading "Surviving the next DNS attack" to get more insight into tuning DNS.

Quite often the project becomes overwhelming as you realize all of the features that have been added to the site over the years. A lot of things happen when a site is in "maintenance mode" and what seems like a small project can quickly become large when this is factored in.

A common question we were asked in the 2010s was "can't we just go part-way with this, upgrade one feature, and move onto the next" but in the early 2010s most CMS software would not allow this approach or would make it quite difficult. Each few years a new major version would come out and force a bunch of re-working to occur.

Enter the Microservices Era

In the late 2010s a new trend started: to split up the functions of a website into "microservices". This essentially means dividing up the functionality of a site into logical parts and dealing with them separately - either embedded in the page or hosted on a different domain.

The main idea is to isolate the development and testing of that "component" so that it is easier and less risky to make changes.

So what is an "app" exactly in this context?

For a typical brochure site you may have something like this in terms of "apps" running the site:

• Contact form
• Shopping cart

...but the rest of the site is largely static.

A microservices architecture would suggest a collection of apps that are presented together as one website. Our collection of apps for the above site would be the following:

• a CMS application - used for managing content and visual assets
• a static front-end application - used to generate static pages for content
• the contact form app
• the shopping cart app

The benefit of rendering a static site is that if any of these other services goes down the users may not even notice. CMS applications often require frequent security updates so rendering static files from the CMS acts as an extra layer of security by not exposing that part of the application to the public at all.

Another benefit of this strategy is that we could swap out different components of the site for different solutions, including commercial options. So unlike getting WooCommerce because you are using Wordpress, you could alternatively use a service like Snipcart or Shopify or many others.

Faster Dev With Some Additional Complexity

As you embrace microservices you will start to enjoy that each of your little "apps" is now self-contained and much smaller than the original beast of a project you were dealing with before. It is all rainbows from here on out, right? Well, not really. As is usual with programming tasks you are really just choosing to have a different problem to solve.

In this case the new problem to solve is how to make sure everything is always running together and running well.

Testing that the website is "up" is a great starting point. That might only test your static files though, so it is also important to note if our shopping cart widget loaded. What happens if it fails to load? Do the users see the contact form? Is everything available and functional?

All of the individual apps need to have their own tests to ensure they do what they are supposed to do. An additional layer of integration tests is necessary to ensure that the different apps are working together in the way you want them to.

Keeping Good Records

Once you have all of your tests in place things are going to start feeling really good and you're going to be able to iterate quickly on new challenges and deploy faster due to less scope of testing when you have made a change. That's great! With automated tests this gets even better.

What happens when you try to onboard a new team member though?

If you were doing CMS-based development there would be a "typical way" things are done and that would enable developers to get to work quickly without having to put much thought into the architecture. If you are developing multiple apps from scratch things can go awry really fast when a new developer isn't aware that part of your app already does something that they are in the process of re-implementing.

The way around this issue, of course, is to write documentation. Specifically to write good requirements even if it means documenting the software after it is written. It should be factual and to-the-point so other developers can rely on it to answer questions without involving the project architect for each change.

Some projects may opt to keep the documentation as part of the code repository, while others may want to keep it internal to their team on some form of wiki. Either way, getting a high-level overview of your project "in writing" will help you work with other developers so they can understand the intention of how your project is supposed to work.

Another tip on the documentation front: keep a changelog. For some of our smaller projects we manage the changelog as more of a "monthly report" that captures requests and bugs. This will help other developers see where project is being made so they can focus their energy appropriately.

Potential for Cost Reductions

One final word about microservices... as you embrace microservices your hosting costs will likely start trending downward, perhaps all the way to zero.

Going back to our list of services our site offers:

• the CMS (no longer serving data to the public - therefore could use "free" tier of some hosts)
• the front-end static rendering process - could be generated by a continuous deployment pipeline, some of these services offer a free tier as well, and deploy the files to a free static host like GitHub pages or others
• the contact form could use a hosted service (formspree.io, etc)
• the shopping cart could use a hosted service (snipcart, shopify, etc)

For those last two you might choose to pay for a service. Or perhaps you will set it up in a smaller more focused hosting account. Or... perhaps you design that part of your application to run once and be done, so you could make a "serverless" script that processes what you need which can cost a lot less than managing to run a server 24x7 and having to make sure it is always running. The best part of this is that you get to make the choice of what strategy will be best and if it isn't working for you it will be possible to swap out that one part for another service without disrupting the things that are already working and which you are happy with.

• Sender Policy Framework, or SPF, which is an open standard
• Sender ID, a Microsoft variant that tries to do what SPF does, even going as far as copying the syntax and calling itself "SPF2"(sigh)
• Domain Keys Identified Mail, or DKIM, a variant pushed by Yahoo which requires each message to be modified before it goes out, and lastly:
• DomainKeys, the original Yahoo creation that spawned DKIM

Whew. That's a lot of stuff to configure. If you go through the process though you will get less "false positives" going into your recipients junk mail folder so it is worth the effort. If you are sending out newsletters, these technologies might be a vital step to increase your open rates.

Who is using email authentication? I know almost all webmail service providers use at least one form of email authentication to verify incoming mail. I know of some offices with off the shelf firewalls that also check this information as it comes in. What is surprising though, is that many email marketing companies, the ones who send things out, such as Constant Contact and Cvent, do not provide a means of authenticating email. This is probably why they are so cheap compared to providers like Responsys which publish their email authentication records so their client messages get through to all of their subscribers. You can test your configuration by following the verification steps on this page or by sending an email to a test server which is also discussed in this article about SPF records. For a server without any of these services you should always get "neutral" as a response for each test. If you have these services configured, you should see "passed" for each method you setup. Currently I have implemented Sender Policy Framework and Sender ID for all of my hosting clients. These two are the most widely deployed and also the most easy to implement. If you could only do one I would recommend Sender Policy Framework as many services should respect those settings as they are similar to Sender ID. If you absolutely must get into Hotmail inboxes I will always recommend both. I am not certain that DKIM & DomainKeys will last due to the complexity of the setup. If they do last awhile longer I will probably add these methods to the server too, but for now I am taking a wait-and-see approach.

The setup was relatively easy, I checked out a few websites, figured out the easiest, most debian of ways to do this. Kwiki was my answer. I checked the features: minimal. This is software zen. Add what you want, or take nothing more.

What's less is more

Years ago I became really excited by the wiki concept, utilized it to do a few specific work-related tasks and moved on. Using it as a document management system is a creative person's delight. Whenever you need a page, JoinWords with capital letters and it's there. You also get a save button, edit button, and rudimentary formatting. What could be simpler than this?

A large part of the creative process is simply getting information on to paper before you move on to something else. If you can cut down on reformatting documentation over and over again it makes much more sense to use a wiki.

Consider what happens when you use a Word document to manage this process. You are writing a manual for a machine. Do you start with turning the machine on if most people are not responsible for powering up the engine? How do you decide where this content best fits in the documentation? You really don't know, it's a small bit of information that is basically filler text with a logical flow of events.

That content goes in a place that is relative to the document structure. In a product like Microsoft Word, or OpenOffice you have to configure placeholders for titles and this and that, copy things here and there, update the table of contents, perhaps split the entire document into two at some point. Copy additional data, rebuild menus, rebuild the other things, rebuild rebuild rebuild. Also consider that Word is not a particularly fun product to use these days if the formatting gets messed up.

The Kwiki has very simple rules for formatting. If you need to branch off somewhere for technical users or people in a hurry you can instantly hyperlink out of a document without overwhelming people with technical details they may not need to know. Often I find as I browse through that sub-headings are often not necessary as you are better off rearranging the content than getting that deep with the information. For people developing a company website for the first time a wiki would be an interesting way to go.

Do you want a wiki? I can set you up with one if you like. If you do go with another host keep in mind what country it is hosted in because your content will be subject to those laws.

After spending countless hours copying files back and forth from disks to tapes and back again I started to think there could be a better way of doing all of this. Networking has advanced a lot since sound research has been going on, but distribution has been lacking. I'm sure something could be said about the RIAA and it's tactics but let's leave that one alone for now.

Differing approaches to online media distribution

Currently sound is delivered to users through a number of methods. One methodology is through file transfer, where users connect either by http, ftp, or p2p networks to exchange files with one another. Instant messaging works similarly by acting as a p2p framework where users can send files right off the desktop without having to "repackage" the media into different forms. Since it is just a copy little new creative work is put into the transfer.

The other emerging framework is that of push technology, the technology buzzword of yesteryear. In this camp you have two common offerings: streaming media and podcast-style downloadable shows. Both of these share a common thread in that they require some kind of "programming" (not in the technical sense) to go on prior to the music being put into production. This usually involves some kind of radio announcer and/or DJ to coordinate the pieces that make up the "show". These technologies cross the line into composition where much more creativity is possible.

Configuration steps required to get running

Having a personal webserver at my disposal I decided to make an attempt at constructing my very own Internet radio station. Over the course of an evening I was able to setup a prototype site, streaming some media over my home network to my Linux laptop without ever copying a single file. The most important thing I learned was the division of responsibility for the programs that run the station. You need to run something like a web server, which is fed the streams by a streaming server (there are a few out there), and finally you need some kind of playlist-generating script to load up the content.

After the initial configuration was setup I had a chance to review my work. The low bitrates were a little disappointing at first, and the sound quality cannot compare to an original file since the compression takes a lot of the fun away. What really excited me about this setup is that with LiveIce I was able to perform some light mixing tasks as the files are being encoded. The two-channel mixer has speed, volume and source file commands that will excite any novice that is new to mixing. When I'm done playing with this program I will likely switch to "shout", a streaming program that I hear may have some additional options for playlist controls.

Looking for more information on how to setup your own server? Be sure to check out the MP3 Howto for a general overview of the configuration steps.

The server accepts the message with the request, uses some "closest match" logic to know what you're talking about and then plays the music. In my 3 years using Linux I don't think I developed anything so cool (imho) so fast.

As a novice user my favorite feature was the detaching menus, so you can leave the filters menu and others on the screen for convenient clicking. This is great for new users who just need to get things done. Another love is the right mouse button menu. It lists all of the menu options from the top of the screen. Much quicker if you have been working in a tiny area of the screen for some time as artists often do.

The downfall with the Gimp is the text rendering. It doesn't seem to support letter spacing at all, the only movement allowed is line spacing. Scribus does a better job of doing this for print layouts but for web work I almost want to stay in the Gimp as much as possible. Designing website comps outside of a graphics suite would shock and amaze the people in my office. We only get Photoshop files. This file dependency also causes a problem as the newer versions of Photoshop don't layer right. Always get a Tiff to verify you're looking at the same thing as your clients.

Most people don't use Photoshop that much but they still need some basic graphics editing. If this sounds like you the Gimp will do your task. For the super math kids in the crowd, you probably already know how fun this software suite is. Have fun everybody.

As you probably already know I have a web server I own that I maintain as a hobby. Recently a few friends who have accounts on the server started requesting new features and I figured it was probably time to bit the bullet and start the upgrades.

So in the past few weeks I did a lot of research and performed the following upgrades:

• Migrated 12 live websites to a newer web server
• Rolled out a secure webmail platform
• Deployed an IMAP server to synchronize mail and read/replied status across devices
• Added or improved secure mechanisms for hosting content and backing up file

Upon doing this I discovered another great thing I have recently accomplished:

• I am no longer dependent on the "Blackberry Internet Service" provided by Rogers in partnership with RIM. This service is horribly slow for personal email accounts and is better served by running an IMAP client like LogicMail, or to use another device altogether. Periodic or "on demand" message checking works better than the "push" message service on the Blackberry which really only checks your messages every 15 minutes.

Good luck planning lunch with that kind of delay.

I would highly recommend that small businesses skip the fanfare around the Blackberry and get themselves access to an IMAP server, it's much more accommodating than I had expected. Synchronization is a great tool to have at your disposal. Doing the same tasks over and over again is definitely not in most creative people's interest so I'm happy to be settling into the new configuration.

Oh yeah, and for those of you on the RSS feed, yes it's been awhile. Now you know why! Stay tuned for more fun in the comng weeks.

• Learning the shell - everything you need to know to get up and running with Linux as a power user or administrator
• Useful commands for the command line - how to get around the system
• Advanced Bash Scripting Guide - for those times when you would rather script an activity
• VIM editor cheat sheet - Instructions on using the powerful "vim" (vi, iMproved) editor that ships with Unix & Linux (use nano if you do not like vi)
• Another Vi/Vim overview - The ins & outs of editing text files the advanced way
• How to be a good and lazy (Linux) system administrator - An entertaining read about approaches to system management

• Migrate my iBook and media server to Linux 2.6.8 kernel
• Install a new sound subsystem, alsa, on iBook
• Configure wacom tablet to work with Linux on PowerPC (still not perfect)
• Update file share configurations for the media server
• Move all the laptops to the router (as opposed to the media server)
• Setup Macintosh OSX to run within Linux, like Windows runs in Parallels
• Configure networking for the virtual Macintosh
• Change firewall configurations on three of the servers

Migration of the kernel spurred all this activity. Back in 2004 I had a similar configuration but lost it all due to a hardware failure. With all of the advances in the kernel and Debian packages this will finally enable me to do some serious audio routing on Linux.

None of this would be possible using the "bleeding edge" technology that many Linux fans love. I stick with a "stable" version of Debian because by the time I get around to doing these tasks most situations (even on a freaky iBook configuration) have been well documented.

After visiting Montréal I have a new found desire to learn more languages so I'm back at the table. Working in the Gimp I started typing. Rather than use full pinyin (romanized words) I started only with the first character. The words suddenly appeared. It makes typing Chinese faster.

I also ran some tests in Inkscape, the vector graphics program for Linux. I drafted some signs, outlined the fonts, and sent them out to the printer. Nice work. I was surprised it all worked so well. Now I'm going to make some flash cards to send out to my blackberry and desktops.

The Upgrade Process

Moving up to the new version of Debian was as simple as running the distribution update command:

apt-get install dist-upgrade

... except for the fact that my OS partition is now getting quite full.  So eventually it would stop, complain about disk space and ask me to resume later.  Apparently Lenny, the codename for Debian 5, requires more space.  Go figure.

apt-get clean

Ok, we've dumped all the installer files for these packages that have been installed.  Resume installation.  Everything goes pretty smoothly from here.

At the end I need to run Lilo and I really need a new Kernel (I had been shamefully running an ancient Kernel on this box - over a year old at least).  So I asked the system for a new kernel and got it. 

One blip: the video driver.  I have a strange Intel-based motherbord - the IntelD201GLY - which has an integrated SIS graphics card with little/no support anywhere to be found.  I had to compile the drivers myself... (against the new kernel of course) and now having done this process twice I will be more diligent about kernel updates because it really doesn't take that long to fix.

The User Experience

The difference in performance was profound.  The combination of a new kernel, a more modern browser (Firefox 3 - packaged as Iceweasel 3) and the vast array of other updates have made the first few minutes an amazingly refreshing adventure.  The few hours I spent debugging were well worth the effort.

The icons changed, themes changed, and in some cases syntax changed (which of course means fixing a variety of scripts).  Overall the system feels more integrated, nicer to look at and less invasive.  I can see myself getting very comfortable with this.

The Missing Link

The one change that did catch me really off guard is the disappearance of the original Xmms.  I had been using it for years as a secondary player and I apprecaited the support it had for changing output devices on the fly.  No other gui player seems to have that feature implemented as far as I have been able to tell.  Why not just use Xmms2 you ask?  As far as I am concerned Xmms2 is a nightmare.

With Xmms2 it seems the developers wanted to make a server-based player.  Fair enough, but I already use Moosic for this and it does a good job.  I wanted to drag and drop from Nautilus - no more.  Of all the gui interfaces to Xmms2 none of them seem to support drag and drop from external apps.  Further to this, Gxmms2 can't seem to load the queue with files from within it's own interface.  Abraca is the same.  Esperanza will load the files into the list but runs the KDE interface - the only app on my desktop that does.

The main issue I have with Xmms2 is the lack of support for changing devices in the gui apps.  I have to run commands to update text files.  Frustrating!  The volume controls do not seem to want to associate with the proper device no matter what I try and no features other than the bare minimum are documented.  What seems to have happened here is that Xmms2 became a radio streaming program and lost sight of why it was created in the first place.  I don't mean to be harsh - I'm sure I will like all those new features when I get to them... but in the meantime, how do I listen to tunes on my second audio card without a bunch of hassle?  Suggestions?

Last Word

Debian 5 is a nice update to a great OS.  There are small improvements everywhere and it makes my old desktop a lot more fun to use.  Hardware support seems to be gradually improving over time and this is a good thing.  I was thinking about jumping ship to Ubuntu a few months ago but it was worth the wait.  I use Debian on the desktop and the server so the consistency is a huge advantage to me.

The reason OS X has been so successful is based on the culture it has created. When I studied in Computer Science at the beginning of my degree OS X was something new. Perhaps it was my shiny blue iBook that looked like a hello kitty purse or perhaps it was the beta operating system, but either way the general feedback I received was "weak computer" when in the halls of the computing concourse.

Back then OSX was difficult to use and the real savvy Mac users were all in the graphics arena. The Macintosh interface was nowhere near finished and documentation was sparse. Arguably, compared to Linux it still lacks documentation given all the howtos out there. The philosophy is different than Linux or Windows too... in most online guides you will find that there is in fact only one way to do "that" on a Macintosh. For many users this is a sigh of relief. To nerds, this is a burden.

So what changed? Was it the legions of devoted Unix-types that jumped the Microsoft's ship when Apple unmasked the largest widespread Unix implementation in history? Or, did the disparity of tools available spark a creative chord with those who had given up on the Windows world? Was it just the viruses?

The nerds took over.

Years later I am looking at the current version of the operating system and a little awestruck at what has happened. The Unix underpinnings of the system made thousands of new software titles available at the flip of a switch. The programming environment enabled cross-platform development which supported Apple's move to Intel. Then they made a boot loader that supports Windows. In short order Apple will likely support running Windows programs within the Mac interface, and this is where things start to get crazy.

Following these moves it is now clear that the Macintosh is positioned to be "everything to everyone" by supporting both Windows and Linux/Unix applications in addition to the easy to use Mac fare everyone loves. Market forecasters are predicting that with some agressive pricing strategy the Mac is slated to take off. Some say that changes are happening but we don't know the direction yet while others are claiming that Apple's sour times are over. Others in the Mac community are even a little more excited about the prospects.

For a long time that better way was paper. I like to have a 7-day spread in front of me at all times in meetings and at play. During my hour-long commutes to Simon Fraser this was a flexible way of managing time. Since then my needs have changed only slightly but the nature of my work has changed too.

At the office I have fewer meetings than I had back in my university days. Hard to believe, but it's true. I was at a lot of meetings when I was studying. The other major shift is that I am now often chained to a desk like most workers. And this means Internet. The same thing goes for home, and at the cafe, so I'm noting this trend: the Internet seems to be everywhere. I bet you noted it too. So why is it that most people with calendars do things the same old way?

Familiarity is one reason. Having the ability to see things as a seven day spread is is vital to the success of any scheduling system I use. Simplicity is another. I prefer paper because there is only ever one way to use it: by writing. For me only the shell compares, so I wrote some scripts that utilize the shell and a software package called mhc to come up with a new way of scheduling.

How does my schedule work? First, I input things into my calendar using a command line tool. It prompts me for the necessary information and then quickly disappears when everything is ready to go. Then, I produce a listing of this week's entries by using the "today" command. I made one modification to the default behaviour by overriding the reporting of Sunday. I prefer Monday as the start of my week.

The last step in my configuration is to display the information out somewhere I can find it. I have a special program on my computer which I created as a "dashboard" for the shell. It displays random information so I extended it and added some additional sections. Here's what it looks like, by section:

• Time - current to North American time zones, in 24 hour format
• Week - the official week of the year, counting up to 52
• Schedule - listing from Monday to Friday ending this story on a good note
• Calendars - the three most relevant months, this one, the last, and the next one
• Fun - stuff that is fun, like the music currently playing

Why did I select these items? Primarily because they are related to time. The 24-hour clock makes things easy to work with, and time zones are necessary for my work. The week of the year is a great planning tool for newsletters. The seven day schedule is a requisite but it is sorted to create the maximum level of personal excitement. The rest is all relative to what you think is important. For me this is the current song, making only the very top and bottom of the listing to have frequently updated information.

To accomplish this I owe one to Irek Szczesniak, who had created a script to put a calendar onto an image. Irek's code figured out what the week of the month was in some simple statements so I expanded upon these to produce some creative output.

Use 1: Finding the week of the month

This one is pretty simple, run the command and it will produce the current week of the month, assuming Monday as the first day and that the first "full week" is the one that counts.

kappa@verbosity.ca:~$ <b>wom</b><br /> 1<br /> <br /> kappa@verbosity.ca:~$

Here the script has checked the calendar and determined that the 9th of July in 2006 is the last day of the first full week. Again, rememeber that Monday is the first day of the week you crazy North Americans!

Use 2: Executing commands based on the week of month

This is what I really wrote the script for. The "wom" script also accepts one parameter which is the "desired week" you are seeking. If you input a value here it will be compared against the current week of the month and set the exit status accordingly.

kappa@verbosity.ca:~$ <b>wom 1 && date</b><br /> Sun Jul 9 21:06:42 PDT 2006<br /> <br /> kappa@verbosity.ca:~$ <b>wom 2 && date</b><br /> <br /> kappa@verbosity.ca:~$<br />

So, if you have a command that you want to put into your crontab all you need to do is prefix it with "wom 1 &&" to make sure it only happens on the first week of the month. Enjoy.

In the computer world we celebrate uptime like some kind of cult. mcluhan has been running 196 days until now. This rivals the uptime that I had when living on 1086 Bute Street when my uptime was also around 200 days. I am always recoiling over the amount of time I spend confiugring computers and this outage is no different. I will have to reopen a few documents I have had open for months, and restart a couple system services. Crazy times. It makes me glad I settled with Linux after brief forays with Macintosh and Windows failed to excite me.

Forthcoming upgrades will provide extended battery backup and some network redundancy in another city. That will allow this system to serve out busy sites and develop out some creative email services. Look for within about a year's time. You know I only restart once or twice a year.

Once I've run the commands to compile the modules I will just need to load them into the system and the card should become active. At least that's my expectation. Rebooting should not be required unless a core-level kernel change is required. If that's the case I will likely take the opportunity to get a new kernel to run the drivers. Should everything go according to plan the next step will be adopting the ALSA subsystem for my environmental audio.

Music rotation is now being handled by "moosic", a free program that can slice and dice a playlist any way you like. A few of my script wrappers are getting updated to reflect the changes. This system should allow me to create multiple ongoing streams, like radio stations, and push them out on any device (to multiple sets of speakers) or to the Internet in real time.

UPDATE! The server experienced it's first crash ever. The user space halted as a result of either overheating or electrical disturbances caused by the tuner card. This card is pretty much junk, so it has been removed from the network. Good thing I didn't invest all day in this. Whew!

Now Microsoft plans to strike a deal with Novell. They distribute a version of Linux, called Suse, which competes with Red Hat in the enterprise desktop and server markets. Up until now Microsoft has been denying Linux any headway by promoting a "Get the facts" campaign that seeks to dispel any studies that find Linux to be more efficient than Windows. So this move is shocking but familiar territory for the software giant.

Maybe they'll finally release a stable and secure version of Windows based on Linux. We can only dream.

Here's that clip from Macworld 1997 in case you missed it -  

Part of our objective here is to use the Linux/Unix file permissions scheme as intended in a way that limits access to the bare-minimum necessary. We presume each developer has their own SSH account on the git server and that is all they should ever need.

Requirements

This recipe is for any Linux host that has Git installed. It requires SSH as it will be used for managing the connections with your users. By default, SSH uses the Linux system's user accounts as an authentication system (known as "auth" method) but if your needs require it, you can also use SSH modules to plug into your local LDAP or ActiveDirectory® authentication systems. One thing that will be of great importance in this tutorial is permissioning the users correctly and setting up a deployment action that suits your needs best. The strategies we use here may be adapted to your own use cases.

Getting Started with Git as a Host

By now I'm sure you've probably heard the philosophy behind Git is that every repo contains all the history of a project and any copy can become the master copy if the original is lost. While this is great in principle, in reality, to share Git with others we will need to setup a special type of repository that is accessible to your system's users.

Installing Applications

First, let's make sure we have Git and SSH installed. On Debian or Ubuntu the command to install Git is as follows: apt-get install git-core openssh-server There is no special version of Git to do shared repositories, the standard one will do it all.

Storage

You will need to create yourself a folder where your repositories will be stored. In my case I'm creating a new directory right in the root of the server so that my users will have a nice path to work with when I give them access to the server. The storage should *not* be your production webserver. You need to put it somewhere that is not live to your users as the shared repository has a bunch of files you don't want to put into a production environment. mkdir /projects

I actually created my projects folder under /var/projects and just created a symlink here, to better integrate with our existing backup processes.

Grouping the Users

Make sure that we have a group for our users. I'm using the group name "webmasters" but you may already have a group established for your team. If that is the case, use the group you are already using. addgroup webmasters We will have to do additional work on the user account to make this work... but for now this is enough.

Initializing the Shared Repository

Now we will create a new project called "newsite". When your colleagues connect to the site the path will be /projects/newsite.git with this configuration. cd /projects git init --bare --shared newsite.git chgrp -R webmasters newsite.git

Adding Users to the Mix

If you already have users on your site, great. If not... adduser newdeveloper webmasters usermod -a -G webmasters newdeveloper The usermod step is necessary so that each time your user, newdeveloper, creates a file, that it will be permissioned to the entire group. This will allow other users to modify the file if it was created by another user. There is one last step to get the permissions structure just right. By default, most Linux systems only allow user files to be edited by the user who created it, even though you have put the file into the group. There are many strategies for how to override this. My personal favourite is to change the system umask value to apply the same permission for the owner to the group as well.

To make a global change to enable "group writeable" by default in Debian or Ubuntu do the following: Edit the file /etc/profile with your favourite text editor. Add umask 002 to the end of the file. If you already have a umask value, you can change it rather than adding a new line. You can also add the umask 002 line to the user's ~/.bashrc file if you wish to do per-user setup for this. Be sure to test that this is working by logging in as your new user by doing su newdeveloper and then typing cd to go to their home folder (note, be sure to login after making the change), then in the user's home directory try doing touch testfile followed by ls -la | grep testfile.

You should see the following output: -rw-rw-r-- 1 newdeveloper webmasters 0 2012-12-20 13:17 testfile In particular: look at the codes at the start. If you see -rw-r--r-- then umask is not set correctly for some reason. You should also see newdeveloper and webmasters as the user and group respectively. If not, go back to the step where you set the user's group to be set to new files by default. Does it all look ok? Then rm testfile and log out of your new user's account. The Control-D key will get you out of their account fast. ;) Keep in mind there are other methods for doing this. If you already have a different system for managing group ownership of files, you will probably want to stick to the system you are already using if it is appropriate for your use case.

Accessing the Repository

Your repository can now be accessed using the following paths. Keep in mind, if it is the first time you clone your repository it will warn you that you are cloning an empty repository. That is ok! You can add some files later and push up to the server so that the next person to clone does not get that message. From the same (local) machine: git clone file:///projects/newsite.git cd newsite From a remote computer anywhere on the Internet: git clone newdeveloper@example.com:/projects/newsite.git cd newsite If you are using a remote computer, you will be asked for your password unless you have added your public key from your remote computer to the user's account on the server.

For Bonus Points, auto-checkout into stage

There is one critical thing that you will want to consider before you go live. How are you going to update your staging environment? By default there IS an action performed when users push new updates to git, defined in the shared repository's hooks folder (under /projects/newsite.git/hooks in the file system), in the post-commit file. One word of warning here though - it will run as the user who does the commit. So your staging environment will constantly have permission errors. Ideally your stage environment probably has one user who is in control of it.

To fix this, a really crude way, I rigged up a script that checks for updates every 30 seconds. Eventually I'll come up with something better, an action that can be taken by any user that doesn't involve giving everyone sudo access to the stage user. Run this "daemon" as a script from cron as the user you want to be responsible for stage: #!/bin/bash cd /stage while [ 'FALSE' != 'TRUE' ] do git pull origin master sleep 30 done

WARNING: it should be obvious that this code won't scale... and will waste some resources unnecessarily; you've been warned!

In my second crude attempt at solving this issue I have taken the following approach:

1. Create a hooks/post-receive file inside your repo
2. Set this file to echo your destination path into a queue file: echo "/var/www/newsite" >> /projects/queue
3. Create the /projects/queue file: touch /projects/queue && chown root:webmasters /projects/queue && chmod 660 /projects/queue
4. Create a checkout script: while read PATH; do cd $PATH; /bin/su target_username -c "/usr/bin/git pull origin master" done
5. Then create a watcher to trigger that checkout script: echo "" >/projects/queue # empty the queue first tail -f /projects/queue | /usr/local/bin/checkout This solves the issue of having multiple users accessing the repository because you specify a user to run the checkout. All the users are able to write to the queue file, and the watcher just keeps an eye on that file. Since the watcher must sudo into another user's account to do the checkout, we can run the watcher as root and there is no possibility to any of our users figuring out they can sudo as someone else - because we don't use sudo at all. You should add your watcher to your startup scripts.

More Bonus Points, disable SSH interactive mode for some users, and allow logins without passwords

This can be considered a sort of cruel and unusual punishment by some... however, in some cases it is handy, for example, when you have a designer changing theme files but who shouldn't be able to get into all your databases and other things. This is really simple to accomplish: usermod newdeveloper -s /usr/bin/git-shell

The recommendation for dealing with public keys is to have the user login to SSH normally, then drop the user into git-shell. I'll be rolling this out soon so I can collect some of these bonus points. Have your user generate the public key. You may wish to avoid RSA because some server-wide sshd_config files have it off by default. I have used DSA in this example, if you use a different encoding, just make sure you use the associated id_XXX.pub file for that.

On the developer's machine, grab the existing .ssh/id_dsa.pub or generate one using: ssh-keygen -t dsa Be sure to leave the challenge response blank. Then copy the contents of the id_dsa.pub file to the server. The contents of the file should be appended to the .ssh/authorized_keys file on the server... then... the important stuff: Back on the server: chmod 700 /home/newdeveloper/.ssh chmod 600 /home/newdeveloper/.ssh/authorized_keys That is it! Now the user should be able to log in automatically, and they will not be able to SSH into the host... only to use git to post the files.

Once again, without much sluething I was able to find the solution (though clearly it has not been implemented yet). As always, notes related to the software were located in /usr/share/doc, a directory tree I used to ignore but has become my lifeline.

Typically when my Linux servers go down my instinct is to search Google for hours. It's a behavioural legacy from using Windows and the Macintosh. When something goes wrong in Debian, check your /usr/share/doc and you will likely find the answer to the problem you are having.

When a site is ready to become an archive it can be a good idea to convert it to a static site. Security updates are no longer necessary, but interactive features of the site disappear... which is usually a good thing in this scenario.

Creating a site mirror

Long before I used Drupal this was all possible with wget, and it continues to work today:

#!/bin/bash
wget -o download.log -N -S --random-wait -x -r -p -l inf -E --convert-links --domains="`echo $1`" $1

I call this script "getsite", you use it by typing "getsite example.com"

This is a simple script that I place in the /usr/local/bin folder of the computer I will be using to create the site mirror.

This script will probably take awhile to run. You can run tail -f download.log in another terminal to watch the progress.

What does it do?

This is a simple web crawler that will follow all links on the page that you provided, but ONLY the links that are on the same domain.

It will try to fetch ALL the assets that come from this exact domain name you provide.

While doing so, it changes all of the paths to be relative to the root.

I also have it set to crawl slowly so as not to scare any firewalls we may be traversing.

You can look up all of the command line options by typing man wget on your system.

After running the command you will have a folder with the name of the domain and all of the files for the site, in addition to a download.log file that you can use to audit the download.

It can be very useful to use the utility tree to see all of the files.

Oh noes! All my paths have .html appended now!

Relax. Just like we can do clean URLs with index.php files we can specify some rules on our webserver to mask that ugly file extension.

In Nginx you can do this as follows:

location / {
  root   /var/www/html
  index  index.html index.htm;
  try_files $uri $uri/index.html $uri/ =404;
}

The "try_files" patterns will match what used to be our Drupal clean URLs.

You may also want to add some kind of htpasswd-style restriction if your content is not intended to be available to the public.

It is as simple as that! Wget is a great utility for making site mirrors or legal archives.

Cleaning up loose ends

Your Drupal site is going to have some interactive components that will no longer work.

In particular:

• User login form
• Webforms
• Commenting
• Anything else using a form and/or a captcha (maybe disable captcha too)

It may be simpler to disable these before taking the snapshot, or alternatively opening the resulting HTML in a text editor and removing the form components after the fact.

You may also want to enable or disable caching of different things depending on what results you get. By default you are probably going to see a lot of security tokens in the downloaded paths, so you may want to disable that... on the other hand, you may want to bundle your CSS to make fewer requests. Review your downloaded archive to see what will be best before you shut down your source site.

Other uses

My team has used variations of this script for a variety of other needs as well:

• to estimate the size and scope of a migration project;
• to get a complete list of paths we may want to alias or redirect after a migration;
• to make an archive of a site for legal proceedings (ie, gathering evidence of copyright infringement);
• to migrate data from a static archive when source databases do not contain fully rendered content;
• and finally: to "pepper" the caches of large sites by hitting each URL after a migration when the caches are all cold.

In that last example we use the spider option to "not" download the files, but simply request them and then move on.

Wget is an extremely powerful tool for mirroring entire sites and provides us an easy way to archive old dynamically-rendered sites without much hassle, and zero ongoing maintenance.

To find out what other things you can do with wget just type man wget on your console and read all the options that are available.

This article assumes that you are working on a Debian Linux installation, but would probably work on Ubuntu and other modern Linux distributions as well.

Generate the Keys

In every article you read you are going to see that same heading.  Keys are the things that keep your privacy, and you must generate them on your own.   In this example my certificate signing authority wanted an AES-256 type of key (that's the type of encryption) so I ran the following command in Debian:

openssl genrsa -aes256 -out personal.key 1024

Update 2011-11-02: new recommended command from my provider: openssl req -nodes -newkey rsa:2048 -keyout personal.key -out personal.csr

As you may have guessed, that creates a file called personal.key.  That's your key!  Keep it safe.  You're probably wanting to do more with it though, because if you go this route you will need to get the certificate signed as well.  If you aren't looking to get a signed key you could alternatively generate a self-signed certificate, but always keep in mind that modern browsers do not like these.  They throw all kinds of errors, security warnings and other messages about your security settings.  So if you want that, go check it out, otherwise let's get that key signed:

openssl req -new -nodes -keyout personal.key -out request.csr

If you're using the italics version above, you can probably skip this step!

In this example you take that key you just created and you generate a signing request.  It wants information - so be honest!  This is your online reputation when someone clicks to view the certificate so make sure it is accurate.

The Waiting Game

After your SSL signing request is ready to go it is time to contract your signing authority.  I have gone to two different ones in the past - a Vancouver company called Omegasphere issues these and I have also signed up for one through Godaddy.  You could buy one from Verisign too, but you will pay a lot more for the privelege.

Once your certificate signing authority gets your request they will do some kind of verification with you and/or your site.  Typically you can pay more for deeper inspection of records.  I'm not sure exactly how that benefits the site owner, but it could make users feel better (provided they understand SSL really well - I'm going to assume most users don't give a F*#$).

All they want is your request.csr file with accurate information.  Be available to respond to their inquiries and requests.  After waiting you should get two new files if your request is approved:

personal.crt
ca-bundle.crt

The file names will certainly differ but the concept is always the same.  You need to install three files when you get into your SSL setup (coming up next, read on!).  First, you need your personal.key.  That has your secret data.  Then you have your personal.crt.  That has your information and signature that says your key is legit.  Lastly, the Certificate Authority Bundle (ca-bundle.crt) is a file that says who vouched for you, and I believe it might also provide a way of authenticating them.  Rest assured, no matter what the case, that last file is about your service provider, not you, but you still need it.

Confirm the Goods

The biggest challenge when you are debugging these files that you just paid for is knowing if you got what you paid for before you spend countless hours trying to force the key into your system(s). I found the solution to this one here, but I'll warn you that the forum is all in French.  Here is a translation.

Run these two commands, they should provide the same answer:

openssl x509 -noout -modulus -in personal.crt | openssl md5
openssl rsa -noout -modulus -in personal.key | openssl md5

Do those look the same?  Good.  Ok, you got what you paid for.

Just in case though, let's do one more test.

openssl x509 -noout -text -in server.crt
openssl rsa -noout -text -in server.key

This one is more complicated.  Check that the exponent is the same.  Typically it is 65537 in most configurations.  So long as you have a match you are gold.

Installing Away

Using Apache2

Installation on Apache2 is "simple" in the sense that there are lots of howto documents out there for you to sink your teeth into.  Sometimes you will chip a tooth.  I took one recommendation and accidentally removed PHP5 while trying to setup SSL.  Be wary of what you copy and paste, it is a mixed bag out there.

Your server will not restart if you do not get this configuration right.  So do it when the sun is not shining.  Also, be sure you checked out the test steps above, they will save you some headaches here.

Enable mod-ssl

Apache2 needs modules to get SSL support configured.  So install it with this command:

a2enmod ssl

Edit ports.conf

You need to enable Port #443 to get SSL in gear.  Update your ports.conf as such:

Listen 80
Listen 443

There, the "Listen 80" line was there so we added a second entry.

Virtual Hosts

This is where virtual hosts get weird.  You need to add the NameVirtualHost directive and define at least one new site.  You can only have one SSL certificate per IP address so start adding some to your account if you intend to run a few different certificates on your server.

NameVirtualHost 127.0.0.5:80
NameVirtualHost 127.0.0.5:443

Be sure to replace 127.0.0.5 with your server's IP address. 

Now define your new site.  Copy your existing virtual host entry and paste it into the same file, making sure to paste it after the end of the primary virtual host.

Now, change your IP address and port in the new entry:

From: 
To: 

Be sure to skip the from/to part that I typed in there, that was just for reference.  You can customize the other settings in here now or come back to it later.  You might want to setup a different log file for SSL errors, for example.

Now, somewhere in the definition of the ssl virtual host add the following lines. 

SSLEngine on
SSLCertificateFile /etc/apache2/personal.crt
SSLCertificateKeyFile /etc/apache2/personal.key
SSLCertificateChainFile /etc/apache2/ca-bundle.crt

Note that you must change the path to these files to match where you have stored them on your system.  As mentioned earlier, if this statement fails your server will not start any more.  You can comment out "SSLEngine on" by putting a # in front of it if you need to get your site back up in a hurry (obviously without SSL).

If you added a password during the setup process you will now need to enter it each time you start Apache2.  It is possible to remove this "feature" if you automate updates and can't predict when your server restarts itself.

For other services

When it comes time to expand your SSL presence beyond Apache2 you can start plugging these three files into your other servers.  You need to keep in mind that the IP address and domain name should be the same as the Apache2 server. 

In my experience certain programs requrie you to change the format of the information to convert it to a PEM file.  This will also probably reduce the number of files you need to install the key, but your mileage may vary.  Ideally, you still want your CA-Bundle in there but sometimes it isn't necessary. 

For Drupal

If you plan on using your SSL certificate with Drupal setup Apache as noted above and then install the Secure Pages Drupal module in your site.  I had to add "user" as one of the paths to encrypt so that visitors know their data is safe.  Otherwise it handled the web side of things quite gracefully.

For those of you still reading my ramblings about technology this marks the end of a great project to learn Linux. Now I'm getting down to business. If you don't hear from me you can trust I'm working away learning new tricks and helping people with their online marketing projects.

Enjoy the weekend!