Drupal Planet

Hosting your own Git-based shared repositories using SSH

Git has become one of the most important tools in a developer's toolkit. To a Drupal developer, it is even more critical as nearly everyone in the community has standardized on it. While there are many great Git hosting services out there, sometimes clients need to have only local copies and Git is all about making each copy a distinctive repo to itself... so why not create your own Git host on your own server? That is what we are here to do today.

Requirements

This recipe is for any Linux host that has Git installed. It requires SSH as it will be used for managing the connections with your users. By default, SSH uses the Linux system's user accounts as an authentication system (known as "auth" method) but if your needs require it, you can also use SSH modules to plug into your local LDAP or ActiveDirectory® authentication systems.

One thing that will be of great importance in this tutorial is permissioning the users correctly and setting up a deployment action that suits your needs best. The strategies we use here may be adapted to your own use cases.

Getting Started with Git as a Host

By now I'm sure you've probably heard the philosophy behind Git is that every repo contains all the history of a project and any copy can become the master copy if the original is lost. While this is great in principle, in reality, to share Git with others we will need to setup a special type of repository that is accessible to your system's users.

Installing Applications

First, let's make sure we have Git and SSH installed. On Debian or Ubuntu the command to install Git is as follows:

apt-get install git-core openssh-server

There is no special version of Git to do shared repositories, the standard one will do it all.

Storage

You will need to create yourself a folder where your repositories will be stored. In my case I'm creating a new directory right in the root of the server so that my users will have a nice path to work with when I give them access to the server.

The storage should *not* be your production webserver. You need to put it somewhere that is not live to your users as the shared repository has a bunch of files you don't want to put into a production environment.

mkdir /projects

I actually created my projects folder under /var/projects and just created a symlink here, to better integrate with our existing backup processes.

Grouping the Users

Make sure that we have a group for our users. I'm using the group name "webmasters" but you may already have a group established for your team. If that is the case, use the group you are already using.

addgroup webmasters

We will have to do additional work on the user account to make this work... but for now this is enough.

Initializing the Shared Repository

Now we will create a new project called "newsite". When your colleagues connect to the site the path will be /projects/newsite.git with this configuration.

cd /projects
git init --bare --shared newsite.git
chgrp -R webmasters newsite.git

Adding Users to the Mix

If you already have users on your site, great. If not...

adduser newdeveloper webmasters
usermod -a -G webmasters newdeveloper

The usermod step is necessary so that each time your user, newdeveloper, creates a file, that it will be permissioned to the entire group. This will allow other users to modify the file if it was created by another user.

There is one last step to get the permissions structure just right. By default, most Linux systems only allow user files to be edited by the user who created it, even though you have put the file into the group. There are many strategies for how to override this. My personal favourite is to change the system umask value to apply the same permission for the owner to the group as well.

To make a global change to enable "group writeable" by default in Debian or Ubuntu do the following:

Edit the file /etc/profile with your favourite text editor.
Add umask 002 to the end of the file. If you already have a umask value, you can change it rather than adding a new line.

You can also add the umask 002 line to the user's ~/.bashrc file if you wish to do per-user setup for this.

Be sure to test that this is working by logging in as your new user by doing su newdeveloper and then typing cd to go to their home folder (note, be sure to login after making the change), then in the user's home directory try doing touch testfile followed by ls -la | grep testfile. You should see the following output:

-rw-rw-r-- 1 newdeveloper webmasters    0 2012-12-20 13:17 testfile

In particular: look at the codes at the start. If you see -rw-r--r-- then umask is not set correctly for some reason. You should also see newdeveloper and webmasters as the user and group respectively. If not, go back to the step where you set the user's group to be set to new files by default.

Does it all look ok? Then rm testfile and log out of your new user's account. The Control-D key will get you out of their account fast. ;)

Keep in mind there are other methods for doing this. If you already have a different system for managing group ownership of files, you will probably want to stick to the system you are already using if it is appropriate for your use case.

Accessing the Repository

Your repository can now be accessed using the following paths. Keep in mind, if it is the first time you clone your repository it will warn you that you are cloning an empty repository. That is ok! You can add some files later and push up to the server so that the next person to clone does not get that message.

From the same (local) machine:

git clone file:///projects/newsite.git
cd newsite

From a remote computer anywhere on the Internet:

git clone newdeveloper@example.com:/projects/newsite.git
cd newsite

If you are using a remote computer, you will be asked for your password unless you have added your public key from your remote computer to the user's account on the server.

For Bonus Points, auto-checkout into stage

There is one critical thing that you will want to consider before you go live. How are you going to update your staging environment? By default there IS an action performed when users push new updates to git, defined in the shared repository's hooks folder (under /projects/newsite.git/hooks in the file system), in the post-commit file. One word of warning here though - it will run as the user who does the commit. So your staging environment will constantly have permission errors. Ideally your stage environment probably has one user who is in control of it.

To fix this, a really crude way, I rigged up a script that checks for updates every 30 seconds. Eventually I'll come up with something better, an action that can be taken by any user that doesn't involve giving everyone sudo access to the stage user. Run this "daemon" as a script from cron as the user you want to be responsible for stage:


#!/bin/bash
cd /stage
while [ 'FALSE' != 'TRUE' ]
do
  git pull origin master
  sleep 30
done

WARNING: it should be obvious that this code won't scale... and will waste some resources unnecessarily; you've been warned!

In my second crude attempt at solving this issue I have taken the following approach:

1. Create a hooks/post-receive file inside your repo
2. Set this file to echo your destination path into a queue file:
echo "/var/www/newsite" >> /projects/queue
3. Create the /projects/queue file: touch /projects/queue && chown root:webmasters /projects/queue && chmod 660 /projects/queue
4. Create a checkout script:

while read PATH;
do
    cd $PATH;
    /bin/su target_username -c "/usr/bin/git pull origin master"
done

5. Then create a watcher to trigger that checkout script:

echo "" >/projects/queue  # empty the queue first
tail -f /projects/queue | /usr/local/bin/checkout

This solves the issue of having multiple users accessing the repository because you specify a user to run the checkout. All the users are able to write to the queue file, and the watcher just keeps an eye on that file. Since the watcher must sudo into another user's account to do the checkout, we can run the watcher as root and there is no possibility to any of our users figuring out they can sudo as someone else - because we don't use sudo at all.

You should add your watcher to your startup scripts.

More Bonus Points, disable SSH interactive mode for some users, and allow logins without passwords

All the Drupal people in the house be rolling their eyes right now. This can be considered a sort of cruel and unusual punishment by some... however, in some cases it is handy, for example, when you have a designer changing theme files but who shouldn't be able to get into all your databases and other things.

This is really simple to accomplish, but it comes with another caveat:

usermod newdeveloper -s /usr/bin/git-shell

So what is the caveat? It ignores your public keys! So if you get the user to generate a public key on their desktop/laptop/whatever, and they give it to you to put in their /home/newdeveloper/.ssh/authorized_keys on the git server... it will totally not do a thing. Jerk!

If you use this method the user must type their password every single commit, even if you setup public keys. Can be annoying...

The recommendation for dealing with public keys is to have the user login to SSH normally, then drop the user into git-shell. I'll be rolling this out soon so I can collect some of these bonus points.

Have your user generate the public key. You may wish to avoid RSA because some server-wide sshd_config files have it off by default. I have used DSA in this example, if you use a different encoding, just make sure you use the associated id_XXX.pub file for that.

On the developer's machine, grab the existing .ssh/id_dsa.pub or generate one using:

ssh-keygen -t dsa

Be sure to leave the challenge response blank. Then copy the contents of the id_dsa.pub file to the server. The contents of the file should be appended to the .ssh/authorized_keys file on the server... then... the important stuff:

Back on the server:

chmod 700 /home/newdeveloper/.ssh
chmod 600 /home/newdeveloper/.ssh/authorized_keys

That is it! Now the user should be able to log in automatically, and they will not be able to SSH into the host... only to use git to post the files.

Solving your multilingual navigation issues with Entity Translation in Drupal 7

UPDATE: the D7 entity module has changed, and now has support for menus. See my latest post on the topic for more up-to-date information.

This blog post has been a long time coming. I have been using Entity Translation, Drupal 7's new interface to doing translations, for the better part of this past year. It has been an exciting adventure, but with every new way of doing things there are some dragons along the path. In this post, I think I have solved one of my last great challenges with it by using some new contrib modules, so I'm documenting my experience here so you do not have to have the same troubles.

Entity Translation, for those who are not aware, is Drupal's transition to translating fields rather than translating nodes. You can read more about it on Gábor's site on Drupal7 Multilingual if you wish (you've probably already seen this page if you were searching around and landed here).

So what is wrong with Drupal's navigation menus when you use Entity Translation?

  • By default, there is only one navigation menu configuration for the node - on the source translation only.
  • The modules relating to entity translation are very new - some of them even lack a user interface!
  • Menus become a difficult problem because they are not entities like everything else
  • "Language neutral" (aka 'LANGUAGE_NONE' or 'und') nav menu entries must be translated from the default language, even if the source node is not in that language, potentially making your translation tables very messy (not to mention this is complicated to setup).

What is the best way to address these issues?

This new module solves a lot of problems by allowing us to add a field to each node that we will use in place of Drupal's standard navigation "menu_links". It then generates Drupal's standard links managed by this module, so that you can use menus in the standard way.

There is one limitation which had me scratching my head for a couple days. If you set the menu field to "translatable" you can get the menu setting on all versions of the translation. This is fabulous. What doesn't work is taking the language from the translated entity and putting that language setting into Drupal's standard "menu_links" table. Not ideal right? Well, there is a workaround.

You see, it is possible to create a navigation menu in Drupal that is language-specific. So, if you take a step back and think about this, since each language version of an entity-node has it's own menu configuration, why not just localize the menu entirely? That way we don't care if the menu_link is language neutral. With this structure, as an example, the French editor when doing the menu settings on a node must only remember to place the French translation within the context of the other French postings. That is simple enough to explain to your users.

Other dragons? Other awesome things?

There are not that many issues I have found yet since this module generates real Drupal menu_links for the navigation menu you can use any modules that support using Drupal's nav menu system, but you might have to do it twice (or more, depending on how many languages you are supporting.

Additional things you might want to think about:

What about the future?

Eventually, when we get around to adopting Drupal8, menu navigation will probably be based on entities. So we're already inching toward the future by embracing entities for everything now!

UPDATE: You may also be interested in my new Entity Translation Tabs module!

UPDATE 2: Keep in mind you don't need many i18n modules with this, and that some i18n modules cause a conflict. Only "Block languages" (i18n_block) and the core i18n modules are necessary for this configuration.

Be sure to disable i18n multilingual select, i18n field translation, i18n path translation and i18n menu translation.

For your convenience:
drush pm-disable i18n_select i18n_field i18n_path i18n_menu

UPDATE 3:

i18n_taxonomy is OK. i18n_taxonomy might cause troubles too, but I have not tested this yet.

I recommend the "Localize" option on your taxonomy, which requires you to put the default language version first, and the "title" module to replace the taxonomy term name interface. You may have to hook_form_alter your Views if you output your taxonomy as an exposed filter to put the translation.

I put the following into my hook_form_alter function to make taxonomy make sense:

   
  if ('taxonomy_form_term' == $form_id) {
    drupal_set_message('Taxonomy terms must be set in English first.  Then click "save and translate" and you can put the French version.  You can then manage existing translations by going to the term using the translate tab.');
  }
  if ('views_exposed_form' == $form_id) {
    global $language;
    // this uses French, but you could abstract it a bit.
    if ($language->language == 'fr') {
      $result = db_query("select * from {field_data_name_field} where language='fr'");
      foreach ($result as $row) {
        if(isset($form['field_series_code_tid']['#options'][$row->entity_id])){
          $form['field_series_code_tid']['#options'][$row->entity_id] = $row->name_field_value;
        }
      }
    }
  }

Deploy Drupal's blocks on a different instance using Features without having to re-map your CSS

Get the fe_block module as part of Features Extra.

drush dl features_extra
drush pm-enable fe_block

Go to your blocks page. Configure any custom blocks you are planning to deploy. When you edit a custom block you will now have a "machine_name" variable. Give your block a machine name.

Now the block is available for Features export. You can now go to Features in your Structure menu and generate a new feature with the custom block(s) and their settings (placement in the theme, if it will be the same theme on the target system).

This is great so far, but how will we keep our CSS consistent when the block's ID will change? There is a block_class module but it does not export data to Features. Sad face... but wait! We have a solution:

We can do a custom block template in our theme to add the machine_name as a custom class for this block. So the machine_name for each block will now (1) enable the export and (2) get a class of the same name if the theme supports it.

Copy over your block.tpl.php file to your theme's templates folder.

If you have this file in your theme already, great! If you're using a starter kit theme, look in the starter kit's templates folder for this file, and copy it to templates/block.tpl.php in your theme. If you are coding your theme from scratch, grab the original block.tpl.php file from modules/block/block.tpl.php under the root of your site.

Now that you have added that file to your theme, using the name templates/block.tpl.php :

Find the line that defines the 'class' in HTML. It is probably really close to the top.

Put the following line before it. This code will add the machine name as a class while leaving the rest of the code in-place:
<?php $classes .= ' ' . _fe_block_get_machine_name($block->delta); ?>

Be sure to test this code before you put it live! Otherwise, get to work on your CSS using the machine_name defined by fe_block module! Enjoy never copying and pasting your blocks ever again!

Overriding Drupal7 node display with Views

Back in the Drupal6 days overriding node displays with a View was a pretty common thing for me but since using Drupal7 most of the time it is not necessary. That time has finally come, and I did a bunch of searching around for a better/automatic way but I did not find anything that did not involve over-engineering so I did it in my theme.

First, create a view, as a block (to not to have extra pages floating around).

  • Set the title to %1 to set your view title the same as the node
  • Do not use a pager
  • Display only one item
  • Add a Contextual Filter using the NID field, set the fallback behaviour to "hide"

Then, override the node.tpl.php file in your theme. If you do not have this file in your theme, find out which theme your custom theme is based on, and copy over the relevant file to the correct location for your theme.

In your node.tpl.php file, look for the following code:

print render($content);

...and replace it with a view_embed_view.

print views_embed_view('node','block_1', $node->nid);

In this example, the first parameter, node, is the machine name for my view, and block_1 is the first block in the view. When you are editing your view you will probably see both of these parameters in your URL. Finally, the last parameter is the node ID. This last parameter will probably be the exact same that you will use unless you are doing something truly crazy (like showing the wrong node on purpose? har har).

I only wanted to apply the view to nodes which use the 'article' content type so I combined the old behaviour and the new behaviour with a test for the content type:

if ($node->type == 'article') {
      print views_embed_view('node','block_1', $node->nid);
    }
    else {
      print render($content);
    }

There you have it! Now you can use Views to render your fields for a specific node type... and using Views you can customize your HTML as much as you like using the awesomeness of Views without the overhead of Panels or Display Suite.