import itertools

def string_patterns(pattern, *args):
  '''generator for string patterns'''

  for data in itertools.product(*args):
    s = pattern.format(*data)
    yield s

# generate some urls!
for url in string_patterns('http://example.com/{0}/{1}', ['page'], range(1,5)):
  print url

Output

http://example.com/page/1

http://example.com/page/2

http://example.com/page/3

http://example.com/page/4

Your First Log-In

The first time ssh’ing to your new instance a warning will appear letting you know that you haven’t seen this hostname and/or IP before and asks you if you really want to connect (i.e. do you trust this server):

The authenticity of host 'xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx)' can't be established.
ECDSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'xxx.xxx.xxx.xxx' (ECDSA) to the list of known hosts.
root@xxx.xxx.xxx.xxx's password:
Welcome to Ubuntu 11.10 (GNU/Linux 3.0.18-linode43 i686)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@lixxx-xxx:~#

If you answer in the affirmative and supply the correct password, your new server’s public key will be stored under two possible entries in your local ~/.ssh/known_hosts file: one for the IP address and the other for the hostname (if used). FYI: you can delete an entry by using:

$ ssh-keygen -R xxx.xxx.xxx.xxx

Create a new user

In later steps we’re going to disallow remote logins by the root user, so this step is CRITICAL!

$ sudo adduser NEWUSERNAME

... lots of easy steps here ...

Add you’ll probably want to add this new user to the sudoers list so they can do things like install system-wide packages and perform upgrades. Edit (as root) the file /etc/sudoers and add your new username to the list under User privilege specification:

# User privilege specification
root,NEWUSERNAME      ALL=(ALL:ALL) ALL

Go Password-Less!

Create a private/public key pair on the client machine. The default type is ‘rsa’, but let’s use the -t option for demonstration purposes. Also, to stop a default email address from getting into the comment, we’ll be specific and use a good email (doesn’t have to be an email, could be a location like “mylaptop”, etc…):

$ ssh-keygen -t rsa -b 4096 -C name@example.com

... answer the questions

When asked to enter the filename, you can accept the default or go with a custom name. I’ll use “MYKEY_rsa” for this workflow. Also, I’ve NOT used a passphrase. You may add a passphrase for added security…

Transfer your public key to the remote machine

The remote machine needs your public key to decrypt the data you send it that’s encrypted with your private key. Below are steps to do this both manually and then quickly with the ssh-copy-id program.

Manually

Create the ~/.ssh directory on the server and protect it:

$ mkdir ~/.ssh
$ chmod 700 ~/.ssh

Transfer your public key from the client to remote machine:

$ cat ~/.ssh/MYKEY_rsa.pub | ssh NEWUSERNAME@xxx.xxx.xxx.xxx 'cat >> ~/.ssh/authorized_keys'

root@xxx.xxx.xxx.xxx's password: XXXXXX

Using ssh-copy-id

This method is a lot more bullet-proof.

$ ssh-copy-id -i ~/.ssh/MYKEY_rsa NEWUSERNAME@xxx.xxx.xxx.xxx
root@xxx.xxx.xxx.xxx's password: XXXXXX
Now try logging into the machine, with "ssh 'root@xxx.xxx.xxx.xxx'", and check in:

  ~/.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Make yourself known to ssh-agent

You’ve given your public identity to the remote server and you’ve added it to your known_hosts. Now, if you attempt an ssh connection, you’ll see this error if you’ve used a non-default name for your keys:

Agent admitted failure to sign using the key.

This is because ssh didn’t know which identity you to use. To remedy this, let the ssh agent know about your new identity so it can provide the correct credentials behind the scenes when needed.

$ ssh-add ~/.ssh/MYKEY_rsa

FYI: to delete all your identities, use this command:

$ ssh-add -D

Ubuntu will prompt you for the password, if the private key has one (I didn’t supply one in this workflow).

Disallow Keyboard-based And Remote Root Logins:

Now that authentication can happen using public/private keys, we can now safely disable standard keyboard password authentication on the server. On the server machine, open the file /etc/ssh/sshd_config and make the following changes:

PasswordAuthentication no
ChallengeResponseAuthentication no

ONLY if you’ve created a new user and given them sudo permissions should you modify this line:

PermitRootLogin no

Reload your sshd config file (using Ubuntu’s upstart):

$ service ssh reload

Now (just to test things) back on the client, tell our agent to forget our key, and attempt a regular ssh connection. You should get a ‘Permission denied’ error:

$ ssh NEWUSERNAME@xxx.xxx.xxx.xxx
Agent admitted failure to sign using the key.
Permission denied (publickey).

Add the key back to the agent, and all should be good!

$ ssh-add MYKEY_rsa
Identity added: MYKEY_rsa (MYKEY_rsa)

$ ssh NEWUSERNAME@xxx.xxx.xxx.xxx

Try Something!

Now to test your new user and their sudo permissions, try installing a cool database!

$ sudo apt-get install redis-server

While scraping acquiring multi-lingual (Unicode) data for a project, I had to make sure I kept the correct utf-8 encoding all along the processing pipeline. One screw-up and it’s garbage out. Here are some tips:

Acquire Content in Perl

Use the Perl modules pQuery and Redis to fetch and store the content. Everything will work in utf-8, no worries.

Normalize the Data as CSV

Use the Perl modules Text::CSV::Unicode, your CSV file will be encoded in utf-8.

Read CSV into MySQL Database

Using Text::CSV::Unicode again, insert the data into a MySQL database. But here’s the kicker, use:

$dbh->{'mysql_enable_utf8'} = 1;
$dbh->do(qq{SET NAMES 'utf8';});

… if you don’t, your utf-8 strings will be re-encoded into the local character set. Not good.

Also create a file at ~/.my.cnf with this content:

[mysql]
default-character-set=utf8

Serving Up The Data

I’m using Express with a node.js server. I found that the documented default behaviour of responding with utf-8 wasn’t working, so I needed to set it manually:

res.charset = 'utf-8';
res.send(output);

There you go… I hope this helps somebody! It gave me quite a headache for a day… =D

Your project may look like the one below, a node server with support files:

my_project
├── .git
├── admin
├── assets
└── server
    ├── resources
    ├── server.js
    ├── static
    ├── views

Create a ‘staging‘ folder outside your project folder.

$ mkdir my_project_staging

In your favorite editor, create a new bash script to do the work for us.

# set staging folder
STAGEDIR=/home/rockstar/my_project_staging

# set your website host
HOST=example.com

# set the destination folder at your host
HOSTDIR=/home/user/my_project/server/

echo "staging locally, then syncing with server..."
GIT_WORK_TREE=${STAGEDIR}

echo "git working tree is now $GIT_WORK_TREE"
export GIT_WORK_TREE

echo "checking out to staging folder..."
git checkout -f master -- server

#echo "installing updates..."
#rsync -r -a -v -u -e "ssh -l ubuntu" --delete ${STAGEDIR}/server/ ${HOST}:${HOSTDIR}

Run this script in your git repository.

The checkout will ONLY check out the /server folder to your staging folder.

Your staging folder should look like this:

my_project_staging
└── server
    ├── resources
    ├── server.js
    ├── static
    ├── views

Run rsync to install the updates to your server

Uncomment the last two lines on the bash script if your staging folder looks correct and then execute the script again. Customize the rsync options to work with your host.

Wrap Up

The big problem with this method is that the timestamps are all NEW in your staging folder so rsyns send everything up to the server every time. I’m looking into a solution, probably using one of these methods mentioned here.

Good Luck!

Use underscore.date instead… it avoids the Date object altogether.

require.paths is removed in latest node

After tracking down the issue, I needed a way to install the latest cradle code from github. This is how you do it, installing into the node_modules folder:

$ git clone git@github.com:<username>/cradle
$ npm remove cradle
$ npm install ./cradle


First you clone the cradle repository. This creates the /cradle folder. Second, you remove existing cradle package using npm. Lastly, you use npm again to install from a folder. Easy as pie!

I’d like to say I’m back in business, but you know how it goes: full-time job, multiple side-projects, team call-ins, mouths to feed, etc… at least it’s now easier to publish something when the mood strikes!

stop and remove the old supervisord script

Previously, we created this file in our home directory, so you might still have that first copy in case you want to go back doing it the old-fashioned way

$ sudo /etc/init.d/supervisord stop
$ sudo rm /etc/init.d/supervisord

create an upstart script

Create a new file /etc/init/supervisor.conf. Its content should look like this:

description     "supervisor"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf

Note that we’re using the same supervisord configuration file we used before. No changes there…

here’s the gotcha!

This upstart script is modified from one found at lincoln loop. The problem with using ‘supervisord.conf’ as the upstart script filename is that the command supervisorctl picks that up and attempt to use it as a supervisor configuration file, which it is NOT. By changing the upstart script and service description to simply supervisor sans ‘d’, we can use supervisorctl normally.

using supervisord with upstart

We can now start and stop supervisord with the following commands

$ sudo stop supervisor
$ sudo start supervisor

To check the status of supervisor

$ initctl list
$ initctl status supervisor