One of the reasons I think I’m having trouble grokking Expression Engine (EE) is because it has a slightly different take on the MVC paradigm than I’m used to from its cousin CodeIgniter, or other web frameworks like Rails or Django. ExpressionEngine seems to focus on the View (or template in EE-speak), where those other frameworks have you spend more time on the Controller or Model.

Let’s look at URL routing…how does the URL map to the code that gets executed and rendered in the browser? Let’s make it a simple example where we’re implementing a site with a structure like this – A Home/Overview page with a few sections, and each section has a couple pages:

Home
|-Section1
|
|-DetailPage1
|
|-Section2
|
|-DetailPage1
.
.
.

I want to have URLs like this:

/home/

“Home page” of the site, with it’s own glossy template with prominent links to each section:

/home/section1
/home/section2

Section home pages. These should have the same look and feel, although populated with different data. (And I don’t want to hear from you REST people saying it should be /home/section/1 — this is just a generic example)

/home/section1/detailpage1
/home/section2/detailpage1

Details pages for those that really want to dig into the subject. These should have the same look and feel even though their in different sections.

Got it? Great…let’s compare how to do that in various frameworks:

Expression Engine (v1.6.8 EE Core, although v2.0 is in beta)

Routing Method: Template Based

EE maps urls to template groups and templates in this format: /template_group/template/parameter. I can get the Home Page URL I want by setting Home as the name of my template_group, and define a default template named index.

I haven’t found a good way to get the section page URLs I want.  The few ways I tried all seemed suboptimal:

• I could create a new template for each Section, but that seems like overkill — may as well write plain HTML for each page.
• I could overload the the Home/index template to show something different based on the second segment of the url (if segment_2 =! ” do something.  Now I’ve complicated my template.
• I could make categories for each of the sections, and create a single entry for each category to show the details.  But using the category tag enforces that you have a category delimiter in your URL, so now my url path will be something like /Home/category/section1/list

The solution I’m driving toward is changing my around my URL structure so its:

/home/list/section1

/home/list/section2

I added a new segment to my URL structure. I have a template named list, which then uses the section name as a parameter to pull the correct weblog entry.  I can do /home/detailspage1/section1 in a similar manner.

What if I want to add more levels to my hierarchy, like /home/section/detail/moredetail/errata/? I don’t see how to do that in EE Core. I need the Pages Module, which comes with the paid version, which allows you to explicitly set the page URI and template.  But my budget doesn’t allow me to sink cash into this for an experiment.  I could also try out the Structure 3rd party Add-On.

CodeIgniter

Routing Method: Filesystem Based

CodeIgniter maps URLs to controller classes and methods.  So given the url /home/section1/detail, it’s going to find the controller class named ‘home’, and execute the ’section1′ method with the ‘detail’ passed as a parameter.  In the section1 you can return arbitrary HTML, or render a view — normally an HTML file with variable placeholders.  You can make a deeper hierarchy by 1) placing controllers in a nested directory tree, or 2) customizing the URI Routing rules, similar to how its done in Django and Rails.

Django and Rails (and CodeIgniter advanced)

Routing Method: Explicit URI Routing Rules

In this paradigm, the developer must open a URI routing file and write code that tells the framework to run a particular controller method when a URI is requested.  Very often you use regular expressions to make routing less verbose/more generic.  Let’s say we were doing this example with Django – I’d write a couple lines in urls.py like:

(r’^home/?P<sectionname>(\w)+’, project.app.controllername.function)

Which would tell Django to run the app.section.controller.function, passing in sectionname as an argument (yes, I know in Django they call them views instead of controllers).  Inside the controller is the code to pull the appropriate data from the database.  Rails is similar, although with Scaffolding it makes a lot more assumptions about how you structure your pages and controllers, and do some of this routing for you.

With this style, the extra complexity buys you more flexibility.  You can define any URL or URL structure of arbitrary hierarchy.  Here, your coding efforts are in the controller rather than the view/template — the view just renders the data it is given.  One benefit of this is that your template developer doesn’t have to know as much about the data structure, and can work more independently of the coding/data.

Conclusion

Once again, pick the right tool for the job…

I guess EE’s focus on templates speaks to its target audience. It started life as a simple blogging system, and grew/morphed into a more general CMS. So it has a relatively simple data model and controller, and instead makes you spend more time on templates, where the target audience is probably most comfortable. CodeIgniter, Rails, and Django on the other hand are targeted to a more technical audience…people that built web applications by hand but now want tools to make that job easier. So in those frameworks you spend most of your time customizing the data model, and specifying the controller behavior. The templates are relatively dumb in comparison.

Also, I’m guessing EE is or was targeted to single or small teams of developers/designers building a site either for themselves or a client. The result is that there is a lot of interplay between the data and the templates, and teammates have to be in pretty close contact to make sure everything works. Contrast this with Django which grew up in a newspaper environment with lots of contributors. They needed a system where designers, developers, and content writers could work relatively independently in order to meet a short deadline. So I feel like that framework better separates the concerns.

It would probably be a more fair to compare ExpressionEngine to something like WordPress or MovableType — other blog engines that have blossomed into content management systems.  WordPress even won 2009 Open Source CMS of the year, so that really is something to investigate.

This whole diatribe is predicated on the assumption that the URL structure is important to anyone. Maybe I’m trying to uphold some strange kind of RESTful ideal that isn’t practical. Certainly the client for this project doesn’t care. I only care so far as the framework acknowledges that my information architecture is going to change and grow over time — categories will come and go, new subsections will be added.  I see the RESTful pattern as a way to organize the information structure in a scalable way.  Frameworks that facilitate the RESTful style, and follow a more traditional MVC build approach,  more cleanly separate the various parts of the system, and make it easier to change and grow the IA over time.

I work for a corporate website team.  When our internal ‘clients’ want a new web project — new page, contact or registration form, major page redesign, etc — they download, print, fill out, and mail us a paper form.  We recognized the waste involved with this paper form, and the irony that the website team does not have a web enabled project request form.  We do most of our work in base PHP, but we’re also evaluating various web toolsets/frameworks/CMSs to adopt for more of our projects going forward.  So we used this small form as a little case study.  Let’s implement this form in a number of different tools.  That way we get practical experience in these different technologies, and we can see first hand what attributes we’re looking for in a web framework.

Project Requirements

We met for a half-hour just to make our project requirements explicit.  We documented the requirements in a story map, and made a quick wireframe:

• Client enters their contact details (name, department, phone, email)
  • Optional: Pull the contact details from the enterprise LDAP server
• Client enters project details (name, description, budget code, deadline date, and whether the project has VP approval)
• On submission, show confirmation page and email confirmation to client.  Also send email notification to our project managers that a new request has been made
• Members of our team may view the list of projects (especially budget codes, which we need for our time sheets), and see project details
• Project managers can add an estimate (number of hours we think the project will take), and estimate description
• When the estimate is created, automatically email an estimate to the client

Now, to build it…

Expression Engine

I got the sense that Expression Engine started its life as a blogging only platform, and someone said, “hey…we could probably use this as a general purpose CMS too!”  I felt that if I wanted a blog or multi-blog site, EE would be great, but we clearly did not understand the “EE Way.”  EE does not come with a form builder out of the box, but we found the FreeForm module which got us on our way.  There’s no GUI form builder or admin panel; it requires a developer to build the form, which was a major downside for our team.  At the end of a 3 hour sprint, we had an expression engine form, styled how we wanted, submitting project requests and sending the project request emails to our managers, but not a confirmation to the client.  We didn’t get to any edit estimate functionality, and certainly didn’t have a way to email the client when the estimate was made.

We never really felt like we ‘got’ ExpressionEngine. It would definitely take a bit more time to figure out if this is the right tool for our CMS needs. The developers on our team felt like it would’ve been much simpler to just write the thing in plain-old PHP.

Verdict: With 4 people working for 3 hours, we got some minimal functionality, but we must be missing something.

Acquia Drupal

Ahh, Drupal.  Apparantly THE open source CMS.  Yet, to paraphrase Peter Parker’s uncle, with great power comes great complexity — and a steep learning curve.  Acquia is a batteries-included Drupal distribution, with optional paid technical support, which is attractive to our company.  Acquia includes the Webform module, which is a GUI, drag/drop form builder from inside the Drupal admin interface.  While the team was at lunch, I built the form by myself.  Furthermore, the Webform module provides a nice admin interface that allows for viewing all the results, updating entries, even downloading the submissions into a CSV file. One thing I really liked about the Webform module was that it comes with an area where you can post your own custom PHP code to be run after a submission…very cool.  Another nice feature was the conditional emails:  if the submission selects A, email to person A; selectB, email person B etc.

I had it in the default Acquia Marina template.  I don’t think we really get how to use Drupal’s templating system yet, so I’m not sure how we would apply our own custom styles/templates.  But I’m confident that we’ll be able to understand Druapl templating before Expression Engine internals (which is based on CodeIgniter — which by the way I really like as a web framework).

Verdict: The webform module is the bomb: 1 person, 15 minutes, done.  Not sure how to bend Drupal templates to make it look how we want, though.

Ruby on Rails

I’ve been looking for an opportunity to try a non-blog/wiki type rails app.  I checked out the Heads First Rails book from the local library, which taught me a lot.

It turns out Rails is tailor made for this type of job.  Rails scaffolding gives you an Apple-esque out of the box experience.  One command line basically got the full app functionality.  Given the name of the database model, and a list of field names and types, rails generates a database model, and the controller and view code for basic CRUD functionality:

ruby script/generate scaffold project first_name:string last_name:string email:string phone:string project_name:string project_description:text date_needed:date budget_code:integer vp_approval:boolean

Rails just really gives lots of tools to do the things you probably want to do on a web app. Need to add/change the database schema after the initial generate?  Migrations are your friend.  Send emails that get triggered on code events?  Use Active Mailer (ruby script/generate mailer) and you’re mostly done.  The views/templating/partials makes sense.  Scaffolding also gives you infrastructure for unit tests and acceptance tests (via the Cucumber behavior driven development framework).

So I was super excited about how quickly I could get something up and running.  I call over one of our graphic artists and one of our project managers to give them a demo of how to create, scaffold, migrate, and configure a rails app.  And I got mixed reactions.  Yes, scaffolding is impressive,  but you still have to be a bit of at techie/programmer to understand how to do it.  I think they were a bit intimidated by the command line knowledge required, as well as the need to learn a new programming language.  As I created a number of apps to show different parts of the generate/scaffolding script, the PM noted, “well, it looks like sometimes it’s easier just to build a whole new application rather than fix one.”  I thought that was insightful…she recognized that the auto-generated code makes it really ‘cheap’ to build a new application.

Verdict: Convention over configuration is great, as long as you can learn the conventions.  Once I started to learn the conventions, I could do a lot in a very little time.

Conclusion

At the risk of sounding cliché, you have to pick the right tool for the job, for your team.  For this single, stand-alone project, we’ll probably roll with the Rails app.  We can get that up and running super quickly.  I don’t know that we would ‘bet the farm’ for our entire site on Rails just yet, though.  Big learning curve for our department.  We’re really looking for a content management system.  We’ve got several instances of WordPress we use, so we’re used to that.  Personally, I’m leaning toward Drupal.  Yes, high learning curve, but it has a really flexible content model, lots of modules, an active developer community, support for workflow and fine-grained authentication/authorization.  And, with Acquia you have commercial support.

But of course the door is still open, and we’re looking at other toys to play with.  I’m going to try doing this form in Pylons, which looks heavily influenced by Ruby on Rails, with a couple advantages 1) written in python which I already know, 2) less opinionated about which tools you use at various points in the web stack.  It scaffolds a rest style controller, but not the views, models, or tests.

Textmate has the ability to create new files based on a template you create.  It had a few templates for Python, but nothing exactly like I want.

When I start a new python script or module, I want to:

• Follow the Pythonista style
• Parse some command line arguments – usually an input file
• Enable logging (either file based, or to the console)

I created a new TextMate template to do those things.

1. Select Bundles > Bundle Editor > Show Bundle Editor
2. Select and Open the Python Bundle
3. Find the Python Bundle Templtes
4. Copy one into a new Template.  Give it a sensible name
5. Replace the template.py text with the following…

#!/usr/bin/env python
# encoding: utf-8
"""
untitled.py

Created by Jerry Steele on 2009-08-12.
Copyright (c) 2009 ACT. All rights reserved.
"""
import os
import sys
import logging
import optparse

LOG = None

def process_command_line(argv):
 """
 Return a 2-tuple: (settings object, args list).
 `argv` is a list of arguments, or `None` for ``sys.argv[1:]``.
 """
 global LOG
 if argv is None:
 argv = sys.argv[1:]

 # initialize the parser object:
 parser = optparse.OptionParser(
 formatter=optparse.TitledHelpFormatter(width=78),
 add_help_option=None)

 # define options here:
 parser.add_option("-f", "--file", dest="filename",
 help="read data from FILENAME")
 parser.add_option("-v", "--verbose", dest="verbose", default=False,
 action='store_true', help="write debug log to FILENAME")
 parser.add_option("-L", "--log", dest="logfile", help="write debug log to FILENAME")
 parser.add_option(      # customized description; put --help last
 '-h', '--help', action='help',
 help='Show this help message and exit.')

 options, args = parser.parse_args(argv)

 # check number of arguments, verify values, etc.:

 # set up logging
 if options.verbose:
 LOG = setlogging(options.logfile)

 if not options.filename:
 pass
 #LOG.error("Input filename not specified")
 #parser.error("You must supply an input file")
 # further process settings & args if necessary

 return options, args

def main(argv=None):
 settings, args = process_command_line(argv)

 # application code here, like:
 # run(settings, args)
 return 0        # success

def setlogging(logfile=None):
 consolelevel = logging.DEBUG
 logger = logging.getLogger(__name__)
 logger.setLevel(consolelevel)
 # create formatter and add it to the handlers
 formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
 # create console handler with a higher log level
 ch = logging.StreamHandler()
 ch.setLevel(consolelevel)
 ch.setFormatter(formatter)
 # add the handlers to logger
 logger.addHandler(ch)

 # create file handler which logs error messages
 if logfile:
 filelevel = logging.ERROR
 fh = logging.FileHandler(logfile)
 fh.setLevel(filelevel)
 fh.setFormatter(formatter)
 logger.addHandler(fh)

 #test logging
 logger.debug("debug message")
 logger.info("info message")
 logger.warn("warn message")
 logger.error("error message")
 logger.critical("critical message")

 return logger

if __name__ == '__main__':
 status = main()
 sys.exit(status)

Even if you don’t use Textmate, you can still use this to quickstart python modules.  Just remove/replace the $TM_ variables

About a month ago a wrote a simple web response timer, because I didn’t quickly find a tool out there that could do it.

I should have looked harder.

I finally found pylot — a python program for running HTTP load tests.  Pylot does what I needed to do (calculate some statistics around page response time) and a whole lot more.  Pylot is designed to benchmark/load test HTTP web services, so you can profile arbitrary URL’s with either HTTP GET or POST requests.  Or, you can just give it a simple URL or two like I did.  You define a simple XML file with your “test cases” (the URLs you want to profile, along with any parameters), and give it some runtime parameters (number of virtual users that will hit the URL, request interval, rampup time, whether or not to launch a GUI to watch the stats in real time).

Much simpler than what I did.  And it generates much prettier reports and graphs, without the need for loading into a separate statistics program.

Generator Tricks for System Programmers really opened my eyes to the utility of python generators.  Yesterday I took an opportunity to use one, and bone up on my regular expression ninja skills to boot.

I received an email from a colleague containing some references to reports our company had written, so that we can post them in appropriate places on our website.  He clearly went through some pains to organize the information in a very human-readable manner:

CATEGORY 1
Report Title1 (report type year)
Report Title2 (report type year)
CATEGORY 2
...

Which is great.  The trouble is, I also need the existing URL associated with each of these reports.  And it may make sense to pull this list from a database, so I’d like to treat each one of those as a row with certain attributes:

NAME  | TYPE | YEAR | CATEGORY

Oh sure, I could copy the text into Excel and put everything into columns by hand, but where’s the fun of that?  Python to the rescue!

Python generators are kind of like a list, but without the list.  Like a list, it is a sequence of things (ints, strings, other lists, objects, etc).  Like a list, you can iterate through each value of the generator.  But instead of storing the entire list in memory, it evaluates some function to generate the next value.  This means that generators can more efficiently deal with very large sets of inputs.  You’re not loading the whole input set into memory, instead you just ask for the ‘next’ value, and the generator decides what to spit out.  It might mean reading the next line of a file, adding or modifying an object, or calculating the next value in the Fibonacci sequence.

Check out the System Admin’s Guide to Generators for a more full explanation, or look up the documentation on the yield statement.  Python also has a nice shorthand for generator expressions, that is very similar to how list comprehensions are done.  It often leads to pretty clean, readable code.  Here’s the key section of code from this example:

regex = re.compile(r"(?P<name>.*) \((?P<type>.*) (?P<year>\d*)\)")    

with open ('researchreports.txt') as infile:
    lines = (l.rstrip() for l in infile)   
    matches = ((regex.search(l),l) for l in lines)   
    newline = (matcherfunction(m) for m in matches)

First, we define the regular expression used to parse the line, and extract the report name, type, and year.  The next four lines do the actual work:

• Open the file for reading
• From the open file, spit out each line, stripping off whitespace from the end
• For each of those lines, run the regular expression.  Spit out a tuple of (Match Object, original line).
• For each of those tuples, run the matcher function, which spits out the tuple (name, type, year), or the original line in the event where the original line wasn’t in the right format.

That’s basically the end of the magic.  The rest is just writing out to a csv file.  Python’s CSV module to the rescue.  Here’s the whole code in case you’re interested.

#!/usr/bin/env python

from csv import writer
import re

def matcherfunction(m):
    """if we have a MatchObject, return the parsed output.  if not, return the original line"""
    if m[0]:
        return (m[0].group('name'), m[0].group('type'), m[0].group('year'))
    else:
        return m[1],

regex = re.compile(r"(?P<name>.*) \((?P<type>.*) (?P<year>\d*)\)")       
with open ('researchreports.txt') as infile:
    lines = (l.rstrip() for l in infile)   
    matches = ((regex.search(l),l) for l in lines)   
    newline = (matcherfunction(m) for m in matches)

    with open ('researchreports.csv', 'w') as outf:
        csvfile = writer(outf)
        headers = ['','TYPE','YEAR', 'URL', 'CATEGORIES']
        csvfile.writerow(headers)
        csvfile.writerows(newline)

Did I over complicate the problem?  Probably.  I could’ve just read in the whole file as a string, and then done a global regex search/replace.  But that would be problematic if I were dealing with a huge input file.  The advantage of this approach is that it doesn’t matter how many rows there are; it’ll march through them with no worries about memory limitations.  Second, it’ll be easier to modify and reuse this approach than a custome RegEx.  Finally, it apparantly fits my mental model of how to solve the problem.

What I really want is for someone to show me how to do this in one line with awk/sed.  =)

For my day job, I’m creating a series of HTML pages that each have a table that shows how our various services and solutions map onto problems our customers are likely to have.  The main site is currently thousands of static HTML pages, with a bit of PHP thrown in a few pages to do page footers.  We’re working on upgrading to a dynamic CMS type site.  In the meantime, I used the opportunity to learn a bit more about PHP and I wrote a small function to generate the table HTML given a JSON document describing the table headers, rows, and content.

As I was debugging the sites, I felt like there was sometimes a noticeable delay in rendering the page that wasn’t there on the existing static pages.  Was this my imagination, or something that our users might notice and complain about.  Hmm, I don’t have any web profiling software, and I couldn’t find anything that I could quickly install and run.  And I had some time.  Looks like I have to write some code.  In the immortal words of Leeroy Jenkins, Let’s Do This!”

(Update: I found a simpler way to profile pages in python)

Python timeit Module

Python’s mantra is Batteries Included, implying that for whatever coding task you have, there’s probably something in the standard library that will do muct of what you want.  You shouldn’t have to go and write something completely from scratch.  I knew about python’s time module.  I was planning on using it to mark the time before fetching my webpage, mark the time after fetching the page, and comparing the two.  But I stumbled onto the timeit module, which makes it even a bit easier.  Timeit basically wraps up that logic of marking time before and after some bit of code in convenient package.  You give the timeit.Timer() class a bit of code that you want to time.  The timeit() method will run the code a specified number of times (default 1,000,000) and return the average time for code execution.  The repeat() method will run the timeit() method a specified number of times, and return a list of the average times.

In action, it looks like this:

import timeit

# Request the page 100 times, time the response time

t = timeit.Timer("h.request('http://PAGE/URL',headers={'cache-control':'no-cache'})", "from httplib2 import Http; h=Http()")
times_p1 = t.repeat(100,1)

Three lines of code…not bad.  The Timer() class takes two strings as parameters: 1) The python code you would like repeated and timed, 2) Python code required to run before each run of the test code.  If you’re familiar with Unit Testing, then the 2nd parameter is like the setUp() method.  Notice I’m using the httplib2 library instead of the standard urllib library.  I like httplib2 for requesting urls because I’m familiar with it, it combines requesting the url and reading its contents, and its really good about dealing with caching.  In this case, I don’t want the server to cache.

The second line instructs my Timer() to run 100 sets of my test code, with 1 trial per set.  The output is a list of 100 times.

The documentation for timeit.repeat() gives some good advice on how much stock to put into these numbers, and using mean/standard deviation to describe the performance.  But what I really wanted to know was whether or not my page took significantly longer to load than a similar page with no dynamic content.  I expanded my code to repeatedly time a second, static page, and the two lists in two columns of a csv file.

import timeit
from csv import writer

# Hit the dynamic page 100 times, time the response time

t = timeit.Timer("h.request('http://PAGE1/URL',headers={'cache-control':'no-cache'})","from httplib2 import Http; h=Http()")
times_p1 = t.repeat(100,1)

# Now hit a similar static page 100 times
t = timeit.Timer("h.request('http://PAGE2/URL', headers={'cache-control':'no-cache'})","from httplib2 import Http; h=Http()")
times_p2 = t.repeat(100,1)

# the times to a CSV file
times = zip(times_p1,times_t2)

with open('times.csv','w') as f:
    w = writer(f)
    w.writerows(times)

Note we’re using the python with statement from Python 2.5+, which encapsulates some of the try/except/finally logic you’d normally write when opening a file.    Because I had even more spare time, I imported my new times.csv file into a statistics program (SPSS) to calculate mean, and perform a T-Test to see if the means of the two columns they are statistically different.  I also could have used various statistics scripting tools: scipy, R, for example.  But I didn’t have THAT much time. 

There was a statistically significant difference.  The dynamic page was, on average, about 1.2 ms slower than the static page.  This makes practically no difference to the user experience of the page, and makes my development life much easier (and also illustrates how practical significance may differ from statistical significance).  I’ll continue to generate pages dynamically.

As I mentioned last week, I’m using Pinax to test a new website for my research lab.  The site is now in private alpha, which means I’m getting some internal users to log in, test it out, and give me feedback, and I don’t think it will fall over.  I’m writing up my experience to 1) help others get started quickly, and 2) start talking about how the pinax community can get people ramped up faster.

My background:

• Journeyman Python Programmer. Can independently write code, diagnose most problems.
• Journeyman HTML/CSS/Javascript. Can independently write code, diagnose most problems.
• Some experience doing apache configuration, though not recently. No experience with mod_python or mod_wsgi. I largely rely on documentation and examples found on the interwebs.
• No major prior experience with django, and none deploying it via Apache/mod_{python,wsgi}. I almost exclusively rely on documentation and examples found on the web (although my python experience helps)

The nice thing about pinax is that it has an active community; in particular an active IRC channel (#pinax on freenode). The pinax maintainers have been there, patiently answered my newbie questions, and got me going.

Step 1 – Get the base pinax software running.

Download and install django and pinax.   I first did this on a WinXP desktop. Crap. The templates aren’t rendering properly. I try commenting out the offending lines in the templates, but problem after problem comes up. Jump on #pinax.  Did you do seed the database with manage.py syncdb?   Uhh, no.  I do that, but still having problems.  Back to #pinax.  Are you running WinXP?  Why yes!  Oh, pinax on Windows has a known bug (which really is caused by one of the external apps they rely on).  Copy some files to a new location, or install on an OS which supports soft links.  I choose door 2, recheckout on my Ubuntu machine, sync the database, run the django dev server ( manage.py runserver) and we’re good to go.

Step 2 – Make sure everything in the base site works as expected.

The pinax project comes with a complete working example under the directory projects/complete_projects.  I copy this to a new directory so that I can refer back to the complete_project in case I screw something up.  This means you have to change the ROOT_URLCONF in the settings.py

Added some users via the admin interface…looks great.  Added an account via the Sign-Up page.  Accounts are created (with or without OpenID), but no confirmation emails are being sent.  Hmm  Check out the included django-mailer app, which uses the same configuration as the standard django.core.mailer.  We need to add some settings to settings.py: EMAIL_HOST, EMAIL_HOST_USER, EMAIL_HOST_PASSWORD.  I set these to use my gmail account to do email.  But really, it’d be better to have the emails coming from the local machine, and I really don’t want to hardcode my gmail password into the config file.  I need sendmail on this machine (sudo apt-get install sendmail).  Now that the local machine has its own SMTP server, I can remove the EMAIL_* settings from settings.py.

Hmm…emails still not going out.  Let’s look at the django-mailer docs.  Oh, I see…email messages aren’t actually sent until manage.py send_mail is run from the command line.  The recommended solution is to add a couple lines to the system crontab so that they are run regularly (this is now much clearer in the pinax deployment docs).  Now we’re sending emails!

Step 3 – Configure templates, images, and site text for our lab.

Ok…first thing is that the default site says ‘Pinax’ everywhere.  I need to change that to the name of our lab, GROK Lab.  There’s an outstanding ticket to use a SITE_NAME variable in settings.py which would be displayed throughout the site.  Until that gets resolved, we just have to find all the hardcoded [Pp]inax strings are replace them with the SITE_NAME variable.  I just used grep to find them all, and then I replaced the ones I needed to replace (I left things like Built with Pinax, About Pinax, James Tauber is the awesomest creater of Pinax ever, etc).  Pinax has several translations available, which I think means that I should have been making most of my changes to externalized text strings in locale/en/LC_MESSAGES.  But I didn’t, and just changed the template files.  I’m not sure what will happen when another language is selected.  Should learn more about django internationalization.

Update 11/14: Fernando Correia did some more digging on how to properly externalize strings in django.

One particular place this got me into trouble was, again, the django-mailer app.  Invitations to join the site are generated via two templates: templates/friends/join_invite_message.txt and templates/friends/join_invite_subject.txt.  I wanted the subject of my email to say “You have been invited to join the new SITE_NAME.”  I tried to use the variable substitution syntax {% site_name %} but got an exception that I couldn’t use that tag inside a trans tag.  Read more django docs to learn about template tags and internationalization.  I also tried changing the join_invite_subject.txt to use {%blocktrans %}, similar to the other template file in that directory.  Oops.  I accidentally added newlines to the subject template, which django-mailer does not like.  I revert to the original join_invite_subject file, and simply hardcode ‘GROK Lab’ into the subject string.  Hmm…still getting the same error messages about newlines in the header, even though I’ve changed the template.  Oh, the old messages with the bad header are still in the email queue…its trying to send those, not render new ones based on this template.  I clear out the emails with the bad header out of the database via the admin interface, and email is working again.  Whew…

The logo image says Pinax.  That’s no good…I want to replace that with our lab logo.  Everything is nicely styled with CSS.  My image is a slightly different shape than the pinax logo, so I need to change the height/width in the base.css #tabhead .logo img selector to match my logo file (I made our logo the same height as the original pinax logo to simplify).

This is supposed to be a serious research site, so we probably don’t need games (although I played some on cloud27 — they’re pretty fun, and might be a nice way to encourage a more active community).  I commented out the arcade and games apps in the settings.py, all the related ARCADE_* variables, and the URLpattern in urls.py.  Oops…many of the templates expect to have the arcade app installed (for example, the navigation tabs are hard coded).  Use grep again to find all mentions of ‘arcade,’ and comment them all out.

Finally, there were some functionality that the default installation hides if the user is not logged in.  I wanted to be able to have some read views of the people, projects, blogs, tweets available to anyone who visited the site, but only allow writing if they were logged in.  This involved again grepping through the templates for the right section of code, and adding {% if user.authenticated %} … {% endif %} tags.

Step 4 – Configure site to run through Apache instead of the dev server.

So I had the site up and running using the django development server on the default port 8000.  But I needed it deployed through apache so that in case of a server restart, I did not have to restart the site by hand.  The most commonly referenced post for deploying pinax is here.  It gives two sample configurations depending on whether you want to use apache modules mod_wsgi or mod_python.  Which one do I want to use?  What’s the difference between the two?  No idea.  Look up django docs on deployment; it seems to recommend mod_python, so I’ll try that. I wrestle with that for a few hours, but little success.  Back to #pinax, where the consensus is mod_wsgi good, ignore mod_python.  Ok, after another few minutes of messing with my apache site config file, and the pinax.wsgi file, we got it up and running.  A few gotchas:

• Added the appropriate WSGIScriptAlias to my apache site config.  I could view the site; however, I got errors that apache didn’t have permissions to write to the db.  Added WSGIDaemonProcess to the site config and that fixed it.
• I could access the admin site, but it did not have the correct styles/media.  Needed to add an Alias and Directory commands to the apache site.
• Changed the name of the pinax.wsgi to match our site name; need to change the DJANGO_SETTINGS_MODULE variable inside this file

So the appropriate section of my apache VirtualHost configuration looks like this:

# attempt to setup pinax
WSGIScriptAlias / /path/to/your/pinax/project/deploy/pinax.wsgi
WSGIProcessGroup name
WSGIDaemonProcess name user=unixusername group=unixgroup threads=25

Alias /media/ "/path/to/django/contrib/admin/media/"
<Directory "/path/to/django/contrib/admin/media/">
Order allow,deny
Options Indexes
Allow from all
IndexOptions FancyIndexing
</Directory>

When debugging this, it was helpful to watch all the appropriate log files to see what error messages were being generated where: tail -f apache.log apacheerror.log projectsend_mail.log project.send_mail_deferred.log.

Conclusions

Clearly one area where we can improve pinax is the installation experience.  Compare it to the wordpress install experience…download into a web accessible directory, open a web page, short configuration, done.

More difficult to deal with is the fact that, in order to use pinax, you need to acquire at least a passing knowledge of django convention and configuration, apache convention and configuration, and all the external apps bundled with pinax.  This can be a tall order.

Still, I got my site up and running, with much more integrated functionality than I would have had with django alone.  I learned alot about pinax, django, and apache, and should have less trouble on my next site.  And I get the desired ‘oohs’ and ‘ahhs’ and ‘damn, how long did it take you to do this?’ from people in the demo.

The next step is a bit more fun.  Now that I’ve got a basic feel for the technology, I can really think about how to design the site and workflows to best serve the users.

I’ve had fun this afternoon playing with Pinax, and new social media framework based on Django.  I got it up and running very quickly, and with very little configuration (changing template files here and there, styles, etc) I’ve got a brand new website for my research lab that features user profiles, project sites complete with wikis and task assignment, social bookmarking, discussion boards, and a host of other features. I showed it to a couple of my labmates.  They were able to log in and get up and running without any prompting for me, so it seems to have a familiar and easy-to-use-and-learn interface.  Those that were running their own projects enjoyed the project interface, and the ability to link and follow activities of different team members.

I think James and the rest of the Pinax team have hit the nail right on the head.  Out of the box I get about 90% of the functionality I was considering building for our lab.  I’d like to create and and in a django-references app, which would store citation information, who’s read what, who’s written what, etc.

I’m looking forward to seeing how much functionality I can add in how much time.  This type of thing could really transform scholarly communication:)

Not sure why this was so hard for me to use the sorted() method this morning.

I’m playing with the FriendFeed API, I wanted to retrieve the entries of a FriendFeed room, and then sort the returned entries by the number of comments…

Getting the latest 30 entries is a simple HTTP call.  FriendFeed returns a json structure which is nicely parsed with simplejson (soon to be part of the standard python library).

import simplejson
import urllib2

r = urllib2.urlopen('http://friendfeed.com/api/feed/room/science21')
json = simplejson.loads(r.read())

# I just want to look at the entries in the room
e = json['entries']

I want to sort the entries by the number of comments.  In other words, I want to use the number of comments as a sort key.  Luckly, Python’s sorted() method has a key attribute, which takes a callable that returns a single value to be used as a key.  We use the operator module in order to generate the sorting key method:

ln = lambda x: len(operator.getitem(x,'comments'))
esorted = sorted(e,key=ln,reverse=True)</code>

for i in esorted:
print i['title'].encode('utf-8'), len(i['comments'])

That took me all morning to get, which shows some of my Python programming limitations.  My first attempt, I tried this:

In [94]: es = sorted(e, key=len(operator.itemgetter('comments')))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

TypeError: object of type 'operator.itemgetter' has no len()

Now I recall that an object must have a .__len__() method in order to work with len().  But operator.itemgetter() returns a function, not an object with a .__len__(), hence the TypeError exceptin I got.

Then I tried this:

esorted = sorted(e, key=lambda x: len(operator.getitem(x,'comments')))

Which is very similar to the solution I ended up with, but this one returns the list in different order.  I’m not sure why yet. Update: Duh…we get a different order because the first one has reverse=True, and the second one doesn’t.