Fictitious Nonsense

I have heard it said that Django reusable apps are not. On some level, I agree with this. On another, I disagree. Reusable apps need can be reusable if done right.

The biggest thing that make a non-reusable app is bad documentation. If documentation doesn’t say how to integrate the app, it is not reusable. There are several important things to document with reusable apps.

• Are any settings needed in settings.py?
• Are there any signals raised by the app? How are they used?
• Does the app catch any signals?
• Does the app include middleware that needs to activated?
• If middleware is needed, how does it fit in the middleware order?
• Does it define its own URLs, views, models, and templates?
• If I have to define any of these, what is needed to make them work?
• Does the app provide any management commands?

Templates are a big concern when I look at reusable apps. Many times provided templates don’t fit into my site, and I have to rewrite them. When I do this, I want to know what template tags are available. I want to know what context variables are set in the template. If there is a form, I want to know about the fields. I want to know which fields are required and how they are validated. If I have to process the form in my view, I want to know what needs to be done.

Ok, I thought I had more to say, but I guess that is all I can think of. I wanted to write more about if the app tries to do more than a single thing or if the app has dependencies on a lot of other apps that conflict with other apps that I already have, like using a different migration app than I am. But I can’t think of worthwhile arguments for  these issues. Just trust me that it matters.

In the end, a well documented app makes it more reusable. Just like any other piece of software.

Last Thursday* was the 3rd PyMNtos meeting.

Kevin Marshall gave a good talk on using Windmill to test through the browser. Windmill makes testing through the browser extremely easy by recording what you do in the browser to Python files. Then you can automate running those tests to ensure all Ajax/Javascript/etc works correctly on different platforms and browsers. If you are not testing through the browser, one way or another, you should.

After that, Matt Westerburg gave a talk on meta classes. Who knew Python was so powerful! For those that don’t know, meta classes allow you specific how a class is created. Not to be confused with __init__ which is how class instances are created. It is all very meta, which is rather suiting.

There are plans for 3 topics at the next meeting.

• Matt Westerburg will give a more indepth talk on meta classes
• I (John Shimek) will talk about virtualenv and virtualenvwrapper
• Nick Bauman will be talking about Pyjamas

There was more talked about, but it is hard to remember everything since I am bad about posting these things on time. It was a great time, and I hope the next meeting will be as well.

*If I would have finished this post on time, last Thursday would have been correct, but I didn’t so it isn’t. In fact, it was May 18, 2009, which is Thursday of last week.

At work recently, our team was asked what would make us more productive. I said drop ClearCase and embrace something, anything better. My manager asked what Git (my preferred tool) has to offer that would make us more productive. I replied with the normal things like generally faster and branching.

Today’s Lack of Productivity:

A coworker and I were working on something that had to be added to another group’s code. We had about 4 new files and 4 changed files. The procedure is to create a branch, check-in, have the other group review, and merge to main. Due to the painfulness of branches in ClearCase, we had been hijacking the files and emailing them between ourselves for a couple days. Today was the day to bite the bullet and create the branch.

The ClearCase Process:

We created the branch with the command:

cleartool mkbrtype -nc -pbranch my-shiny-new-branch

Then we updated our config specs to pull in files off the branch and to add files to the branch when checking out. Then we waited 30 minutes for our views to update. Then I tried to checkout a file I had changed to the new branch. The branch didn’t exist on the VOB. We had accidently created the branch on the wrong VOB. Time to run the command to create the branch again but in the correct VOB.  At this time, I checked out some files I had hijacked and check them in on the new branch.

Then I move on to the new files. I checked out a directory and tried to add one file to ClearCase. It sort of succeeded, except where it lost my file. Any time I tried asking ClearCase the status of the file or directory, it complained that the file is not specified in my config spec, so it can’t tell me anything. It lost my file! I couldn’t check-in, I couldn’t delete it, and try again (never try this with ClearCase, it does not like it). I pulled up my dynamic view, played around with the config spec for a bit, and still didn’t find the file. I went back to the snapshot view and checked in the folder. Now the dynamic view could find the file. It was not on the branch and it was empty. I updated the config spec so it I could check it out in the dynamic view onto the branch. I copied everything into the file, and checked it in. Updated the folder on the snapshot view, and viola, my file appeared.

I spend another 45 minutes fiddling away like this with the rest of the files. One by one, adding them through the slow as molasses dynamic view, updating a folder in the snapshot view.

The last thing I had to do was recompile everything and test to ensure I got the right versions of every file checked in. Then I sent an email to my manager telling her how this is much easier in Git.

The Git Process:

We started with some new files and some changed files. From there I could run following commands:

git branch my-shiny-new-branch
git checkout my-shiny-new-branch
git commit -a

And then celebrate the wonderfulness of Git!

The Ultimate Conclusion:

Branching is incredibly difficult in ClearCase. I had a mistake in my config spec; I was missing “element * /main/0″ at the bottom, and I lost over 2 hours because of it. If I had been using Git, I would not need a config spec, I could branch lightening fast, and be done in 5 minutes. Even if I ran into no problems in ClearCase, it was still an hour task! I doubt we will move off of ClearCase anytime soon, but it I can feel the rumble of a change coming. I just hope it is for the better.

Note: I prefer Git, but I have nothing against the other VCS; I just want to be done with ClearCase.

Background:

Over the past year and a half, I have been learning and using Python. I found that I really love Python, but was bemoaning the fact that there is no Python Users Group in the Twin Cities. There had been a Zope/Python group back in 2003/2004, but they are not active and the server with the mailing is down.

Then a couple months ago on Twitter, I saw Ian Bicking is planning to move to Minneapolis. We chatted some about him moving to the area, and he asked about the Zope group. This started gears turning in my head. Maybe, just maybe, I could start up a Python Users Group here in the Twin Cities. I mean, how hard can it be? Get a bunch of people together to talk about the same thing, should be easy.

Then fast forward to early this month at MinneDemo. I talked to Luke Francl who helped organize MinneDemo hoping to get pointers on starting an users group. He then introduced me to Scott Vlaminck who organized GUM. Luke and Scott both gave me great advice. First of all, just get together regularly every month and talk about Python. Even if it is just 3 of us. Then get a mailing list going, Google Groups is the easiest. Third, advertise.

I thought about it, and thought about it, and thought about it. I want this. I want to be part of an active community, and I don’t want to learn Ruby or Groovy (sorry Luke and Scott:). My distaste of Java is rising (after enterprise legacy app that is as agile as a…um…really un-agile thing, I prefer more dynamic languages). So that meant I need to follow through.

The Group

So on February 17, 2009, I created the MinnePy Google Group. Then I invited a few people I know and tweeted it. The response was good. There is now 19 members, plus someone said they started another group last month, but they haven’t met yet. Looks like we will be merging because the more the merrier.

My idea is that the group is open to all Python topics. I myself am Django guy, but I hold nothing against TG, web2py, SqlAlchemey, Py3k, or anything else. The more knowledge the better.  Also, feel free to post any Python related topic to the mailing list.

Logistics

Currently, we are planning to meet the second Thursday of the month, so the first meeting will be March 12 from  7 to 8 PM. For the first meeting, Luke suggested just having everyone introduce themselves talk about what we like and would want from MinnePy. The Sierra Bravo office has been offered for the meetup, but I want to talk off-line with the kind folks there before making that official.

Todo

• Have first meetup
• Choose a name (some like PyMNtos, others like MinnePy)
• Get a website
• Get a domain
• Have fun

If you or anyone you know is interested in Python and in the Twin Cities area (or care to come to the Twin Cities area once a month), please join and spread the word!

Ah, yes: things in the environment can wink out of existence as well–newtorks, files’[sic] URLs, license keys, users, printers–you name it.

Pragmatic Unit Testing

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

WARNING: DO NOT REPRODUCE THIS BUG WITHOUT REALIZING THE CONSEQUENCES!

I recently had a good need for using virtualenv due working on multiple Django projects.  With that, I decided to use virtualenvwrapper as well. I absolutely love these too programs since they make managing different environments extremely easy. Virtualenv provides a great sandbox area where I can install all the python packages a want without messing up any other environment. Virtualenvwrapper provides some commands to easily work with multiple virtualenvs.

While first testing virtualenvwrapper, I created several virtualenvs, switched to different ones, and then tried deleting one. Then I found the nastiest of bug. If no arguments are given to rmvirtualenv, it deletes $WORKON_HOME and all environments.

I am bad a running a command without the arguments and I really don’t want to lose everything. I am just glad I found this before actually putting real stuff in any enviroments.

I updated virtualenvwrapper to print an error message instead of deleting everything and emailed it to Doug Hellmann, the creator of virtualenvwrapper. He said he will put out a new version this weekend.

Other than this issue, I love virtualenv and virtualenvwrapper.

In my last post, I showed a way to override a widget on the change/add page in the Django admin site. Now I will show how to process the marked up text.

First, here is what my model looks like:

#models.py
from django.db import models

class Post(models.Model):
   title = models.CharField(max_length=100)
   slug = models.SlugField(unique=True)
   raw_body = models.TextField()
   html_body = models.TextField()
   #other random non important attributes

As you can see, I am doing a blog application. The raw_body and html_body fields are the important ones, the former one holds the unprocesses marked up text, the later holds the processed text. Since a blog is read heavy application, I opted not to run the text through the markup processed for every view. I am keeping the original so it can be edited multiple times.

#utils.py
from django.utils.html import escape
from markdown import markdown

def mark_it_down(raw):
    return markdown(escape(raw))

#models.py
from blog.utils import mark_it_down

class Post(models.Model):
    #model fields from above
    def save(self, **kwargs):
        self.html_body = mark_it_down(self.raw_body)
        super(Post, self).save(**kwargs)

The Post’s save method is overridden so that html_body can be set with the processed marked up text. I actually process the text in the mark_it_down method. This method first runs the raw text through django.utils.html.escape escape any <, >, &, “, etc with &lt;, &gt;, &amp;, &quot;, etc. Then it is run the the Markdown processor which generates the appropraite HTML. By the way, I am not displaying html_body field on the admin change/add page.

The last part is the preview with the MarkItUp! editor. If you set the previewParserPath variable in markitup/sets/markdown/set.js to a URL such as ‘/blog/markdown/preview/’, the MarkItUp will make an Ajax call to the URL with the markup text, and display the results. So I created a view to handle this:

#views.py

from django.http import HttpResponse
from moore.util import mark_it_down

def markdown_preview(request):
    processed = ''
    if request.method == 'POST':
        processed = mark_it_down(request.POST.get('data'))
    return HttpResponse(processed)

The text is in the data POST variable. The preview is displayed below the MarkItUp editor. I think this can be configured with the settings, but I haven’t played with that yet. I would like to not hard code the URL into the settings file, but then I would need to write a view for the js file and make the js file a template with the URL generated using a tag. Not hard, but I didn’t feel like doing that yet.

I believe that is everything I did, but if I missed anything or was unclear, please let me know.

In an app I am creating in Django, I decided I want to be able to format some text. There are several ways I could do this: allow all HTML, some HTML, or some other markup syntax. I decided I don’t want HTML for security reasons, either all or just a subset. So I decided to go with another markup syntax. I am looking at both Markdown and Textile. To make this work, I want an “editor” because I will never remember all the syntax.

The big part is then getting the editor incorporated into the Django admin site. It took some work, several tries, and some pure luck but in the end I am happy with the results.

First, I chose the markItUp! editor because it looks nice, it is a jQuery plugin (I am alreay using jQuery), and it is not tied to any markup syntax. To use this editor, I needed to configure Django to use a different widget than the normal textarea widget for my model. I found this post and followed its directions. I worked great, but just didn’t feel right. Then by pure chance I found that you can specify which form to use for the ModelAdmin. To do this, I created a widget, MarkItUpWidget:

class MarkItUpWidget(forms.Textarea):
    class Media:
        js = (
            'js/jquery.js',
            'js/markitup/jquery.markitup.js',
            'js/markitup/sets/markdown/set.js',
            'js/markItUp_init.js',
        )
        css = {
            'screen': (
                'js/markitup/skins/simple/style.css',
                'js/markitup/sets/markdown/style.css',
            )
        }

The widget subclasses Textarea and includes the JavaScript and CSS files needed for MarkItUp plus a JavaScript file, markItUp_init.js, that initilizes MarkItUp:

$(document).ready(function()    {
    $('textarea').markItUp(mySettings);
});

Then a ModelForm, PostAdminForm is needed that uses the MarkItUpWidget:

class PostAdminForm(forms.ModelForm):
    raw_body = forms.CharField(widget=MarkItUpWidget())

    class Meta:
        model = Post

PostAdminForm uses the Post model and specifies the MarkItUpWidget for raw_body, a model attribute on the Post model which will holds the unprocessed markup text. The last thing is setting the form for the PostAdmin and registering it.

class PostAdmin(admin.ModelAdmin):
	form = PostAdminForm

admin.site.register(Post, PostAdmin)

After all this, here is what the admin site looks like for the Post.

The top text area is for the input. The toolbar is added automatically by MarkItUp. The check mark on the toolbar generates a preview of the marked up text which MarkItUp automatically places in the box below.

Then the plain text (top area) is send to Django when saved. I have some server side hooks to process the save and preview, but I will get to that in another post.

There are still some issues being worked out, like the fact that the links are not working (notice the non-blue and non-underlined word “link” in the preview), but that is probably something with my server side processing.

The Django documentation tells how to do each bit of this, but doesn’t show to how put it all together, so it took a while and luck to randomly find the right parts (like setting form in PostAdmin). I am just happy there such a nice simple way to accomplish this. Also, any custom widget could be added to the admin change/add page like this, not just the markItUp! widget for text areas.  The main thing getting the custom widget class setup the way you need it.

I can’t wait to play with it and see what is new! I have been really digging Python lately, and am extremely excited.

I have been closely following Django as it nears the 1.0 release.  I have been using Django for about 6 months now.  It is the first framework that I have actually grokked.  Before Django, I never understood the idea of frameworks.  I don’t know if Django is the best, but since I really like Python, it suits me.

Also, my favorite changeset is #8862 because I wrote the patch used and I was thanked in the commit message.  It makes me happy:)