#Django #TransactionTestCase with REUSE_DB=1 of #django-nose

Lately, I found out that Django’s TransactionTestCase leaves test data in database after the test case is executed. It’s not until the next execution of _pre_setup method of a TransactionTestCase instance that the database is flushed. This is troublesome when tests are run with Django Nose’s test runner with REUSE_DB =1.

An easy fix to this is to customize the TransactionTestCase so that it deletes the test data on exit. I wrote a simple wrapper around Django’s TransactionTestCase and extend it to write other transaction test cases.

from django.test import TransactionTestCase
from django.db import connections, DEFAULT_DB_ALIAS

def flushdb(cls):
    if getattr(cls, 'multi_db', False):
        databases = connections
    else:
        databases = [DEFAULT_DB_ALIAS]
    for db in databases:
        management.call_command('flush', verbosity=0,
            interactive=False, database=db)

class BaseTransactionTestCase(TransactionTestCase):
    @classmethod
    def tearDownClass(cls):
        flushdb(cls)

Advertisements

App specific logging in Transifex

Yesterday, I was working on adding app specific loggers in Transifex. By app specific logger I mean a logger which shows the app name which generated the log. As of now, the logs in Transifex look something like this:


2012-06-29 13:01:43,300 tx DEBUG Saved: Project Avant Window Navigator
2012-06-29 13:01:43,312 tx DEBUG Saved: Project Switchdesk
2012-06-29 13:01:43,324 tx DEBUG Saved: Project Usermode
2012-06-29 13:01:43,342 tx DEBUG Saved: Project desktop-effects
2012-06-29 13:01:43,349 tx DEBUG Saved: Project im-chooser
2012-06-29 13:01:43,355 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:43,364 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,704 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:45,717 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,731 tx DEBUG Resource Resource1: New ResourcePriority created.

It does not tell anything about which app generated the logs. In a first glance, fixing this looks pretty straight forward and dumb. All it needs it to customize this https://github.com/transifex/transifex/tree/devel/transifex/txcommon/log module for each app and instead of importing the logger from txcommon.log, import it from the log module inside the app.
But this would lead to a lot of code duplication and a lot of boring changes in the code. So, I decided to customize transifex.txcommon.log module itself so that it can detect the function calling the logger. It was pretty straight forward to do this for the handler at https://github.com/transifex/transifex/blob/devel/transifex/txcommon/log/receivers.py#L6: def model_named() in the following way:

import re

tx_module_regex = re.compile(
    r'transifex(\.addons)?\.(?P<app_name>\w+)(\..*)?')
def model_named(sender, message='', **kwargs):
    """
    Receive signals for objects with a .name attribute.
    """
    from txcommon.log import _logger as logger
    sender_module = sender.__module__
    m = tx_module_regex.search(sender_module)
    app_name = '.' + m.group('app_name') if m else ''
    logger.name = 'tx' + app_name
    obj = kwargs['instance']
    logger.debug("%(msg)s %(obj)s %(name)s" %
            {'msg': message,
            'obj': sender.__name__,
            'name': getattr(obj, 'name', '')})

sender is the object or instance for which the log is being generated. In our case, it’s a model instance. So, sender.__module__ gives the parent module for sender. Using regular expressions, we extract the app name from the module name and we set the name of the logger as ‘tx.<app_name>‘. And we are done here (for now)! But when we do something like

from transifex.txcommon.log import logger
logger.debug('foo bar')

we do not have a sender instance to allow us to find the calling module name. After some searching, I found about the inspect python module. And all I needed was inspect.stack(). Here’s what I did in https://github.com/transifex/transifex/tree/devel/transifex/txcommon/log/__init__.py:

  1. Write a wrapper around logger instance,
  2. find the caller calling the logger using stack.inspect(),
  3. accordingly set the logger name,
  4. and finally, log the event.

import logging, re, inspect

_logger = logging.getLogger('tx')

# regex to extract app name from a file path to a TXC app
tx_app_path_regex = re.compile(
r'txc/transifex(/addons)?/(?P<app_name>\w+)/(\..*)?')
class Logger:
    """
    A wrapper class around _logger. This is used to log events
    along with app names.
    """
    @classmethod
    def get_app_name_from_path(cls, path):
        """
        Extracts app name from a file path to a TXC app

        Args:
            path: A string for the file path
        Returns:
            A string for the app name or ''
        """
        m = tx_app_path_regex.search(path)
        return m.group('app_name') if m else ''

    @classmethod
    def set_logger_name(cls):
        """
        Sets logger name to show calling app's name.
        """
        # inspect.stack()[2] since cls.debug() method has now become the
        # immediate caller in of this method in the stack. We want the caller
        # of cls.debug() or other logging method wrappers.
        caller_module_path = inspect.stack()[2][1]
        app_name = cls.get_app_name_from_path(caller_module_path)
        _logger.name = 'tx' + '.%s' % app_name if app_name else ''

    @classmethod
    def debug(cls, *args, **kwargs):
        """Wrapper for _logger.debug"""
        cls.set_logger_name()
        _logger.debug(*args, **kwargs)

    # And similarly for other logger methods like info(), waring(), error(), critical()

logger = Logger

Now, this is sweet! No one need to bother about logging events with app names. I am saved from editing hundreds of files and duplicating code 😉 It’s transparent and scalable. The logs now seem like:

2012-06-29 20:39:03,635 tx.projects DEBUG Saved: Project Foo Project
2012-06-29 20:39:05,575 tx.projects DEBUG Saved: Project Avant Window Navigator
2012-06-29 20:39:05,587 tx.projects DEBUG Saved: Project Switchdesk
2012-06-29 20:39:05,599 tx.projects DEBUG Saved: Project Usermode
2012-06-29 20:39:05,612 tx.projects DEBUG Saved: Project desktop-effects
..........
..........
..........
2012-06-29 22:15:07,088 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: ReleaseNotifications
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: Release All Resources
2012-06-29 22:15:07,466 tx.txcommon DEBUG Running low-level command 'msgfmt -o /dev/null --check-format --check-domain -'
2012-06-29 22:15:07,469 tx.txcommon DEBUG CWD: '/home/rtnpro/transifex/rtnpro/github/txc/transifex'
2012-06-29 22:15:07,661 tx.releases DEBUG release: Checking string freeze breakage.
2012-06-29 22:15:07,702 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (pt_BR)
2012-06-29 22:15:07,707 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,740 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (ar)
2012-06-29 22:15:07,745 tx.webhooks DEBUG Project project1 has no web hooks

Thanks for reading. If you have any suggestions or query, please feel free to comment.

FUDCON KL Day 3

The 3rd day of FUDCON KL started a bit sluggishly for me. May be because of brainstorming and hacking till late night. We (Kushal, Soumya and me) decided to work on a new app to display system logs in a user friendly manner. We named the application Tower log tower, in short, tlogt, after Twin towers of Kuala Lumpur 😉

During the first few hours of the day, we went to visit some tourist spots in Kuala Lumpur: Aquaria and Petronas towers. After we returned, we settled down for the on going talks. Amidst of various talks on the 3rd day of FUDCON, I was sometimes in listening mode, but for most time I was in coding mode. We decided to try something different in TlogT. The UI would be rendered by a Django daemon with all the WOW factor of HTML, CSS and JS. I was to code the Django server code, while Kushal and Soumya were working on writing the parsers for extracting the logs for various processes. In a few hours, we had a decent Django based functional desktop app ready. Although, quite some work remains to be done on the UI part.

There was Kushal’s session on Python for newbies in the afternoon. It’s always nice to see Kushal talk on Python. I don’t have the exact count, but I am sure that Python sessions by Kushal inspired many (including me and my friends) to start coding in Python. The ending keynote for the day was given by Abu Mansur Manaf. This should really boost newbies to become contributors 🙂

I spent the evening in the hotel room listening about various functional programming languages and features of the languages from hircus and Kushal. Later, we went out for dinner with the rest of the event crew members to a local food joint. I stayed up at night to see off other Kushal, Soumya and others who had to catch an early morning flight back to India. After bidding them good bye, I packed my bags and went to sleep.

Me & FUDCON India 2011

I arrived in Pune for FUDCON on November 2, 2011. On November 3, 2011, I had an opportunity to visit the Red Hat office in the city and hang around with Fedora community members. I learnt more about the mechanism and importance of localization from Runa Bhattacharjee. In my free time, I also helped with some FUDCON related work. I  was able to meet Jared Smith, Joerg Simon and Robert Scheck for the first time in my life, talk with them. I couldn’t ask for anything better.

Day 1: November 4, 2011

Day 1 began with a speech from the Director of COEP followed by the keynote speech by Fedora’s Project Leader Jared Smith. Jared explained to people the various aspects of Fedora and how Fedora takes the lead in pushing the limits of FOSS development. Following that was my talk on Transifex. The audience included people mainly from the L10n domain. There were some mishaps during the talk. First, my Dell XPS laptop wasn’t able to use the projector; second, I had to use another netbook which I wasn’t used to; third, there were some power issues, and the netbook turned off and finally, there was power cut for a few minutes. What a chain of mishaps! I had to resort to reading the slides from my phone during the power cut. Finally, I used Kishan Goyal’s laptop to continue with the presentation. In  my talk, I explained the current features and upcoming features of Transifex. I explained it with a use case, starting from registering to advanced usage of www.transifex.net. The audience appreciated our upcoming market place idea and the Translation Memory feature. I also got feature request for having a global glossary for a particular language in the language page. It was really nice that the audience were so actively communicating during the session. I also told how to start contributing to the Transifex project and shared my experience working on Transifex so far: from a contributor to an employee.

After the lunch, I attended Heherson Pagcaliwagan‘s session on “Fedora web of trust” and got more insight into the use of GPG keys. We did a small workshop with Heherson on how to get introduced to each other, verify identity, share GPG key and sign it. Heherson also showed us how to encrypt mails using GPG key. Then I attended Joerg Simon’s talk on Fedora Security Lab and OSSTMM. Kital showed us a variety of security tools that can be found in the FSL, and mentioned others that need to be packaged. I must try out the tools in the FSL now. They are so cool.

Then I went back to the speaker’s lounge and started writing some code on Transifex. It was a great first day at FUDCON for me.

Day 2: November 5, 2011

The Second day of FUDCON began with Harish Pillay speaking about the community. Unfortunately, I was not able to turn up during the keynote, as I had to re create my slides on my talk on “Testing your Django app”. Because I had accidentally, deleted the folder which contained the slides. After I was done, I hurried to attend Arun Sag’s talk on “Creating web apps using Django”. I liked Arun’s way of presenting things to the newbies in a very lucid way. He used his classic Blog example for this.

Next was my talk on “Test your Django app”. As during Day 1, the projector did not work with my laptop, so I used Arun’s laptop for the purpose. I explained why tests are necessary, different testing frameworks in Django (doctest, unittest). Then I went forward to explain how to write simple unittests. For this, I wrote some tests for Arun’s blog example and used it so that the audience could relate things with the previous session taken by Arun. I showed some simple test cases and ran the tests. Then, I introduced the Django Test Client and spoke about its importance and features. After explaining things about the Test Client, I showed the relevant code and ran the tests again. Finally, I explained about coverage: what is it? why is it required? How to use coverage? I again ran the tests I had run before, but this time I ran them with coverage and explained how to read the coverage report. I haven’t been too happy with this session of mine. Now, when I think of it, for students who just got introduced to Django, the session on Django testing might have been asking too much. Anyway, the students now at least know something called Django testing exists. So, when they need it, they can learn.

After having lunch, I attended Siddhesh’s talk on “Security Exploits”. During the session, I could not but think that why I did not have someone like Siddhesh teaching me OS in my college. He was awesome. Back in my college days, I had tried reading about security exploits, but I did not get far. But during this session, thanks to Siddhesh’s explanations and my earlier reading, I have gained a better understanding of security exploits, especially stack smash attack, overwriting nearby entities in data region by string overflow, etc.

Following were two lightning talks on 1) How to deal with kernel panic? 2) Running external commands from Postgres. Yogesh in his talk on “kernel Panic” showed how to collect relevant data when there is a kernel panic. This data can be used for creating useful bug reports or for fixing the bug itself.

After attending Siddhesh’s talk on autotools, I joined mether’s talk on ask.fedoraproject.org or askbot. Mether discussed its roadmap and mentioned various feature requests. I picked up to implement a few of the feature requests. After the session, we had a group photo session of the almost all the people involved in FUDCON. Then all the speakers, volunteers and organizers went for the FUDPUB. I enjoyed a lot at the FUDPUB. I spoke with the community members, danced with them, drank Mirinda and ate some delicious food. It was just awesome.

Day 3: November 6, 2011

It’s the hackfest day. I decided to run a Transifex Testathon. I pitched the topic on stage and invited people to join me. I helped some of my friends install and setup transifex in their machines and showed them how to run tests. I started writing new tests for the watches addon in Transifex. I came across a chain of undiscovered bugs while doing so. It took me some time to write a proper test case for the watches and accordingly fix the bugs in the code. I also helped Kushal to get him logged into his Transifex account and creating a Transifex project for “Python for you and me”. Jared Smith, set up the tx client for “Python for you and me” and now PYM is hosted at https://www.transifex.net/projects/p/pym/ for localization. Shreyank, Vaidik and I had discussions on the roadmap for Dorrie. I setup Dorrie on my machine and played with it for some time. I decided to write tests for Dorrie the coming weekend.

Then, at the end of Day 3, in the auditorium, a cake was cut to celebrate this FUDCON along with quite a few photo shoots. The FUDCON organizing group and the volunteers from COEP did a great job to make this event go on smoothly.

FUDCON India 2011 is the first ever FUDCON in India and the largest FUDCON in terms of the number of participants. Apart from learning new stuffs and hacking, FUDCON provided a great platform for Fedora contributors to meet with each other and make new friends. It is also a nice experience to work with people whom I had known only in the IRC until now. I am carrying sweet memories of this FUDCON with me. These memories will help me focus more on contributing to open source and be a better contributor.

Start testing Transifex

 

How do you setup Transifex?

Here is all you need to know to setup Transifex: http://help.transifex.net/server/install.html

http://fosswithme.wordpress.com/2011/10/20/setup-transifex-in-virtualenv/ is another good write-up on how to setup and run Transifex in virtualenv. So, I’d be building on top of that to show you how to start testing Transifex using django-nose.

What packages will you need?

django-nose, nose, nose-exclude, coverage

You can install the above packages using

pip install <package_name>

Configure Transifex settings to enable django-nose Test Runner

cd <transifex source code's root directory>

cd transifex/settings

cp  90-local.conf.sample 90-local.conf

open and edit 90-local.conf and add the following lines:

INSTALLED_APPS += [

‘django_nose’,

]

TEST_RUNNER = ‘django_nose.NoseTestSuiteRunner’

cd ..

Now, save the file. Now you are good to go.

Start testing

python manage.py test <app_name>

for example:

python manage.py test resources

You can also test a particular test class like below:

python manage.py test resources.tests:TestJavaProperties

You can even run a particular test method:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser

Run coverage on Transifex tests

django-nose has a plugin for coverage. So, you can run the above tests and collect coverage data.

For example:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser --with-coverage --cover-package=resources.tests.formats

All the coverage results are saved in a .coverge file by default in the current directory. Although, running tests with coverage plugin of django-nose shows the coverage results by default. You can also see the coverage results in the usual way:

coverage -rm

You can also use grep along with the above command to filter the results displayed.

Well, that’s all you need to know to start testing Transifex.

What’s next?

Start testing transifex. If you find a test fails, try to find the reason why it failed. Read the traceback info properly. Find where the error took place. There are various reasons why a test may fail:

  • Test is not updated according to updates in code
  • Bug in code
  • A wrong test case

and others…

You can report the issues or any bug you find at http://trac.transifex.org/newticket. Feel free to submit a patch that fixes the issue. The patch will be reviewed by the Transifex upstream and if it is ok, it will be merged with Transifex’s code at http://code.indifex.com/transifex/.

Keep hacking 🙂

Add plug-n-play functionality to your Django project using Django-addons

What is Django-addons?

A Django app used to add true plug-n-play functionality to your own Django applications and projects. Django-addons is brought to you by Indifex, the company behind Transifex.

Django-addons is a bunch of code that makes writing addon/plugins for your Django project much easier. Add django-addons to your Django project and you can drop all the addons to ‘/addons’ directory.

How to install Django-addons?

You can install the latest version of django-addons running
pip install django-addons
or
easy_install django-addons

You can also install the development version of django-addons with
pip install django-addons==dev
or
easy_install django-addons==dev.

Source code

http://code.indifex.com/django-addons/

Features

  • Addons overview page
  • Automatic signal connecting of addons
  • Automatic URL discovery of addons
  • Template hooking system (inject code from addons to your main project)
  • Django-staticfiles to serve site media from each addon
  • Django-notifications support (automatic registration of noticetypes)
  • Per addon localization
  • Per addon settings
  • Disabling addons via ./manage.py addons

Transifex implements related tag cloud

Lately, I have been working on a bunch of exciting new stuffs for Transifex. I have worked on a tag-cloud implementation which not only shows the popular tags, but also shows tags related to a tag selected by the user. It is pretty useful. It directs the user to select more relevant tags. The tag cloud is refreshed each time the user makes a selection to show the related tags.

I built this on top of the django-tagging module. I wrote a model to represent a tag as a node in a graph. The model includes all the tags related to it (that is tags which appear with the tag in concern) as adjacent nodes along with the weight (that is number of times the two tags appear together) of each edge between two related tag nodes. This data is updated and synced as necessary, e.g, after a project is added or updated. Now, whenever a tag is selected, the tag-cloud is refreshed to show the related tags. The font-size of a related tag is decided by taking into consideration both the weight of an edge it shares with the selected tag and its count. Below is a sample use case for related tag-cloud in Transifex.

Let’s say there are two projects, p1 with tag ‘foo1’ and p2 with tags ‘gui’, ‘graphics’, ‘imaging’ and ‘photography’. For sake of simplicity, I am showing only 3 most popular tags: ‘foo1’, ‘gui’, ‘graphics’. So, now when the maintainer for prohect p1 goes to edit the project, he sees the following tag-cloud:

Initial tagcloud

Now, he selects a new tag ‘graphics’ and the tagcloud is refreshed to show the tags related to ‘graphics’.

Tagcloud with related tags.

Such small things together can really take the user experience to a new level. By implementing related tag-clouds, we enable the user to choose relevant tags in a better way. At Transifex, we innovate to help people localize in a better way :).