App specific logging in Transifex

Yesterday, I was working on adding app specific loggers in Transifex. By app specific logger I mean a logger which shows the app name which generated the log. As of now, the logs in Transifex look something like this:


2012-06-29 13:01:43,300 tx DEBUG Saved: Project Avant Window Navigator
2012-06-29 13:01:43,312 tx DEBUG Saved: Project Switchdesk
2012-06-29 13:01:43,324 tx DEBUG Saved: Project Usermode
2012-06-29 13:01:43,342 tx DEBUG Saved: Project desktop-effects
2012-06-29 13:01:43,349 tx DEBUG Saved: Project im-chooser
2012-06-29 13:01:43,355 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:43,364 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,704 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:45,717 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,731 tx DEBUG Resource Resource1: New ResourcePriority created.

It does not tell anything about which app generated the logs. In a first glance, fixing this looks pretty straight forward and dumb. All it needs it to customize this https://github.com/transifex/transifex/tree/devel/transifex/txcommon/log module for each app and instead of importing the logger from txcommon.log, import it from the log module inside the app.
But this would lead to a lot of code duplication and a lot of boring changes in the code. So, I decided to customize transifex.txcommon.log module itself so that it can detect the function calling the logger. It was pretty straight forward to do this for the handler at https://github.com/transifex/transifex/blob/devel/transifex/txcommon/log/receivers.py#L6: def model_named() in the following way:

import re

tx_module_regex = re.compile(
    r'transifex(\.addons)?\.(?P<app_name>\w+)(\..*)?')
def model_named(sender, message='', **kwargs):
    """
    Receive signals for objects with a .name attribute.
    """
    from txcommon.log import _logger as logger
    sender_module = sender.__module__
    m = tx_module_regex.search(sender_module)
    app_name = '.' + m.group('app_name') if m else ''
    logger.name = 'tx' + app_name
    obj = kwargs['instance']
    logger.debug("%(msg)s %(obj)s %(name)s" %
            {'msg': message,
            'obj': sender.__name__,
            'name': getattr(obj, 'name', '')})

sender is the object or instance for which the log is being generated. In our case, it’s a model instance. So, sender.__module__ gives the parent module for sender. Using regular expressions, we extract the app name from the module name and we set the name of the logger as ‘tx.<app_name>‘. And we are done here (for now)! But when we do something like

from transifex.txcommon.log import logger
logger.debug('foo bar')

we do not have a sender instance to allow us to find the calling module name. After some searching, I found about the inspect python module. And all I needed was inspect.stack(). Here’s what I did in https://github.com/transifex/transifex/tree/devel/transifex/txcommon/log/__init__.py:

  1. Write a wrapper around logger instance,
  2. find the caller calling the logger using stack.inspect(),
  3. accordingly set the logger name,
  4. and finally, log the event.

import logging, re, inspect

_logger = logging.getLogger('tx')

# regex to extract app name from a file path to a TXC app
tx_app_path_regex = re.compile(
r'txc/transifex(/addons)?/(?P<app_name>\w+)/(\..*)?')
class Logger:
    """
    A wrapper class around _logger. This is used to log events
    along with app names.
    """
    @classmethod
    def get_app_name_from_path(cls, path):
        """
        Extracts app name from a file path to a TXC app

        Args:
            path: A string for the file path
        Returns:
            A string for the app name or ''
        """
        m = tx_app_path_regex.search(path)
        return m.group('app_name') if m else ''

    @classmethod
    def set_logger_name(cls):
        """
        Sets logger name to show calling app's name.
        """
        # inspect.stack()[2] since cls.debug() method has now become the
        # immediate caller in of this method in the stack. We want the caller
        # of cls.debug() or other logging method wrappers.
        caller_module_path = inspect.stack()[2][1]
        app_name = cls.get_app_name_from_path(caller_module_path)
        _logger.name = 'tx' + '.%s' % app_name if app_name else ''

    @classmethod
    def debug(cls, *args, **kwargs):
        """Wrapper for _logger.debug"""
        cls.set_logger_name()
        _logger.debug(*args, **kwargs)

    # And similarly for other logger methods like info(), waring(), error(), critical()

logger = Logger

Now, this is sweet! No one need to bother about logging events with app names. I am saved from editing hundreds of files and duplicating code 😉 It’s transparent and scalable. The logs now seem like:

2012-06-29 20:39:03,635 tx.projects DEBUG Saved: Project Foo Project
2012-06-29 20:39:05,575 tx.projects DEBUG Saved: Project Avant Window Navigator
2012-06-29 20:39:05,587 tx.projects DEBUG Saved: Project Switchdesk
2012-06-29 20:39:05,599 tx.projects DEBUG Saved: Project Usermode
2012-06-29 20:39:05,612 tx.projects DEBUG Saved: Project desktop-effects
..........
..........
..........
2012-06-29 22:15:07,088 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: ReleaseNotifications
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: Release All Resources
2012-06-29 22:15:07,466 tx.txcommon DEBUG Running low-level command 'msgfmt -o /dev/null --check-format --check-domain -'
2012-06-29 22:15:07,469 tx.txcommon DEBUG CWD: '/home/rtnpro/transifex/rtnpro/github/txc/transifex'
2012-06-29 22:15:07,661 tx.releases DEBUG release: Checking string freeze breakage.
2012-06-29 22:15:07,702 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (pt_BR)
2012-06-29 22:15:07,707 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,740 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (ar)
2012-06-29 22:15:07,745 tx.webhooks DEBUG Project project1 has no web hooks

Thanks for reading. If you have any suggestions or query, please feel free to comment.

Advertisements

FUDCON KL 2012 Day 2

Day 2 of FUDCON KL started with a talk on Fedora book by Joshua Wulf (for me Sitapati Prabhu). This idea is very intuitive for any one (especially newbies) to start contributing to Fedora documentation. Although, contributors need to know some basics of docbook, I guess that’s not tough.

There were many interesting talks for the day on topics like Ask Fedora, Transifex, Fedora Tour, etc. Soumay‘s talk on Fedora in Education was really inspiring. He shared how he initially started as a contributor and many things about DGPLUG‘s Bijra project. Following this talk, there was another talk by Soumya on Ask Fedora. This one explained to the attendees what is Ask Fedora, why was it needed and how it can be used. Soumya also explained to the attendees that Ask Fedora runs on Askbot (an Open Source QA forum) and encouraged people to contribute to Askbot.

After lunch, it was time for my and Mahay’s talk on Effective localization Crowdsourcing (using Transifex). Mahay stared the talk explaining to the attendees about localization, internationalization and it’s importance. This helped set the scene for the entire talk. Then I spoke on the various gotchas in the traditional localization workflow and how Fedora tried to get rid of these gotchas using Transifex. I explained to people about Transifex, what it does and why it’s so awesome. I also explained about the various super cool features in Transifex like crowdsourcingproject management, release management, Translation Memory, glossary, etc. I also told the attendees about how to contribute to Transifex.

Well, that was not all. It was followed by another session on How to internationalize and localize softwares. I took example of a simple Django app and explained how to i18n (using gettext) the app and extract the source POT file. Then I showed how to localize it using Transifex. I gave the attendees a walk through from creating a Transifex account to creating a project, resources, releases, forming teams, translating and finally to downloading translations and deploying it in their app. I also mentioned about other i18n methods available for different languages and directed people to the necessary resources. With this, I finally concluded my talk.

After the talk, it was time for some tea and then lightning talks. Christoph Wickert‘s talks on LXDE and Clouds were super cool. Michel also spoke on the ROX DE and ROX filer. We had some post session discussions and some group pics.

Today was the day for FUDPub. It was supposed to start at 8 PM at Sri Petaling Hotel. We reached the hotel and had some rest. Then we moved downstairs to join FUDPub. It was hell lot of fun out there.

FUDCon KL 2012 Day 1

I arrived at Hotel Sri Petaling at around 3 AM on May 18, 2012. After very less sleep, I woke up at 6:30 AM. After having our breakfast at the hotel, we left for UCTI (the venue for FUDCON KL) in a bus arranged by the FUDCON KL organizers. After reaching the venue, we got ourselves registered at the registration booth for FUDCON and we received our coupons for lunch and tea. It didn’t take long for the other attendees to arrive at the venue.

The first day of FUDCON began with a key note speech by Christoph Wickert on “Leadership in leaderless organizations” and how this is in action in Fedora.

Christoph Wickert speaking on “Leadership in leaderless organizations”

Christoph Wickert with his key note

After the keynote, it was time for the  bar camp. Although, many were skeptical about barcamp in the 1st day of a FUDCon, it turned out to be awesome. Many a people pitched their topics for the bar camp. Among others, I also pitched two topics:

  • Transifex: A developer’s perspective
  • Agile system tests in Django

FUDCON Attendees

Since there were a lot of topics proposed and there was limited time, the topics were shortlisted based on votes by the attendees. One of my pitched topics: “Agile system tests using Django” finally made it into the shortlist. The bar camp sessions were scheduled to start at 2:30 PM. Over lunch, I was having some nice conversation with Kushal, Michel and Soumya on various topics related to Open Source and the LUG related activities in our places.

Among the many interesting lighting talks I managed to attend some (though not all) talks: Improving collaboration with other open source projects, Fedora for students, Git, Fedora and packages, etc. I had my talk at 4 PM. I spoke on:

  • System tests are needed
  • Why we need fast system tests?
  • How the Django test framework makes system tests slow?
  • How can we remove unnecessary overheads?
  • How we run faster system tests at Transifex?

For this, I used this post from blog.transifex.com.

It was an awesome 1st day of FUDCON KL. I participated in my first bar camp and it turned out to be a lot of fun and exciting. We had some post-event chats and discussions while leaving the venue. I also had some long conversation with Joshua Wulf while heading back to the hotel on matters related to spirituality. This was an eye opener and I had many of my doubts resolved by the grace of an elevated devotee like him.

So, it was an enlightening day for me, both in terms of FOSS and spirituality.

Dear Turkish translators

Transifex usually defines plural rules for languages according to http://unicode.org/repos/cldr-tmp/trunk/diff/supplemental/language_plural_rules.html. So, the plural rule for Turkish language in Transifex is other → everything. However, lately there has been some requests that the Turkish language should have two plural forms:

nplurals=2; plural=(n>1)

The requests have been with reference to http://translate.sourceforge.net/wiki/l10n/pluralforms.

Here is a quote from a user at https://bitbucket.org/indifex/transifex/issue/26/turkish-plural-forms:

Turkish behaves like Akan for example. The rule should be:

One: 0, 1 Other: 2-999

It is only when including a count that there are no plural forms. For example:

“You posted a photo”, “You posted several photos”

is correct in Turkish, as is:

“You posted 1 photo”, “You posted 6 photo”.

So, dear Turkish translators, please share your opinion on this issue. This will help a lot to resolve this issue at Transifex and fix plural translations in Turkish language.

rtnpro @ Mukti 2012

Mukti is the annual FOSS festival organised by the GNU/Linux Users Group of NIT Durgapur. Mukti 2012 was held on 3-5th February 2012. I have attended every Mukti in NIT Durgapur from 2008 to 2011 as a student and this time (in 2012) as a speaker. My talk was on Localization and Transifex. NITDGPLUG, as always, put a lot of effort in making Mukti a grand FOSS event in the region. It was a packed with a plethora of events and had a large number of participants. Mukti serves as a great means to get together people interested in FOSS in the Eastern and North Eastern part of India. It helps newbies get more insight into FOSS.

Day 1, February 3, 2012

The first day of Mukti began with an inauguration programme. After the inauguration programme, students queued at the registration desk for registering themselves. Sayan and Gaurav came there with a small group of 1st year students (interested in FOSS) from Dr. B. C. Roy Engineering College. I spoke to them for a 1-2 hours on FOSS, how to contribute, my experience with FOSS and how I made to Transifex. After bidding good bye to the 1st year students from BCREC, we (me, Sayan, Gaurav and a few others) settled in my room at the Guest House, NIT Durgapur and started discussing on various stuff like Transifex, Django, unit testing, some college news, etc. There was also a workshop on KDE development that day by Smit Shah. After the workshop was over, the Transifex community guys from Durgapur crashed in my place and we kept hacking till late night.

Day 2, February 4, 2012

For the 1st half of the day, I came to BCREC to talk with the students on FOSS and meet my teachers and other friends. After returning to NIT Durgapur, I had a discussion with some folks interested in web development and Transifex. I discussed with them about Transifex, what it is, why it is created, how it works and how it is written. Also, we discussed on other stuffs like contributing to FOSS, python, Django, etc.  We spent the entire evening hacking on Transifex. We fired our local Transifex instance and started discussing about bugs and areas of improvement. I also explained in details to the Transifex contributors on how to write unit tests for Transifex. I also showed to them how to write a handler for a file format in Transifex.

In between, I had a good conversation with Smit Shah. We shared our views on FOSS and contributing to it, and also our experience and excitement in working for a startup. We also discussed on Manga: Naruto, Fullmetal Alchemist Brotherhood and One piece 😉 Even after dinner, we kept hacking, till midnight. The day was quite eventful. We triagged some tickets at trac.transifex.org, fixed some bugs, found new bugs to work on, etc.

Day 3, February 5, 2012

This was the final day of Mukti and my talk on Localization, Transifex and FOSS contribution in general was scheduled for this day. In this talk, I started with “What” and “Why” of localization and how it helps the global usage of a software. Also, I explained that localization is one of the easiest way to start contributing to FOSS and get the feet wet in community, learn new technologies, etc.  Then, I discussed the workflow of localization and its pros and cons.

Then, I came to Transifex, why was it needed, how and when did it start, and how it takes localization to an all new level. I discussed the technologies used behind Transifex and gave the audience a tour through Transifex. Transifex is no small thing now. It has grown over the years and it takes a lot to explain its features. Enough with technical jargon. To make it interactive, I called Sayan to share his experiences about his contribution to Transifex. Also, I shared our story that how a group of 3 newbie translators made http://www.transifex.net available in Hindi just in a few days.

Then, I told the people that how they can start contributing to Transifex and any open source project in general. But, still there was the impression that contributing is a VERY DIFFICULT task. So, I decided to hack live in front of all the audience and fix a few Transifex bugs (bugs on which we worked on the previous day, during the hackfest). I fixed 2-3 small bugs, showed what is a patch and how to commit a patch. The patches had just 1-2 lines of change. I hope the audience got my point, that fixing bugs is not a very difficult job.

Then, I shared my experiences with FOSS, how I came into the FOSS community, how I started contributing and how I made into Transifex. With this, I concluded my talk. After the session, a few students came to me with queries and we had a kind of group discussion with them.

You can find the slide deck I used for my talk at http://rtnpro.fedorapeople.org/Transifex-Mukti2012/presentation.pdf

After the talk, we headed back to the guest house and had some gossip and masti with my college juniors. In the evening, we attended the prize distribution function and then headed back to the guest room. After dinner, we started discussing about things like how to boys should proceed in their open source endeavours, brainstormed some crazy project ideas, etc.

It was an awesome experience at Mukti this year. Met with many people, made new friends, had lots of fun and a lot of hacking.

#Transifex now supports comments in Apple .strings i18n files

#Transifex now supports comments in Apple .strings i18n files. Only /* foo */ style comment in the line preceding the key value pair in the source file is saved as a comment for the key. The example below will explain this in a better way:

/*Comment for key1*/
"key1" = "value 1";

/* This comment will not be
included in key2*/

/* comment for
key2*/
"key2" = "value 2";

/* this comment will not be included in key3*/

"key3" = "value 3";

Well, I’m pretty sure that the above snippet explains which comments from source Apple .strings file are saved by Transifex. You can see the comment for a source string in its “Details” section in Lotte.

Comment for a source string imported from a source Apple .strings file

Me & FUDCON India 2011

I arrived in Pune for FUDCON on November 2, 2011. On November 3, 2011, I had an opportunity to visit the Red Hat office in the city and hang around with Fedora community members. I learnt more about the mechanism and importance of localization from Runa Bhattacharjee. In my free time, I also helped with some FUDCON related work. I  was able to meet Jared Smith, Joerg Simon and Robert Scheck for the first time in my life, talk with them. I couldn’t ask for anything better.

Day 1: November 4, 2011

Day 1 began with a speech from the Director of COEP followed by the keynote speech by Fedora’s Project Leader Jared Smith. Jared explained to people the various aspects of Fedora and how Fedora takes the lead in pushing the limits of FOSS development. Following that was my talk on Transifex. The audience included people mainly from the L10n domain. There were some mishaps during the talk. First, my Dell XPS laptop wasn’t able to use the projector; second, I had to use another netbook which I wasn’t used to; third, there were some power issues, and the netbook turned off and finally, there was power cut for a few minutes. What a chain of mishaps! I had to resort to reading the slides from my phone during the power cut. Finally, I used Kishan Goyal’s laptop to continue with the presentation. In  my talk, I explained the current features and upcoming features of Transifex. I explained it with a use case, starting from registering to advanced usage of www.transifex.net. The audience appreciated our upcoming market place idea and the Translation Memory feature. I also got feature request for having a global glossary for a particular language in the language page. It was really nice that the audience were so actively communicating during the session. I also told how to start contributing to the Transifex project and shared my experience working on Transifex so far: from a contributor to an employee.

After the lunch, I attended Heherson Pagcaliwagan‘s session on “Fedora web of trust” and got more insight into the use of GPG keys. We did a small workshop with Heherson on how to get introduced to each other, verify identity, share GPG key and sign it. Heherson also showed us how to encrypt mails using GPG key. Then I attended Joerg Simon’s talk on Fedora Security Lab and OSSTMM. Kital showed us a variety of security tools that can be found in the FSL, and mentioned others that need to be packaged. I must try out the tools in the FSL now. They are so cool.

Then I went back to the speaker’s lounge and started writing some code on Transifex. It was a great first day at FUDCON for me.

Day 2: November 5, 2011

The Second day of FUDCON began with Harish Pillay speaking about the community. Unfortunately, I was not able to turn up during the keynote, as I had to re create my slides on my talk on “Testing your Django app”. Because I had accidentally, deleted the folder which contained the slides. After I was done, I hurried to attend Arun Sag’s talk on “Creating web apps using Django”. I liked Arun’s way of presenting things to the newbies in a very lucid way. He used his classic Blog example for this.

Next was my talk on “Test your Django app”. As during Day 1, the projector did not work with my laptop, so I used Arun’s laptop for the purpose. I explained why tests are necessary, different testing frameworks in Django (doctest, unittest). Then I went forward to explain how to write simple unittests. For this, I wrote some tests for Arun’s blog example and used it so that the audience could relate things with the previous session taken by Arun. I showed some simple test cases and ran the tests. Then, I introduced the Django Test Client and spoke about its importance and features. After explaining things about the Test Client, I showed the relevant code and ran the tests again. Finally, I explained about coverage: what is it? why is it required? How to use coverage? I again ran the tests I had run before, but this time I ran them with coverage and explained how to read the coverage report. I haven’t been too happy with this session of mine. Now, when I think of it, for students who just got introduced to Django, the session on Django testing might have been asking too much. Anyway, the students now at least know something called Django testing exists. So, when they need it, they can learn.

After having lunch, I attended Siddhesh’s talk on “Security Exploits”. During the session, I could not but think that why I did not have someone like Siddhesh teaching me OS in my college. He was awesome. Back in my college days, I had tried reading about security exploits, but I did not get far. But during this session, thanks to Siddhesh’s explanations and my earlier reading, I have gained a better understanding of security exploits, especially stack smash attack, overwriting nearby entities in data region by string overflow, etc.

Following were two lightning talks on 1) How to deal with kernel panic? 2) Running external commands from Postgres. Yogesh in his talk on “kernel Panic” showed how to collect relevant data when there is a kernel panic. This data can be used for creating useful bug reports or for fixing the bug itself.

After attending Siddhesh’s talk on autotools, I joined mether’s talk on ask.fedoraproject.org or askbot. Mether discussed its roadmap and mentioned various feature requests. I picked up to implement a few of the feature requests. After the session, we had a group photo session of the almost all the people involved in FUDCON. Then all the speakers, volunteers and organizers went for the FUDPUB. I enjoyed a lot at the FUDPUB. I spoke with the community members, danced with them, drank Mirinda and ate some delicious food. It was just awesome.

Day 3: November 6, 2011

It’s the hackfest day. I decided to run a Transifex Testathon. I pitched the topic on stage and invited people to join me. I helped some of my friends install and setup transifex in their machines and showed them how to run tests. I started writing new tests for the watches addon in Transifex. I came across a chain of undiscovered bugs while doing so. It took me some time to write a proper test case for the watches and accordingly fix the bugs in the code. I also helped Kushal to get him logged into his Transifex account and creating a Transifex project for “Python for you and me”. Jared Smith, set up the tx client for “Python for you and me” and now PYM is hosted at https://www.transifex.net/projects/p/pym/ for localization. Shreyank, Vaidik and I had discussions on the roadmap for Dorrie. I setup Dorrie on my machine and played with it for some time. I decided to write tests for Dorrie the coming weekend.

Then, at the end of Day 3, in the auditorium, a cake was cut to celebrate this FUDCON along with quite a few photo shoots. The FUDCON organizing group and the volunteers from COEP did a great job to make this event go on smoothly.

FUDCON India 2011 is the first ever FUDCON in India and the largest FUDCON in terms of the number of participants. Apart from learning new stuffs and hacking, FUDCON provided a great platform for Fedora contributors to meet with each other and make new friends. It is also a nice experience to work with people whom I had known only in the IRC until now. I am carrying sweet memories of this FUDCON with me. These memories will help me focus more on contributing to open source and be a better contributor.