Validate Python string translation in Transifex

Transifex already supported validating translations of old styled Python strings, e.g.,

"A sample string with a %(keyword)s argument." % {'keyword': 'key word'}

The validation is done by checking if all the positional and keyword arguments are present in the translation string and the translation string does not contain any extra argument which is not in the source string. You can have a look at the validator code here.

However, the existing validator is not able to check for replacement fields in new style Python format strings, e.g.

"This is a sample string with different replacement fields: {} {1} {foo["bar"]:^30}".format(
"arg0", "arg1", foo={"bar":"a kwarg"})

I tried to devise a regex to extract the replacement fields in the Python format string based on the grammar defined here.

# Regex to find format specifiers in a Python string

import re

field_name = '(?P<field_name>(?P<arg_name>\w+|\d+){0,1}'\
conversion = '(?P<conversion>r|s)'
align = '(?:(?P<fill>[^}{]?)(?P<align>[<>^=]))'
sign = '(?P<sign>[\+\- ])'
width = '(?P<width>\d+)'
precision = '(?P<precision>\d+)'
type_ = '(?P<type_>[bcdeEfFgGnosxX%])'
format_spec = ''\
    ')' % {
        'align': align,
        'sign': sign,
        'width': width,
        'precision': precision,
        'type': type_
replacement_field = ''\
    '\}' % {
        'field_name': field_name,
        'conversion': conversion,
        'format_spec': format_spec

printf_re = re.compile(
    '(?:' + replacement_field + '|'

Well, with the above, I was able to parse almost all the cases discussed here except for this one:

import datetime
d = datetime.datetime(2010, 7, 4, 12, 15, 58)
s = '{:%Y-%m-%d %H:%M:%S}'.format(d)

I was not sure how I could fit the above case to my regex. After some discussions in #python on IRC, I found some limitations of regular expressions and that it is not Turing complete. People suggested me to use some parser tools.

I, being a strong supporter of “Never re invent the wheel”, gave another shot to find some existing solution and lucky I was to come across _formatter_parser() of a Python string object.  It correctly found all replacement fields in python format strings properly and returned  an iterable of tuples (literal_textfield_nameformat_specconversion). All I needed then was to convert this info to a list of replacement fields in a format string. A simple script below would is all that I needed to extract replacement fields in a format string in Python:

replacement_fields = []
s = "{foo:^+30f} bar {0} foo {} {time:%Y-%m-%d %H:%M:%S}"

for literal_text, field_name, format_spec, conversion in \
    if field_name is not None:
        replacement_field = field_name
        if conversion is not None:
            replacement_field += '!' + conversion
        if format_spec:
            replacement_field += ':' + format_spec
        replacement_field = '{' + replacement_field + '}'
print replacement_fields
["{foo:^+30f}", "{0}", "{}", "{time:%Y-%m-%d %H:%M:%S}"]

That’s all. Simple and easy, isn’t it?

A year at Transifex

It’s more than a year now that I have been working at Transifex. It’s a great experience to be a part of the Transifex team. Well, it’s been a roller coaster ride for me at Transifex. I had to go through steep learning curves, work with new stuffs, deliver great features, meet strict deadlines. It was fun, because of being part of an awesome team. I am very much thankful to Apostolis, Konstantinos, John and Diego for guiding me and helping me.

During the course of this journey, I have made many friends. It’s always great to speak with the OpenTranslators folk. They all rock. I am also very much thankful to Youversion, Eventbrite, Pinterest, Dropbox folks for their queries and feedback. It makes me happy to help resolve issues with Transifex and add new features for our users. It gives me a sense of satisfaction that I am able to play my part for the greater good.

Mozilla has always fascinated me since a long time. I love Mozilla, it’s logos, it’s products and it’s goals. At Transifex, I worked on developing new API features to support Pontoon use Transifex as a backend. Following this, I also got involved with Pontoon’s development and have made a few commits. Finally, I contributed in some way to a Mozilla project.

A long road lies ahead. I hope it to be full of new challenges and excitement.

App specific logging in Transifex

Yesterday, I was working on adding app specific loggers in Transifex. By app specific logger I mean a logger which shows the app name which generated the log. As of now, the logs in Transifex look something like this:

2012-06-29 13:01:43,300 tx DEBUG Saved: Project Avant Window Navigator
2012-06-29 13:01:43,312 tx DEBUG Saved: Project Switchdesk
2012-06-29 13:01:43,324 tx DEBUG Saved: Project Usermode
2012-06-29 13:01:43,342 tx DEBUG Saved: Project desktop-effects
2012-06-29 13:01:43,349 tx DEBUG Saved: Project im-chooser
2012-06-29 13:01:43,355 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:43,364 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,704 tx DEBUG Saved: Project Test Project
2012-06-29 13:01:45,717 tx DEBUG Saved: Project Test Private Project
2012-06-29 13:01:45,731 tx DEBUG Resource Resource1: New ResourcePriority created.

It does not tell anything about which app generated the logs. In a first glance, fixing this looks pretty straight forward and dumb. All it needs it to customize this module for each app and instead of importing the logger from txcommon.log, import it from the log module inside the app.
But this would lead to a lot of code duplication and a lot of boring changes in the code. So, I decided to customize transifex.txcommon.log module itself so that it can detect the function calling the logger. It was pretty straight forward to do this for the handler at def model_named() in the following way:

import re

tx_module_regex = re.compile(
def model_named(sender, message='', **kwargs):
    Receive signals for objects with a .name attribute.
    from txcommon.log import _logger as logger
    sender_module = sender.__module__
    m =
    app_name = '.' +'app_name') if m else '' = 'tx' + app_name
    obj = kwargs['instance']
    logger.debug("%(msg)s %(obj)s %(name)s" %
            {'msg': message,
            'obj': sender.__name__,
            'name': getattr(obj, 'name', '')})

sender is the object or instance for which the log is being generated. In our case, it’s a model instance. So, sender.__module__ gives the parent module for sender. Using regular expressions, we extract the app name from the module name and we set the name of the logger as ‘tx.<app_name>‘. And we are done here (for now)! But when we do something like

from transifex.txcommon.log import logger
logger.debug('foo bar')

we do not have a sender instance to allow us to find the calling module name. After some searching, I found about the inspect python module. And all I needed was inspect.stack(). Here’s what I did in

  1. Write a wrapper around logger instance,
  2. find the caller calling the logger using stack.inspect(),
  3. accordingly set the logger name,
  4. and finally, log the event.

import logging, re, inspect

_logger = logging.getLogger('tx')

# regex to extract app name from a file path to a TXC app
tx_app_path_regex = re.compile(
class Logger:
    A wrapper class around _logger. This is used to log events
    along with app names.
    def get_app_name_from_path(cls, path):
        Extracts app name from a file path to a TXC app

            path: A string for the file path
            A string for the app name or ''
        m =
        return'app_name') if m else ''

    def set_logger_name(cls):
        Sets logger name to show calling app's name.
        # inspect.stack()[2] since cls.debug() method has now become the
        # immediate caller in of this method in the stack. We want the caller
        # of cls.debug() or other logging method wrappers.
        caller_module_path = inspect.stack()[2][1]
        app_name = cls.get_app_name_from_path(caller_module_path) = 'tx' + '.%s' % app_name if app_name else ''

    def debug(cls, *args, **kwargs):
        """Wrapper for _logger.debug"""
        _logger.debug(*args, **kwargs)

    # And similarly for other logger methods like info(), waring(), error(), critical()

logger = Logger

Now, this is sweet! No one need to bother about logging events with app names. I am saved from editing hundreds of files and duplicating code 😉 It’s transparent and scalable. The logs now seem like:

2012-06-29 20:39:03,635 tx.projects DEBUG Saved: Project Foo Project
2012-06-29 20:39:05,575 tx.projects DEBUG Saved: Project Avant Window Navigator
2012-06-29 20:39:05,587 tx.projects DEBUG Saved: Project Switchdesk
2012-06-29 20:39:05,599 tx.projects DEBUG Saved: Project Usermode
2012-06-29 20:39:05,612 tx.projects DEBUG Saved: Project desktop-effects
2012-06-29 22:15:07,088 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: ReleaseNotifications
2012-06-29 22:15:07,177 tx.releases DEBUG Deleted: Release All Resources
2012-06-29 22:15:07,466 tx.txcommon DEBUG Running low-level command 'msgfmt -o /dev/null --check-format --check-domain -'
2012-06-29 22:15:07,469 tx.txcommon DEBUG CWD: '/home/rtnpro/transifex/rtnpro/github/txc/transifex'
2012-06-29 22:15:07,661 tx.releases DEBUG release: Checking string freeze breakage.
2012-06-29 22:15:07,702 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (pt_BR)
2012-06-29 22:15:07,707 tx.webhooks DEBUG Project project1 has no web hooks
2012-06-29 22:15:07,740 tx.resources DEBUG resource: Checking if resource translation is fully reviewed: Test Project: Resource1 (ar)
2012-06-29 22:15:07,745 tx.webhooks DEBUG Project project1 has no web hooks

Thanks for reading. If you have any suggestions or query, please feel free to comment.

FUDCON KL 2012 Day 2

Day 2 of FUDCON KL started with a talk on Fedora book by Joshua Wulf (for me Sitapati Prabhu). This idea is very intuitive for any one (especially newbies) to start contributing to Fedora documentation. Although, contributors need to know some basics of docbook, I guess that’s not tough.

There were many interesting talks for the day on topics like Ask Fedora, Transifex, Fedora Tour, etc. Soumay‘s talk on Fedora in Education was really inspiring. He shared how he initially started as a contributor and many things about DGPLUG‘s Bijra project. Following this talk, there was another talk by Soumya on Ask Fedora. This one explained to the attendees what is Ask Fedora, why was it needed and how it can be used. Soumya also explained to the attendees that Ask Fedora runs on Askbot (an Open Source QA forum) and encouraged people to contribute to Askbot.

After lunch, it was time for my and Mahay’s talk on Effective localization Crowdsourcing (using Transifex). Mahay stared the talk explaining to the attendees about localization, internationalization and it’s importance. This helped set the scene for the entire talk. Then I spoke on the various gotchas in the traditional localization workflow and how Fedora tried to get rid of these gotchas using Transifex. I explained to people about Transifex, what it does and why it’s so awesome. I also explained about the various super cool features in Transifex like crowdsourcingproject management, release management, Translation Memory, glossary, etc. I also told the attendees about how to contribute to Transifex.

Well, that was not all. It was followed by another session on How to internationalize and localize softwares. I took example of a simple Django app and explained how to i18n (using gettext) the app and extract the source POT file. Then I showed how to localize it using Transifex. I gave the attendees a walk through from creating a Transifex account to creating a project, resources, releases, forming teams, translating and finally to downloading translations and deploying it in their app. I also mentioned about other i18n methods available for different languages and directed people to the necessary resources. With this, I finally concluded my talk.

After the talk, it was time for some tea and then lightning talks. Christoph Wickert‘s talks on LXDE and Clouds were super cool. Michel also spoke on the ROX DE and ROX filer. We had some post session discussions and some group pics.

Today was the day for FUDPub. It was supposed to start at 8 PM at Sri Petaling Hotel. We reached the hotel and had some rest. Then we moved downstairs to join FUDPub. It was hell lot of fun out there.

FUDCon KL 2012 Day 1

I arrived at Hotel Sri Petaling at around 3 AM on May 18, 2012. After very less sleep, I woke up at 6:30 AM. After having our breakfast at the hotel, we left for UCTI (the venue for FUDCON KL) in a bus arranged by the FUDCON KL organizers. After reaching the venue, we got ourselves registered at the registration booth for FUDCON and we received our coupons for lunch and tea. It didn’t take long for the other attendees to arrive at the venue.

The first day of FUDCON began with a key note speech by Christoph Wickert on “Leadership in leaderless organizations” and how this is in action in Fedora.

Christoph Wickert speaking on “Leadership in leaderless organizations”

Christoph Wickert with his key note

After the keynote, it was time for the  bar camp. Although, many were skeptical about barcamp in the 1st day of a FUDCon, it turned out to be awesome. Many a people pitched their topics for the bar camp. Among others, I also pitched two topics:

  • Transifex: A developer’s perspective
  • Agile system tests in Django

FUDCON Attendees

Since there were a lot of topics proposed and there was limited time, the topics were shortlisted based on votes by the attendees. One of my pitched topics: “Agile system tests using Django” finally made it into the shortlist. The bar camp sessions were scheduled to start at 2:30 PM. Over lunch, I was having some nice conversation with Kushal, Michel and Soumya on various topics related to Open Source and the LUG related activities in our places.

Among the many interesting lighting talks I managed to attend some (though not all) talks: Improving collaboration with other open source projects, Fedora for students, Git, Fedora and packages, etc. I had my talk at 4 PM. I spoke on:

  • System tests are needed
  • Why we need fast system tests?
  • How the Django test framework makes system tests slow?
  • How can we remove unnecessary overheads?
  • How we run faster system tests at Transifex?

For this, I used this post from

It was an awesome 1st day of FUDCON KL. I participated in my first bar camp and it turned out to be a lot of fun and exciting. We had some post-event chats and discussions while leaving the venue. I also had some long conversation with Joshua Wulf while heading back to the hotel on matters related to spirituality. This was an eye opener and I had many of my doubts resolved by the grace of an elevated devotee like him.

So, it was an enlightening day for me, both in terms of FOSS and spirituality.

Dear Turkish translators

Transifex usually defines plural rules for languages according to So, the plural rule for Turkish language in Transifex is other → everything. However, lately there has been some requests that the Turkish language should have two plural forms:

nplurals=2; plural=(n>1)

The requests have been with reference to

Here is a quote from a user at

Turkish behaves like Akan for example. The rule should be:

One: 0, 1 Other: 2-999

It is only when including a count that there are no plural forms. For example:

“You posted a photo”, “You posted several photos”

is correct in Turkish, as is:

“You posted 1 photo”, “You posted 6 photo”.

So, dear Turkish translators, please share your opinion on this issue. This will help a lot to resolve this issue at Transifex and fix plural translations in Turkish language.