My talk got selected for #Pycon India 2012

My proposed talk titled Develop for an international audience got selected for Pycon India, 2012. It’s time to start working on the slides. I am thinking to use rst to write my slides. Also, I have booked by flight tickets for Pycon ūüôā

Thanks everyone who voted for my talk.

Validate Python string translation in Transifex

Transifex already supported validating translations of old styled Python strings, e.g.,

"A sample string with a %(keyword)s argument." % {'keyword': 'key word'}

The validation is done by checking if all the positional and keyword arguments are present in the translation string and the translation string does not contain any extra argument which is not in the source string. You can have a look at the validator code here.

However, the existing validator is not able to check for replacement fields in new style Python format strings, e.g.

"This is a sample string with different replacement fields: {} {1} {foo["bar"]:^30}".format(
"arg0", "arg1", foo={"bar":"a kwarg"})

I tried to devise a regex to extract the replacement fields in the Python format string based on the grammar defined here.

# Regex to find format specifiers in a Python string

import re

field_name = '(?P<field_name>(?P<arg_name>\w+|\d+){0,1}'\
                '(?:(?P<attribute_name>\.\w+)|'\
                '(?P<element_index>\[(?:\d+|(?:[^\]]+))\]))*)'
conversion = '(?P<conversion>r|s)'
align = '(?:(?P<fill>[^}{]?)(?P<align>[<>^=]))'
sign = '(?P<sign>[\+\- ])'
width = '(?P<width>\d+)'
precision = '(?P<precision>\d+)'
type_ = '(?P<type_>[bcdeEfFgGnosxX%])'
format_spec = ''\
    '(?P<format_spec>'\
        '%(align)s{0,1}'\
        '%(sign)s{0,1}#?0?'\
        '%(width)s{0,1},?'\
        '(?:\.%(precision)s){0,1}'\
        '%(type)s{0,1}'\
    ')' % {
        'align': align,
        'sign': sign,
        'width': width,
        'precision': precision,
        'type': type_
}
replacement_field = ''\
    '\{'\
    '(?:'\
        '%(field_name)s{0,1}'\
        '(?:!%(conversion)s){0,1}'\
        '(?:\:%(format_spec)s){0,1}'\
    ')'\
    '\}' % {
        'field_name': field_name,
        'conversion': conversion,
        'format_spec': format_spec
}

printf_re = re.compile(
    '(?:' + replacement_field + '|'
        '%((?:(?P<ord>\d+)\$|\((?P<key>\w+)\))?(?P<fullvar>[+#-]*(?:\d+)?'
            '(?:\.\d+)?(hh\|h\|l\|ll)?(?P<type>[\w%])))'
    ')'
)

Well, with the above, I was able to parse almost all the cases discussed here except for this one:

import datetime
d = datetime.datetime(2010, 7, 4, 12, 15, 58)
s = '{:%Y-%m-%d %H:%M:%S}'.format(d)

I was not sure how I could fit the above case to my regex. After some discussions in #python on IRC, I found some limitations of regular expressions and that it is not Turing complete. People suggested me to use some parser tools.

I, being a strong supporter of “Never re invent the wheel”, gave another shot to find some existing solution and lucky I was to come across _formatter_parser() of a Python string object. ¬†It correctly found all replacement fields in python format strings properly and returned ¬†an iterable of tuples (literal_text,¬†field_name,¬†format_spec,¬†conversion). All I needed then was to convert this info to a list of replacement fields in a format string. A simple script below would is all that I needed to extract replacement fields in a format string in Python:

replacement_fields = []
s = "{foo:^+30f} bar {0} foo {} {time:%Y-%m-%d %H:%M:%S}"

for literal_text, field_name, format_spec, conversion in \
        s._formatter_parser():
    if field_name is not None:
        replacement_field = field_name
        if conversion is not None:
            replacement_field += '!' + conversion
        if format_spec:
            replacement_field += ':' + format_spec
        replacement_field = '{' + replacement_field + '}'
        replacement_fields.append(replacement_field)
print replacement_fields
["{foo:^+30f}", "{0}", "{}", "{time:%Y-%m-%d %H:%M:%S}"]

That’s all. Simple and easy, isn’t it?

A year at Transifex

It’s more than a year now that I have been working at Transifex. It’s a great experience to be a part of the Transifex team. Well, it’s been a¬†roller coaster¬†ride for me at Transifex. I had to go through steep learning curves, work with new stuffs, deliver great features, meet strict deadlines. It was fun, because of being part of an awesome team. I am very much thankful to Apostolis, Konstantinos, John and Diego for guiding me and helping me.

During the course of this journey, I have made many friends. It’s always great to speak with the OpenTranslators folk. They all rock. I am also very much thankful to Youversion, Eventbrite, Pinterest, Dropbox folks for their queries and feedback. It makes me happy to help resolve issues with Transifex and add new features for our users. It gives me a sense of satisfaction that I am able to play my part for the greater good.

Mozilla has always fascinated me since a long time. I love Mozilla, it’s logos, it’s products and it’s goals. At Transifex, I worked on developing new API features to support Pontoon use Transifex as a backend. Following this, I also got involved with Pontoon’s development and have made a few commits. Finally, I contributed in some way to a Mozilla project.

A long road lies ahead. I hope it to be full of new challenges and excitement.

Updating all your fedora package git directories

Ankur's Tech blog

Most package maintainers already know this (or use something way easier), but this is how I go about updating all my fedora package git folders:

 [ankur@ankur ~]$ cd Documents/work/repos/others/fedora-packages/ [ankur@ankur fedora-packages]$ ls aeskulap dcm4che-test fpaste hiran-perizia-fonts libmpd memaker OpenNL python-hl7 subversion trash-cli cmake detex freemedforms Inventor libtorrent mpdas OSGi-bundle-ant-task rssdler suitesparse vtk comps django-authopenid gdcm klt libtpcimgio nifticlib pyclutter rtorrent toothchart xmedcon curlpp evolution-rss gnumed libgexiv2 libtpcmisc oldstandard-sfd-fonts pystache scout transmission zlib [ankur@ankur fedora-packages]$ for i in `ls`; do cd "$i"; git pull; cd ..; done remote: Counting objects: 5, done. remote: Compressing objects: 100% (3/3), done. Unpacking objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 0 (delta 0) From ssh://pkgs.fedoraproject.org/aeskulap * [new branch] f17 -> origin/f17 ca0fa88..ef4f943 master -> origin/master Updating ca0fa88..ef4f943 Fast-forward aeskulap.spec | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) remote: Counting objects: 20, done. remote: Compressing objects: 100% (17/17), done…

View original post 12 more words

Ouch! My Bad

Subhodip's Blog

JavaScript is powerful and Backbone.js is a nice small utility that makes it even more powerful. I was playing with Backbone and Open layers thrown over a Django system.
However there is something weird that I faced and took a good amount of my time to figure out the problem.
This blog post is kind of self note so that I notice this the first place when I face such problem again.

What was I trying to do : Write a small backbone code to display Open layers(OSM) Map.
How was I trying to do :

(function($) {
var Map = Backbone.Model.extend({});

var MapView = Backbone.View.extend({

initialize: function() {
_.bindAll(this, 'initMap');
this.initMap();
},

initMap: function() {
// Initialize Basic Openlayers;
var center = new OpenLayers.LonLat(8110203.9998955, 2170000.4068373);
map = new OpenLayers.Map(this.el, {
projection: new OpenLayers.Projection("EPSG:900913"),
displayProjection: new OpenLayers.Projection("EPSG:4326")
});
var layers = [];
layers[0] = new OpenLayers.Layer.OSM(); //some more layer will…

View original post 183 more words

#Transifex now supports comments in Apple .strings i18n files

#Transifex now supports comments in Apple .strings i18n files. Only /* foo */ style comment in the line preceding the key value pair in the source file is saved as a comment for the key. The example below will explain this in a better way:

/*Comment for key1*/
"key1" = "value 1";

/* This comment will not be
included in key2*/

/* comment for
key2*/
"key2" = "value 2";

/* this comment will not be included in key3*/

"key3" = "value 3";

Well, I’m pretty sure that the above snippet explains which comments from source Apple .strings file are saved by Transifex. You can see the comment for a source string in its “Details” section in Lotte.

Comment for a source string imported from a source Apple .strings file

Start testing Transifex

 

How do you setup Transifex?

Here is all you need to know to setup Transifex: http://help.transifex.net/server/install.html

http://fosswithme.wordpress.com/2011/10/20/setup-transifex-in-virtualenv/ is another good write-up on how to setup and run Transifex in virtualenv. So, I’d be building on top of that to show you how to start testing Transifex using django-nose.

What packages will you need?

django-nose, nose, nose-exclude, coverage

You can install the above packages using

pip install <package_name>

Configure Transifex settings to enable django-nose Test Runner

cd <transifex source code's root directory>

cd transifex/settings

cp  90-local.conf.sample 90-local.conf

open and edit 90-local.conf and add the following lines:

INSTALLED_APPS += [

‘django_nose’,

]

TEST_RUNNER = ‘django_nose.NoseTestSuiteRunner’

cd ..

Now, save the file. Now you are good to go.

Start testing

python manage.py test <app_name>

for example:

python manage.py test resources

You can also test a particular test class like below:

python manage.py test resources.tests:TestJavaProperties

You can even run a particular test method:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser

Run coverage on Transifex tests

django-nose has a plugin for coverage. So, you can run the above tests and collect coverage data.

For example:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser --with-coverage --cover-package=resources.tests.formats

All the coverage results are saved in a .coverge file by default in the current directory. Although, running tests with coverage plugin of django-nose shows the coverage results by default. You can also see the coverage results in the usual way:

coverage -rm

You can also use grep along with the above command to filter the results displayed.

Well, that’s all you need to know to start testing Transifex.

What’s next?

Start testing transifex. If you find a test fails, try to find the reason why it failed. Read the traceback info properly. Find where the error took place. There are various reasons why a test may fail:

  • Test is not updated according to updates in code
  • Bug in code
  • A wrong test case

and others…

You can report the issues or any bug you find at http://trac.transifex.org/newticket. Feel free to submit a patch that fixes the issue. The patch will be reviewed by the Transifex upstream and if it is ok, it will be merged with Transifex’s code at¬†http://code.indifex.com/transifex/.

Keep hacking ūüôā

A brief introduction to coverage.py

Coverage.py is a tool for measuring code coverage of Python programs. It monitors your program, noting which parts of the code have been executed, then analyzes the source to identify code that could have been executed but was not.

Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not.

Getting started:

1. Install coverage:

  • pip install coverage
  • easy_install coverage

2. Use coverage to run your program and gather data:

$ coverage run my_program.py arg1 arg2
blah blah ..your program's output.. blah blah

3. Generate reports with coverage:

$coverage -rm

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
my_program                   20      4    80%   33-35, 39
my_other_module              56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76      10    87%

4. You can also use coverage to generate reports in other presentation oriented formats like HTML:

$coverage html

You can also use coverage.py with Django. You can run your Django tests along with coverage to check which codes in your app have been tested by your tests. With the coverage data, you can write new tests to test the codes which have not been tested so far by your tests. For example:

  • $coverage -e¬†¬†¬†¬†¬†¬†¬† //This deletes previous coverage data
  • $coverage -x manange.py test foo_app.FooTest.foo_method¬†¬†¬†¬†¬†¬†¬† //Execute manage.py from coverage to collect coverage data
  • $coverage -rm | grep ‘foo_app’ ¬† ¬†¬† // to filter the report to show the coverage of foo_app
  • coverage run –include=”*foo_app*” –omit=”*tests* manage.py test foo_app¬†¬†¬†¬†¬†¬†¬† //This will include *foo_app* pattern and omit *tests* pattern from your coverage report.

You can also write custom Test Runners using the coverage API to measure code coverage in a more controlled manner. You can find more detailed documentation about coverage here.

More fine tuned auto save feature in Lotte for Transifex

Lotte is the component of Transifex providing the web UI for the translators to translate online. Well, some time back, I made some changes in the Lotte code so that it has the auto-save feature on by default. It was running fine then and I was happy. Then I added the spellcheck feature in Lotte. Then the things started going not as expected.

Actually, the auto-save triggers when the contents of a text area has been edited and it loses focus. The first problem I faced is that the spellcheck button did not respond on the first click, but on the second. As expected, on the first click, auto-save was triggered. But this was not the case when I clicked the Undo button. First the auto-save and then the undo function would be executed. I was not able to come up with an answer to explain this. I was tinkering with the code and then accidentally I solved the problem. I just changed the order of the buttons. Initially I had – Spellcheck button , Save Button and Undo Button. Now, I have – Save button, Spellcheck button, Undo button. Weird! You can have a look at the change at https://bitbucket.org/rtnpro/transifex/changeset/b59880cadf67 . If you have an explanation for this, please comment.

One problem gets fixed and another comes to your mind. The auto-save function was being an overhead. The function would be triggered even if I press the spellcheck button, auto translate and copy source buttons in the same row. I am still editing the same string! The way it should be is  that it should trigger when I am finished with editing the string.

So, what I did is that I moved the code saving the string from the function being called by the blur event of the textarea to a new function. Whenever a textarea being edited receives the blur event, I save the edited string, its id and a must_push flag for future use. Now, whenever any other textarea receives focus, depending on the must_push flag, the string save function is called. This was just one part of the problem I fixed. Now, I have to deal with the click events that might take place in various parts of the page and it should save the edited string from the textarea that just lost focus. My save function is already ready and it deals with the required checks on whether to save or not. I just need to call the new save function. I just browsed through the html tree, found a list of elements and bound their click event to the save function. As for the table rows, I bound the click event of all the rows other than the current row to the save function. Here is the commit implementing the new auto-save feature : https://bitbucket.org/rtnpro/transifex/changeset/5a7c2a2e13ea¬† . I had quite some jquery drill to fix these issues ūüôā .

Well, now Lotte with new auto-save feature is ready. Nice and elegant. I hope it makes up to the expectations of the translators. That’s what people strive at Transifex.