#Transifex now supports comments in Apple .strings i18n files

#Transifex now supports comments in Apple .strings i18n files. Only /* foo */ style comment in the line preceding the key value pair in the source file is saved as a comment for the key. The example below will explain this in a better way:

/*Comment for key1*/
"key1" = "value 1";

/* This comment will not be
included in key2*/

/* comment for
key2*/
"key2" = "value 2";

/* this comment will not be included in key3*/

"key3" = "value 3";

Well, I’m pretty sure that the above snippet explains which comments from source Apple .strings file are saved by Transifex. You can see the comment for a source string in its “Details” section in Lotte.

Comment for a source string imported from a source Apple .strings file

NVIDIA issues fixed on Fedora 16

This week, I upgraded from Fedora 15 to Fedora 16 on my Dell XPS M1530 laptop. This laptop has a 256 MB NVIDIA 8600M GT graphics card. The default driver for NVIDIA cards that came with the installation was nouveau. Nouveau is an open source driver for NVIDIA graphics cards and is under development. Things are becoming better and better with nouveau.

I ran gnome-shell for some time with the nouveau driver. 3d rendering worked nicely and without any latency. I did not find any other issue except for some overheating issues. So, I decided to switch to the NVIDIA’s proprietary driver.

Here is a good tutorial to disable nouveau and install NVIDIA’s proprietary driver in Fedora 16: http://www.if-not-true-then-false.com/2011/fedora-16-nvidia-drivers-install-guide-disable-nouveau-driver/

But, this would work perfectly if this was for Fedora 15. Fedora 16 comes with glibc-2.14.90-14 and the NVIDIA proprietary driver (the latest stable driver as of now is NVIDIA-Linux-x86-285.05.09.run). This issue has been reported at https://bugzilla.redhat.com/show_bug.cgi?id=737223. The issues I faced after installing the proprietary NVIDIA driver in my Fedora 16 machine were:

• window manager behaving sluggishly
• tab switch in applications like gnome-terminal, nautilus browser taking around 3-4 seconds (no such issues with KDE’s konsole)
• System getting overheated
• Increased latency in gnome-shell effects
• Similar issues with window manager and tab switch on XFCE too

I guess everything depended on glibc were affected.

This issue could be fixed  by downgrading glibc to  glibc-2.14.90-4. I tried to do this to find that there are quite a few applications depended on glibc-2.14.90-14 in F16. So, I gave up the idea. I was looking for nvnews for any news from NVIDIA about fix for the above issue. And I came across this thread http://www.nvnews.net/vbulletin/showthread.php?t=122606 where I found about the current releases of NVIDIA graphics driver and the beta driver at http://www.nvnews.net/vbulletin/showthread.php?p=2498046. I was desperate enough to try the beta driver. I did:

• $ wget -ivh ftp://download.nvidia.com/XFree86/Linux-x86/290.06/NVIDIA-Linux-x86-290.06.run
• $ su
• # init 3
• # sh NVIDIA-Linux-x86-290.06.run

and then follow through the on screen instructions.

• # init 5

and logged in. To my surprise, everything was perfect this time. No latency in gnome-shell, no overheating issue. Everything is just fine. Now, I have been running gnome-shell on Fedora 16 in my laptop for over 24 hours. I did not find any issue with the NVIDIA beta graphics driver so far.

Start testing Transifex

 

How do you setup Transifex?

Here is all you need to know to setup Transifex: http://help.transifex.net/server/install.html

http://fosswithme.wordpress.com/2011/10/20/setup-transifex-in-virtualenv/ is another good write-up on how to setup and run Transifex in virtualenv. So, I’d be building on top of that to show you how to start testing Transifex using django-nose.

What packages will you need?

django-nose, nose, nose-exclude, coverage

You can install the above packages using

pip install <package_name>

Configure Transifex settings to enable django-nose Test Runner

cd <transifex source code's root directory>

cd transifex/settings

cp  90-local.conf.sample 90-local.conf

open and edit 90-local.conf and add the following lines:

INSTALLED_APPS += [

'django_nose',

]

TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'

cd ..

Now, save the file. Now you are good to go.

Start testing

python manage.py test <app_name>

for example:

python manage.py test resources

You can also test a particular test class like below:

python manage.py test resources.tests:TestJavaProperties

You can even run a particular test method:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser

Run coverage on Transifex tests

django-nose has a plugin for coverage. So, you can run the above tests and collect coverage data.

For example:

python manage.py test resources.tests:TestJavaProperties.test_properties_parser --with-coverage --cover-package=resources.tests.formats

All the coverage results are saved in a .coverge file by default in the current directory. Although, running tests with coverage plugin of django-nose shows the coverage results by default. You can also see the coverage results in the usual way:

coverage -rm

You can also use grep along with the above command to filter the results displayed.

Well, that's all you need to know to start testing Transifex.

What's next?

Start testing transifex. If you find a test fails, try to find the reason why it failed. Read the traceback info properly. Find where the error took place. There are various reasons why a test may fail:

• Test is not updated according to updates in code
• Bug in code
• A wrong test case

and others...

You can report the issues or any bug you find at http://trac.transifex.org/newticket. Feel free to submit a patch that fixes the issue. The patch will be reviewed by the Transifex upstream and if it is ok, it will be merged with Transifex's code at http://code.indifex.com/transifex/.

Keep hacking

Regex pattern for c style comments

Today, I am going to discuss my attempts to parse c style comments.

For example,

//This is a comment


/***This is also
*** a comment ***/

Initially, I came up with a regex for /*…*/ style comments :
/\*.*\*/
Well, the above expression was not able to parse comments like:

/*** This is a comment ***/


I googled and came across http://ostermiller.org/findcomment.html where I found the regex:
/\*(.|[\r\n])*?\*/
This was able to match comments like the above one. But it’d also match the following /*…*/ comments which are not really comments:

s = "This is a string: /* with a comment */";
//comment1 /*
foo();
//comment2 */


I then worked on a regex for //… style comments: //[^\n]*\n

Then I combined the two regexes by or and my regex pattern becomes:

//[^\n]*\n|/\*(.|[\r\n])*?\*/

Now, this pattern is able to search for both: //… and /*…*/ style comments and avoid matches for patterns like:

//comment1 /*
foo();
//comment2 */

One caveat that remains is the /*…*/ pattern in
s = "This is a string: /* with a comment */";
getting matched. If any one has a work around this issue, please comment.

I hope this helps.

Add plug-n-play functionality to your Django project using Django-addons

What is Django-addons?

A Django app used to add true plug-n-play functionality to your own Django applications and projects. Django-addons is brought to you by Indifex, the company behind Transifex.

Django-addons is a bunch of code that makes writing addon/plugins for your Django project much easier. Add django-addons to your Django project and you can drop all the addons to ‘/addons’ directory.

How to install Django-addons?

You can install the latest version of django-addons running
pip install django-addons
or
easy_install django-addons

You can also install the development version of django-addons with
pip install django-addons==dev
or
easy_install django-addons==dev.

Source code

http://code.indifex.com/django-addons/

Features

• Addons overview page
• Automatic signal connecting of addons
• Automatic URL discovery of addons
• Template hooking system (inject code from addons to your main project)
• Django-staticfiles to serve site media from each addon
• Django-notifications support (automatic registration of noticetypes)
• Per addon localization
• Per addon settings
• Disabling addons via ./manage.py addons

Transifex implements related tag cloud

Lately, I have been working on a bunch of exciting new stuffs for Transifex. I have worked on a tag-cloud implementation which not only shows the popular tags, but also shows tags related to a tag selected by the user. It is pretty useful. It directs the user to select more relevant tags. The tag cloud is refreshed each time the user makes a selection to show the related tags.

I built this on top of the django-tagging module. I wrote a model to represent a tag as a node in a graph. The model includes all the tags related to it (that is tags which appear with the tag in concern) as adjacent nodes along with the weight (that is number of times the two tags appear together) of each edge between two related tag nodes. This data is updated and synced as necessary, e.g, after a project is added or updated. Now, whenever a tag is selected, the tag-cloud is refreshed to show the related tags. The font-size of a related tag is decided by taking into consideration both the weight of an edge it shares with the selected tag and its count. Below is a sample use case for related tag-cloud in Transifex.

Let’s say there are two projects, p1 with tag ‘foo1′ and p2 with tags ‘gui’, ‘graphics’, ‘imaging’ and ‘photography’. For sake of simplicity, I am showing only 3 most popular tags: ‘foo1′, ‘gui’, ‘graphics’. So, now when the maintainer for prohect p1 goes to edit the project, he sees the following tag-cloud:

Initial tagcloud

Now, he selects a new tag ‘graphics’ and the tagcloud is refreshed to show the tags related to ‘graphics’.

Tagcloud with related tags.

Such small things together can really take the user experience to a new level. By implementing related tag-clouds, we enable the user to choose relevant tags in a better way. At Transifex, we innovate to help people localize in a better way .

A brief introduction to coverage.py

Coverage.py is a tool for measuring code coverage of Python programs. It monitors your program, noting which parts of the code have been executed, then analyzes the source to identify code that could have been executed but was not.

Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not.

Getting started:

1. Install coverage:

• pip install coverage
• easy_install coverage

2. Use coverage to run your program and gather data:

$ coverage run my_program.py arg1 arg2
blah blah ..your program's output.. blah blah

3. Generate reports with coverage:

$coverage -rm

Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
my_program                   20      4    80%   33-35, 39
my_other_module              56      6    89%   17-23
-------------------------------------------------------
TOTAL                        76      10    87%

4. You can also use coverage to generate reports in other presentation oriented formats like HTML:

$coverage html

You can also use coverage.py with Django. You can run your Django tests along with coverage to check which codes in your app have been tested by your tests. With the coverage data, you can write new tests to test the codes which have not been tested so far by your tests. For example:

• $coverage -e        //This deletes previous coverage data
• $coverage -x manange.py test foo_app.FooTest.foo_method        //Execute manage.py from coverage to collect coverage data
• $coverage -rm | grep ‘foo_app’      // to filter the report to show the coverage of foo_app
• coverage run –include=”*foo_app*” –omit=”*tests* manage.py test foo_app        //This will include *foo_app* pattern and omit *tests* pattern from your coverage report.

You can also write custom Test Runners using the coverage API to measure code coverage in a more controlled manner. You can find more detailed documentation about coverage here.

More fine tuned auto save feature in Lotte for Transifex

Lotte is the component of Transifex providing the web UI for the translators to translate online. Well, some time back, I made some changes in the Lotte code so that it has the auto-save feature on by default. It was running fine then and I was happy. Then I added the spellcheck feature in Lotte. Then the things started going not as expected.

Actually, the auto-save triggers when the contents of a text area has been edited and it loses focus. The first problem I faced is that the spellcheck button did not respond on the first click, but on the second. As expected, on the first click, auto-save was triggered. But this was not the case when I clicked the Undo button. First the auto-save and then the undo function would be executed. I was not able to come up with an answer to explain this. I was tinkering with the code and then accidentally I solved the problem. I just changed the order of the buttons. Initially I had – Spellcheck button , Save Button and Undo Button. Now, I have – Save button, Spellcheck button, Undo button. Weird! You can have a look at the change at https://bitbucket.org/rtnpro/transifex/changeset/b59880cadf67 . If you have an explanation for this, please comment.

One problem gets fixed and another comes to your mind. The auto-save function was being an overhead. The function would be triggered even if I press the spellcheck button, auto translate and copy source buttons in the same row. I am still editing the same string! The way it should be is  that it should trigger when I am finished with editing the string.

So, what I did is that I moved the code saving the string from the function being called by the blur event of the textarea to a new function. Whenever a textarea being edited receives the blur event, I save the edited string, its id and a must_push flag for future use. Now, whenever any other textarea receives focus, depending on the must_push flag, the string save function is called. This was just one part of the problem I fixed. Now, I have to deal with the click events that might take place in various parts of the page and it should save the edited string from the textarea that just lost focus. My save function is already ready and it deals with the required checks on whether to save or not. I just need to call the new save function. I just browsed through the html tree, found a list of elements and bound their click event to the save function. As for the table rows, I bound the click event of all the rows other than the current row to the save function. Here is the commit implementing the new auto-save feature : https://bitbucket.org/rtnpro/transifex/changeset/5a7c2a2e13ea  . I had quite some jquery drill to fix these issues .

Well, now Lotte with new auto-save feature is ready. Nice and elegant. I hope it makes up to the expectations of the translators. That’s what people strive at Transifex.

Testing coverage of your Django code

Just writing tests for your Django codebase is not enough. You need to check how much of the code is covered in your tests. For this, there are some tools available. Again, it is not just the number of lines of code tested that matters. What matters is “Are these lines important?”. Well, for this, we have to use our head.

covergae is a tool for checking test coverage of python applications. django-test-coverage was built on top of coverage.py to meet requirements of django tests.

I was able to plugin django-test-coverage in Transifex. For some test cases it ran, for others, it failed. It raised a warning saying that a module was being imported more than once. But, the stats generated by it were misleading. Except for files with 0 lines of code (like __init__.py), it showed code coverage % as 0 and 100 only for the files having 0 line of code. I hacked into its code and was able to run it for cases where it had failed previously. But still the statistics were misleading. Time was running out. So, I decided that I would revisit its code some time later.

I resorted to use coverage.py. You can find an introduction about coverage.py at here. Using coverage boils down to 3 steps:

1. Erase previously collected data : $coverage -e
2. Execute the necessary tests and collect data: $coverage -x <test module>
3. Display the result: $coverage -r -m

If the test module is large, the report generated by coverage is also large. I usually save the report to a file : $coverage -r -m > report.txt. Now, I can use grep to shortlist the report to see the details of the files which concerns me now. That’s pretty easy.

coverage.py gives you very useful informations like percentage code coverage of a file, missing statements, etc. Although a higher percentage code coverage is better, but the importance of the lines also matters a lot. You could increase the code coverage by including 10 not so important lines rather than including 1 important line. So, code coverage statistics helps us to write tests to cover more codes, but it is not a replacement to thinking. The final judgement is to be done by us.

Fixing unittests for transifex

Last week, I have been working on fixing the existing unittests for Transifex. The Transifex codebase is constantly being updated. So, some of the previous test cases failed. Before this, I had already written a unittest or two for some of the codes that I contributed to Transifex. But I did not study the Django unittest framework deeply for that.

This time, it was different. I took my time to go through the Django documentation on testing. And then I started to work on fixing the existing tests. To avoid confusion, I ran test on each component separately. Only a few were containing errors like projects, lotte, release, resources, projects, txcommon, suggestions, charts, etc.

I started working on one component at a time. After fixing it, I committed and pushed my changes. Then I proceeded to the next one. In between, I was always in touch with diegobz, kbairak, messas. They helped a lot. I once ran into a bug due to haystack module. I reported this to diegobz. He fixed the issue with haystack quickly.

There was another instance when a particular test was showing errors in my box but not for anyone else. I struggled with it for sometime. Then, kbairak told me that the Transifex code is now compatible with Django-1.2.5, but will become compatible with Django-1.3 soon. And here I was running Django-1.3. kbairak fixed some code to make it compatible with Django-1.3.

The Transifex upstream ROCKS!!!