Integrating Unicode CLDR: Adding Locale Support

So far translation data is ready to be used for translation of date strings. This includes combined data from unicode CLDR and supplementary data contributed by many individuals. Initially the data from unicode CLDR was stored as json files and supplementary data was stored as yaml files and data from both these sources were to be combined after loading each separately which was then supposed to be used for translating dates. But after discussion with my mentor we agreed upon a better approach suggested by him that we could store data directly in python modules. Storing data in python modules directly has the advantage that importing data from python modules is faster than loading from json and yaml files. And considering the fact that we don't have to combine data from cldr and supplementary data files at run time, storing data in Python modules proves to be much more efficient.

As the data is ready, now is the time to make necessary modifications in the codebase to support locales for date parsing. The modifications would be adding arguments locales, and region in addition to existing languages argument in parse function as I mentioned in my proposal; creating Locale class to create locale instances for translation; making changes to loader to support lazy importing of languages' data from python modules and later yielding required locale instances for translation; and finally making changes to parser to support translation with new data. I am currently working on the first three parts, and after completing it I will make relevant changes to parser. 

Popular posts from this blog

GSoC 2017: Work Summary

So finally Google Summer of Code 2017 is coming to an end. It was an amazing experience to be a part of this prestigious program and I learned a lot in these three months about open source development. Work Summary I worked on dateparser , a project under sub-org Scrapinghub of Python Software Foundation (PSF) , that deals with parsing dates in various languages and formats. The objective of the project was to integrate translation data of all locales in Unicode Common Locale Data Repository(CLDR) which is a standard repository of locale specific data with the existing translation data in dateparser. Here is a brief outline of the work done on my project on dateparser during GSoC 2017: Work Completed 1.   Retrieved Translation data from Unicode CLDR Scripts were written to retrieve translation data from unicode CLDR  github repository. The translation data for dates and numerals were separately stored. 2. Ordered languages by population The languages were...
Read more

Integrate Unicode CLDR: Working with Retrieved Data

I have been working on scripts and modifying data for quite some time and it seems the data is ready to be used for parsing dates. The data retrieved from Unicode CLDR  has been divided into two parts: numeral_translation_data , that will be specifically used to parse numerals, and date_translation_data ,   that will be used to parse date strings after modifying it by parsing numerals if included in the date string. The existing data that was contributed by various individuals to dateparser has been modified to supplement the data retrieved from CLDR, i.e., only the portion of data that is not included in data retrieved from CLDR remains as supplementary data , which will continue to be modified by contributors in future. The data for date translation consists of translations for months, weekdays and periods, date order, and translations for relative-type dates. This data is stored for different languages and for each language, locale-specific data is stored for ...
Read more

GSoC 2017 : My Project

So I have been finally selected for Google Summer of Code 2017  at Python Software Foundation . Python Software Foundation(PSF) serves as an umbrella organisation comprising various sub-orgs that have projects that contribute to the development of the Python language. The sub-org I am going to work with is Scrapinghub . About Scrapinghub Scrapinghub, as its name clearly suggests, deals primarily with scraping the web. Scrapinghub is a company concerned with Information Retrieval and its later manipulation, i.e., it deals with both data extraction and data processing after retrieval. It has various projects that deal with these topics and the one I am going to work on deals with data processing after retrieval. I am going to work with dateparser , which is a Python library that primarily deals with parsing dates in various languages and formats. About dateparser dateparser is a Python library that is used to parse various forms of dates in different languages to a common for...
Read more