Integrating Unicode CLDR: Parsing with Retrieved Data

As I mentioned in the last blog post, I have been working to modify the codebase so that dateparser could use new data to parse date strings. Most of the new data for date translation is similar to previous data used for translation, the only major difference is in dealing with relative type date strings. The previous data used translations for ago and in to translate relative type date strings like "10 years ago",  "15 hours 10 minutes ago", "in 10 minutes and 8 seconds". But the data retrieved from unicode CLDR contains translation for complete date strings so that they translate directly to formats like "** year ago" or "in ** month" which can be used further for parsing. Initially I thought direct translations like these would work but then I realised that many relative dates are not covered in these translations. These include dates that contains more than one date fields, like "1 year 3 months ago" and dates that include time as well like "in 3 days 2 pm". Currently dateparser is able to parse such dates and many more dates that contain relative type dates as substrings, and if such dates are not translated correctly it would mean such dates would no longer be parsed.

The obvious solution to the problem is to use regex to translate relative type substrings in date string and translate other portions of date string separately, and later combine to form the complete translated date. I have used regex to translate relative type substrings as well as split the date string to separate out other parts of date string which are then separately translated using the current approach for translation. Another change that has been made is translation of numeral digits, for which the Python built-in function int() itself comes in handy. Instead of using translations for digits stored in unicode CLDR, it is more convenient and practical to use int() itself as it is able to translate most of the numeral digits, it is able to convert digits used for positional numeral systems, so it cannot be used to translate numeral systems that are not positional, like Chinese numerals. So first the numeral digits are converted in date string before translating rest of the date string. I am still working and I will be able to complete my work by the end of this week so that dateparser will be able to parse  dates without numeral words. After that I will start my work on numeral parser and dates with numeral words.

Popular posts from this blog

GSoC 2017: Work Summary

So finally Google Summer of Code 2017 is coming to an end. It was an amazing experience to be a part of this prestigious program and I learned a lot in these three months about open source development. Work Summary I worked on dateparser , a project under sub-org Scrapinghub of Python Software Foundation (PSF) , that deals with parsing dates in various languages and formats. The objective of the project was to integrate translation data of all locales in Unicode Common Locale Data Repository(CLDR) which is a standard repository of locale specific data with the existing translation data in dateparser. Here is a brief outline of the work done on my project on dateparser during GSoC 2017: Work Completed 1.   Retrieved Translation data from Unicode CLDR Scripts were written to retrieve translation data from unicode CLDR  github repository. The translation data for dates and numerals were separately stored. 2. Ordered languages by population The languages were ordered on th
Read more

Integrate Unicode CLDR: Working with Retrieved Data

I have been working on scripts and modifying data for quite some time and it seems the data is ready to be used for parsing dates. The data retrieved from Unicode CLDR  has been divided into two parts: numeral_translation_data , that will be specifically used to parse numerals, and date_translation_data ,   that will be used to parse date strings after modifying it by parsing numerals if included in the date string. The existing data that was contributed by various individuals to dateparser has been modified to supplement the data retrieved from CLDR, i.e., only the portion of data that is not included in data retrieved from CLDR remains as supplementary data , which will continue to be modified by contributors in future. The data for date translation consists of translations for months, weekdays and periods, date order, and translations for relative-type dates. This data is stored for different languages and for each language, locale-specific data is stored for all locales with th
Read more

GSoC 2017 : My Project

So I have been finally selected for Google Summer of Code 2017  at Python Software Foundation . Python Software Foundation(PSF) serves as an umbrella organisation comprising various sub-orgs that have projects that contribute to the development of the Python language. The sub-org I am going to work with is Scrapinghub . About Scrapinghub Scrapinghub, as its name clearly suggests, deals primarily with scraping the web. Scrapinghub is a company concerned with Information Retrieval and its later manipulation, i.e., it deals with both data extraction and data processing after retrieval. It has various projects that deal with these topics and the one I am going to work on deals with data processing after retrieval. I am going to work with dateparser , which is a Python library that primarily deals with parsing dates in various languages and formats. About dateparser dateparser is a Python library that is used to parse various forms of dates in different languages to a common format.
Read more