GSoC 2017 @ dateparser

Finally I have successfully implemented translation of date strings with new integrated data. I have been working on it for a long while now, much more than I expected. I expected to be completely finished with translation of date strings without numeral words much earlier and I was planning to get started with numeral parser as early as possible. But it took me a while longer as I did not think clearly and therefore started on with a complicated solution and then moved on to simpler solutions. As I mentioned in my last blog post, the problem was to translate relative date strings, which are of two types, one that have no digits like 'yesterday' and the other that have digits and are stored as regex patterns like '(\\d+) day ago'. Currently dateparser uses translations for 'ago' and 'in' along with other words and relative dates are translated in a similar way as other date strings. First dates are splitted by numbers within dates and then by known wo...

As I mentioned in the last blog post, I have been working to modify the codebase so that dateparser could use new data to parse date strings. Most of the new data for date translation is similar to previous data used for translation, the only major difference is in dealing with relative type date strings. The previous data used translations for ago and in to translate relative type date strings like "10 years ago",  "15 hours 10 minutes ago", "in 10 minutes and 8 seconds". But the data retrieved from unicode CLDR contains translation for complete date strings so that they translate directly to formats like "** year ago" or "in ** month" which can be used further for parsing. Initially I thought direct translations like these would work but then I realised that many relative dates are not covered in these translations. These include dates that contains more than one date fields, like "1 year 3 months ago" and dates that includ...

So far translation data is ready to be used for translation of date strings. This includes combined data from unicode CLDR and supplementary data contributed by many individuals. Initially the data from unicode CLDR was stored as json files and supplementary data was stored as yaml files and data from both these sources were to be combined after loading each separately which was then supposed to be used for translating dates. But after discussion with my mentor we agreed upon a better approach suggested by him that we could store data directly in python modules. Storing data in python modules directly has the advantage that importing data from python modules is faster than loading from json and yaml files. And considering the fact that we don't have to combine data from cldr and supplementary data files at run time, storing data in Python modules proves to be much more efficient. As the data is ready, now is the time to make necessary modifications in the codebase to support l...

There has been some time since I last wrote and I have been working on my project. As I was free during the GSoC bonding period, I started working on my project early so as to have a head start. I started working on scripts to retrieve translation data from unicode CLDR. That came out good as by the time GSoC period started, I had already written a major part of the scripts and had retrieved most of the data required for parsing. At the same time I started to write tests for dateparser and coverage has increased fairly. After the GSoC period started I made further changes to the script to resolve some issues like correcting date order of languages and storing data in order. As of now since I have added numeral data as well, data that has been retrieved is complete with all components required to translate dates as initially proposed in my proposal. The challenge now is to use this data to effectively and efficiently translate dates. Currently dateparser uses a dictionary based metho...