Implement library to get texts

[arielgreen]

If you haven’t already, check out our contributing guidelines for onboarding!

---

We are implementing the structure for internationalisation and localisation.

As part of that, we need to implement a library to get texts. We will pass it a translation key, a locale and optionally an object with variables and it will return the correct text to use.

Let’s first add some clarity on what each language file will look like. By default, we will only have the en file (which will be the one used by default when no translation is found). Let’s use the file src/languages/en.js as an example, it would look something like this and would just export that object:

{
	localizationConfig: {
	phoneCountryCode: ‘1’,
}
	signOut: “Sign Out”,
	chatSwitcher: {
		searchInput: “Find or start a chat”
	}
	IOUReport: {
		oweRow: ({payerName, payeeName, amount, currency}) => `${payerName} owes` + currency(‘en’, amount, currency, 2) + `to ${payeeName}`;
	}
}

There will be a main language file called translations.js, which will import each of the existing defined language files into one big object and export that. This is what will be used by the library to get the texts.

This library will only export one method translate(locale, key,variables = {}) and it will:

1. Lookup that key in the translations object:

• Try to find it using the full locale
• Try to find it using the language part of the locale only (ie: the first part of the locale. For es-ES will get es)
• If the language requested is not en, then log an alert to the server (so we know we are missing a translation) and then try to find it using the default language en.
• In dev throw if the key does not exist and in production would use the key name and log an alert to the server.

2. So now we found the key and see the value, if it is a string, it will return it as is. If the value is a function, it will call the function, passing the variables to it as a parameter (JS-Lib’s Str.result does this)

Please include JUnit tests for the added methods.

Apply on Upwork: https://www.upwork.com/jobs/~019d8c473a29e6f464
https://github.com/Expensify/Expensify/issues/151944

[tugbadogan]

Hi @arielgreen,

I would like to work on this issue. Here is my proposal:

1. I'm going to create en.js and en-GB.js sample translation files in src/languages folder. These files will contain object with translation data and I will export these objects.
2. Then, I will create translations.js file in the same folder. This file will import all available locales, merge them into a single object keyed with locale and export this single object.
3. I will create a library named translate.js in src/libs folder. This module will export translate(locale, key,variables = {}) method.
4. In translate method, I'm going to extract corresponding language object from translations object. If full locale or first part of the locale is not found in the translations object, I will log an alert to the server like this:

https://github.com/Expensify/Expensify.cash/blob/490d56553556e614e0d3c15cf4496e8fabb7bd3c/src/Expensify.js#L34

5. I'm assuming key parameter will be a string. I have a question here. For nested values how are we going to pass the key? In the translation example, are we going to pass chatSwitcher.searchInput as key parameter to get translation for searchInput?
6. If key is found in translation object and the value is string, I will return the string. If it's a function I will call that function with given variables. If key is not found, I will return the key itself and log an alert to the server.

[parasharrajat]

Proposal

1. Create a file libs/translate.js which exports the only method translate(locale, key,variables = {}).
2. translate function will import the translation.js file which import each language files and exports a object like below.

en:{
   localizationConfig: {
   phoneCountryCode: ‘1’,
      }
   signOut: “Sign Out”,
   chatSwitcher: {
   	searchInput: “Find or start a chat”
   }
   IOUReport: {
   	oweRow: ({payerName, payeeName, amount, currency}) => `${payerName} owes` + currency(‘en’, amount, currency, 2) + `to ${payeeName}`;
   }
},
`other-locale`: {
 ...same set of keys but in the current locale.
 }
 ...so on

3. Translate function will use lodash.get to find the keys (eg 'chatSwitcher.signup') based on the locale full name, if not found lookup using the language part of the locale.
4. If found, use result method from expensify-common/str.js to get the final result.
5. If not found, check the language, and if it's not en use libs/log.js Log method to log to the server. Now find the same key with en.
6. Throw if none of the above passing in dev mode (!IS_IN_PRODUCTION from CONFIG.js), otherwise Log to the server.
7. I will add the Test to cover all the cases.

Questions

1. Shouldn't we use the locale automatically based on the browser locale and optionally pass the locale? So that the new syntax be like translate(key, locale?, variables?). As we have to pass the locale while calling it and we would get the locale from the language settings. so we can just set the language setting in Onyx store and use it in the translate function.

[iwiznia]

I'm assuming key parameter will be a string. I have a question here. For nested values how are we going to pass the key? In the translation example, are we going to pass chatSwitcher.searchInput as key parameter to get translation for searchInput?

@tugbadogan great question! The key should be either a string like "chatSwitcher.searchInput" OR an array like ["chatSwitcher", "textInput"]. Both should work.

@tugbadogan also, keep in mind this, which I don't see in your proposal:

If the language requested is not en, then log an alert to the server (so we know we are missing a translation) and then try to find it using the default language en.
In dev throw if the key does not exist and in production would use the key name and log an alert to the server.

---

Shouldn't we use the locale automatically based on the browser locale and optionally pass the locale? So that the new syntax be like translate(key, locale?, variables?). As we have to pass the locale while calling it and we would get the locale from the language settings. so we can just set the language setting in Onyx store and use it in the translate function.

@parasharrajat, no, that will be handled at a higher level, this library is not tasked to detect the browser's locale.

[tugbadogan]

@tugbadogan also, keep in mind this, which I don't see in your proposal:

If the language requested is not en, then log an alert to the server (so we know we are missing a translation) and then try to find it using the default language en.

For missing translations for given language, I'm going to do the following to handle that scenario (4th item in my proposal).

4. In translate method, I'm going to extract corresponding language object from translations object. If full locale or first part of the locale is not found in the translations object, I will log an alert to the server like this:

In dev throw if the key does not exist and in production would use the key name and log an alert to the server.

And missing translations for given key, I'm going to do the following (6th item in my proposal).

6. If key is found in translation object and the value is string, I will return the string. If it's a function I will call that function with given variables. If key is not found, I will return the key itself and log an alert to the server.

Am I missing something here?

[iwiznia]

Not sure we are saying the same thing or not 😅

There's 2 scenarios:

If the language requested is not en, then log an alert to the server (so we know we are missing a translation) and then try to find it using the default language en.

This is to fallback to english if someone requests a text in for example spanish that the spanish transalation does not have. That will let us know that we missed translating a key to spanish.

In dev throw if the key does not exist and in production would use the key name and log an alert to the server.

This comes after the previous check and it applies only to english (or when we are falling back to english). This has 2 goals:

1. In dev, we will throw an exception if you are trying to use a key that has not been added to the default language (en). That will hopefully prevent us from shipping code that references a key that was never added in the first place
2. Since we don't want to throw an exception (and break the app) in production, we would just log it (and use the key as the translation). This is just in case 1 above did not catch the missing key.

I'll give you a an example which might be easier:
en translation file is: {greeting: "hello"}
es translation file is empty: {}

I ask for key greeting with es locale. So what it should do is:

• Log that greeting was not found in the es locale
• Fallback to english and return hello

I ask for key farewell with es locale. So what it should do is:

• Log that farewell was not found in the es locale
• If we are on dev, throw an exception saying farewell was not found in the default language
• If we are on prod, log that farewell was not found in the en locale and return farewell

[tugbadogan]

Thanks for explanation @iwiznia. I got it now 👍 Is there a common way to distinguish dev from prod in the code?

[iwiznia]

Yes, you can use this

[kidroca]

Regarding bundling and loading all the supported languages

This will add some maintenance limitations:

1. Translations are bundled inside the app -> you will have to resubmit the app to the store for translation fixes
2. All translations are loaded while only one is used
3. All translation have to be loaded before using the app

As the app grows these factors will increase proportionally

Instead I can offer you a solution where your translations are provided dynamically from you backend.
During you app initialization (splash screen): the proper locale is fetched (and cached) - only 1 locale
If any problems (like being offline) arise you can fallback to the last cached version of that locale or the default locale (en)
This is a cross platform solution.

You will be free to update or add more translations at any time without having to re-submit the app to the store
It's a flexible solution which allows you to pivot - if you decide you want to keep translations on the device you load the translation from the device instead of a server.

---

Currently proposed structure for translation mappings is not optimal

IOUReport: {
	oweRow: ({payerName, payeeName, amount, currency}) => `${payerName} owes` + currency(‘en’, amount, currency, 2) + `to ${payeeName}`;
}

This couples implementation details - how specific parameters work (currency).
It's not translation tools friendly (Ask for details if you want to know more)
Code will repeat for each language you support e.g. currency('fr', amount, currency, 2)

Instead You can use json or any other popular format which would allow you to extract translations from you main code and provide it dynamically from a server:

{
  "IOUReport": {
    "oweRow": "{payerName} owes {amount} to {payeeName}"
  }
}

Functionality like currency('en', amount, currency, 2) can be moved out of translations and be applied by the translate(key, params) function. Something like:

Sample translate() interface/usage

translate('IOUReport.oweRow', {
  payerName,                 // currency('en', amount, currency, 2)
  amount: (localCtx) => formatCurrency(localeCtx.locale, amount, currency, 2),
  payeeName,
})

When a param in params to translate(key, params) is a function, translate will invoke that function with a locale context and use the result to substitute text in the retrieved mapping.
This way specifics and code repetition are decoupled from translation mappings, while translate() would only delegate context to the caller in a couple free way.

A related topic in date internationalization: #1856 (comment)

---

You can contact me for more details.