[Proposal] Invest in a new translation service

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Invest in a new translation service

vmassol
Administrator
Hi devs,

We started discussing this yesterday on the xwiki chat.

Here’s the rationale for changing:

* l10n is slow and is preventing users to contribute translations (it’s much harder than it should be)
* it’s costing us maintenance that we shouldn’t have to do (it’s not our role to maintain a translation service, even though it was nice to eat our own dog food initially). I see this very similar to when we switched from our GWT-based WYSIWYG editor to CKEditor.
* It’ll allow us to benefit from new features/bug fixes the external service develops
* Right now it’s taking an hour or more every time we release to integrate the translation changes and this slows us down to increase our release delivery speed
* We’re putting the onus on the RM only to validate that the proposed translations are good. So only 1 pair of eyes.
* We have no time to fix any translation if they’re not correct. A system where when someone proposes a new translation generates a PR that we apply as they come in would be much better and solve both the review and lateness issues.

Of course it’ll mean some plugins/customizations to develop in the service we will use since it’s not going to support some custom formats we have such as our XAR XML format. So we need to pick a tool that allows for this.

The proposal:
* Start investing in implementing XWiki translation using an external tool.
* Start  by looking at weblate: https://weblate.org/en/ since
** it seems to offer lots of features we need (automatic PR - see https://docs.weblate.org/en/latest/vcs.html#github, quality checks, plugins). See https://weblate.org/en/features/
** it’s open source itself and in case the service goes down we can ask XWiki SAS to host it for us (see https://github.com/WeblateOrg/weblate).
** We need to ask them if they can host us, see https://weblate.org/en/hosting/ and the part about "Hosting for free software”.
** We could also ask XWiki SAS if they’d accept paying a “Basic” plan (200 euros/year which is affordable IMO vs hosting it ourselves)
** There’s a REST API so that if we want we can provide a view/status of the translations from xwiki.org: https://docs.weblate.org/en/latest/api.html . We could even imagine using this API to convert from a format to our own format, if need be.
* Propose to have a developer work on this in an upcoming roadmap (to be defined but as early as possible since l10n is not in good shape)
* Fix l10n as much as we can without spending too much time on it, until we have the new translation service ready to be used

Things to look at:
* Ability to register custom formats or more generally how to handle our custom format
* How do we handle deprecated translations keys
* How do we handle global rename/move of keys from one resource file to another
* <add more here>

WDYT?

I’d like to have especially Thomas’s POV since he’s the one who spent the most time on l10n and I’d like to make sure he’s ok with this.

Thanks
-Vincent




Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Invest in a new translation service

Eduard Moraru
Hi,

I am a bit skeptical about such a platform that:
* we'd need our users to register to (yet another account for the XWiki
contributor)
* we'd need to twist and turn so that it bends to our needs
* (and probably other things caused by the fact that I did not get to read
too much about it yet :) )

You mentioning PRs made me wonder if we couldn't simply ask contributors to
submit a standard PR for translation changes. That would take care of many
things and also give the author proper attributions.

The minimum requirements for this would be, IMO:
* the list of all translation files (i.e. links on GitHub; either manually
managed, automatically extracted or a mix) and
* the ability to search for a key or translation and find the relevant file
(most likely this would involve us indexing or caching the translation
files content somewhere on xwiki.org, kept up to date with webhooks? --
maybe on a very light translations dedicated wiki; a light l10n?).
* the ability to create a new language for a translation page, if it does
not already exist (to help the edge case when the language file is not
already there).

Something like this could even be done without an xwiki.org account, as
long as the user has a github account for editing (even with GitHub's web
edit UI) and submitting the PR.

Most importantly, IMO, the translations could become part of the
development process and not just a step at release time. A clear
consequence would be that snapshots would benefit from new and integrated
translations, added since the last release and we won't be seeing them just
after the release. AFAIU, weblate supports this as well, but by using its
own github account and submitting PRs in the name of the author (which
would not be the same effect as the actual author, but then again, only
actual GitHub users would care about this).

WDYT? Would it be too hard to use for non-techical users? Would we need
more that the described, in terms of UI (and would it justify the
effort/cost)?

Thanks,
Eduard


On Fri, Mar 23, 2018 at 11:56 AM, Vincent Massol <[hidden email]> wrote:

> Hi devs,
>
> We started discussing this yesterday on the xwiki chat.
>
> Here’s the rationale for changing:
>
> * l10n is slow and is preventing users to contribute translations (it’s
> much harder than it should be)
> * it’s costing us maintenance that we shouldn’t have to do (it’s not our
> role to maintain a translation service, even though it was nice to eat our
> own dog food initially). I see this very similar to when we switched from
> our GWT-based WYSIWYG editor to CKEditor.
> * It’ll allow us to benefit from new features/bug fixes the external
> service develops
> * Right now it’s taking an hour or more every time we release to integrate
> the translation changes and this slows us down to increase our release
> delivery speed
> * We’re putting the onus on the RM only to validate that the proposed
> translations are good. So only 1 pair of eyes.
> * We have no time to fix any translation if they’re not correct. A system
> where when someone proposes a new translation generates a PR that we apply
> as they come in would be much better and solve both the review and lateness
> issues.
>
> Of course it’ll mean some plugins/customizations to develop in the service
> we will use since it’s not going to support some custom formats we have
> such as our XAR XML format. So we need to pick a tool that allows for this.
>
> The proposal:
> * Start investing in implementing XWiki translation using an external tool.
> * Start  by looking at weblate: https://weblate.org/en/ since
> ** it seems to offer lots of features we need (automatic PR - see
> https://docs.weblate.org/en/latest/vcs.html#github, quality checks,
> plugins). See https://weblate.org/en/features/
> ** it’s open source itself and in case the service goes down we can ask
> XWiki SAS to host it for us (see https://github.com/WeblateOrg/weblate).
> ** We need to ask them if they can host us, see https://weblate.org/en/
> hosting/ and the part about "Hosting for free software”.
> ** We could also ask XWiki SAS if they’d accept paying a “Basic” plan (200
> euros/year which is affordable IMO vs hosting it ourselves)
> ** There’s a REST API so that if we want we can provide a view/status of
> the translations from xwiki.org: https://docs.weblate.org/en/
> latest/api.html . We could even imagine using this API to convert from a
> format to our own format, if need be.
> * Propose to have a developer work on this in an upcoming roadmap (to be
> defined but as early as possible since l10n is not in good shape)
> * Fix l10n as much as we can without spending too much time on it, until
> we have the new translation service ready to be used
>
> Things to look at:
> * Ability to register custom formats or more generally how to handle our
> custom format
> * How do we handle deprecated translations keys
> * How do we handle global rename/move of keys from one resource file to
> another
> * <add more here>
>
> WDYT?
>
> I’d like to have especially Thomas’s POV since he’s the one who spent the
> most time on l10n and I’d like to make sure he’s ok with this.
>
> Thanks
> -Vincent
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Invest in a new translation service

Guillaume Delhumeau
2018-03-26 12:05 GMT+02:00 Eduard Moraru <[hidden email]>:

> Hi,
>
> I am a bit skeptical about such a platform that:
> * we'd need our users to register to (yet another account for the XWiki
> contributor)
>

Maybe we could use an LDAP/SSO connector, just as we do for the Forum.


> * we'd need to twist and turn so that it bends to our needs
> * (and probably other things caused by the fact that I did not get to read
> too much about it yet :) )
>
> You mentioning PRs made me wonder if we couldn't simply ask contributors to
> submit a standard PR for translation changes. That would take care of many
> things and also give the author proper attributions.
>
> The minimum requirements for this would be, IMO:
> * the list of all translation files (i.e. links on GitHub; either manually
> managed, automatically extracted or a mix) and
> * the ability to search for a key or translation and find the relevant file
> (most likely this would involve us indexing or caching the translation
> files content somewhere on xwiki.org, kept up to date with webhooks? --
> maybe on a very light translations dedicated wiki; a light l10n?).
> * the ability to create a new language for a translation page, if it does
> not already exist (to help the edge case when the language file is not
> already there).
>
> Something like this could even be done without an xwiki.org account, as
> long as the user has a github account for editing (even with GitHub's web
> edit UI) and submitting the PR.
>

You just said it was a problem to use an other account that the xwiki.org
account :) You assume most people have a Github account, but I'm not sure
concerning translators.


>
> Most importantly, IMO, the translations could become part of the
> development process and not just a step at release time. A clear
> consequence would be that snapshots would benefit from new and integrated
> translations, added since the last release and we won't be seeing them just
> after the release. AFAIU, weblate supports this as well, but by using its
> own github account and submitting PRs in the name of the author (which
> would not be the same effect as the actual author, but then again, only
> actual GitHub users would care about this).
>

> WDYT? Would it be too hard to use for non-techical users? Would we need
> more that the described, in terms of UI (and would it justify the
> effort/cost)?
>
> Thanks,
> Eduard
>
>
> On Fri, Mar 23, 2018 at 11:56 AM, Vincent Massol <[hidden email]>
> wrote:
>
> > Hi devs,
> >
> > We started discussing this yesterday on the xwiki chat.
> >
> > Here’s the rationale for changing:
> >
> > * l10n is slow and is preventing users to contribute translations (it’s
> > much harder than it should be)
> > * it’s costing us maintenance that we shouldn’t have to do (it’s not our
> > role to maintain a translation service, even though it was nice to eat
> our
> > own dog food initially). I see this very similar to when we switched from
> > our GWT-based WYSIWYG editor to CKEditor.
> > * It’ll allow us to benefit from new features/bug fixes the external
> > service develops
> > * Right now it’s taking an hour or more every time we release to
> integrate
> > the translation changes and this slows us down to increase our release
> > delivery speed
> > * We’re putting the onus on the RM only to validate that the proposed
> > translations are good. So only 1 pair of eyes.
> > * We have no time to fix any translation if they’re not correct. A system
> > where when someone proposes a new translation generates a PR that we
> apply
> > as they come in would be much better and solve both the review and
> lateness
> > issues.
> >
> > Of course it’ll mean some plugins/customizations to develop in the
> service
> > we will use since it’s not going to support some custom formats we have
> > such as our XAR XML format. So we need to pick a tool that allows for
> this.
> >
> > The proposal:
> > * Start investing in implementing XWiki translation using an external
> tool.
> > * Start  by looking at weblate: https://weblate.org/en/ since
> > ** it seems to offer lots of features we need (automatic PR - see
> > https://docs.weblate.org/en/latest/vcs.html#github, quality checks,
> > plugins). See https://weblate.org/en/features/
> > ** it’s open source itself and in case the service goes down we can ask
> > XWiki SAS to host it for us (see https://github.com/WeblateOrg/weblate).
> > ** We need to ask them if they can host us, see https://weblate.org/en/
> > hosting/ and the part about "Hosting for free software”.
> > ** We could also ask XWiki SAS if they’d accept paying a “Basic” plan
> (200
> > euros/year which is affordable IMO vs hosting it ourselves)
> > ** There’s a REST API so that if we want we can provide a view/status of
> > the translations from xwiki.org: https://docs.weblate.org/en/
> > latest/api.html . We could even imagine using this API to convert from a
> > format to our own format, if need be.
> > * Propose to have a developer work on this in an upcoming roadmap (to be
> > defined but as early as possible since l10n is not in good shape)
> > * Fix l10n as much as we can without spending too much time on it, until
> > we have the new translation service ready to be used
> >
> > Things to look at:
> > * Ability to register custom formats or more generally how to handle our
> > custom format
> > * How do we handle deprecated translations keys
> > * How do we handle global rename/move of keys from one resource file to
> > another
> > * <add more here>
> >
> > WDYT?
> >
> > I’d like to have especially Thomas’s POV since he’s the one who spent the
> > most time on l10n and I’d like to make sure he’s ok with this.
> >
> > Thanks
> > -Vincent
> >
> >
> >
> >
> >
>



--
Guillaume Delhumeau ([hidden email])
Research & Development Engineer at XWiki SAS
Committer on the XWiki.org project
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Invest in a new translation service

Eduard Moraru
On Mon, Mar 26, 2018 at 1:11 PM, Guillaume Delhumeau <
[hidden email]> wrote:

> 2018-03-26 12:05 GMT+02:00 Eduard Moraru <[hidden email]>:
>
> > Hi,
> >
> > I am a bit skeptical about such a platform that:
> > * we'd need our users to register to (yet another account for the XWiki
> > contributor)
> >
>
> Maybe we could use an LDAP/SSO connector, just as we do for the Forum.
>
>
> > * we'd need to twist and turn so that it bends to our needs
> > * (and probably other things caused by the fact that I did not get to
> read
> > too much about it yet :) )
> >
> > You mentioning PRs made me wonder if we couldn't simply ask contributors
> to
> > submit a standard PR for translation changes. That would take care of
> many
> > things and also give the author proper attributions.
> >
> > The minimum requirements for this would be, IMO:
> > * the list of all translation files (i.e. links on GitHub; either
> manually
> > managed, automatically extracted or a mix) and
> > * the ability to search for a key or translation and find the relevant
> file
> > (most likely this would involve us indexing or caching the translation
> > files content somewhere on xwiki.org, kept up to date with webhooks? --
> > maybe on a very light translations dedicated wiki; a light l10n?).
> > * the ability to create a new language for a translation page, if it does
> > not already exist (to help the edge case when the language file is not
> > already there).
> >
> > Something like this could even be done without an xwiki.org account, as
> > long as the user has a github account for editing (even with GitHub's web
> > edit UI) and submitting the PR.
> >
>
> You just said it was a problem to use an other account that the xwiki.org
> account :) You assume most people have a Github account, but I'm not sure
> concerning translators.
>

Yes, I did realize that :), but I did assume that the github account has
more value and has a higher chance of already existing for the contributor
(code or translations contributor), than some weblate service account.

Even comparing xwiki.org ot github accounts, there is still a higher chance
that some contributor has the github account already in place :)

Thanks,
Eduard

>
>
> >
> > Most importantly, IMO, the translations could become part of the
> > development process and not just a step at release time. A clear
> > consequence would be that snapshots would benefit from new and integrated
> > translations, added since the last release and we won't be seeing them
> just
> > after the release. AFAIU, weblate supports this as well, but by using its
> > own github account and submitting PRs in the name of the author (which
> > would not be the same effect as the actual author, but then again, only
> > actual GitHub users would care about this).
> >
>
> > WDYT? Would it be too hard to use for non-techical users? Would we need
> > more that the described, in terms of UI (and would it justify the
> > effort/cost)?
> >
> > Thanks,
> > Eduard
> >
> >
> > On Fri, Mar 23, 2018 at 11:56 AM, Vincent Massol <[hidden email]>
> > wrote:
> >
> > > Hi devs,
> > >
> > > We started discussing this yesterday on the xwiki chat.
> > >
> > > Here’s the rationale for changing:
> > >
> > > * l10n is slow and is preventing users to contribute translations (it’s
> > > much harder than it should be)
> > > * it’s costing us maintenance that we shouldn’t have to do (it’s not
> our
> > > role to maintain a translation service, even though it was nice to eat
> > our
> > > own dog food initially). I see this very similar to when we switched
> from
> > > our GWT-based WYSIWYG editor to CKEditor.
> > > * It’ll allow us to benefit from new features/bug fixes the external
> > > service develops
> > > * Right now it’s taking an hour or more every time we release to
> > integrate
> > > the translation changes and this slows us down to increase our release
> > > delivery speed
> > > * We’re putting the onus on the RM only to validate that the proposed
> > > translations are good. So only 1 pair of eyes.
> > > * We have no time to fix any translation if they’re not correct. A
> system
> > > where when someone proposes a new translation generates a PR that we
> > apply
> > > as they come in would be much better and solve both the review and
> > lateness
> > > issues.
> > >
> > > Of course it’ll mean some plugins/customizations to develop in the
> > service
> > > we will use since it’s not going to support some custom formats we have
> > > such as our XAR XML format. So we need to pick a tool that allows for
> > this.
> > >
> > > The proposal:
> > > * Start investing in implementing XWiki translation using an external
> > tool.
> > > * Start  by looking at weblate: https://weblate.org/en/ since
> > > ** it seems to offer lots of features we need (automatic PR - see
> > > https://docs.weblate.org/en/latest/vcs.html#github, quality checks,
> > > plugins). See https://weblate.org/en/features/
> > > ** it’s open source itself and in case the service goes down we can ask
> > > XWiki SAS to host it for us (see https://github.com/WeblateOrg/weblate
> ).
> > > ** We need to ask them if they can host us, see
> https://weblate.org/en/
> > > hosting/ and the part about "Hosting for free software”.
> > > ** We could also ask XWiki SAS if they’d accept paying a “Basic” plan
> > (200
> > > euros/year which is affordable IMO vs hosting it ourselves)
> > > ** There’s a REST API so that if we want we can provide a view/status
> of
> > > the translations from xwiki.org: https://docs.weblate.org/en/
> > > latest/api.html . We could even imagine using this API to convert from
> a
> > > format to our own format, if need be.
> > > * Propose to have a developer work on this in an upcoming roadmap (to
> be
> > > defined but as early as possible since l10n is not in good shape)
> > > * Fix l10n as much as we can without spending too much time on it,
> until
> > > we have the new translation service ready to be used
> > >
> > > Things to look at:
> > > * Ability to register custom formats or more generally how to handle
> our
> > > custom format
> > > * How do we handle deprecated translations keys
> > > * How do we handle global rename/move of keys from one resource file to
> > > another
> > > * <add more here>
> > >
> > > WDYT?
> > >
> > > I’d like to have especially Thomas’s POV since he’s the one who spent
> the
> > > most time on l10n and I’d like to make sure he’s ok with this.
> > >
> > > Thanks
> > > -Vincent
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> Guillaume Delhumeau ([hidden email])
> Research & Development Engineer at XWiki SAS
> Committer on the XWiki.org project
>