https://gist.github.com/1685724 is a diff showing all the places we currently have non-ASCII msgids (as of current mb_server.pot used on transifex as of this writing)

Assigning to Oliver since he told me to in order to solicit feedback.

Note: this may in fact be two bugs; we should probably be using US-ASCII for msgids for the sake of ease of use for coders, regardless of the solution to the issue where non-US-ASCII msgids don't get translated by Locale::gettext_pp.

Things shouldn't be breaking, working around that is not a solution. I don't mind moving to US-ASCII for msgids, that seems logical, but accidentally copying a certain character in should not break things. I don't like the sound of munging $ENV to change the language - that sounds wrong, so I'd look into a different approach there.

All available i18n libraries for perl have this or other problems. It really seems like we have exactly 0 good options.

I'll defer to pronik's rant on the topic: http://rassie.org/archives/247

http://stackoverflow.com/questions/4441399/perl-utf-8-problems-with-localemaketext is also an interesting read, linking at one point to https://metacpan.org/module/Locale::Maketext::Gettext which points out how Locale::Maketext, by default, isn't multibyte safe (even for translations!). Falling back to Gettext message catalogs avoids this, but means we have to use Maketext's format, which has problems (see pronik's post) that make Gettext a better idea.

Gettext messes with the environment (either way – nl_putenv for gettext_xs (which seems to have other lurking problems) or playing with $ENV for gettext_pp, which can't do non-ascii msgids), which makes a problem with anything multithreaded (as noted in a comment on pronik's rant).

Clearly we need to write Locale::Maketext::Gettext::Maketext::Gettext::Maketext::Gettext, that passes our strings back and forth between the two as many times as possible in order to avoid all the bugs in both implementations.

We may be able to circumvent the environment problem by using dcgettext and dcngettext instead of dgettext and dngettext, ref: http://search.cpan.org/dist/libintl-perl/lib/Locale/Messages.pm

Okay, so, found a solution: Locale::Util has a web_set_locale (the web_ is a bit deceptive – it just takes a list of languages and a character set) that seems to work correctly using gettext_xs, that doesn't explode with non-ASCII msgids.

I'll submit a patch once http://codereview.musicbrainz.org/r/1941/ is done.

This is in beta testing with the origin/split-domains branch (one merge conflict addressed since Ian's branch).