SWeTE Users Guide

SWeTE stands for Simple Website Translation Engine. It is a reverse HTTP proxy that provides internationalization of the content that it serves. It allows you to easily convert your web site (or web application) from one language into another. It also provides a full administration console with forms to translate your site content. In addition it provides some powerful developer tools to allow you to manipulate the look and feel of your site in locale-specific ways using CSS, HTML, and Javascript.

SWeTE would generally be set up on a web server and configured to proxy content from another site. For example, if we wanted to translate the site xataface.com into French, we might set up SWeTE in the directory xataface.com/fr/. Then, if we point our web browser to http://xataface.com/fr/index.html it would serve the page located at http://xataface.com/index.html except that it would be translated into French. The translation itself is performed by parsing the HTML of the requested page into individual strings, then checking a translation memory for translations of those strings.

SWeTE combines two simple, yet powerful, concepts to great effect: HTTP reverse proxying, and Translation Memories. These two concepts extrapolate into enormous feature potential as a platform, but they all boil down to a few key areas:

The preferred installation method is using git to check out the repository, and then run the setup.sh script found in the bin directory.

Once you have SWeTE installed, you can go ahead and set up a proxy website. As an example, let’s set up Wikipedia as our first translated website. We’ll create a French version of Wikipedia using SWeTE. The steps are as follows:

This chapter introduced SWeTE Server as a reverse-proxy that provides seamless content translation and localization. SWeTE will run on a fairly typical LAMP server stack (Linux, Apache, PHP, MySQL). It can be downloaded either as an archived, all-in-one distribution or via the subversion repository. The Subversion repository is recommended for keeping up-to-date with patches and bug fixes.

We listed many of the features of SWeTE in Features. The core functionality centers around the translation and serving of web content. It includes an advanced translation memory that can be used to import and export to many formats, including XLIFF, TMX, CSV, and XML.

The chapter also includes a step-by-step tutorial on:

At this point we have only scratched the surface. The remaining chapters will go through the features of SWeTE in greater depth. The initial release (0.2) is also meant to be a bare-bones starting point for development. Development is active and ongoing, and many more features are planned.

SWeTE does a pretty good job of parsing HTML content and extracting a set of phrases to be translated. In cases where the HTML markup doesn’t make structure explicit, you can add your own HTML tags and attributes to the content to help it along.

A request-response cycle in SWeTE works roughly as follows:

This chapter will focus mostly on step 3 (parsing a webpage into strings). It is possible to use SWeTE successfully without understanding much about how the parsing works, but you can achieve much better results if you know how it works, and thus, know how to manipulate it.

SWeTE’s parsing algorithm is designed to figure out how the text of a page should be partitioned into strings. By default, SWeTE will look for what it considers to be "block-level" HTML elements. It treats the text content of each "block-level" as an atomic string (phrase) for use in translation. Block-level elements include such elements as:

In fact, SWeTE considers any tag that is not an "inline" tag to be a "block-level" tag. The definitive set of "inline" tags are:

<a>, <em>, <abbr>, <i>, <u>, <b>, <span>, <strong>, <acronym>, <font>, <sup>, <sub>

In addition, you can also explicitly specify that a tag should be treated as "inline" by adding the data-swete-inline="1" HTML attribute to it.

++

Example Parse Tree.

Consider the following snippet of HTML:

This would be parsed into the following strings:

This break-down seems to follow the semantic flow of the text quite well. It keeps sentences together, and groups "inline" tags into a single string (e.g. The <a> +`and `+<em> tags in the first paragraph are just included inside the parent string rather than being partitioned into their own strings). This is largely because the HTML mark up is clean and following the semantics of the content. Well-written HTML should maintain this characteristic, but you may find exceptions where the HTML markup does not break-down well semantically.

One common HTML construct that doesn’t fare well under the default rules are menus of <a> tags. E.g., consider the following HTML menu:

Because <a> tags are inline elements, SWeTE will parse this into a single string for translation. Semantically, though, each of the <a> tags should be treated as separate.

The difference is apparent when you place this menu into a larger context. For example, consider the following page that includes menus at the head and foot that are almost the same but have a slightly different order:

When SWeTE parses this, it will produce three strings:

Notice that it extracts the menus as being distinct strings even though their content is largely the same. It would be much better if it just extracted the individual menu labels separately. In fact, this is how it would work if <a> tags were block-level. But SWeTE has no way of knowing that we intend for these particular tags to be block level, since, most of the time, <a> tags are just included in other content without disrupting the flow. Unless, that is, you give SWeTE a hint.

In some cases you may want to be very explicit about what gets translated and how a document is partitioned into strings. SWeTE supports the translate and notranslate HTML attributes for this purpose. notranslate tells SWeTE not to parse anything inside of its tag. translate specifies that the contents of the tag should be treated as a single string, even if it contains sub elements. Using these attributes in tandem provides an alternative to the SWeTE parser. Indeed, in some cases you may want to mark the <body> tag with notranslate and then just explicitly mark the sections that you with to translate within the body.

For example, consider the menu with inline <a> tags from the previous section:

As we saw before, SWeTE will parse all menu items into a single string, which is not optimal. Previously we saw a solution to this problem involving the data-swete-block attribute. You could, alternatively solve this problem using a combination of the translate and notranslate attributes as follows:

This instructs SWeTE to "not translate" anything inside the "menu" div. It then further instructs SWeTE to translate the specific individual <a> tag contents. SWeTE will parse this into the following strings:

After SWeTE parses the strings out of an HTML page, it checks the translation memory for the site to see if there are any translations for the extracted strings. It will only use a translation if the string is an exact match. However it does have a hashing algorithm for string equivalence that factors out HTML tags, and variables (inline sections that should either be translated or not depending on the type of variable). This allows for maximum translation reuse without the possibility for incorrect translations bleeding into the output.

According to SWeTE’s hashing algorithm, the following strings are equivalent:

But these are not the same as:

The reason why these last 3 are not equivalent to the first three or to each other is that they are missing key elements. The first one is missing the HTML tag, so it will be recognized as a distinct string. The second one is missing the trailing punctuation point. And the third one has a lower case "h" in "hello" rather than the upper case version in the previous three. And the last one has the span tag around "is Steve" instead of just "Steve".

This example shows the following rules of string equivalence:

The following page has 3 strings, but since they are all equivalent unders SWeTE’s string equivalence rules, it will only store the first one in the translation memory.

Hence if we capture strings on this page, then look at the "Strings" tab in SWeTE we would only see the first string pulled in:

"Hello my name is <span>Steve</span>"

If we then translate this string into French using the translation form, e.g. using the following translation:

"Bonjour mon nom est <span>Steve</span>",

Then SWeTE would translate the page as follows:

Notice that SWeTe preserved the tags. (i.e. even though the translation memory only contained a translation for the <span> tag version, it presered the <a> and <name> tags properly on translation).

In this chapter, we discussed the SWeTE translation parser and showed how it can be manipulated to partition content in the ways of our choosing. We also discussed string equivalence and introduced some strategies for minimizing the number of unique strings that are required to be held by our translation memories (e.g. using inline variables and automatic date formatting).

Using a combination of source content design (i.e. adding SWeTE meta tags directly into your source content), preprocessing, and post-processing it should be possible to efficiently manage the internationalization and translation of the most complex web application.

As of version 0.3, SWeTE now includes a REST (Representational State Transfer) web service API for translating strings. Using this API, you can pass HTML or plain text content to SWeTE over HTTP and retrieve translated versions of the content as output.

Before you can start using the REST API for a website, you’ll need to create a secret web-service key for the site in the site profile. This key will be used by your REST clients as a sort of password to access the API.

The steps are as follows:

Once you have activated the REST API for a site, you should be able to send HTTP requests to the service with strings that you wish to translate. The HTTP requests should be POST requests to a URL in the proxy site with the following POST parameters:

REST requests are processed by SWeTE in exactly the same way that regular web requests are processed. The only difference is that the content to be translated is fetched from the POST parameters instead of making a background request to the equivalent page on the source site. This means that the content provided in a REST request goes through all of the same preprocessing steps that standard content goes through. It also means that string capturing works exactly the same way. I.e. if you have "log translation misses" enabled in the site configuration, then you perform a REST API request with some content that hasn’t yet been translated, then the untranslated strings will be added to the translation miss log and appear under the "strings" tab to be translated.

Generally, SWeTE will do a pretty good job of parsing and translating website content out of the box. Sometimes, however, you may feel the need to make changes to the page content to help make it easier to deal with. Some examples of such situations include:

There are three ways to modify input content to make it more amenable to SWeTE’s preferences:

This chapter focuses on the last option: PHP Event Handlers.

SWeTE allows you to define a delegate class corresponding to each site in the SWeTE instance. This delegate class can contain the following methods:

Using these three hooks, you should be able to arbitrarily modify the input content to suit your purposes.

One of the most difficult aspects of managing a multilingual web site is managing the translation of the content. This includes such activities as:

SWeTE provides several features that can help to make these activities painless (or at least less painful than they would normally be). Some of these include:

This chapter presents some of SWeTE’s translation management features and suggests some strategies managing translations in different types of web sites and applications.

One of the most fundamental concepts in SWeTE is the translation memory. A translation memory is just a dictionary that maps strings in one language to the equivalent string in another language. A translation memory may include more than one translation for a string, but only one of those translations can be active at a time. When SWeTE processes web requests, it parses out all of the strings in the page and compares them to the site’s translation memory. If it finds any matching translations, it replaces the original string with its translation, and then outputs the page to the client.

SWeTE assigns one translation memory per website. It is possible to share the same translation memory between two websites as long as those websites have the same source and target languages. Future versions of SWeTE may add the ability to use more than one translation memory in a single site (e.g. different translation memory for different pages), but currently (as of version 0.3) you are limited to a single translation memory. This should be sufficient for most websites.

The "Strings" tab of SWeTE shows all of the strings that have been loaded into the system. This table is sometimes referred to as the "Translation Miss Log" because the primary way that strings are loaded into it is by SWeTE inserting strings for which it cannot find a translation during its web page translation cycle. Over time it became apparent that this "log" was very useful for being able to manage strings in the system, so the tab label was changed to "Strings". Some of the useful activities that can be performed in this tab include:

The translation miss log is insulated from the actual translation memories. If you delete a string from the log, it won’t have any effect on the string’s translation, which is stored inside a translation memory. In fact you can delete the entire translation miss log table (i.e. every record in the "Strings" tab), and your proxy site will still function perfectly.

There are three ways to add new "strings" and translations into SWeTE:

CSV and XLIFF importing will be covered in a later section. In this section I will focus on how the capture process works (when translating a web page).

There are 3 primary ways to translate strings in SWeTE:

If your translator prefers to use his own CAT (computer assisted translation) tool for translating site content you will need to export the strings you wish to have translated to an external file format. SWeTE allows you to export in CSV (comma-separated value) and XLIFF (XML Localisation Interchange File Format). Your choice of format will depend on the tool-chain you and your translator intend to use. CSV is handy because it is simple. It can be opened and edited by any spread-sheet program.

XLIFF is an XML-based format created to standardize the way localizable data are passed between tools during the localization process. The problem with XLIFF is that it isn’t yet fully supported by many of the most popular CAT tools. SWeTE uses XLIFF standard placeholder tags in place of HTML tags that may be embedded in its strings and translations. These placeholders need to be maintained throughout the translation process if you want to be able to import the resulting translations back into SWeTE and have them "work" properly. If your CAT tool only supports XLIFF as an import source and export target, then these tags are likely to get lost in translation (pardon the pun). Even if the CAT tool is advertised to work with XLIFF natively it may not support the full specification, and thus may strip out these special tags.

In order to provide a workflow between external CAT tools and SWeTE, you must be able to re-import the files that had previously been exported - but have had translations added to them. SWeTE supports importing of both CSV and XLIFF files as long as they are in the same format as SWeTE produces in its export operations.