Discussion:
Proposal: Transform blocks
Micah Stetson
2010-02-17 19:02:32 UTC
Permalink
I have mostly implemented an idea in the PHP version of json-template
that I think is generally useful. I am using it for i18n, but it's
more widely applicable than that. "On Design Minimalism" suggests
that json-template is a functional programming language. If that's
so, then what I'm proposing is analogous to Lisp's macros. Here's
what I'm doing:

Parsing a json-template program produces a JSON (or language
equivalent) value as follows:

A program is a statement or an array of statements.
A formatter list is an array of strings.
A statement is either a string or an object in one of the following
forms:
{"type":"section", "name":<string>, "contents":<program>,
"or":<program>}
{"type":"repeated section", "name":<string>,
"contents":<program>, "or": <program>, "alternates_with":<program>}
{"type":"substitution", "name":<string>, "formatters":<formatter
list>}
{"type":"trans", "formatters":<formatter list>, "contents":<program>}

Type trans, short for transformation, is for a new block type:
{.trans(|formatter)*}...{.end}, which I'll talk more about in a
moment. So this template:

Members present: {.repeated section members}{@|html}{.alternates
with}, {.or}None{.end}

Would be parsed into this value:

["Members present: ",
{"type":"repeated section","name":"members",
"contents":[{"type":"substitution","name":"@","formatters":["html"]}]
"or":["None"],
"alternates_with":[", "]}]

Evaluation of these structures is done as you might expect: an array
of statements evaluates to the concatenation of the evaluation of each
element; a string evaluates to itself; section, repeated section, and
substitution evaluate according to the current rules. Trans, however,
passes the parsed representation of its contents to a formatter
pipeline, and then evaluates the result. That's the key: evaluation
after processing.

In addition, the json-template module makes a few new functions
public: one which evaluates a program in a given scoped context, one
that parses json-template source code into a program, and one that
converts a program into equivalent json-template source code. An
additional formatter is installed by default named 'eval' that
evaluates its input as a program in the current scoped context -- by
the definitions above, this formatter is a no-op on strings. Also,
the parser accepts a new option, default-trans, specifying the
formatter to use in transformations, when none is specified.

This mechanism isn't complicated to implement, and it enables a lot of
new function. First, translation: write a formatter that converts its
program input into json-template source code, uses that source code as
a key for lookup in a translation table, then parses and returns the
translated value as the program to be evaluated. Set this as the
default transform, and {.trans}My name is {name}{.end} will translate
the string before evaluating the substitutions. Since formatters can
take arguments, you could make a variant of the translate formatter
called plural that looks up a number in context and translates a
string using the appropriate plural form for that value: {.trans|
plural count}There are {count} items.{.end} would be translated to
"There is one item." if count==1.

Using eval, simple formatters can be used as transforms as well:
{.trans|eval|html}a<b && c>d{.end} would become "a&lt;b &amp;&amp;
c&gt;d".

A formatter that implements a simple expression language: {.trans|eval|
expr}{quantity}*{cost}{.end}

True inclusion (for good or evil): {.trans|include}path/to/
template.jsont{.end}

If you like the idea, I can polish up my code and submit it for
review.

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Martijn Faassen
2010-02-18 14:47:14 UTC
Permalink
Hi there,

I'm not sure I follow your description, but I have a question about {.trans}.

Could this be used to modify the behavior of sections and repeated
sections to include an extra bit of text in the rendered template? I
have a use case there it's useful to include a hidden <div> into the
rendered template that specifies which section (and which index, in
case of repeated section) we're currently in. A javascript program can
then use this to do lookups in the underlying JSON structure that are
correct for particular elements in the DOM. In effect, it supports the
existence of values in the JSON that may not actually be rendered to
HTML directly, but can still be looked up for that element from
Javascript.

Regards,

Martijn
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Micah Stetson
2010-02-18 17:35:23 UTC
Permalink
Post by Martijn Faassen
Could this be used to modify the behavior of sections and repeated
sections to include an extra bit of text in the rendered template? I
have a use case there it's useful to include a hidden <div> into the
rendered template that specifies which section (and which index, in
case of repeated section) we're currently in.
Yes, you could write a filter function, maybe called divmaker, that
walked the JSON structure of the parsed template looking for sections
and repeated sections and inserted your <div>s as it went. You'd have
to place the whole section that needed processing inside {.trans} like
this:

{.trans|divmaker}
{.section foo}
blah blah
{.repeated section bar}
blah blah
{.end}
{.end}
{.end}

And the divmaker filter would do the work.

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Martijn Faassen
2010-02-18 17:43:29 UTC
Permalink
Hi there,
Post by Micah Stetson
Yes, you could write a filter function, maybe called divmaker, that
walked the JSON structure of the parsed template looking for sections
and repeated sections and inserted your <div>s as it went.  You'd have
to place the whole section that needed processing inside {.trans} like
{.trans|divmaker}
 {.section foo}
   blah blah
   {.repeated section bar}
     blah blah
   {.end}
 {.end}
{.end}
And the divmaker filter would do the work.
Cool. One issue is that the divmaker needs information about the state
of processing. Accessing the particular section names shouldn't be too
hard, but in repeated sections, I'd need the index of the current
iteration. The goal is to construct a path along the lines of this:

foo/bar[10]

that's section foo, section bar, entry 10

and the index would differ for each repetition of 'bar'.

Would the divmaker have access to the index information? If I recall
correctly, json-template doesn't expose this.

I'm so interested as I have a patched version of the javascript
json-template right now that does something like this, but this would
be a cleaner solution. It's also be an argument for .trans. :)

Regards,

Martijn
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-18 18:21:00 UTC
Permalink
Does this solve the same problem as formatters on sections? I think
it looks similar:

http://code.google.com/p/json-template/issues/detail?id=36

See the group thread. The basic idea is that sections have
formatters, and then they are applied after the body of the section is
expanded.

The basic idea is that you can have little dialects in the template:

{.section paragraph1|markdown}
Use *markdown* with {var}
{.end}

{.section code|source-code-highlight}
for (var i=0,...) {}
{.end}


{.section figure|dot}
digraph {
// dot syntax
}
{.end}

Andy
Post by Micah Stetson
I have mostly implemented an idea in the PHP version of json-template
that I think is generally useful.  I am using it for i18n, but it's
more widely applicable than that.  "On Design Minimalism" suggests
that json-template is a functional programming language.  If that's
so, then what I'm proposing is analogous to Lisp's macros.  Here's
Parsing a json-template program produces a JSON (or language
A program is a statement or an array of statements.
A formatter list is an array of strings.
A statement is either a string or an object in one of the following
       {"type":"section", "name":<string>, "contents":<program>,
"or":<program>}
       {"type":"repeated section", "name":<string>,
               "contents":<program>, "or": <program>, "alternates_with":<program>}
       {"type":"substitution", "name":<string>, "formatters":<formatter
list>}
       {"type":"trans", "formatters":<formatter list>, "contents":<program>}
{.trans(|formatter)*}...{.end}, which I'll talk more about in a
with}, {.or}None{.end}
["Members present: ",
{"type":"repeated section","name":"members",
       "or":["None"],
       "alternates_with":[", "]}]
Evaluation of these structures is done as you might expect: an array
of statements evaluates to the concatenation of the evaluation of each
element; a string evaluates to itself; section, repeated section, and
substitution evaluate according to the current rules.  Trans, however,
passes the parsed representation of its contents to a formatter
pipeline, and then evaluates the result.  That's the key: evaluation
after processing.
In addition, the json-template module makes a few new functions
public: one which evaluates a program in a given scoped context, one
that parses json-template source code into a program, and one that
converts a program into equivalent json-template source code.  An
additional formatter is installed by default named 'eval' that
evaluates its input as a program in the current scoped context -- by
the definitions above, this formatter is a no-op on strings.  Also,
the parser accepts a new option, default-trans, specifying the
formatter to use in transformations, when none is specified.
This mechanism isn't complicated to implement, and it enables a lot of
new function.  First, translation: write a formatter that converts its
program input into json-template source code, uses that source code as
a key for lookup in a translation table, then parses and returns the
translated value as the program to be evaluated.  Set this as the
default transform, and {.trans}My name is {name}{.end} will translate
the string before evaluating the substitutions.  Since formatters can
take arguments, you could make a variant of the translate formatter
called plural that looks up a number in context and translates a
string using the appropriate plural form for that value: {.trans|
plural count}There are {count} items.{.end} would be translated to
"There is one item." if count==1.
{.trans|eval|html}a<b && c>d{.end} would become "a&lt;b &amp;&amp;
A formatter that implements a simple expression language: {.trans|eval|
expr}{quantity}*{cost}{.end}
True inclusion (for good or evil): {.trans|include}path/to/
template.jsont{.end}
If you like the idea, I can polish up my code and submit it for
review.
Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Martijn Faassen
2010-02-18 19:00:23 UTC
Permalink
Post by Andy Chu
Does this solve the same problem as formatters on sections?
I'm trying to see how formatters on sections would solve *my* problem,
where I want to insert information about where in the JSON structure
the output comes from in the output.

I can see how a translation action could modify the template tree to
include these (granting access to indexes of repeated sections
somehow). I'm not sure whether section formatters can do this.

Could this be made to work with a section formatter?

{.section foo|insert-path-comments}
<div>
{.repeated section bar}
<div>{qux}</div>
{.end}
</ul>
{.end}

where the output could be something like:

<!-- foo -->
<div>
<!-- foo/bar[0] -->
<div>One</div>
<!-- foo/bar[1] -->
<div>Two</div>
</div>

Regards,

Martijn
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Micah Stetson
2010-02-18 22:17:23 UTC
Permalink
Does this solve the same problem as formatters on sections?  I think
http://code.google.com/p/json-template/issues/detail?id=36
See the group thread.  The basic idea is that sections have
formatters, and then they are applied after the body of the section is
expanded.
The key difference is that with {.trans} you can choose whether the
formatter is applied before or after expansion. Formatters on
sections can't support the translation system because 'Hello, {name}'
needs to be translated, rather than 'Hello, Andy'. Martijn's problem
is similar:

He wants this

{.trans|insert-path-comments}
{.section foo}
<div>
{.repeated section bar}
<div>{qux}</div>
{.end}
</div>
{.end}
{.end}

to be translated to this

{.section foo}
<!--foo-->
<div>
{.repeated section bar}
<!--foo/bar[{@index}]-->
<div>{qux}</div>
{.end}
</div>
{.end}

before expansion. If a formatter can process the parsed template
structure, this is pretty easy, but I don't see a way to do it with
post-processing.

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-19 08:20:04 UTC
Permalink
OK I see what you guys are getting at. You're right it's like
metaprogramming (macros).

So you would want to standardize a JSON parse tree for JSON Template?

On the one hand, that makes future evolution of the language
difficult. This is exposing all if its guts. Also, it makes the
templates harder to understand for non-programmers, which is an
important goal.

On the other hand, it's not that far from a JSON structure already.
Also, the JSON boundary is a natural one for compilation to native JS
code (or any other language). I posted a message recently about how
JSON Template can be used to format JavaScript parse trees in JSON
format (from Narcissus).

What are you going to use this for? For localization? (i.e. english
-> spanish)

For Martin's use case, there is @index already. And I proposed @name
and @value I think (in the bug list). That goes partway I think.
What is the real use case behind it?

thanks,
Andy
Post by Micah Stetson
Does this solve the same problem as formatters on sections?  I think
http://code.google.com/p/json-template/issues/detail?id=36
See the group thread.  The basic idea is that sections have
formatters, and then they are applied after the body of the section is
expanded.
The key difference is that with {.trans} you can choose whether the
formatter is applied before or after expansion.  Formatters on
sections can't support the translation system because 'Hello, {name}'
needs to be translated, rather than 'Hello, Andy'.  Martijn's problem
He wants this
{.trans|insert-path-comments}
 {.section foo}
   <div>
   {.repeated section bar}
     <div>{qux}</div>
   {.end}
   </div>
 {.end}
{.end}
to be translated to this
{.section foo}
 <!--foo-->
 <div>
 {.repeated section bar}
   <div>{qux}</div>
 {.end}
 </div>
{.end}
before expansion.  If a formatter can process the parsed template
structure, this is pretty easy, but I don't see a way to do it with
post-processing.
Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Martijn Faassen
2010-02-19 13:25:59 UTC
Permalink
Hi there,
Post by Andy Chu
What is the real use case behind it?
Ah, I didn't realize this existed, I thought it didn't exist at some point.

Anyway, we had long discussions about my use cases last year, I'm not
going to go into them again.

Wrapping my use case in a block makes some of the issues go away: it's
more controllable where the transformation happens, and it's clear
that a transformation happens.

Regards,

Martijn
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Micah Stetson
2010-02-20 20:02:59 UTC
Permalink
Post by Andy Chu
What are you going to use this for? For localization? (i.e. english
-> spanish)
Yes, this is my real use case. All I really need is a way to make
"{.trans}{color} suspenders{.end}" become "bretelles {color}" before
substituting "rouges" for {color}, and a way to make "{.trans|plural
count}{count} people{.end}" become "{count} person" before
substituting 1 for {count}.

I'm of the opinion that if making strings translatable requires more
than a very small amount of effort, people will say, "I'll fix it up
later after everything's working." Later often doesn't come. All of
the ways of doing translation that I can think of with the current
syntax involve writing the strings to be translated in some separate
location, outside the flow of the main templates. That crosses the
"I'll do it later" threshold.
Post by Andy Chu
On the one hand, that makes future evolution of the language
difficult.  This is exposing all if its guts.
Language evolution could break formatter functions that walk the
parsed data structure. But I think those functions will be the
exception, rather than the rule. For my use case, it's enough to have
a public function that converts an opaque parsed structure back into
source code -- the translation formatters need know nothing about the
structure.

If it was needed, one way to allow syntax-walking formatters to be
more future proof would be to go even more lispy with the data
structure. Say a statement list is an array that starts with an
identifying string or object followed by any number of statements and
statement lists. Then

{.section foo}{@|html|js}{.or}No Foo{.end}

would be parsed as

[{"type":"section","name":"foo"},
["contents",{"type":"substitution","name":"@","formatters":["html","js"]}],
["or", "No Foo"]]

Given a rule that no object can contain a statement list, a
syntax-walking formatter could easily walk the tree, leaving
unrecognized objects alone. Such a formatter would likely survive
pretty serious language changes.
Post by Andy Chu
Also, it makes the
templates harder to understand for non-programmers, which is an
important goal.
{.trans} does enable creation of hard-to-understand template
sublanguages, but you don't have to use it that way. My translation
use case above is not complicated. This is a feature that could be
misused, but it need not be.

In any event, you're right that we should focus on real-world use, and
what I'm really looking for is a minimal, easy-to-use translation
system for templates. If there's a simpler, or less dangerous way to
get a translation syntax close to what I've suggested above, I'm
interested.

I do like this idea better than section formatters, though, simply
because of the syntax. {.section foo|html} is hard to read as
{(.section foo)|html} and easy to read as {.section (foo|html)}. I
much prefer {.section foo}{.trans|html}...{.end}{.end}

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-23 05:21:58 UTC
Permalink
What are you going to use this for?  For localization?  (i.e. english
-> spanish)
Yes, this is my real use case.  All I really need is a way to make
"{.trans}{color} suspenders{.end}" become "bretelles {color}" before
substituting "rouges" for {color}, and a way to make "{.trans|plural
count}{count} people{.end}" become "{count} person" before
substituting 1 for {count}.
The person/people thing is already addressed by predicates. Do you
know about those? I've been delinquent in documenting them.

I do want a solution for principled localization. In my experience, a
lot of translation is done by simply duplicating templates, which is
nasty.

I think this is the kind of solution you say crosses the "do it later"
threshold, although I'm not sure I agree: You simply separate all
strings into JSON files, and then use 2 template expansions.

english.json:
{
"suspender_desc": "{color} suspenders"
}

spanish.json:
{
"suspender_desc": "bretelles {color}"
}

And then you substitute "suspender_desc" into your template (at
"build" time perhaps), and then do an expansion with JSON data from
the application (at runtime).

While I can see that this breaks the flow, isn't there an issue with
your solution, in that if you change the thing in {trans} -- "{color}
suspenders" -- it will break the translation? How do you associate
the 2 strings together?

I feel like people would just change the main template and never
bother to test the translated ones. Also, when you send files out to
a translator, I think they would much rather work with JSON files than
templates with embedded markup.
Language evolution could break formatter functions that walk the
parsed data structure.  But I think those functions will be the
exception, rather than the rule.  For my use case, it's enough to have
a public function that converts an opaque parsed structure back into
source code -- the translation formatters need know nothing about the
structure.
Well, I think your proposal exposes the parse tree unavoidably, even
if you wouldn't use it for a particular case.
In any event, you're right that we should focus on real-world use, and
what I'm really looking for is a minimal, easy-to-use translation
system for templates.  If there's a simpler, or less dangerous way to
get a translation syntax close to what I've suggested above, I'm
interested.
I'm interested too in seeing what other template systems do. Have you
translated a web app before? I have been tangentially involved in one
site and all they did was duplicate templates. That is of course
really constraining.
I do like this idea better than section formatters, though, simply
because of the syntax.  {.section foo|html} is hard to read as
{(.section foo)|html} and easy to read as {.section (foo|html)}.  I
much prefer {.section foo}{.trans|html}...{.end}{.end}
I think others have pointed it out. But you can always just add
spaces: {.section foo | html}.

thanks,
Andy
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Martijn Faassen
2010-02-23 17:06:00 UTC
Permalink
Hi there,
I'm interested too in seeing what other template systems do.  Have you
translated a web app before?  I have been tangentially involved in one
site and all they did was duplicate templates.  That is of course
really constraining.
I have some experience with it, but it's been a few years so I'm
rusty. In Zope Page Templates, you mark translatable strings with an
i18n:translate attribute (it's an XML-based templating language, not a
text-base one). You can give translatable bits an explicit name, or
you can let the name be derived from the text to translate
automatically. Then the info is extracted into po files and such.

You can also specify i18n:domain for the domain.

One interesting bit is i18n:name:

Name the content of the current element for use in interpolation
within translated content. This allows a replaceable component in
content to be re-ordered by translation. For example:

<span i18n:translate=''>
<span tal:replace='here/name' i18n:name='name' /> was born in
<span tal:replace='here/country_of_birth' i18n:name='country' />.
</span>

would cause this text to be passed to the translation service:

"${name} was born in ${country}."

Regards,

Martijn
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-24 03:46:40 UTC
Permalink
Post by Martijn Faassen
Hi there,
I'm interested too in seeing what other template systems do.  Have you
translated a web app before?  I have been tangentially involved in one
site and all they did was duplicate templates.  That is of course
really constraining.
I have some experience with it, but it's been a few years so I'm
rusty. In Zope Page Templates, you mark translatable strings with an
i18n:translate attribute (it's an XML-based templating language, not a
text-base one). You can give translatable bits an explicit name, or
you can let the name be derived from the text to translate
automatically. Then the info is extracted into po files and such.
You can also specify i18n:domain for the domain.
Name the content of the current element for use in interpolation
within translated content. This allows a replaceable component in
   <span i18n:translate=''>
     <span tal:replace='here/name' i18n:name='name' /> was born in
     <span tal:replace='here/country_of_birth' i18n:name='country' />.
   </span>
   "${name} was born in ${country}."
I wouldn't be opposed to something simple like that. You can just
have some kind of delimiter like {.translate} {.end} and then the text
can be extracted trivially. I think you generally want to generate
templates from templates at build time. I don't think it makes sense
to mix it all up in the same runtime engine.

I guess the difference with {.trans} is that you're passing text
rather than some kind of parse tree. That seems simpler. Tokenizing
a JSON Template string is trivial by design -- a single re.split()
call.

Andy
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Micah Stetson
2010-02-26 06:43:56 UTC
Permalink
I think I understand what you're suggesting, but let me run through an
example to be sure.

There are two parts to the translation system. The first part
processes template source looking for {.translate}{.end} pairs and
returning a list of strings that occur between these tokens. This
list can be used to create or update a language-specific translation
map, mapping the primary text in the templates to translated text.
The second part of the system takes template source and a translation
map and returns template source with the translation strings replaced
as specified in the map. These parts can be thought of as functions
in some sort of pseudo-code:

get_translations(template source) -> list of string
do_translations(template source, map from string to string) ->
template source

So let's say I have a template like this:

{.translate}Hello, {name}{.end}
{.translate}{count} files{.end}.

Passing this to get_translations would return ["Hello,
{name}","{count} files"]. My Polish translator would then produce a
translation map like this:

{
"Hello, {name}": "Cześć, {name}",
"{count} files": "{.section count}{count} {.1?}plik{.or
pl_234?}pliki{.or}plików{.end}{.end}"
}

Note the three plural forms. Several languages have rules more
complicated than, "Is the number equal to 1?". Your predicate system
works pretty well for that. The pl_234? predicate might be specific
to Polish and would return true for every number ending in 2, 3, or 4,
except those ending 12, 13, or 14. Passing this map and the original
template to do_translations would give me a new template like this:

Cześć, {name}
{.section count}{count} {.1?}plik{.or pl_234?}pliki{.or}plików{.end}{.end}

I would then pass that template to JSON-template to get the final output:

Cześć, Andy
12 plików

That's not bad. Would you want a translation scheme like this be
maintained as part of JSON Template, or should it be kept separate?

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-27 04:42:21 UTC
Permalink
Post by Micah Stetson
I think I understand what you're suggesting, but let me run through an
example to be sure.
There are two parts to the translation system.  The first part
processes template source looking for {.translate}{.end} pairs and
returning a list of strings that occur between these tokens.  This
list can be used to create or update a language-specific translation
map, mapping the primary text in the templates to translated text.
The second part of the system takes template source and a translation
map and returns template source with the translation strings replaced
as specified in the map.  These parts can be thought of as functions
Yes. It's basically a couple simple tools to process templates, which
was always an intended use case.
Post by Micah Stetson
   get_translations(template source) -> list of string
   do_translations(template source, map from string to string) ->
template source
   {.translate}Hello, {name}{.end}
   {.translate}{count} files{.end}.
Passing this to get_translations would return ["Hello,
{name}","{count} files"].  My Polish translator would then produce a
Yeah, but I also wonder if it makes sense to have an ID:

{.trans welcome}Hello, {name}{.end}

and then you get:

{"welcome": "Hello, {name}"}

This is addressing the issue where you translate English to Polish,
but then you change the English text again. Also then the strings can
be reordered in the template and you won't get a different order in
the output the file.

Also, I wouldn't want {.end} to be the same thing, because translation
is like a preprocessing step rather than a construct like {.section}.

So at first I thought of:

{!trans}Hello, {name}{!end}

But then this is even better:

{#trans}Hello, {name}{#end}

Because it is already valid, and I have an obsession with parsimony.
It's just using the comment syntax, and # also reminds of preprocessor
directives in C.
Post by Micah Stetson
   {
   "Hello, {name}": "Cześć, {name}",
   "{count} files": "{.section count}{count} {.1?}plik{.or
pl_234?}pliki{.or}plików{.end}{.end}"
   }
That would work, or as mentioned you could send him english.json with
IDs, and he could send you back polish.json with IDs and no English.
Not sure which one is better. I suppose that it's possible to
implement both as it doesn't change the language.
Post by Micah Stetson
Note the three plural forms.  Several languages have rules more
complicated than, "Is the number equal to 1?". Your predicate system
works pretty well for that.  The pl_234? predicate might be specific
to Polish and would return true for every number ending in 2, 3, or 4,
except those ending 12, 13, or 14.  Passing this map and the original
   Cześć, {name}
   {.section count}{count} {.1?}plik{.or pl_234?}pliki{.or}plików{.end}{.end}
Right, and I think this would also work if the English has 2 clauses
for plurality, while Polish has 3, right? The translator just has to
know a little but about the syntax, and with the predicates you can
make it more straightforward for them.
Post by Micah Stetson
   Cześć, Andy
   12 plików
That's not bad.  Would you want a translation scheme like this be
maintained as part of JSON Template, or should it be kept separate?
I think for now it seems like it can be done separately, but if it
proves popular we can fold it in. I don't imagine that the tools are
going to be more than 100-200 lines. It's basically:

re.split() to tokenize
use a stack to match {#trans} and {#end} (I suppose you will disallow
nested {#trans}
And then export a JSON file from those strings

The opposite tool is:

re.split()
Output tokens literally, except whenever you hit {#trans}, look up the
next token in the translation map

It can probably just be a single Python file for both tools. I like
it, it is nice and simple.

thanks,
Andy
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Micah Stetson
2010-02-27 08:42:30 UTC
Permalink
Yes.  It's basically a couple simple tools to process templates, which
was always an intended use case.
I'm happy with this.
{.trans welcome}Hello, {name}{.end}
{"welcome": "Hello, {name}"}
This is addressing the issue where you translate English to Polish,
but then you change the English text again.  Also then the strings can
be reordered in the template and you won't get a different order in
the output the file.
If the meaning of the English text changes, then the Polish text
probably needs to change as well. For minor edits, I would tend to
use an English translation file:

{"Hello, {name}": "Hi, {name}"}

I like not making the template author ask whether he's come up with a
unique text id, and I like keeping the original and translated text
side-by-side so that it's easier for somebody to double-check
translation quality.
{#trans}Hello, {name}{#end}
That looks good to me.
Post by Micah Stetson
   Cześć, {name}
   {.section count}{count} {.1?}plik{.or pl_234?}pliki{.or}plików{.end}{.end}
Right, and I think this would also work if the English has 2 clauses
for plurality, while Polish has 3, right?  The translator just has to
know a little but about the syntax, and with the predicates you can
make it more straightforward for them.
Exactly.
I think for now it seems like it can be done separately, but if it
proves popular we can fold it in.  I don't imagine that the tools are
Sounds good. I'll implement a PHP version for OpenBiblio and start using it.

Micah
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Andy Chu
2010-02-27 18:16:21 UTC
Permalink
Post by Micah Stetson
Yes.  It's basically a couple simple tools to process templates, which
was always an intended use case.
I'm happy with this.
{.trans welcome}Hello, {name}{.end}
{"welcome": "Hello, {name}"}
This is addressing the issue where you translate English to Polish,
but then you change the English text again.  Also then the strings can
be reordered in the template and you won't get a different order in
the output the file.
If the meaning of the English text changes, then the Polish text
probably needs to change as well.  For minor edits, I would tend to
{"Hello, {name}": "Hi, {name}"}
I like not making the template author ask whether he's come up with a
unique text id, and I like keeping the original and translated text
side-by-side so that it's easier for somebody to double-check
translation quality.
{#trans}Hello, {name}{#end}
That looks good to me.
Post by Micah Stetson
   Cześć, {name}
   {.section count}{count} {.1?}plik{.or pl_234?}pliki{.or}plików{.end}{.end}
Right, and I think this would also work if the English has 2 clauses
for plurality, while Polish has 3, right?  The translator just has to
know a little but about the syntax, and with the predicates you can
make it more straightforward for them.
Exactly.
I think for now it seems like it can be done separately, but if it
proves popular we can fold it in.  I don't imagine that the tools are
Sounds good.  I'll implement a PHP version for OpenBiblio and start using it.
Great. I should warn you that the PHP version is not as complete as
Python/JavaScript. You might want to hook up with Steven Roussey as
he's been working on it. I don't think it has predicates yet.

I didn't write it and I know very little about PHP unfortunately. But
I'm happy to answer any questions about the language spec. The tests
are meant to cover most things.

Andy
--
You received this message because you are subscribed to the Google Groups "JSON Template" group.
To post to this group, send email to json-template-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to json-template+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/json-template?hl=en.
Loading...