Harmonize bundle API, expose get_message and format_pattern as API on bundles.

This change changes a lot of tests, as there's a shift from
format() to format_pattern(get_message().value).
With the get_message API, the complex lookups format('msg.attr') also expose
an API surface that's not inline with other impls, so I removed
quite a bit of utility code that served that purpose.
And as I changed all lines referencing bundles, I named the local
variable `bundle` instead of `ctx`.

There's a bit of clean-up sneaking in to this patch, triggered by
reshuffling the referencing code. Notably, all resolvers now
consistently return fluent types. In some cases, they returned
resolvers, which required resolving to get to types.

Author: Pike

fluent.runtime/CHANGELOG.rst

@@ -7,6 +7,7 @@ fluent.runtime next
 * Added ``fluent.runtime.FluentResource`` and
   ``fluent.runtime.FluentBundle.add_resource``.
 * Removed ``fluent.runtime.FluentBundle.add_messages``.
+* Replaced ``bundle.format()`` with ``bundle.format_pattern(bundle.get_message().value)``.
 
 fluent.runtime 0.2 (September 10, 2019)
 ---------------------------------------

fluent.runtime/docs/usage.rst

@@ -45,31 +45,34 @@ file stored on disk, here we will just add them directly:
     ... welcome = Welcome to this great app!
     ... greet-by-name = Hello, { $name }!
     ... """)
-    >>> bundle.add_resource(resource
+    >>> bundle.add_resource(resource)
 
-To generate translations, use the ``format`` method, passing a message
-ID and an optional dictionary of substitution parameters. If the the
-message ID is not found, a ``LookupError`` is raised. Otherwise, as per
-the Fluent philosophy, the implementation tries hard to recover from any
-formatting errors and generate the most human readable representation of
-the value. The ``format`` method therefore returns a tuple containing
-``(translated string, errors)``, as below.
+To generate translations, use the ``get_message`` method to retrieve
+a message from the bundle. If the the message ID is not found, a
+``LookupError`` is raised. Then use the ``format_pattern`` method, passing
+the message value or one if its attributes and an optional dictionary of
+substitution parameters.  As per the Fluent philosophy, the implementation
+tries hard to recover from any formatting errors and generate the most human
+readable representation of the value. The ``format_pattern`` method therefore
+returns a tuple containing ``(translated string, errors)``, as below.
 
 .. code-block:: python
 
-    >>> translated, errs = bundle.format('welcome')
+    >>> welcome = bundle.get_message('welcome')
+    >>> translated, errs = bundle.format_pattern(welcome.value)
     >>> translated
     "Welcome to this great app!"
     >>> errs
     []
 
-    >>> translated, errs = bundle.format('greet-by-name', {'name': 'Jane'})
+    >>> greet = bundle.get_message('greet-by-name')
+    >>> translated, errs = bundle.format_pattern(greet.value, {'name': 'Jane'})
     >>> translated
     'Hello, \u2068Jane\u2069!'
 
-    >>> translated, errs = bundle.format('greet-by-name', {})
+    >>> translated, errs = bundle.format_pattern(greet.value, {})
     >>> translated
-    'Hello, \u2068name\u2069!'
+    'Hello, \u2068{$name}\u2069!'
     >>> errs
     [FluentReferenceError('Unknown external: name')]
 
@@ -105,7 +108,8 @@ When rendering translations, Fluent passes any numeric arguments (``int``,
     >>> bundle.add_resource(FluentResource(
     ... "show-total-points = You have { $points } points."
     ... ))
-    >>> val, errs = bundle.format("show-total-points", {'points': 1234567})
+    >>> total_points = bundle.get_message("show-total-points")
+    >>> val, errs = bundle.format_pattern(total_points.value, {'points': 1234567})
     >>> val
     'You have 1,234,567 points.'
 
@@ -117,14 +121,15 @@ by wrapping your numeric arguments with
 
     >>> from fluent.runtime.types import fluent_number
     >>> points = fluent_number(1234567, useGrouping=False)
-    >>> bundle.format("show-total-points", {'points': points})[0]
+    >>> val, errs = bundle.format_pattern(total_points.value, {'points': points})[0]
     'You have 1234567 points.'
 
     >>> amount = fluent_number(1234.56, style="currency", currency="USD")
     >>> bundle.add_resource(FluentResource(
     ... "your-balance = Your balance is { $amount }"
     ... ))
-    >>> bundle.format("your-balance", {'amount': amount})[0]
+    >>> balance = bundle.get_message("your-balance")
+    >>> bundle.format_pattern(balance.value, {'amount': amount})[0]
     'Your balance is $1,234.56'
 
 The options available are defined in the Fluent spec for
@@ -142,7 +147,8 @@ passed through locale aware functions:
 
     >>> from datetime import date
     >>> bundle.add_resource(FluentResource("today-is = Today is { $today }"))
-    >>> val, errs = bundle.format("today-is", {"today": date.today() })
+    >>> today_is = bundle.get_message("today-is")
+    >>> val, errs = bundle.format(today_is.value, {"today": date.today() })
     >>> val
     'Today is Jun 16, 2018'
 
@@ -171,7 +177,7 @@ To specify options from Python code, use
     >>> from fluent.runtime.types import fluent_date
     >>> today = date.today()
     >>> short_today = fluent_date(today, dateStyle='short')
-    >>> val, errs = bundle.format("today-is", {"today": short_today })
+    >>> val, errs = bundle.format_pattern(today_is, {"today": short_today })
     >>> val
     'Today is 6/17/18'
 
@@ -201,7 +207,8 @@ ways:
        datetime.datetime(2018, 6, 17, 12, 15, 5, 677597)
 
        >>> bundle.add_resource(FluentResource("now-is = Now is { $now }"))
-       >>> val, errs = bundle.format("now-is",
+       >>> now_is = bundle.get_message("now-is")
+       >>> val, errs = bundle.format_pattern(now_is.value,
        ...    {"now": fluent_date(utcnow,
        ...                        timeZone="Europe/Moscow",
        ...                        dateStyle="medium",
@@ -233,7 +240,7 @@ You can add functions to the ones available to FTL authors by passing a
     ...   *[other]    Welcome
     ...   }
     ... """))
-    >>> print(bundle.format('welcome')[0]
+    >>> print(bundle.format_pattern(bundle.get_message('welcome'))[0])
     Welcome to Linux
 
 These functions can accept positional and keyword arguments (like the

fluent.runtime/fluent/runtime/__init__.py

@@ -10,7 +10,7 @@
 from .builtins import BUILTINS
 from .prepare import Compiler
 from .resolver import ResolverEnvironment, CurrentEnvironment
-from .utils import ATTRIBUTE_SEPARATOR, TERM_SIGIL, ast_to_id, native_to_fluent
+from .utils import native_to_fluent
 
 
 def FluentResource(source):
@@ -20,12 +20,14 @@ def FluentResource(source):
 
 class FluentBundle(object):
     """
-    Message contexts are single-language stores of translations.  They are
-    responsible for parsing translation resources in the Fluent syntax and can
+    Bundles are single-language stores of translations.  They are
+    aggregate parsed Fluent resources in the Fluent syntax and can
     format translation units (entities) to strings.
 
-    Always use `FluentBundle.format` to retrieve translation units from
-    a context.  Translations can contain references to other entities or
+    Always use `FluentBundle.get_message` to retrieve translation units from
+    a bundle. Generate the localized string by using `format_pattern` on
+    `message.value` or `message.attributes['attr']`.
+    Translations can contain references to other entities or
     external arguments, conditional logic in form of select expressions, traits
     which describe their grammatical features, and can use Fluent builtins.
     See the documentation of the Fluent syntax for more information.
@@ -38,7 +40,8 @@ def __init__(self, locales, functions=None, use_isolating=True):
             _functions.update(functions)
         self._functions = _functions
         self.use_isolating = use_isolating
-        self._messages_and_terms = {}
+        self._messages = {}
+        self._terms = {}
         self._compiled = {}
         self._compiler = Compiler()
         self._babel_locale = self._get_babel_locale()
@@ -47,30 +50,33 @@ def __init__(self, locales, functions=None, use_isolating=True):
     def add_resource(self, resource, allow_overrides=False):
         # TODO - warn/error about duplicates
         for item in resource.body:
-            if isinstance(item, (Message, Term)):
-                full_id = ast_to_id(item)
-                if full_id not in self._messages_and_terms or allow_overrides:
-                    self._messages_and_terms[full_id] = item
+            if not isinstance(item, (Message, Term)):
+                continue
+            map_ = self._messages if isinstance(item, Message) else self._terms
+            full_id = item.id.name
+            if full_id not in map_ or allow_overrides:
+                map_[full_id] = item
 
     def has_message(self, message_id):
-        if message_id.startswith(TERM_SIGIL) or ATTRIBUTE_SEPARATOR in message_id:
-            return False
-        return message_id in self._messages_and_terms
-
-    def lookup(self, full_id):
-        if full_id not in self._compiled:
-            entry_id = full_id.split(ATTRIBUTE_SEPARATOR, 1)[0]
-            entry = self._messages_and_terms[entry_id]
-            compiled = self._compiler(entry)
-            if compiled.value is not None:
-                self._compiled[entry_id] = compiled.value
-            for attr in compiled.attributes:
-                self._compiled[ATTRIBUTE_SEPARATOR.join([entry_id, attr.id.name])] = attr.value
-        return self._compiled[full_id]
-
-    def format(self, message_id, args=None):
-        if message_id.startswith(TERM_SIGIL):
-            raise LookupError(message_id)
+        return message_id in self._messages
+
+    def get_message(self, message_id):
+        return self._lookup(message_id)
+
+    def _lookup(self, entry_id, term=False):
+        if term:
+            compiled_id = '-' + entry_id
+        else:
+            compiled_id = entry_id
+        try:
+            return self._compiled[compiled_id]
+        except LookupError:
+            pass
+        entry = self._terms[entry_id] if term else self._messages[entry_id]
+        self._compiled[compiled_id] = self._compiler(entry)
+        return self._compiled[compiled_id]
+
+    def format_pattern(self, pattern, args=None):
         if args is not None:
             fluent_args = {
                 argname: native_to_fluent(argvalue)
@@ -80,12 +86,11 @@ def format(self, message_id, args=None):
             fluent_args = {}
 
         errors = []
-        resolve = self.lookup(message_id)
         env = ResolverEnvironment(context=self,
                                   current=CurrentEnvironment(args=fluent_args),
                                   errors=errors)
         try:
-            result = resolve(env)
+            result = pattern(env)
         except ValueError as e:
             errors.append(e)
             result = '{???}'

fluent.runtime/fluent/runtime/resolver.py

@@ -1,8 +1,6 @@
 from __future__ import absolute_import, unicode_literals
 
 import contextlib
-from datetime import date, datetime
-from decimal import Decimal
 
 import attr
 import six
@@ -92,12 +90,28 @@ class Literal(BaseResolver):
     pass
 
 
-class Message(FTL.Message, BaseResolver):
-    pass
+class EntryResolver(BaseResolver):
+    '''Entries (Messages and Terms) have attributes.
+    In the AST they're a list, the resolver wants a dict. The helper method
+    here should be called from the constructor.
+    '''
+    def _fix_attributes(self):
+        self.attributes = {
+            attr.id.name: attr.value
+            for attr in self.attributes
+        }
 
 
-class Term(FTL.Term, BaseResolver):
-    pass
+class Message(FTL.Message, EntryResolver):
+    def __init__(self, id, **kwargs):
+        super(Message, self).__init__(id, **kwargs)
+        self._fix_attributes()
+
+
+class Term(FTL.Term, EntryResolver):
+    def __init__(self, id, value, **kwargs):
+        super(Term, self).__init__(id, value, **kwargs)
+        self._fix_attributes()
 
 
 class Pattern(FTL.Pattern, BaseResolver):
@@ -174,12 +188,26 @@ def __call__(self, env):
         return self.value
 
 
-class MessageReference(FTL.MessageReference, BaseResolver):
+class EntryReference(BaseResolver):
     def __call__(self, env):
-        return lookup_reference(self, env)(env)
+        try:
+            entry = env.context._lookup(self.id.name, term=isinstance(self, FTL.TermReference))
+            if self.attribute:
+                pattern = entry.attributes[self.attribute.name]
+            else:
+                pattern = entry.value
+            return pattern(env)
+        except LookupError:
+            ref_id = reference_to_id(self)
+            env.errors.append(unknown_reference_error_obj(ref_id))
+            return FluentNone('{{{}}}'.format(ref_id))
+
 
+class MessageReference(FTL.MessageReference, EntryReference):
+    pass
 
-class TermReference(FTL.TermReference, BaseResolver):
+
+class TermReference(FTL.TermReference, EntryReference):
     def __call__(self, env):
         if self.arguments:
             if self.arguments.positional:
@@ -189,36 +217,7 @@ def __call__(self, env):
         else:
             kwargs = None
         with env.modified_for_term_reference(args=kwargs):
-            return lookup_reference(self, env)(env)
-
-
-class FluentNoneResolver(FluentNone, BaseResolver):
-    def __call__(self, env):
-        return self.format(env.context._babel_locale)
-
-
-def lookup_reference(ref, env):
-    """
-    Given a MessageReference, TermReference or AttributeExpression, returns the
-    AST node, or FluentNone if not found, including fallback logic
-    """
-    ref_id = reference_to_id(ref)
-    try:
-        return env.context.lookup(ref_id)
-    except LookupError:
-        env.errors.append(unknown_reference_error_obj(ref_id))
-
-        if ref.attribute:
-            # Fallback
-            parent_id = reference_to_id(ref, ignore_attributes=True)
-            try:
-                return env.context.lookup(parent_id)
-            except LookupError:
-                # Don't add error here, because we already added error for the
-                # actual thing we were looking for.
-                pass
-
-    return FluentNoneResolver(ref_id)
+            return super(TermReference, self).__call__(env)
 
 
 class VariableReference(FTL.VariableReference, BaseResolver):
@@ -230,7 +229,7 @@ def __call__(self, env):
             if env.current.error_for_missing_arg:
                 env.errors.append(
                     FluentReferenceError("Unknown external: {0}".format(name)))
-            return FluentNoneResolver(name)
+            return FluentNone(name)
 
         if isinstance(arg_val, (FluentType, six.text_type)):
             return arg_val
@@ -312,7 +311,7 @@ def __call__(self, env):
             return function(*args, **kwargs)
         except Exception as e:
             env.errors.append(e)
-            return FluentNoneResolver(function_name + "()")
+            return FluentNone(function_name + "()")
 
 
 class NamedArgument(FTL.NamedArgument, BaseResolver):

fluent.runtime/fluent/runtime/utils.py

@@ -3,7 +3,7 @@
 from datetime import date, datetime
 from decimal import Decimal
 
-from fluent.syntax.ast import Term, TermReference
+from fluent.syntax.ast import TermReference
 
 from .types import FluentInt, FluentFloat, FluentDecimal, FluentDate, FluentDateTime
 from .errors import FluentReferenceError
@@ -12,15 +12,6 @@
 ATTRIBUTE_SEPARATOR = '.'
 
 
-def ast_to_id(ast):
-    """
-    Returns a string reference for a Term or Message
-    """
-    if isinstance(ast, Term):
-        return TERM_SIGIL + ast.id.name
-    return ast.id.name
-
-
 def native_to_fluent(val):
     """
     Convert a python type to a Fluent Type.
@@ -39,7 +30,7 @@ def native_to_fluent(val):
     return val
 
 
-def reference_to_id(ref, ignore_attributes=False):
+def reference_to_id(ref):
     """
     Returns a string reference for a MessageReference or TermReference
     AST node.
@@ -55,7 +46,7 @@ def reference_to_id(ref, ignore_attributes=False):
     else:
         start = ref.id.name
 
-    if not ignore_attributes and ref.attribute:
+    if ref.attribute:
         return ''.join([start, ATTRIBUTE_SEPARATOR, ref.attribute.name])
     return start