freemarker-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Woonsan Ko <>
Subject Re: [FM3] Solving FM2 caching problems due to locale etc.
Date Sun, 26 Mar 2017 04:44:02 GMT
On Tue, Mar 21, 2017 at 1:29 PM, Daniel Dekany <> wrote:
> Tuesday, March 21, 2017, 3:12:38 PM, Woonsan Ko wrote:
>> On Sat, Mar 18, 2017 at 6:47 PM, Daniel Dekany <> wrote:
>>> The problem in FM2
>>> ------------------
>>> In FM2, if you write `cfg.getTemplate("foo.ftl", Locale.GERMANY)`,
>>> then FreeMarker will return a Template where template.getLocale() is
>>> Locale.GERMANY, even if there's no foo_de.ftl or such, and it has just
>>> fallen back to the most generic foo.ftl template. In the template
>>> cache, we will have an entry like
>>>   ("foo.ftl", GERMANY) -> Template where getLocale() returns GERMANY
>>> Then if you do `cfg.getTemplate("foo.ftl", Locale.ITALY)`, then you
>>> will have yet another cache entry:
>>>   ("foo.ftl", ITALY) -> Template where getLocale() returns ITALY
>>> So far you loaded (I/O!) and parsed foo.ftl twice, and you also store
>>> the parsed template (the AST, which includes all the static text too)
>>> in the memory twice. You see the problem; you don't reuse a single
>>> Template object, despite there's not foo_de.ftl or foo_it.ftl. It's
>>> wasteful, and users has complained about it too (if you use the
>>> preferred locale of the browser of the visitor, the bloat can be
>>> disastrous).
>>> Actually, the are multiple Template properties that have the same
>>> problem (cache inefficiency):
>>> - locale: See earlier
>>> - name: This stores the name used for the lookup (such as for the
>>>   localized lookup, but note that the lookup strategy is pluggable).
>>>   This is the normalized form of what you pass to
>>>   Configuration.getTemplate(...).
>>> - customLookupCondition: Used by some custom lookup strategies. Also
>>>   passed to Configuration.getTemplate(...)
>>> - (There's also `encoding` and `parsed`, but those will get out of the
>>>   way according the proposal from 2017-02-06.)
>>> Possible solution  in FM3
>>> -------------------------
>>> We could introduce something like LookupIndependentTemplate, which
>>> stores:
>>> - The parsed template (the AST)
>>> - The sourceName of the template (it's the name used as the parameter
>>>   of the TemplateLoader that has actually loaded the template "file",
>>>   so this is past lookup)
>>> - The template specific configuration settings (provided by
>>>   the Configuration.templateConfigurations setting, also by the #ftl
>>>   header in the template)
>>> - Reference to the Configuration
>>> Then there's Template, which is familiar from FM2, but this time it
>>> only contains:
>>> - The lookup parameters:
>>>   - locale used for the lookup
>>>   - name (the one used for the lookup)
>>>   - customLookupCondition
>>> - The LookupIndependentTemplate. Multiple Template-s may use the same
>>>   LookupIndependentTemplate.
>>> The template cache split to two layers:
>>> - Level 1: Looks up the Template with the lookup parameters. So this
>>>   is very similar to the FM2 cache.
>>> - Level 2: Looks up the UnboundTempalate based solely on the
>>>   sourceName (the TemplateLoader parameter).
>> Is UnboundTemplate the same as LookupIndependentTemplate mentioned above?
>>> So if you have a level 1 cache miss (or stale entry hit), then during
>>> the lookup, we get the templates from the level 2 cache. So in the
>>> original example, you will still have two Template-s (one for GERMANY
>>> and one for ITALY), but they are quite light and share the single
>>> UnboundTemplate that belongs to "foo.ftl". You have two entries in the
>>> level 1 cache merely to speed up the lookup. (Though the lookup on top
>>> of a level 2 cache that has already warmed up is quite fast, because
>>> just as in FM2, we would cache negative results, on both levels. Like
>>> we cache the fact that there was no "foo_de_DE.ftl" and "foo_de.ftl".)
>>> A tricky corner case is when LookupIndependentTemplate itself stores a
>>> non-null Locale (it can, as a TemplateCofiguration can contain all
>>> runtime configuration settings), which differs from the Locale for
>>> which the template was successfully looked up. I guess then that
>>> should take precedence over Template.locale. (A possible use case: you
>>> might don't use localized lookup, and instead the template itself
>>> dictates the locale. After all, the static text in the template uses
>>> some language.)
>>> Any thoughts?
>> Everything sounds good. Your judgement on the tricky corner case makes
>> sense, too.
>> One consideration might be naming. It sounds like Template 'contains'
>> LookupIndependentTemplate. But the names sound like 'inheritance' to
>> me. ;-)
> I agree, it's confusing.
>> So I wonder if there could be a better / more intuitive naming.
> I was thinking more about this, and I think that probably we just
> shouldn't store information about the lookup in Template, and then,
> naturally, we don't need LookupIndependentTemplate. (The two level
> caching would still of course remain, only it doesn't add an extra
> class to the published API, which is good.)
> So how would that work? My idea is that we first create the
> Environment first, set the locale and the customLookupCondition in it,
> and after that the Environment invokes the TemplateResolver to look up
> the main template, with env.locale and env.customLookupCondition as
> parameters (in additionally to the template name, of course). That's
> the opposite order compared to how FM2 does it; there you look up the
> template first, the resulting Template remembers the lookup locale an
> the customLookupCondition, and then the Template creates the
> Environment, and env.locale defaults to template.locale. If we do it
> in the other way around, then the Template doesn't have to remember
> the lookup parameters, and so Template is lookup independent. (Also,
> in case the TemplateLookupStrategy ignores the locale, or the
> customLookupCondition, now you don't have to instantiate a Template
> for each permutations of their values.)
I think this is a more logical choice. Great thinking! :-)
Environment by definition provides a runtime context on each template
processing, so it does make more sense to let Environment have the
lookup strategy and choose a proper template in that level.

> But of course, there's at least a catch with this idea... If Template
> is lookup independent, then it can only store the `sourceName`, but
> not the `name`. (`name` is the lookup parameter after some
> normalization, which is then possibly transformed by the
> TemplateLookupStrategy to the `sourceName`. `sourceName` is the
> parameter to the TemplateLoader invocation that has actually managed
> to load the template file.) But we need the `name` to resolve relative
> paths in the template. So when a Template is loaded into the
> Environment, that information will have to be tracked somehow. Because
> the same template can be included for multiple times with different
> `names` (for some lookup strategies), it's not entirely trivial. But
> it certainly adds less complexity than having a separate
> LookupIndependentTemplate and Template class, or at least less
> complexity that's visible for the users.
> The other thing with this choice is that it changes some of our most
> fundamental API-s. In FM2 you do this:
>     Template t = cfg.getTemplate(name, locale, customLookupCond);
>     t.process(dataModel, out);
> So things happen in the wrong order. Thus we had to change that to
> something like:
>     cfg.processTemplate(name, dataModel, out, locale, customLookupCond);
> There would be overloads of processTemplate, where you omit the last
> or last two parameters. Also, there should be overloads where instead
> of a `name` you specify a Template object, that you have created from
> a String yourself with new Template(...), so it couldn't have lookup
> information anyway.
> Template.process(...) would be removed, as it's not the duty of the
> Template to create the Environment anymore.
> On the level of the more expert Environment API, this translates to
> something like this:
>     Environment env = new Environment(cfg, dataModel, out);
>     env.setLocale(aLocale);
>     env.setCustomLookupCondition(aLookupCondition);
>     ... // Addjust env further
>     env.loadMainTemplate(name); // After setting locale and customLookupCond!
>     env.process();
>     ... // Get back variables from env if you want
This looks a lot clearer and more intuitive to me!

> You might notice that Environment now stores the
> customLookupCondition, while in FM2 the Template stores it. Any
> template loading from the Environment will happen with that condition
> (and with env.locale, just as in FM2). In FM2 the
> customLookupCondition is inherited from the Template that contains the
> #import/#include statement, but if you think about it, the end results
> is practically the same.
Yeah, it's really cool concepts now. Very exciting!!



>> Regards,
>> Woonsan
>>> --
>>> Thanks,
>>>  Daniel Dekany
> --
> Thanks,
>  Daniel Dekany

View raw message