I was thinking what should we call these files, which cause the generation
of an output file (a "transformation" as you called it int that gist),
taking the CLI into account as well. Maybe, "seeds", as those are the
things output generation starts out from. That is, for a
freemarker-generator run, you may want to specify:

   - *Seed templates*: That's what you have called "aggregation" in the
   gist. So, one output file per template. Normally you also want to specify a
   data source for this, but that's not a "seed data source" (see next point).
   It's not strictly necessary though, as the template can just run in itself.
   - *Seed data sources*: This is when you generate one output per data
   source file. You must also specify a *transform template* for this, and
   note that that's not called a "seed template".
   - Seed data source entries: When you generate one output file per entry
   (like per table row) inside the data source. (We had to introduce data
   loaders for this to work, but I try to focus on one thing per mail, and for
   now I assume that we just load and parse data sources inside the template,
   like it's done currently. But I also wanted to show that there are more
   than 2 kind of seeds.)

So in a CLI call, that might looks like this:

One output per template (this is a single command, I juts use line break
for readability):

freemarker-generator
    --seed-template *.ftl
    --data-source main.csv
    --shared-data-source:helper helper.csv

I imagine it so that --seed-... starts a section, so that everything up
until to the next --seed-... or --shared-..., configures to that group of
seeds:


freemarker-generator
    --seed-template foo/*.ftl
    --data-source foo/main.csv

    --seed-template bar/*.ftl
    --data-source bar/main.csv

    --shared-data-source:helper common/helper.csv

One output per data file:

freemarker-generator
    --seed-data-source logs/*.csv
    --transform-template report.ftl
    --shared-data-source:helper bar/helper.csv


Based on above, naturally you can also mix one-output-per-template, and
one-output-per-data-source (again, this is a single command line command,
line breaks are only for readability):

freemarker-generator
    --seed-template foo/*.ftl
    --data-source foo/main.csv

    --seed-template bar/*.ftl
    --data-source bar/main.csv

    --seed-data-source logs/*.csv
    --transform-template report.ftl

    --shared-data-source:helper common/helper.csv


So there you generate output based on each ftl file in foo/, then you also
generate output based on each ftl file in bar/, and then you generate
output based on each csv files in logs/. All these will use
commons/helper.csv, which therefore has to be loaded only once. (That again
will be more helpful with data loaders, but that's for another time.)

Of course, this can be extended with specifying output file name per
--seed-..., and so on.

For the most common uses we could allow a sort hands in CLI, by assuming an
implicit --seed-template before the first argument. So if you want just try
something in FreeMarker, you can write this (as --data-source is optional
for --seed-template):

freemarker-generator adhoc.ftl

Or if you just need to transform a single data file (note that it doesn't
really matter then if it's the template that's the "seed", or the data
source):

freemarker-generator adhoc.ftl --data-source some.csv


This is just my idea anyway. What do you think?


On Fri, Mar 6, 2020 at 12:11 AM Daniel Dekany <daniel.dekany@gmail.com>
wrote:

> About the world choice of "aggregation" and "generation" on
> https://gist.github.com/sgoeschl/b09b343a761b31a6c790d882167ff449. We
> are "generating" files even if we are "aggregating". The project
> itself is called freemarker-generator for that reason actually. (Also
> "aggregation" is not really aggregating in the typical IT sense, I
> think.) I think to present the concept in an easy to understand way,
> we should express that you will have either one "transformation" per
> template file (which you called "aggregation"), or one "tranformation"
> per data-source (which you called "generation").
>
> But the concept is even more important. Again, at very least because
> of shared data (see mail before the previous), you will need to be
> able to use both "aggreagation" and "generation" in the same
> freemarker-cli run. Also there's the use case when you run one
> "transformation" per row in a table. So, *for each* something you run
> a "transformation", where that "something" can be a template, or a
> data file, or an entry (a row) in a data file.
>
> The tricky question is how to put all that in a sane way into CLI... I
> will think about that. But we really shouldn't cripple the core
> concept just because it's hard to express in a primitive interface
> like CLI. Anyway, pretty certainly only the simplest use-cases will
> look good on CLI, and for others, you will want to use a configuration
> file, or Maven, or Gradle.
>
> BTW, the "freemarker-cli" command should be really called
> "freemarker-generator", which is the project name. The Maven task
> isn't called "freemarker-maven" either. :) You are calling
> freemarker-generator via CLI, or via Maven.
>
> Also, you mentioned in last mail this idea of adding query parameters
> to configure that data source. That I think will be difficult to do in
> practice. Both the front-ends (CLI and Maven) has the concept of
> collecting file paths for you. But they won't attach query parameters
> to them for you. Also, data source configs can be long winded and
> having nested structures, so you would often end up with ?setup=<some
> JSON here>, which is quite horrible to URL encode and all.
>


-- 
Best regards,
Daniel Dekany