Add the Embedded Puppet templating language

[hashar]

This was originally proposed in 2016 by @binford2k as #1345

Puppet configuration management supports two kinds of templating system to generate files resources based on Puppet variables:

• Embedded Ruby (erb) which is already supported by highlightjs
• Embedded Puppet (epp) which this pull request add support for

epp is similar with some differences:

• variables are decorated with $
• curly braces are used instead of do / end blocks
• arguments of blocks are in slightly different position
• interpolated strings use ${...} instead of the #
• etc

It is similar to erb but the backing language is Puppet rather than plain ruby.

[detect]

The detect test is erroneously recognized as mel and some attempts to tweak the input test gave me awk or csharp. Based on #1213, claim auto-detection to be too broad and disable it for epp.

[markup]

Epp can represent any other language configure it with subLanguage: [] and provide a markup test based on XML (which is properly detected by the auto-detector).

Additional tests cover epp comment and containment of Puppet.

More or less related: the erb language has subLanguage: xml when it should be [], I have left a TODO note for later.

Co-authored-by: Wikimedia Foundation Inc.

Changes

Checklist

• Added markup tests, or they don't apply here because...
• Updated the changelog at CHANGES.md

[hashar]

The pull request description is the commit message with the addition of a preliminary note:

This was originally proposed in 2016 by @binford2k as #1345

I ran the test suite under NodeJS 16 with npm run build && npm run test.

[joshgoebel]

This is problematic as it relies on auto-detect which has fallen a bit out of favor (and honestly has never worked super well)... that's going to make the anysublanguage.txt test fragile, as well as working poorly in general cases. Is there a MUCH shorter list of common embedded environments we could list explicitly?

I don't think we're interested in saying:

Just use "epp" and we'll figure it out.

When we know that's not really true and will only lead to incorrect highlighting and support requests.

What you'd really want (I think) is a way to specify you want epp-xml and then handle just that case... But I'm not sure how you could specify that in a CSS class name dynamically without it being confused as the grammar epp-xml rather than epp (embedded within XML).

Thoughts?

[hashar]

The "problem" with those templating languages (epp, erb, jinja from Django) is they can be used to represent anything though maybe it is not going to be used to generate a compiled language source code (csharp, c, delphi etc) there is no real limit as to which target format it can represent (our repo has Apache config, bash, js, json, lua, sql, php, python, ruby, systemd, toml, plain text, xml, yaml to name a fews).

I wrote the first markup test with something looking like an ini file which was indeed erroneously detected (as csharp, potentially due to a <% # this is a comment %>). Retrying today with test/markup/epp/anysublanguage.txt:

<% # this is a comment %>
[section]
foo = <%= ec2_metadata["ami_id"] %>

Is autodetected as erlang so yeah that is really fragile hence why I went with a test using XML which is properly auto detected. But autodetections kick in on each block and test/markup/epp/anysublanguage.expect.txt has a block for mercury:

<span class="language-mercury"><span class="hljs-comment">%&gt;&lt;/ami-id&gt;</span>

So that is clearly wrong and subLanguage: [] is going to be nightmare and it should be removed. So if one process an epp file they will have solely the templating markers and embedded Puppet rendered but the rest of the file untouched (plain text?).

---

From the discussion I had with one of my colleague on that topic, I mentioned vim and its support to mix up syntax for a file. An erb template representing a javascript file can be held in a file like template.js.erb and one can hint vim about it with:

autocmd BuffRead *.js.erb set ft=eruby.javascript

Which tells vim to apply erb first then javascript (a screenshot for an erb template generating a ini file: https://phabricator.wikimedia.org/F37104450 ).

What you'd really want (I think) is a way to specify you want epp-xml and then handle just that case... But I'm not sure how you could specify that in a CSS class name dynamically without it being confused as the grammar epp-xml rather than epp (embedded within XML).

The different languages have their own <span> with the CSS class for the language (language-xml or language-puppet). From test/markup/epp/anysublanguage.expect.txt:

<span class="language-xml">&lt;?xml version=<span class="hljs-string">&quot;1.0&quot;</span>
...
<span class="language-puppet"><span class="hljs-variable">$ec2_metadata</span></span>

Can the sublanguages be given when invoking highlightjs? If the caller knows the file is an erb for a javascript (maybe cause of the convention the filename has a suffix of .js.erb) it could invoke highlightjs with the sublanguage set explicitly, something like: highlightjs(language='erb, sublanguage['js']).

I will rework the pull request to remove the subLanguage: [].

[joshgoebel]

the rest of the file untouched (plain text?).

Well, it's not subLanguage: "plaintext" exactly but it's essentially equivalent since our plain text grammar does nothing.

Can the sublanguages be given when invoking highlightjs?

I was worried more about use in HTML, ie <pre><code class="lang-x"> style usage, which I consider the harder question, how does one specify arguments here. I dunno about "when invoking"... grammars are "compiled" at first use... they aren't things that are very dynamic - and certainly there currently exists no way to pass arguments to a grammar. That said writing an extension that did this would be trivial... something along the lines of:

// get the compiled grammar
var epp = hljs.getLanguage("epp")
// get the raw source function of the grammar before compilation
var epp_xml = l.rawDefinition();
// alter the grammar to specify XML
epp_xml.subLanguage= ["xml"]
// re-register and recompile your new grammar
hljs.registerLanguage("epp-xml", epp_xml)
// use your new grammar to highlight code
hljs.highlight(code, {language: "epp-xml"})

To me someone already using the API explicitly (vs just auto mode) to highlight blocks would just do something along these lines. I'm not yet seeing an approachable simple way of adding this directly to core.

[joshgoebel]

One could then imagine an API that took a function that did a dynamic transform of a specified grammar:

hljs.highlightTransform(code, {language: "epp"}, (def) => { def.subLanguage = ["xml"] })

We could even ship the simple version of "epp" in core and then you could publish a short wrapper on the Wiki that let someone choose the sublanguage of their choice if they were willing to add a little custom JS to their project.

[joshgoebel]

I think (for now) the correct solution here is to publish this as a 3rd party grammar - one with some custom JS glue that can be called to pre-register as many useful combos as one might need (epp_xml, epp_json, etc) and then generates those grammars by simply modifying subLanguage of the original template grammar (as shown above)... and you'd want to provide a function someone could run to add their own - say if they were adding their own 3rd party grammars and wanted to use epp to template them. And perhaps we'll find an elegant solution...

OR...

Lets start a second discussion on what to do about http once we deprecate sublanguage: * auto-detect usage... it's exactly the same problem... I think we'd perhaps like to magically expose http-xml, http-json etc to highlight HTTP requests with various content payloads... if we find an elegant solution there then I imagine it would be easy to circle back around and revisit this and see if it would work on the same foundation.

Does that make sense?

[paladox]

@joshgoebel would you be able to create a hjs epp repo similar to highlightjs-closure-templates please?

[paladox]

Oh... I can create it myself.