Consider implement lambdas instead of making filter, map etc. compiler magic

[danieldaeschle]

i hope the title says everything.

The lambda could look like:

param -> result

i also like =>

[medvednikov]

The decision was to use traditional fn syntax for closures, and a short it syntax for small builtin functions like filter and map.

That keeps the language simple without introducing a new syntax construct.

[danieldaeschle]

hmm, alright, i understand. Do you really want to builtin filter, map? They could be traditional functions instead of compiler magic. What do you think?

[medvednikov]

In order to optimize filter/map chains we need them to be implemented via compiler magic. Otherwise they will always be slower than for loops.

[danieldaeschle]

How does Node/chromium is optimizing them? On chrome forEach is nearly faster than for.

[medvednikov]

The same way it's optimized right now in V. Compiler magic.

[danieldaeschle]

ok.

[dumblob]

I know this issue got cleared up, but still I feel it's important to express my strong preference to use as much compiler magic as possible as that's what differentiates V from writing plain C/JS/... code.

Making a transpiler (which V fundamentally is, though there is native binary output support planned for certain archs) makes practically sense only when one wants to use higher level programming concepts and compile them highly efficiently using compiler magic to lower level concepts.

So please keep going using this enormous opportunity everywhere. There's huge space of possible transpiler magic lurking around allowing V to be extremely efficient.

[ceremony]

I have to disagree with this.

Perhaps the best example of higher-order functions and lambdas in a procedural/OOP language is C# Linq. There is no "compiler magic"; if one wishes they can rewrite the entireity of the LINQ functions independently without any performance degradation.

The key to LINQ are the notions of an Enumerator and IEnumerable, along with types for higher-order functions (Func<TIn, TOut>, etc.). These three concepts - the enumerator, an object that can be enumerated, and functions that operate on enumerations - serve as the bedrock.

With these three concepts, map and filter and merely a small part of what can be achieved. If you combine this with lambda functions, it's possible to create entire chains of operations dynamically.

Eventually, if map and filter become integral to V and widely used, without question developers will want to be able to extend these higher-order functions. At that point, they will face a performance wall because of compiler magic.

[dumblob]

@ceremony maybe I'm missing something, but what you wrote is IMHO in no way a disagreement with compiler magic as compiler magic is used just for special cases, but this doesn't stop you from writing efficient implementations on your own. The only difference is, that compiler magic makes some special cases (e.g. map[string, int]) efficient by default without introducing any complex stuff in the language itself just to squeeze a bit more performance out of it.

Note also, that LINQ by definition uses enumerators (as you pointed out) which are known to be non-optimizable compared to e.g. plain arrays and thus leading to lower performance in critical sections (seach the web for LINQ in games for example). In a JITted VM like C# it doesn't matter and the speed is nearly up to the C# native, but in a compiled lang like V this make a huge difference (starting by factor 10 and going up).

[ceremony]

@dumblob, I'm not suggesting that V implement LINQ (and without a root hierarchy, JIT, etc. that would be infeasible anyway). What I wanted to point out was that the utility of optimizations over a Type or products of Types (<T1, T2, ... TX>) will likely be limited in the middle/long run (and obviously for user-defined types). It would introduce a burden whereby some types for equivalent data structures/algorithms are optimized, and others aren't; and this would be a moving target for various functions/data structures.

Rather than the core V language, these sorts of optimizations would be much better suited to a "vdsa" library - through the use of which the developer is assuming responsibility for knowledge of behind-the-scenes optimizations in the library - as opposed to sometimes-builtin compiler magic.

[dumblob]

@ceremony in other words you're suggesting making V slow by default in all situations and introducing an optimized opt-in library which'll have certain (common) cases highly optimized?

If it is so, then I don't understand why. It'll mean more work (the same code will be written twice - once without optimizations and once with them), it'll violate the V basic principle ("there is only one way of doing things" because the user will have a choice offering exactly the same functionality, but different speed), it'll mean that one can't just use V as a fast language for majority (common) cases, but exactly vice versa (i.e. the user needs to write and maintain more code, more dependencies, etc. just to squeeze the performance out of V).

I hope I'm missing something in the understanding of what you're asking for 😉.

---

Btw. optimizations are by definition always limited to certain cases. So even if V had something like LINQ (I'm aware you do not advocate for that, but let's take this just for this moment as the best possible case imaginable), then it won't reach the speed of compiler magic while being always more complex than compiler magic (adding compiler magic is super easy in V without any need of adjustments on user's side in contrast to LINQ approach which by definition always needs some adjustments on the user's side).

[danieldaeschle]

@dumblob i think he means: if V has compiler magic for filter, map etc. it's only for those. but what if the user want's to define it's own findIndex function (just an example) which works really the same like the others but is not supported by the compiler.

we just should make the compiler magic / optimization more generic instead of just some cases.

[dumblob]

we just should make the compiler magic / optimization more generic instead of just some cases.

I agree. But that's totally in agreement with everything I wrote so far 😉. The first step is actually already there - one can very easily implement own compiler magic directly in the compiler (V is intentionally built as an extendable "library" to allow exactly this). Feel free to add your special case (it shouldn't be that difficult to patch each V release with your modification because as I said V is well structured for that, so your changes will be very local and not scattered).

Note also, that after V 1.0 gets released, the language can be extended if there is such a need. But first I'd stick with the "patching" of V compiler itself as outlined above. Maybe the only needed change in the future will be refactoring of the compiler magic code to allow writing your own modules implementing your own additional compiler magic instead of introducing any language extensions. Who knows.

[danieldaeschle]

Not every developer or company want to add compiler magic for their own functions.

Maybe the only needed change in the future will be refactoring of the compiler magic code to allow writing your own modules implementing your own additional compiler magic instead of introducing any language extensions. Who knows.

Hmm, or make a generic compiler optimization which works for special cases. Let's say a method takes a function as parameter with only one parameter, compiler can make => myfun(it.foo) optimization.

[dumblob]

Not every developer or company want to add compiler magic for their own functions.

Even if it's as easy or easier to do than writing it in your app itself (because that's currently the case)?

Hmm, or make a generic compiler optimization which works for special cases.

That's what V does and will do way more - it's called compiler magic 😉.

Let's say a method takes a function as parameter with only one parameter, compiler can make => myfun(it.foo) optimization.

Hm, C compiler under the hood does this already (V started using -O3, so this is more or less guaranteed). I know this was just a tiny example, but IMHO V should rather go for better optimizations with way more gains (see e.g. #1839 or #1868 ) 😉.

On the other hand, I don't see any harm with extending the current filter/map/... interface with something accepting anonymous functions ("lambdas"). I'd though probably be against swapping the current interface with one heavily using anonymous functions. But as I said, you're free to extend the interfaces yourself already now, it's really easy 😉.

[pr-yemibedu]

Hello,
Nim does a similar thing. They have map (a procedure taking a procedure and a generic array) and mapIt (a template taking a procedure and any type). The map function is used often and keeps your code fast. The mapIt macro keeps you flexible and your code consistently readable when using other types not supported out of the box.

I am not saying to do it like Nim has it exactly but something could be done to have both. Thank you, Good day.

[dedesite]

Hey I'm new to V and sorry if I say something wrong, but I like the idea of having compiler magic, but for me, the way map and filter are handle does not fit the way or syntactic sugar are implemented.

For exemple, the for loop offer you a way to choose variable name, get index and create a scope, this way you can write on multiple line.

Maybe map and filter doesn't have to be generic as using a fn lambda, but at least, could it be something like for ?

eg :

a := [1, 2, 3, 3, 4, 5]
only_three = a.filter(key, value {
    key == 3 && value == 3
})
println(only_three) // [3]

But maybe you don't want people to write lots of code inside filter and map ?

Just my two cents.

[dedesite]

I just found on this other issue that we can pass an anonymous function to filter and map which is great !

Is there a way to have access to the key and the value ?

Like this :

a := [1, 2, 3, 3, 4, 5]
only_three = a.filter(fn(key int, value int) bool {
    return key == 3 && value == 3
})
println(only_three) // [3]

[danieldaeschle]

arrays doesn't have keys. ineed, the index is not passed.

[dedesite]

Your right I should have used index in my example.

[antonkatz]

What is the reasoning for not being able to use it anywhere a single parameter lambda can be used?