Patching an array with assignment on multiple lines corrupts the output file

[tchristiansen-aquaveo]

I've been trying out f90nml on a project, and like the patch functionality a lot.

I encountered what looks like a bug though. If I patch an array that assigns one index per line, it corrupts the output. I made a test case to illustrate.

In template.nml:

&GROUP
x(1) = 1
x(2) = 2
x(3) = 3
x(4) = 4
y = 5
/

In a Python script:

from f90nml import patch

patch_dict = {
  'GROUP': {
    'x': [6, 7, 8, 9],
    'y': 10,
  }
}

patch('template.nml', patch_dict, 'output.nml')

When I ran the script, I expected to get the following in output.nml:

&GROUP
x(1) = 6
x(2) = 7
x(3) = 8
x(4) = 9
y = 10
/

Instead, I got this:

&GROUP
x(1) = 6
7(2) = 2
x(3) = 3
x(4) = 4
y = 10
/

On the third line, the name of the array was replaced with the value, and the value was left unpatched. After that, nothing else in the array got patched, but it recovered afterward for y.

[marshallward]

There are a lot of known problems with the patch feature, some nearly unsolvable without a rewrite (see #77, #80 for some examples). The underlying problem is that patch does a simple token substitution while scanning/parsing the input file, and often lacks sufficient context to do even simple manipulations. Arrays are a perfect example of this.

This one feels slightly plausible, since it does not require backtracking and we could perhaps track both name and index, rather than just name, while checking substitution. But patch is such an odd function that problems could easily pop up.

As with the other issues, the correct solution is to introduce a whitespace-preserving tokenizer, which has been on the TODO list for several years. Unfortunately, I just find it hard to prioritize such projects these days.