Feature Request: improve accessibility of Monaco

[mltony]

Is your feature request related to a problem? Please describe.

Accessibility of Monaco editor is limited despite Microsoft effort. For screenreaders it creates an invisible text area, that is only filled with 10 lines of code. As a result the following functions of NVDA don't work correctly in Monaco:
• SayAll doesn't work
• Indentation navigation in source code (via IndentNav add-on) doesn't work.
• Paragraph navigation with multi line break style (as introduced in Improve Paragraph Navigation - add single and multi line break style #13798) doesn't work correctly in VSCode. Despite the author claiming to have tested it with VSCode, around line 500 it won't work correctly since it is using textInfo API, which would only see the first 500 lines of the file.
• Sentence navigation (via SentenceNav add-on) doesn't work.
• Quick search function (via Tony's enhancements add-on) doesn't work.
• In general, one can come up with many other ideas of new functions that can be implemented via TextInfo API. None of them would work correctly beyond 500 lines of code in VSCode or other Monaco-based web applications.

Describe the solution you'd like

1. I would like to build a new accessibility interface (let's call it JSAccessible) that would be written in JavaScript and I will modify Monaco to implement this interface.
2. I would also like to build JavaScript Accessibility Bridge - we need a way for NVDA to talk to web applications implementing JSAccessible. The most straightforward way to implement JSAccessible seems to be by writing a Chrome extension that can communicate to Monaco running inside browser on one hand and also communicate to NVDA via Native messaging API.
3. Modify NVDAObject to talk to Monaco editor via JSAccessible when available - otherwise fall back to IAccessible2.

Please let me know if you want me to split this into three separate feature requests. However these three are integral parts of a single feature and they are going to be useless when considered separately, so I prefer to keep them together.

To further illustrate this proposed accessibility API, for plain text editors we can think of something like this:

interface IJSAccessiblePlainTextEditable {
    // functions to be implemented by Monaco 
    getTextInRange(startIndex: number, endIndex: number): string;
    getCursorIndex(): number;
    setCursorIndex(index: number);
    /// ....
};

Once Monaco implements JSAccessible API, then I will build a new implementation of NVDAObject that can communicate with target Monaco editor via JavaScript Accessibility Bridge and JSAccessible.

Describe alternatives you've considered

1. @lukaszgo1 suggested to fix IAccessibility implementation in Chrome instead. However relevant chromium issues issue1 and issue2 have been open for many years and nobody from Chromium team tackled them. And I have reasons to believe that tackling those issues is going to be hard for Chromium maintainers and nearly impossible for the outsiders.
   The previous Chromium issue that was related to Text Area performance was indeed closed about 5 years ago; before that issue was resolved, performance of Chromium was even worse, so that Monaco could only show 10 lines to the screenreader without regressing performance for sighted users. However the fact that it took so long to resolve this issue and the fact that it only improved performance to a limited extent suggests that fixing performance properly (so that Monaco can show the entire text file, say up to 100k lines) is not feasible for Chromium maintainers, and much less feasible for outsiders like me.
2. Instead of writing a Chrome extension, we can consider making use of Selenium to allow NVDA to talk to JavaScript environment inside browsers. However my intuition tells me that Chrome extension is a more sound and flexible solution since we'd have full control over it.

Additional context

Previous discussion and big picture thoughts: #15995

[Adriani90]

@mltony is this issue also occuring when you use UIA form Chromium browsers explicitely? You can enable and test that in NVDA advanced settings.

[cary-rowen]

cc @seanbudd
What does NV Access have to say about this, I think it will be a big step forward for screen readers.

[thgcode]

An expandable code textInfo API might be useful for other editors as well, for example we could expose on this API more information about parenthesis levels for example, so the screen reader could alter the pitch or play a beep wyhen the user enters on a parenthesis or square bracket character, it could have sounds when speaking keywords if the API could expose this information. And if it could be used from other languages it could also be implemented by other IDEs as well, for example IntelliJ. I think most features like sound on warnings or errors could be implemented on top of this API, and VSCode wouldn't need to implement it anymore.

[thgcode]

Another idea, maybe this new API could be implemented as an addon, and when opening
supported IDEs, NVDA could suggest to install the addon. This could ease
implementation of new features and would not tie the API to a specific
nvda release.

[mltony]

@Adriani90, I did a quick test and yes, lagging in text areas is also occurring with UIA, actually it's even worse with UIA than IA2.
My Chrome browser didn't want to switch to UIA, so I tested with Edge. I pasted 10000 -lines-long text file into a plain text area. with IA2 any navigation command lags by about 1 second, while with UIA the lag is noticeably worse at about 2 seconds.

[mltony]

@thgcode, the problem with add-on approach is that I would need to convince Monaco devs to allow me to implement JSAccessible and convince Electron maintainers to include Javascript accessibility bridge so that it can be picked up by VSCode. It would be much easier to do so if NVAccess is aligned, so that I can reach out to Monaco and Electron as NVDA dev. But yeah, if not for that, I would've just implemented an add-on prototype first.

[mltony]

Also I just found out that by default Monaco only shows 10 lines to screenreader, not 500 as I thought before. I updated issue description with that. This can be easily tested on Monaco playground. So yeah, Monaco accessibility is even more broken that I initially thought.

[LeonarddeR]

I'm curious to know what @isidorn has to say about this. There were several attempts to replace the hidden text area in Monaco with something else I believe.

[isidorn]

@hediet should have some updates about the thinking here.

[hediet]

We are investigating whether to adopt EditContext, which could solve some issues here, as it basically replaces the hidden text area as input mechanism.

EditContext looks similar to IJSAccessiblePlainTextEditable.

Also I just found out that by default Monaco only shows 10 lines to screenreader, not 500 as I thought before.

We are up to improve that, but the text area gets quite slow quickly, so maybe we can increase that with some IJSAccessiblePlainTextEditable-approach.

---

We should make sure that the improvements for the monaco editor also improve VS Code.

[mltony]

@hediet, thanks for chiming in. I am trying to understand how would edit context improve accessibility.
I am looking at EditContext API and it indeed looks very much like I envisioned IJSAccessiblePlainTextEditable to be. However, I don't see a way for screenreader to communicate directly with EditContext API, since screenreaders don't have a direct way to interact directly with JavaScript code. Does it mean that we would still have to implement JavaScript Accessibility Bridge to make use of EditContext API on NVDA end?
In theory I can try to implement it as chrome extension/electron extension/VSCode extension, but NVAccess is not aligned on that, so that won't be a part of NVDA. Also If I go the VSCode extension route, it already appears to have full access to document being edited, so it can be done in current version without Edit Context.

EditContext, which could solve some issues here, as it basically replaces the hidden text area as input mechanism.

If you replace text area, how screenreaders will be able to see and edit text? From this page I see:

For example, you could maintain a separate view of the text in an offscreen DOM element that's presented to screen readers.

Is this the intended way: If so, then NVDA by default won't see source code as editable, only as read only HTML and there will probably be some work required to make it to work on NVDA side to make it to work as editable. Also won't this cause performance concerns with large files, e.g. 10k+ or 100k lines?

We should make sure that the improvements for the monaco editor also improve VS Code.

Absolutely agree with that. When I created this issue I really meant improving Monaco including the one in VSCode.

[mltony]

FWIW I built a solution that improves accessibility of VSCode. Specifically it provides access to the entire file in VSCode - not only to 500 lines.
This consists of:

• VSCode extension that opens a named pipe and listens to requests. The protocol in that named pipe is basically JSON messages. It can respond to commands like:
  • getStoryText
  • getCaretOffset
  • setCaretOffset
  • and a few similar ones.
• IndentNav extension v2.0 can now talk to that extension via named pipes. It implements VSCodeTextInfo class that has access to the entire document. When IndentNav is installed, other add-ons can make use of this textInfo by calling:

  focus.makeEnhancedTextInfo('caret')
  

I would be happy to contribute this solution to NVDA core if NVAccess is positive.
This only works in VSCode. In the long run I envision a similar solution with Chrome extension or Electron extension to serve as a more universal multipurpose JavaScript accessibility bridge that would be able to improve accessibility of Monaco editors running in browsers. My work with IndentNav can serve as a proof of concept.

[cary-rowen]

Hi @mltony

Sorry, I haven't looked at it yet. Is the VS Code extension you developed effective for the WEB version of VSCode? In other words, can it be installed in vscode.dev?

Thanks

[mltony]

@cary-rowen, most likely that won't work in web VSCode. Most likely it won't be able to create a named pipe from inside sandbox.

[cary-rowen]

The extension can be found in vscode.dev, but no install button appears.
This is a pity. In some environments, only the web version of vscode can be used.

[gerald-hartig]

@mltony have you continued to work on this proposal? Or should it be closed?

[mltony]

Let's close this for now. I still think this would be a good direction for NVDA, but I don't expect to have enough time to work on this in the nearby future.