Skip to content

Latest commit

 

History

History
204 lines (185 loc) · 33.8 KB

TODO.md

File metadata and controls

204 lines (185 loc) · 33.8 KB

This todo list describes ChrisAnt996's current intended roadmap for Clink's future. It is a living document and does not convey any guarantee about when or whether any item may be implemented.


IMPROVEMENTS

Mystery Issue

High Priority

Unit Tests

Normal Priority

  • Randomly hit assert(group == m_prev_group || group == m_catch_group); upon Ctrl-Space. It left input in a weird state with clink-select-complete still active but not handling input. Could not repro again after I got out of the state. It seems likely to be a long-standing issue in some obscure edge case.
  • Event handler enhancements:
    • Allow setting an optional priority when registering event handlers? So that scripts can control the precedence of onbeginedit, onendedit, and so on.
    • Allow adding a ONE-TIME event handler which automatically removes itself upon firing? And clink-diagnostics would need to show any ONE-TIME event handlers until the next beginedit.
      • Watch out for back-compat: Consider making new API functions for adding one-time event handlers. Adding an optional parameter is dangerous because a script author could use it without taking steps to ensure backward compatibility, and then potentially significant malfunctions could occur. And anyway, probably only a small number of events would actually need support for one-time handlers (maybe even only onbeginedit).
  • The oncommand event isn't sent when the command word is determined by chaincommand parsing; line_editor_impl::maybe_send_oncommand_event() needs to let _argreader determine the command word.
  • Some wizard for interactively binding/unbinding keys and changing init file settings; can write back to the .inputrc file.

Low Priority

  • Windows Terminal now has 3 ways of measuring character widths. There's no way for Clink to ask which mode is being used (but it might be possible to deduce on the fly using techniques from the wcwidth-verifier repo). The wcwidth updates in Clink are consistent with the "Grapheme clusters" mode. If the "wcswidth" and "Windows Console" modes turn out to be worth supporting, then that would require a bunch of extra work and configuration.
  • Find a high performance way to detect git bare repos and encapsulate it into a Lua function?
  • clink_reset_line still causes UNDO list leaks. UP until sudo where, then asdf, then ESC, then ENTER. May take several repititions; may repro quicker when varying which history entry is recalled.
  • line_state parsed foo^ bar as a single word "foo^ bar", but CMD parses it as two words "foo" and "bar". The parser is fixed now, but what about downstream edge cases where things check the next character after a word (or try to skip a run of spaces but get confused by foo ^ ^ bar)?
  • Open issue in Terminal repo about bugs in the new shell integration in v1.18.
    • Transient prompt can lead to Terminal getting confused about where prompt markers are.
    • Can the same thing happen with zsh and powerlevel10k transient prompt?
    • Provide a sample .txt file that repros the issue. Maybe multiple .txt files that chain together (or with a pause; is there an escape code for a pause?) to show the UX flow.
  • Consider plumbing lua_State* through all layers to help guarantee things don't accidentally cross from a coroutine into main?
  • Make a reusable wrapper mechanism to create coroutine-friendly threaded async operations in Lua?

Argmatcher syntax

  • See the argmatcher_syntax branch.


"New" commits from Martin

To Be Considered

No



APPENDICES

Known Issues

  • When echo is off, CMD doesn't print a prompt, and Clink can't which ReadConsoleW calls are for reading the input line. In theory, Clink could use RtlCaptureStackBackTrace() to deduce when a call is for the input line (see comment in host_cmd::read_console()), but that API isn't reliable for use in non-debug code.
  • foo bar a/b/c will try to expand a/b/c as an abbreviated path even if foo bar never generates filename matches. This is a case that Clink can't really get perfectly right anymore, because of the automatic deduction of whether to use file matches. Overall, this seems acceptable.
  • Readline's expand_tilde() doesn't handle embedded {space}{tilde}{pathsep} correctly in strings; rl.expandtilde() does, and has an optional parameter to use Readline's original style of tilde expansion.
  • Cursor style may behave unexpectedly in a new console window launched from a Windows Terminal console, or in a console window that gets attached to Windows Terminal. This is because there's no reliable way for Clink to know whether it is running inside Windows Terminal. Related to Terminal #7434.
  • Perturbed PROMPT envvar is visible in child processes (e.g. piped shell in various file editors).
  • #369 and mridgers#531 anti-malware suites sometimes think Clink is malicious [This is likely because of the use of CreateRemoteThread and/or hooking OS APIs. There's no way around that. Signing the binaries might reduce that, but that's financially expensive and there's no way for an indepedent author to get an EV code signing certificate even if they were willing to pay the thousands of dollars per year.]
  • Terminal #10191 Microsoft Terminal does not allow a console application to know about or access the scrollback history, nor to scroll the screen. It blocks Clink's scrolling commands, and also the console.findline() function and everything else that relies on access to the scrollback history.
  • The auto-updater settings are stored in the profile. That makes it more cumbersome to control the auto-updater settings if you use multiple Clink profiles. However, it makes it possible to control the auto-updater settings separately in "portable installs" (e.g. on a USB memory stick).
  • Lua code can check if there is real console input available, and can read real console input. But there is no way for Lua code to check whether there is any input queued for Readline (pending input, pushed input, macro text). That probably makes sense, since there is (correctly) no way for Lua code to read input queued for Readline.

Mystery

  • There's some kind of case where accept-line doesn't print a newline before returning the input line to CMD. Maybe somehow involving modmark. I've hit it several times. [But it only happened for a few days and then stopped. So I wonder if it was actually some kind of regression in Windows Terminal which got fixed.]
    • Maybe after printing the transient prompt and rl_crlf(), something may have gotten a chance to force clink.refilterprompt() and print a normal prompt again out of turn, leaving the cursor at the end of that and thus output from the command began at that cursor position? A command that I definitely only ran once shows up in a transient prompt, immediately followed by a normal prompt plus input line, then followed by command output immediately at the end of the displayed input line. But maybe the transient prompt is present because I hit UP then typed value, then hit CTRL-X,CTRL-R which prints transient prompt, then I hit UP again and typed value again and finally hit ENTER? 
      > xx run -long_flags_that_make_command_wrap args -flags value

 reponame  branchname ↓415  12345 14 hours ago             Tue 11:40
*> xx run -long_flags_that_make_command_wrap args -flags valueThis XX command is using the *XX Workflow* If you are testing changes in xx client code...
    • Instrumentation seems to indicate it cannot hit read_console() again after the rl_crlf(), so on-idle coroutines shouldn't be involved.
    • Could some onfoo callback have forced clink.refilterprompt() out of turn?
    • Could this be a race condition versus reset_stdio_handles()? Doesn't appear to be possible, since it goes through hooked_fwrite.
    • It only happened for a few days and then stopped, without any changes in Clink, so maybe the cause was external.
  • Once in a while raw mouse input sequences spuriously show up in the edit line; have only noticed it when the CMD window did not have focus at the time. [Not fixed by bb870fc494.] [Have not seen for many weeks.] [Likely due to ENABLE_VIRTUAL_TERMINAL_INPUT and largely mitigated by a8d80b752a.] [Root cause is microsoft/terminal#15711]
  • Mouse input toggling is unreliable in Windows Terminal, and sometimes ends up disallowing mouse input. [Might be fixed by bb870fc494?]
  • "qq": "QQ" in .inputrc, and then type qa --> infinite loop. [Was occurring in a 1.3.9 development build; but no longer repros in a later 1.3.9 build, and also does not repro in the 1.3.8 release build.]
  • Windows 10.0.19042.630 seems to have problems when using WriteConsoleW with ANSI escape codes in a powerline prompt in a git repo. But Windows 10.0.19041.630 doesn't.

Punt

  • Some wizard for interactively viewing/modifying color settings. [This is low priority now that Clink supports .clinktheme color themes.]
  • Can git.getstatus() be simplified even further, so it automatically handles clink.promptcoroutine()? Maybe a git.getstatusasync() function? [Not at this time: that could be something to consider for a bunch of various APIs later, but for now scripts should just use the normal clink.promptcoroutine() usage pattern.]
  • Input hinter; need some way for :gethint() to work with coroutines and override the optimization and call it again. [No: that would lead to hints cycling through multiple values at the same cursor position. Once a hint is shown, it shouldn't change until at least another keypress occurs.]
  • ^>nul echo hello behaves strangely: It redirects to echo and tries to run hello. What is going on with that syntax? Any ^ combined with redirection before the command word seems to go awry one way or another. It looks like a bug in the CMD parser. Trying to accurately predict how the bug will behave in all possible contexts seems unrealistic.
    • ^>xyz occurring before the command word loses the xyz and pulls the next token as the redirection target.
    • ^2>&1 whatever fails to duplicate a handle.
    • 2^>&1 whatever says & was unexpected.
    • Etc.
  • #554 Consider adding some way to configure Clink to try to run as a "portable program" in the sense that it doesn't write to any OS default locations (such as %TEMP%) and instead writes only to places that are specifically configured. But Lua scripts and programs they launch would need to also have their own special "portable program" support to avoid writing to OS default locations (especially %TEMP%). And what size of temporary files are ok to redirect to a "portable storage"? And who maintains/purges files from the portable storage? Because Clink runs scripts and other programs, trying to support a "portable program" mode is more complicated than it might sound at first. [No; it's not feasible.]
  • Clink's win_terminal_in keyboard driver generates some things differently than VT220:
    • Ideally it might have mapped CTRL-SPC->0x00(^@), CTRL--->0x0d(^M), CTRL-/->0x1f(^_), CTRL-?->0x7f(^? aka Rubout). But I think Clink's approach is overall better for those keys.
    • Ubuntu in Windows Terminal receives ^? for BACKSPC and ^H for CTRL-BACKSPC. But that seems backwards versus what I've always seen on many systems over the decades, so I think Clink should stick with ^H for BACKSPC and ^? for CTRL-BACKSPC.
    • The CTRL-DIGIT keys intentionally produce unique key sequences instead of CTRL-0 through CTRL-9 mapping to certain control codes or 1, 9, and 0.
  • The : and = parsing has a side effect that flags like -ffile are ambiguous: since parsing happens independently from argmatchers, -fc:\file could be -f and c:\file or it could be -fc: and \file. [Too much complexity for too little benefit too rarely.]
    • Revisit the possibility of allowing line_state to be mutable and argmatchers adjusting it as they parse the input line? No; too messy. E.g. splitting "-fc:\foo bar" gets weird because quoting encloses two adjacent words.
    • But an important benefit of the current implementation is that program_with_no_argmatcher --unknown-flag:filename is able to do filename completion on filename.
    • Maybe a better solution would be to let argmatchers specify getopt-like parsing rules. Then an argmatcher parser could split the word into -f and c:\file, and the second part could be put into a "pending word" variable which the parser could check before trying to advance the parser's word index? And could even potentially recognize -az as two valid flags -a and -z when appropriate (and if either flag is unknown, then color the whole word as unknown).
    • UPDATE: This is handled now by arghelper.lua via concat_one_letter_flags=true.
  • Maybe consider a way to allow piping matches and help into less or other external pager? It would break/interfere with the intended flow and behavior of the complete command and it might not work with ANSI codes. But in clink-completions#500 a user cited a link where someone assumed that bash would use an external pager like less. [No; it would break how completion is designed to work, and ANSI escape codes couldn't be used.]
  • CMD sets =ExitCode = the exit code from running a program. But it doesn't set the envvar for various other things that update CMD's internal exit code variable. So, Clink's tempfile dance to get %ERRORLEVEL% is still necessary.
  • A reliable way for scripts to tell whether they're loaded in clink set versus in cmd. [No. The only case reported that needed this was trying to access key bindings when the script was loaded, and due to a bug in rl.getkeybindings() Clink crashed. The crash has been fixed (now it returns an empty table instead), and the script is better implemented using clink.oninject() anyway.]
  • Provide some kind of "line editor tester" in the clink lua interpreter to facilitate writing unit tests for argmatchers? [No. Too many fundamental incompatibilities with the rest of the code. Completion script authors can do unit testing of their own code, but trying to do end-to-end testing of Clink itself from within Clink itself with being integrated with CMD? Hard no.]
  • #387 requests adding an option to put the cursor at the end of the search text when using history-substring-search-backward (and -forward). But that diverges even more from Readline, and I'm actively trying to instead converge as much as possible. [The request should be made against bash/Readline. If it gets implemented there, Clink will be able to pick up the change.]
  • Consider not redrawing while resizing the terminal, if there is no RPROMPT? Maybe just flag that a full redraw needs to happen, and defer it until the next time a redraw is normally requested? [Defer any further changes to terminal resize behavior until there is further feedback.]
  • Ctrl-Break does not interrupt Lua scripts during onendline or onfilterinput. The Lua engine doesn't support being interrupted; it only supports the application being terminated. The engine could be modified to check for a volatile flag, but that would need to be done carefully to ensure it doesn't interrupt Clink's own internal Lua scripts. [Not needed; it's very rare, and hooking up os.issignaled() should be sufficient, though it does require scripts to explicitly support being interrupted.]
  • Some way for history.save false to not do any disk IO for history, but still enable clink history to show the session's history (probably using Shared Memory). [Unfavorable cost vs benefit; expensive and complicated, while offering very little benefit beyond what could be achieved by simply applying ACLs and/or encryption to the profile directory, which is something that is best done externally from Clink.]
  • Coroutines can call clink.refilterprompt() and it immediately refilters while in the coroutine. Should it instead set a flag to refilter after the coroutines have yielded? [It should be fine because only line_editor_impl has an input idle callback that runs Lua coroutines.]
  • Show time stamps in history popup? [Gets complicated because of horizontal scrolling. Too many edge cases; the benefit is not worth the cost.]
  • Some way to push keys? (Push keys to Clink; not to other processes.) [Can use WScript.Shell.SendKeys when that's needed.]
  • Should coroutines really be able to make Readline redraw immediately? Should instead set a flag that the main coroutine responds to when it gains control again? [For now it seems fine; coroutines run during idle when waiting for input, so it should be safe to let the display code run.]
  • Readline should pass the timeout into the rl_input_available_hook callback function. [Not needed; the timeout is only for systems that need to use select(), and we don't need to.]
  • Fix order that isearch executes the extra pending command in Callback Mode. REPRO: ^R x Right p ==> "p" is inserted, THEN Right is executed. [Readline Callback Mode bug that only malfunctions when isearch-terminators omits ESC; not worth tracking down.]
  • Maybe redefine keyboard driver for Alt+(mod)+Fn key sequences to be like Ubuntu? [Not worth the disruptive impact unless it causes some problem.]
  • Readline doesn't handle certain display cases correctly. Rather than try to fix the Readline display, I've built an alternative display implementation. Here are some notes on the Readline issues:
    • A prompt exactly the width of the terminal seems to add a newline between the prompt and the input line, which might result from a missing _rl_term_autowrap test.
      • _rl_vis_botlin ends up with an incorrect value.
      • Could commit d4f48721ea22258b7239748d0d78d843ba2820f1 be related to that, or maybe it didn't fully fix the problem?
      • Run clink drawtest --width 31 --emulation emulate to observe.
      • Maybe Readline doesn't emit SPC CR if the prompt ends exactly at the screen width?
    • Readline is trying to use simple arithmetic to figure out how to convert byte position to absolute position, but the arithmetic is wrong and the data structures don't facilitate solving the issues.
      • E.g. a prompt whose last line wraps TWICE and has one UTF8 multibyte character in the FIRST wrapped segment; cursor position on the final line is offset wrongly.
        • It looks like this line nleft = cpos_buffer_position - pos; is trying to reset nleft to only include positions on the current screen row, which then throws off the woff arithmetic. It could maybe use modulus on the overall position, but that wouldn't account for double-wide characters that don't fit at the end of a screen row and wrap "early".
        • This code might be relevant: 
          /* This assumes that all the invisible characters are split
   between the first and last lines of the prompt, if the
   prompt consumes more than two lines. It's usually right */
/* XXX - not sure this is ever executed */
_rl_last_c_pos -= (wrap_offset-prompt_invis_chars_first_line);
      • E.g. a prompt whose last line wraps AT the screen width and contains multibyte UTF8 characters; cursor position near the beginning of the input line gets positioned incorrectly.
        • _rl_last_c_pos is negative on entry to rl_redisplay().
        • Because the "yet another special case" logic is triggered incorrectly, and adjusts cpos incorrectly, which carries forward to future calls.
        • After disabling that logic, then the cursor still goes wrong when crossing the woff boundary.
          • Need another iDNA for that...
  • Readline 8.1 has slight bug in update_line; type c then l, and it now identifies 2 chars (cl) as needing to be displayed; seems like the diff routine has a bug with respect to the new faces capability; it used to only identify l as needing to be displayed. [Moot; Clink no longer uses Readline's display implementation.]
  • Optional feature to simplify auto-path-separator after completion, like in zsh: highlight \ in a color, and if Space is the next input then replace the "\" with " ". [Not worth it; there is very little value, and there are many side effects, e.g. wrt autosuggest. I got excited at first, but then I realized what I really need is a better way to signal for menu-complete to accept a directory it's inserted and start a new completion. And then I realized, since completions are normalized as of commit 9ec9eb1b69 in v1.1.24, typing \Tab goes from foo\ to foo\\ to foo\bar, so the scenario I had in mind is already fully solved in a simple and reliable way.]
  • Postpone: Ideally the updater could have a way to run an embedded script in the newly installed version, to do any needed finalization. But there isn't really a way to reliably determine whether it needs to run, nor to handle errors that may occur. And a more reliable mechanism is to do upgrade steps on the next inject.
  • There's no straightforward way to let Lua scripts change the current directory and have CMD pick up the changed state. CMD maintains internal private state about what directory to use when running commands and programs. Running cd is the only way to alter CMD's internal private state.
  • Make clink-show-help call out prefix key sequences, since they can behave in a confusing manner? [Complex present in a non-confusing way, and very rare to actually occur. Not worth the investment at this time.]
  • Maybe deal with timeouts in keyboard input? Could differentiate Esc versus Esc,Esc but is very dangerous because it makes input processing unpredictable depending on the CPU availability. [Too dangerous. And turned out to not be the issue.]
  • Ability to rearrange and edit popup list items? [Can't realistically rearrange or edit history, due to how the history file format works.]
  • Using a thread to run globbers could let suggestions uses matches even with UNC paths. [But ONLY globbers would be safe; if anything else inside match generators tries to access the UNC path then it could hang. So it's not really safe enough.]
  • Make scrolling key bindings work at the pager prompt. Note that it would need to revise how the scroll routines identify the bottom line (currently they use Readline's bottom line, but the pager displays output past that point). [Low value; also, Windows Terminal has scrolling hotkeys that supersede Clink, and it can scroll regardless whether prompting for input. Further, Windows Terminal is deprecating the ability for an app to scroll the screen anyway.]
  • Is it a problem that update_internal() gets called once per char in a key sequence? Maybe it should only happen after a key that finishes a key binding? [Doesn't cause any noticeable issues.]
  • Provide API to show an input box? But make it fail if used from outside a "luafunc:" macro. [Questionable usage pattern; just make the "luafunc:" macro invoke a standalone program (or even standalone Lua script) that can accept input however it likes.]
  • Classify queued input lines? [Low value, high cost; the module layer knows about coloring, but queued lines are handled by the host layer without ever reaching the module layer. Also, the queued input lines ("More?") do not adhere to the current parsing assumptions; it would become necessary to carry argmatcher state across lines. Also, argmatchers do not currently understand ( or ).]
  • Support this quirk, or not? Esc in conhost clears the line but does not reset the history index, but in Clink it resets the history index. Affects F1, F2, F3, F5, F8. [Defer until someone explains why it's important to them.]
  • Additional ANSI escape codes.
    • ESC[?47h and ESC[?47l (save and restore screen) -- not widely supported, so I can't use it, and it's not worth emulating. Which makes me very sad; no save + show popup + restore. 😭
    • ESC[?1049h and ESC[?1049l (enable and disable alternative buffer) -- not worth using or emulating; there's no way to copying between screens, so it can't help for save/popup/restore.
  • Marking mode in-app is not realistic:
    • Windows Terminal is essentially dropping support for console APIs that read/write the screen buffer, particularly the scrollback buffer.
    • Different terminal hosts have different capabilities and limitations, so building a marking mode that behaves reasonably across all/most terminal hosts isn't feasible.
    • One of the big opportunities for terminal hosts is to provide enhancements to marking and copy/paste.
    • So it seems best to leave marking and copy/paste as something for terminal hosts to provide.
  • Lua API to copy text to clipboard (plain text or HTML) is not realistic, for the same technical reasons as for marking mode.
  • Would be nice to complete "%ENVVAR%*" by internally expanding ENVVAR for collecting matches, but not expanding it in the editing line. However, it's difficult to make that work reasonably in conjunction with path normalization.
  • ConsoleZ sometimes draws the prompt in the wrong color: scroll up, then type => the prompt is drawn in the input color instead of in the default color. It doesn't happen in conhost or ConEmu or Windows Terminal. Debugging indicates Clink is not redrawing the prompt, so it's entirely an internal issue inside ConsoleZ.
  • Max input line length:
    • CMD has a max input buffer size of 8192 WCHARs including the NUL terminator.
    • ReadConsole does not allow more input than can fit in the input buffer size.
    • Readline allows infinite input size, and has no way to limit it.
    • There is a truncation problem here that does not exist without Clink.
    • However, even CMD itself silently fails to run an inputted command over 8100 characters, despite allowing 8191 characters to be input.
    • So I'm comfortable punting this for now.
  • A way to disable/enable clink once injected. [Why?] [Available via noclink.cmd in clink-gizmos]
  • #486 Ctrl+C doesn't always work properly [Unrelated to Clink; the exact same behavior occurs with plain cmd.exe]

Chris Antos - [email protected]