__bp_prexec_invoke_exec issue with editing command line in vim

changed the description

What's in your .bash_profile/.profile/.bashrc? It could be a bad interaction between shell integration and another script. Generally, shell integration works best if it is the last one loaded. It is careful not to stomp on existing settings.

I removed my .bashrc and .bash_profile only contains the following line:

test -e "${HOME}/.iterm2_shell_integration.bash" && source "${HOME}/.iterm2_shell_integration.bash"

The issue above still occurs

-bash-4.2$ set -o vi
-bash-4.2$ echo "Test"
Test
-bash-4.2$ echo "Test"
echo "Test"
__bp_preexec_invoke_exec "$_"
Test

In normal usage, the line which loads the shell integration is the last line in .bash_profile which sources .bashrc first, so it is the last thing to load.

Even worse: This will happen for every "line" in every iteration of the command. For example:

$ for f in {1..3}; do echo "This is ${f}"; done
This is 1
This is 2
This is 3
$ for f in {1..3}; do echo "This is ${f}"; done
for f in {1..3}; do echo "This is ${f}"; done
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
This is 1
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
This is 2
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
This is 3

I wonder if the bash that Amazon uses was compiled with some unusual flags. I've seen weird behavior from bash on LTS Ubunutus.

Logging into an AWS machine is probably too difficult, but Fedora would be doable. What version of Fedora is it? I can easily get VMWare images to play with.

Fedora release 29 (Twenty Nine)

But you can actually recreate this on a mac, too.

I'm using Bash 5.0 from homebrew but it also occurs with the provided v3.2

Well this is odd! I can't do it:

Can you attach your copy of .iterm2_shell_integration.bash here so I can make sure I test with the same one you have? Also, what version of macOS are you on? Anything in your .bash_profile, .profile, or .bashrc? Maybe also attach your ~/Library/Preferences/com.googlecode.iterm2.plist file in case there's something there that makes a difference.

Macos 10.15.4

I downloaded a fresh integration before I tested in each environment. I've attached the one currently in use on my macbook

iterm2_shell_integration.bash.tgz

I set my default shell to /bin/bash to see if maybe there was a zsh-to-bash issue but that didn't seem to have any effect.

Since I use zsh normally, I don't have a .bashrc nor .bashprofile. /etc/profile is the default:

System-wide .profile for sh(1)

if [ -x /usr/libexec/path_helper ]; then eval /usr/libexec/path_helper -s fi

if [ "${BASH-no}" != "no" ]; then [ -r /etc/bashrc ] && . /etc/bashrc fi

com.googlecode.iterm2.plist

Forgot to mention, and just realized this now, I'm using oh-my-zsh and the vi-mode plugin. That is what adds the "v to edit in vim" functionality.

I have no idea why I can't reproduce this myself, yet it happens to you on multiple systems.

What's the output of trap; set; echo "set -$-"; shopt -p ; shopt -po when this issue is reproducible? Hopefully that will capture all of bash's state.

Attaching output from my macbook and from an amazon linux server where this is happening.macbook.txt aws.txt

Still no luck. How hard is it to get a shell on AWS? I've never used it before, but I'm willing to give it a go.

George, I've set up an AWS linux instance and tested to ensure this happens. If you can send me a public ssh key I can add it to the server so you can ssh in and test.

Thank you so much!

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFGkapXXH1ZnuAxEXidffeGhWITfB66TLJcfCB4cULN+VVBfR7tiPHl9TVW1AEdByiDPqkrcQDGOyOoDBz98fJX/+eEaqGwO1Iijqthl+GKMMOduG3tlV9aYprn2V7q4IIX2LoaI+HyJXZM0nSf0maYgujsr0uquoQaVNh+79mbE4rT0l/98F5L/zB84zhq6fK7UanTcHgbFQiUzKivaIeXw0OpfucqnlsBRAxkerGu5eQcDjPXt8tJVnH4Do0iGcBooGOiXZh5gWPuHGINd+vrujcfEFCafxKxv2PTOo4sStn2IX8RdFKbsAiuYvatQMPs8qjuBSKAxkb9MlZAEDRbRaIyQf4a61JWiXAAYrOlu83rQmmZEWeYnmDauBdXdhij5aQrWUBj0pY915wznRu3WhrHtbo0NrSD97T4NWjj9xc0lRgEnR6Id28cjKZW9SdgiBgfoxwN5YyEQxc6eWEqGXX5TPaMJO/bPWPFejPFBFfK0jEU/sA1WI2Jj9VwHFHAC8mFg6qlhcQX+njmAU0NQVeyGY7QwMuJk7sK2LK/D1sy7M/LBd6z0vRKvskkkIZYn3LrEGYhmnncVIIQTB1fCFqpUed7JyyBYOvK7Z39BQt+D6bLLEl49hGqCrPT/7D6vxfQkNIiIE4R49Ml1sBD4+fLeYWplf/qrhBPClOpw== gnachman@gmail.com

And just to confirm, the repro steps are:

Install the latest bash shell integration
Log in so shell integration is running
Invoke set -o vi
Invoke echo "test"
Invoke echo "test" again
Bug occurs

You're welcome! You should be able to log on to justthat.strangled.net using your key and the logon name 'gnachman'

Shell integration is already installed for gnachman and set -o vi is in .bashrc so you won't need to worry about those.

The steps to reproduce the issue are:

type: echo "test"
press esc (to enter vi edit mode)
press v (to edit the command line in vi)
type ZZ to save and run the command. Result:

echo "test"
__bp_preexec_invoke_exec "$_"
test

press i to get back into insert mode

This happens after each command iteration so something like this...

type: for i in {1..10}; do echo "test"; done
press esc (to enter vi edit mode)
press v (to edit the command line in vi)
type ZZ to save and run the command. Result:

for i in {1..10}; do echo "test"; done
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test
__bp_preexec_invoke_exec "$_"
__bp_preexec_invoke_exec "$_"
test```

...results in a lot of noise.

Thank you for setting that up. I stupidly missed the last step of your original description which is why I had so much trouble reproducing it. I can get it to happen now and it sure looks like a bash bug. I've opened a question here:

https://unix.stackexchange.com/questions/583927/why-does-bash-print-the-debug-trap-after-edit-and-execute-command

I'll have a look at the bash code to see if I can make any sense of it (I usually cannot) and will file a bug against bash if I can make a case that it's not somehow intended behavior on their party.

Ok, I did some spelunking and what I found is that when you invoke your editor (v in vi mode or C-x C-e in emacs mode) it runs the fc builtin. For example, fc -e vi to invokes vi. This is done in fc_builtin() which blocks on retval = parse_and_execute (command, "fc", SEVAL_NOHIST); until the editor exits. The last thing it does is echo_input_at_read = 1;, which turns on printing of commands. I guess it does this with the intent of printing out the command you entered so you can see what it's about to execute.

Immediately after setting that flag, it does fc_execute_file() to run the command you just entered in your editor. fc_execute_file() eventually calls parse_and_execute() with the tempfile to run its commands. A side-effect of parsing the commands is to print them out when echo_input_at_read is true. This is fine. The trouble comes next: it runs the debug trap, and the echo_input_at_read flag is still 1. Here's a chunk of the relevant stack trace:

  * frame #0: 0x00007fff6d46b20c libsystem_c.dylib`fprintf
    frame #1: 0x000000010c53a205 bash`shell_getc(remove_quoted_newline=1) at parse.y:2494:6
    frame #2: 0x000000010c5384f0 bash`read_token(command=0) at parse.y:3251:23
    frame #3: 0x000000010c533f13 bash`yylex at parse.y:2761:19
    frame #4: 0x000000010c5307c5 bash`yyparse at y.tab.c:1842:16
    frame #5: 0x000000010c530359 bash`parse_command at eval.c:303:7
    frame #6: 0x000000010c5c714b bash`parse_and_execute(string=<unavailable>, from_file="debug trap", flags=5) at evalstring.c:364:11 [opt]
    frame #7: 0x000000010c597a59 bash`_run_trap_internal(sig=32, tag="debug trap") at trap.c:1032:4
    frame #8: 0x000000010c597703 bash`run_debug_trap at trap.c:1117:25
    frame #9: 0x000000010c54d524 bash`execute_simple_command(simple_command=0x00007fa43b601680, pipe_in=-1, pipe_out=-1, async=0, fds_to_close=0x00007fa43c000910) at execute_cmd.c:4196:12
    frame #10: 0x000000010c54ba3d bash`execute_command_internal(command=0x00007fa43b601660, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x00007fa43c000910) at execute_cmd.c:844:4
    frame #11: 0x000000010c5c7300 bash`parse_and_execute(string=<unavailable>, from_file="/var/folders/v3/jxvtby191yj866l8_09rdbxr0000gn/T//bash-fc.65iKNO", flags=16) at evalstring.c:458:17 [opt]
    frame #12: 0x000000010c5c6b95 bash`_evalfile(filename="/var/folders/v3/jxvtby191yj866l8_09rdbxr0000gn/T//bash-fc.65iKNO", flags=163) at evalfile.c:285:14 [opt]
    frame #13: 0x000000010c5c8cb5 bash`fc_builtin(list=<unavailable>) at fc.def:468:12 [opt]

After fc_execute_file() returns it calls run_unwind_frame which restores echo_input_at_read to its normal value (0 unless you ran bash --verbose) with set_verbose_flag().

Looking at run_debug_trap() we see something really interesting:

      old_verbose = echo_input_at_read;
#if 0   /* not yet */
      echo_input_at_read = 0;
#endif

      trap_exit_value = _run_trap_internal (DEBUG_TRAP, "debug trap");

      echo_input_at_read = old_verbose;

The blame log from https://github.com/bminor/bash shows it was added in bash-5.0 on Jan 17, 2019. The bash project doesn't use git and that commit is a quarter million lines of diff so it's hard to draw any inferences from it :).

I've posted a message to the bug-bash mailing list, which ought to show up in the archive tomorrow:

https://lists.gnu.org/archive/html/bug-bash/

Thanks for all the digging, George!

Looks like this came up recently (https://lists.gnu.org/archive/html/bug-bash/2020-04/msg00016.html) and the fix has been moved into the bash devel branch. I can't find any documentation on their release process, but it appears to be pretty slow. I can't blame them for that; maintaining bash must be an insane burden. Anyway, if you feel brave you can build it from http://git.savannah.gnu.org/cgit/bash.git/log/?h=devel or otherwise wait a few years and get bash 5.1 or whatever they call it :)

I have a workaround but I really don't know if it's a good idea to ship it because modifying bash builtins is sure to have unexpected consequences. It turns out that the v command is a shortcut for the builtin command fc. The following seems to solve the problem:

__bp_fc() {
 trap - DEBUG
 command fc "$@"
 trap -- '__bp_preexec_invoke_exec "$_"' DEBUG
}
alias fc=__bp_fc

I've filed an issue with bash-preexec (which is actually the upstream for the code that sets the debug trap) here: https://github.com/rcaloras/bash-preexec/issues/108

Let's see if my anxiety about releasing this can be reduced.

Thanks for all the effort, George!

Looks like there isn't a simple fix that won't have unintended consequences.

closed

__bp_prexec_invoke_exec issue with editing command line in vim

Designs

Child items ...

Activity

System-wide .profile for sh(1)