Commit 8ff4363e authored by adam j hartz's avatar adam j hartz

remove a bunhc of stuff

parent af1f01ce
......@@ -2,6 +2,7 @@
Xonsh Change Log
.. current developments
* ``$PROMPT`` from foreign shells is now ignored.
* ``$RC_FILES`` environment variable now stores the run control files we
attempted to load.
* Only show the prompt for the wizard if we did not attempt to load any run
control files (as opposed to if none were successfully loaded).
* Git and mercurial branch and dirty function refactor to imporve run times.
* Fixed an issue whereby attempting to delete a literal value (e.g., ``del 7``)
in the prompt_toolkit shell would cause xonsh to crash.
* Fixed broken behavior when using ``cd ..`` to move into a nonexistent
* Partial workaround for Cygwin where ``pthread_sigmask`` appears to be missing
from the ``signal`` module.
* Fixed crash resulting from malformed ``$PROMPT``.
* Fixed regression on Windows with the locate_binary() function.
The bug prevented `source-cmd` from working correctly and broke the
``activate``/``deactivate`` aliases for the conda environements.
* Fixed crash resulting from errors other than syntax errors in run control
* On Windows if bash is not on the path look in the registry for the defaults
install directory for GitForWindows.
* Question mark literals, ``?``, are now allowed as part of
subprocess argument names.
* IPython style visual pointer to show where syntax error was detected
* Pretty printing of output and syntax highlighting of input and output can now
be controlled via new environment variables ``$COLOR_INPUT``,
* In interactive mode, if there are stopped or background jobs, Xonsh prompts
for confirmations rather than just killing all jobs and exiting.
* ``which`` now gives a better verbose report of where the executables are
* Tab completion now uses a different interface, which allows new completers
to be implemented in Python.
* Most functions in the ``Execer`` now take an extra argument
``transform``, indicating whether the syntax tree transformations should
be applied.
* ``prompt_toolkit`` is now loaded lazily, decreasing load times when using
the ``readline`` shell.
* RC files are now executed directly in the appropriate context.
* ``_`` is now updated by ``![]``, to contain the appropriate
``CompletedCommand`` object.
* Fixed bug on Windows where ``which`` did not include current directory
* Fixed crashed bash-completer when bash is not avaiable on Windows
* Fixed bug on Windows where tab-completion for executables would return all files.
* Fixed bug on Windows which caused the bash $PROMPT variable to be used when no
no $PROMPT variable was set in .xonshrc
* Improved start-up times by caching information about bash completion
* The --shell-type CLI flag now takes precedence over $SHELL_TYPE specified in
* Fixed an issue about ``os.killpg()`` on OS X which caused xonsh crash with
* Fixed PermissionError when tab completions tries to lookup executables in
directories without read permissions.
* Fixed incorrect parsing of command line flags
* When a subprocess exits with a signal (e.g. SIGSEGV), a message is printed,
similar to Bash.
* Added comma literals to subproc mode.
* ``@$(cmd)`` has been added as a subprocess-mode operator, which replaces in
the subprocess command itself with the result of running ``cmd``.
* New ``showcmd`` alias for displaying how xonsh interprets subprocess mode
commands and arguments.
* Added ``$DYNAMIC_CWD_WIDTH`` to allow the adjusting of the current working
directory width in the prompt.
* Added ``$XONSH_DEBUG`` environment variable to help with debuging.
* The ``${...}`` shortcut for ``__xonsh_env__`` now returns appropriate
completion options
* On Windows the default bash completions files ``$BASH_COMPLETIONS`` now points
to the default location of the completions files used by 'Git for Windows'
* On Cygwin, some tweaks are applied to foreign shell subprocess calls and the
readline import, in order to avoid hangs on launch.
* Special cased code for handling version of prompt_toolkit < v1.0.0
* Show sorted bash completions suggestions.
* Fix bash completions (e.g git etc.) on windows when completions files have
spaces in their path names
* Fixed a bug preventing ``source-bash`` from working on Windows
* Numerous improvements to job control via a nearly-complete rewrite.
* Addressed issue with finding the next break in subproc mode in context
sensitive parsing.
* Fixed issue with loading readline init files (inputrc) that seems to be
triggered by libedit.
* ``$MULTILINE_PROMPT`` now allows colors, as originally intended.
* Rectified install issue with Jupyter hook when installing with pyenv,
Jupyter install hook now repects ``--prefix`` argument.
* Fixed issue with the xonsh.ply subpackage not being installed.
* Fixed a parsing bug whereby a trailing ``&`` on a line was being ignored
(processes were unable to be started in the background)
* ``and``, ``or``, ``&&``, ``||`` have been added as subprocess logical operators,
by popular demand!
* Subprocesses may be negated with ``not`` and grouped together with parentheses.
* New framework for writing xonsh extentions, called ``xontribs``.
* Added a new shell type ``'none'``, used to avoid importing ``readline`` or
``prompt_toolkit`` when running scripts or running a single command.
* New: `sudo` functionality on Windows through an alias
* Automatically enhance colors for readability in the default terminal (cmd.exe)
on Windows. This functionality can be enabled/disabled with the
$INTENSIFY_COLORS_ON_WIN environment variable.
* Added ``Ellipsis`` lookup to ``__xonsh_env__`` to allow environment variable checks, e.g. ``'HOME' in ${...}``
* Added an option to update ``os.environ`` every time the xonsh environment changes.
This is disabled by default but can be enabled by setting ``$UPDATE_OS_ENVIRON`` to
* Added Windows 'cmd.exe' as a foreign shell. This gives xonsh the ability to source
Windows Batch files (.bat and .cmd). Calling ``source-cmd script.bat`` or the
alias ``source-bat script.bat`` will call the bat file and changes to the
environment variables will be reflected in xonsh.
* Added an alias for the conda environment activate/deactivate batch scripts when
running the Anaconda python distribution on Windows.
* Added a menu entry to launch xonsh when installing xonsh from a conda package
* Added a new ``which`` alias that supports both regular ``which`` and also searches
through xonsh aliases. A pure python implementation of ``which`` is used. Thanks
to Trent Mick.
* Added support for prompt toolkit v1.0.0.
* Added ``$XONSH_CACHE_SCRIPTS`` and ``$XONSH_CACHE_EVERYTHING`` environment
variables to control caching of scripts and interactive commands. These can
also be controlled by command line options ``--no-script-cache`` and
``--cache-everything`` when starting xonsh.
* Added a workaround to allow ctrl-c to interrupt reverse incremental search in
the readline shell
* Running scripts through xonsh (or running a single command with ``-c``) no
longer runs the user's rc file, unless the ``--login`` option is specified.
Also avoids loading aliases and environments from foreign shells, as well as
loading bash completions.
* rc files are now compiled and cached, to avoid re-parsing when they haven't
changed. Scripts are also compiled and cached, and there is the option to
cache interactive commands.
* Left and Right arrows in the ``prompt_toolkit`` shell now wrap in multiline
* Regexpath matching with backticks, now returns an empty list in python mode.
* Pygments added as a dependency for the conda package
* Foreign shells now allow for setting exit-on-error commands before and after
all other commands via the ``seterrprevcmd`` and ``seterrpostcmd`` arguments.
Sensinble defaults are provided for existing shells.
* PLY is no longer a external dependency but is bundled in xonsh/ply. Xonsh can
therefore run without any external dependencies, although having prompt-toolkit
* Provide better user feedback when running ``which`` in a platform that doesn't
provide it (e.g. Windows).
* The lexer now uses a custom tokenizer that handles regex globs in the proper
* Fixed bug with loading prompt-toolkit shell < v0.57.
* Fixed bug with prompt-toolkit completion when the cursor is not at the end of
the line.
* Aliases will now evaluate enviornment variables and other expansions
at execution time rather than passing through a literal string.
* Fixed environment variables from os.environ not beeing loaded when a running
a script
* The readline shell will now load the inputrc files.
* Fixed bug that prevented `source-alias` from working.
* Now able to ``^C`` the xonfig wizard on start up.
* Fixed deadlock on Windows when runing subprocess that generates enough output
to fill the OS pipe buffer.
* Sourcing foreign shells will now return a non-zero exit code if the
source operation failed for some reason.
* Fixed PermissionError when running commands in directories without read permissions
* Prevent Windows fixups from overriding environment vars in static config
* Fixed Optional Github project status to reflect added/removed files via git_dirty_working_directory()
* Fixed xonsh.exe launcher on Windows, when Python install directory has a space in it
* Fixed `$CDPATH` to support `~` and environments variables in its items
* Added new valid ``$SHELL_TYPE`` called ``'best'``. This selects the best value
for the concrete shell type based on the availability on the user's machine.
* New environment variable ``$XONSH_COLOR_STYLE`` will set the color mapping
for all of xonsh.
* New ``XonshStyle`` pygments style will determine the approriate color
mapping based on ``$XONSH_COLOR_STYLE``. The associated ``xonsh_style_proxy()``
is intended for wrapping ``XonshStyle`` when actually being used by
* The functions ``print_color()`` and ``format_color()`` found in ````
dispatch to the approriate shell color handling and may be used from
* ```` flag now denotes if pygments is installed and
available on the users system.
* The ``ansi_colors`` module is now availble for handling ANSI color codes.
* ``?`` and ``??`` operator output now has colored titles, like in IPython.
* ``??`` will syntax highlight source code if pygments is available.
* Python mode output is now syntax highlighted if pygments is available.
* New ``$RIGHT_PROMPT`` environment variable for displaying right-aligned
text in prompt-toolkit shell.
* Added ``!(...)`` operator, which returns an object representing the result
of running a command. The truth value of this object is True if the
return code is equal to zero and False otherwise.
* Optional dependency on the win_unicode_console package to enable unicode
support in cmd.exe on Windows. This can be disabled/enabled with the
``$WIN_UNICODE_CONSOLE`` environment variable.
* Updated ``$SHELL_TYPE`` default to ``'best'``.
* Shell classes are now responsible for implementing their own color
formatting and printing.
* Prompt coloring, history diffing, and tracing uses new color handling
* New ``Token.Color`` token for xonsh color names, e.g. we now use
``Token.Color.RED`` rather than ``Token.RED``.
* Untracked files in git are ignored when determining if a git workdir is
is dirty. This affects the coloring of the branch label.
* Regular expression globbing now uses ``re.fullmatch`` instead of
``re.match``, and the result of an empty regex glob does not cause the
argument to be deleted.
* The ```` mapping has been axed, along with all
references to it. This may cause a problem if you were using a raw color code
in your xonshrc file from ``$FORMATTER_DICT``. To fix, simply remove these
* Multidimensional slicing, as in numpy, no longer throws SyntaxErrors.
* Some minor zsh fixes for more platforms and setups.
* The ``BaseShell.settitle`` method no longer has its commands captured by
* ``trace`` alias added that enables users to turn on and off the printing
of source code lines prior to their execution. This is useful for debugging scripts.
* New ability to force callable alias functions to be run in the foreground, i.e.
the main thread from which the function was called. This is useful for debuggers
and profilers which may require such access. Use the ``xonsh.proc.foreground``
decorator on an alias function to flag it. ``ForegroundProcProxy`` and
``SimpleForegroundProcProxy`` classes have been added to support this feature.
Normally, forcing a foreground alias is not needed.
* Added boolean ``$RAISE_SUBPROC_ERROR`` environment variable. If true
and a subprocess command exits with a non-zero return code, a
CalledProcessError will be raised. This is useful in scripts that should
fail at the first error.
* If the ``setproctitle`` package is installed, the process title will be
set to ``'xonsh'`` rather than the path to the Python interpreter.
* zsh foreign shell interface now supported natively in xonsh, like with Bash.
New ``source-zsh`` alias allows easy access to zsh scripts and functions.
* Vox virtual environment manager added.
* The ``foreign_shell_data()`` keyword arguments ``envcmd`` and ``aliascmd``
now default to ``None``.
* Updated alias docs to pull in usage from the commands automatically.
* Hundreds of bugs related to line and column numbers have been addressed.
* Fixed path completion not working for absolute paths or for expanded paths on Windows.
* Fixed issue with hg dirty branches and $PATH.
* Fixed issues related to foreign shell data in files with whitespace in the names.
* Worked around bug in ConEmu/cmder which prevented ``get_git_branch()``
from working in these terminal emulators on Windows.
* New configuration utility 'xonfig' which reports current system
setup information and creates config files through an interactive
* Toolkit for creating wizards now available
* timeit and which aliases will now complete their arguments.
* $COMPLETIONS_MENU_ROWS environment variable controls the size of the
tab-completion menu in prompt-toolkit.
* Prompt-toolkit shell now supports true multiline input with the ability
to scroll up and down in the prompt.
* The xonfig wizard will run on interactive startup if no configuration
file is found.
* BaseShell now has a singleline() method for prompting a single input.
* Environment variable docs are now auto-generated.
* Prompt-toolkit shell will now dynamically allocate space for the
tab-completion menu.
* Looking up nonexistent environment variables now generates an error
in Python mode, but produces a sane default value in subprocess mode.
* Environments are now considered to contain all manually-adjusted keys,
and also all keys with an associated default value.
* Removed ``xonsh.ptk.shortcuts.Prompter.create_prompt_layout()`` and
``xonsh.ptk.shortcuts.Prompter.create_prompt_application()`` methods
to reduce portion of xonsh that forks prompt-toolkit. This may require
users to upgrade to prompt-toolkit v0.57+.
* First prompt in the prompt-toolkit shell now allows for up and down
arrows to search through history.
* Made obtaining the prompt-toolkit buffer thread-safe.
* Now always set non-detypable environment variables when sourcing
foreign shells.
* Fixed issue with job management if a TTY existed but was not controlled
by the process, posix only.
* Jupyter kernel no longer times out when using foreign shells on startup.
* Capturing redirections, e.g. ``$(echo hello > f.txt)``, no longer fails
with a decoding error.
* Evaluation in a Jupyter cell will return pformatted object.
* Jupyter with redirect uncaptured subprocs to notebook.
* Tab completion in Jupyter fixed.
v0.2.1 - v0.2.4
You are reading the docs...but you still feel hungry.
* Rich history recording and replaying
* Naturally typed environment variables
* Inherits the environment from BASH
* Uses BASH completion for subprocess commands
* Regular expression filename globbing
* Its own PLY-based lexer and parser
* xonsh code parses into a Python AST
* You can do all the normal Python things, like arithmetic and importing
* Captured and uncaptured subprocesses
* Pipes, redirection, and non-blocking subprocess syntax support
* Help and superhelp with ? and ??
* Command aliasing
* Multiline input, unlike ed
* History matching like in IPython
* Color prompts
* Low system overhead
The before times, like 65,000,000 BCE.
.. _devguide:
Developer's Guide
Welcome to the xonsh developer's guide! This is a place for developers to
place information that does not belong in the user's guide or the library
reference but is useful or necessary for the next people that come along to
develop xonsh.
.. note:: All code changes must go through the pull request review procedure.
Pull requests will often have CHANGELOG entries associated with. However,
to avoid excessive merge conflicts, please follow the following procedure:
1. Go into the ``news/`` directory,
2. Copy the ``TEMPLATE.rst`` file to another file in the ``news/`` directory.
We suggest using the branchname::
$ cp TEMPLATE.rst branch.rst
3. Add your entries as a bullet pointed lists in your ``branch.rst`` file in
the appropriate category. It is OK to leave the ``None`` entries for later
4. Commit your ``branch.rst``.
Feel free to update this file whenever you want! Please don't use someone
else's file name. All of the files in this ``news/`` directory will be merged
automatically at release time. The ``None`` entries will be automatically
filtered out too!
Style Guide
xonsh is a pure Python project, and so we use PEP8 (with some additions) to
ensure consistency throughout the code base.
Rules to Write By
It is important to refer to things and concepts by their most specific name.
When writing xonsh code or documentation please use technical terms
appropriately. The following rules help provide needed clarity.
* User-facing APIs should be as generic and robust as possible.
* Tests belong in the top-level ``tests`` directory.
* Documentation belongs in the top-level ``docs`` directory.
* Code must have associated tests and adequate documentation.
* User-interaction code (such as the Shell class) is hard to test.
Mechanism to test such constructs should be developed over time.
* Have *extreme* empathy for your users.
* Be selfish. Since you will be writing tests you will be your first user.
Python Style Guide
xonsh uses `PEP8`_ for all Python code. The following rules apply where `PEP8`_
is open to interpretation.
* Use absolute imports (``import``) rather than explicit
relative imports (``import .tools``). Implicit relative imports
(``import tools``) are never allowed.
* Use ``'single quotes'`` for string literals, and
``"""triple double quotes"""`` for docstrings. Double quotes are allowed to
prevent single quote escaping, e.g. ``"Y'all c'mon o'er here!"``
* We use sphinx with the numpydoc extension to autogenerate API documentation. Follow
the `numpydoc`_ standard for docstrings.
* Simple functions should have simple docstrings.
* Lines should be at most 80 characters long. The 72 and 79 character
recommendations from PEP8 are not required here.
* All Python code should be compliant with Python 3.4+. At some
unforeseen date in the future, Python 2.7 support *may* be supported.
* Tests should be written with nose using a procedural style. Do not use
unittest directly or write tests in an object-oriented style.
* Test generators make more dots and the dots must flow!
You can easily check for style issues, including some outright bugs such
as mispelled variable names, using pylint. If you're using Anaconda you'll
need to run "conda install pylint" once. You can easily run pylint on
the edited files in your uncommited git change::
$ pylint $(git status -s | awk '/\.py$$/ { print $$2 }' | sort)
If you want to lint the entire code base run::
$ pylint $(find tests xonsh -name \*.py | sort)
How to Test
If you want to run your "work in progress version" without installing
and in a fresh environment you can use Docker. If Docker is installed
you just have to run this::
$ python
This will build and run the current state of the repository in an isolated
container (it may take a while the first time you run it). There are two
additionals arguments you can pass this script.
* The version of python
* the version of ``prompt_toolkit``
$ python 3.4 0.57
Ensure your cwd is the root directory of the project (i.e., the one containing the
.git directory).
Prep your environment for running the tests::
$ pip install requirements-tests.txt
Running the Tests - Basic
Run all the tests using Nose::
$ nosetests -q
Use "-q" to keep nose from outputing a dot for every test. There are A LOT of tests
and you will waste time waiting for all the dots to get pushed through stdout.
Running the Tests - Advanced
To perform all unit tests::
$ scripts/run_tests.xsh all
If you're working on a change and haven't yet committed it you can run the
tests associated with the change. This does not require that the change
include the unit test module. This will execute any unit tests that are
part of the change as well as the unit tests for xonsh source modules in
the change::
$ scripts/run_tests.xsh
If you want to run specific tests you can specify the test names to
execute. For example to run test_aliases::
$ scripts/run_tests.xsh aliases
The test name can be the bare test name (e.g., ``aliases``), include
the ``test_`` prefix and ``.py`` suffix without the directory
(e.g., ````), or the complete relative path (e.g.,
``tests/``). For example:
Note that you can pass multiple test names in the above examples::
$ scripts/run_tests.xsh aliases environ
As before, if you want to test the xonsh code that is installed on your
system first cd into the `tests` directory then run the tests::
$ cd tests
$ env XONSHRC='' nosetests
Happy testing!
How to Document
Documentation takes many forms. This will guide you through the steps of
successful documentation.
No matter what language you are writing in, you should always have
documentation strings along with you code. This is so important that it is
part of the style guide. When writing in Python, your docstrings should be
in reStructured Text using the `numpydoc`_ format.
Auto-Documentation Hooks
The docstrings that you have written will automatically be connected to the
website, once the appropriate hooks have been setup. At this stage, all
documentation lives within xonsh's top-level ``docs`` directory.
We uses the sphinx tool to manage and generate the documentation, which
you can learn about from `the sphinx website <>`_.
If you want to generate the documentation, first xonsh itself must be installed
and then you may run the following command from the ``docs`` dir:
.. code-block:: bash
~/xonsh/docs $ make html
For each new
module, you will have to supply the appropriate hooks. This should be done the
first time that the module appears in a pull request. From here, call the
new module ``mymod``. The following explains how to add hooks.
Python Hooks
Python documentation lives in the ``docs/api`` directory.
First, create a file in this directory that represents the new module called
The ``docs/api`` directory matches the structure of the ``xonsh/`` directory.
So if your module is in a sub-package, you'll need to go into the sub-package's
directory before creating ``mymod.rst``.
The contents of this file should be as follows:
.. code-block:: rst
.. _xonsh_mymod:
My Awesome Module -- :mod:`xonsh.mymod`
.. currentmodule:: xonsh.mymod
.. automodule:: xonsh.mymod
This will discover all of the docstrings in ``mymod`` and create the
appropriate webpage. Now, you need to hook this page up to the rest of the
Go into the ``index.rst`` file in ``docs/xonsh`` or other subdirectory and add
``mymod`` to the appropriate ``toctree`` (which stands for table-of-contents
tree). Note that every sub-package has its own ``index.rst`` file.
Building the Website
Building the website/documentation requires the following dependencies:
#. `Sphinx <>`_
#. `Cloud Sphinx Theme <>`_
Procedure for modifying the website
The xonsh website source files are located in the ``docs`` directory.
A developer first makes necessary changes, then rebuilds the website locally
by executing the command::
$ make html
This will generate html files for the website in the ``_build/html/`` folder.
The developer may view the local changes by opening these files with their
favorite browser, e.g.::
$ google-chrome _build/html/index.html
Once the developer is satisfied with the changes, the changes should be
committed and pull-requested per usual. Once the pull request is accepted, the
developer can push their local changes directly to the website by::
$ make push-root
Branches and Releases
Mainline xonsh development occurs on the ``master`` branch. Other branches
may be used for feature development (topical branches) or to represent
past and upcoming releases.
All releases should have a release candidate ('-rc1') that comes out 2 - 5 days
prior to the scheduled release. During this time, no changes should occur to
a special release branch ('vX.X.X-release').
The release branch is there so that development can continue on the
develop branch while the release candidates (rc) are out and under review.
This is because otherwise any new developments would have to wait until
post-release to be merged into develop to prevent them from accidentally
getting released early.
As such, the 'vX.X.X-release' branch should only exist while there are
release candidates out. They are akin to a temporary second level of staging,
and so everything that is in this branch should also be part of master.
Every time a new release candidate comes out the vX.X.X-release should be
tagged with the name 'X.X.X-rcX'. There should be a 2 - 5 day period of time
in between release candidates. When the full and final release happens, the
'vX.X.X-release' branch is merged into master and then deleted.
If you have a new fix that needs to be in the next release candidate, you
should make a topical branch and then pull request it into the release branch.
After this has been accepted, the topical branch should be merged with
master as well.
The release branch must be quiet and untouched for 2 - 5 days prior to the
full release.
The release candidate procedure here only applies to major and minor releases.
Micro releases may be pushed and released directly without having a release
When releasing xonsh, make sure to do the following items in order:
1. Review **ALL** issues in the issue tracker, reassigning or closing them as
2. Ensure that all issues in this release's milestone have been closed. Moving issues
to the next release's milestone is a perfectly valid strategy for
completing this milestone.
3. Perform maintenance tasks for this project, see below.
4. Write and commit the release notes.