Git user manual
Daily work flow
Keeping up to date with the master branch
In order to keep up to date with the latest development, you should frequently merge the master branch into your own branch. Make sure you are in your own nickname branch (if not do
git checkout nickname first), then:
git pull origin master
which means: "merge the branch
master from the location
origin into the current branch".
pull command is actually a combination of
git fetch, which fetches all new information from the remote repository on the server, and
git merge origin/master, which will try to merge changes from the master branch into your branch. The branch
origin/master is a local branch which refers to the remote
master branch on
origin, it is your local view of that branch and this information is updated by the
git fetch command.
If you work from multiple repositories on different computers, you should also keep your own branch in sync as well with a single
git pull, as the default is to pull the branch of the same name. In summary, for such a typical development setup, the commands you would run are:
git pull git merge origin/master
IMPORTANT Some people have globally set up git to do non-fast-forward merges. If this is the case then you should deactivate this for Molcas, as our merging system will see your commit as different, even though you did not make any modifications.
DO NOT merge other people's branches into your own, since that will make your branch dependent on possibly unstable code, and you cannot undo that (reverting a merge is absolutely not allowed). Use separate "nickname-stuff" branches for trying out these things. Only put your own changes into your nickname branch or merge origin/master (stable, verified code).
It can happen that two branches have started to diverge since the latest origin/master version. In that case, you will be notified by the conflict, and the proper thing to do is to wait for another update to master, and then merge
origin/master as usual and resolve any conflicts (see further down). So you shouldn't merge other people's branches directly. Technically, merging other developer's branches isn't a problem, as long as you don't try to revert it, because that causes changes in the original branch to disappear (if you want to know why, read up on git and reverting merges here). It is highly recommended to coordinate your development as much as possible with other developers, because that avoids conflicts in the first place.
Making and uploading new changes
When you are finished making any kind of changes and tested them properly, prepare to record them by:
git add path/to/file(s) # for any added/modified files git rm path/to/file(s) # for any deleted files git commit -m "informative message about the changes"
Any changes are allowed, but be considerate about making changes that affect other developers (if you change e.g. their module, it might be good to inform them about it). Additionally, you should NEVER COMMIT BINARY FILES!!! Especially large binary files could end up being stuck in the git history and cannot be removed from the git objects unless the history is rewritten, which requires every developer to re-clone the repository. Think before you do such things!
Now, your changes should have been be recorded. You can check this by showing a summary of the things that changed in the last commit in the branch:
IMPORTANT Check your commits before uploading them!! This way you avoid adding erroneous files or garbage to commits. When you have recorded one or more changes, and you are confident that the commits are fine, you can send them to central repository with:
Pushing your nickname branch is final, meaning that once you pushed your changes, they will be merged at night into a daily snapshot and if testing succeeds will be integrated into the master branch. If you made a mistake, you have to make a new commit and push it.
Therefore, it's a very good practice to run tests locally before pushing. For experimental stuff that you wish to share with others, use a branch called
NOTE When creating new tests/checks, you have to do this in a separate commit. Imagine you make a change but do not commit it, then you run make. The build will now be labeled as "dirty", since there are changes in the work tree that have not been committed. If you use binaries built that way, there is no way to know later on what version the test/check was run. Therefore, first commit any new changes that affect a test, then build, then rerun the check file generation, and then commit that. The police script will warn about this.
Establishing a repository on your own remote server
NOTE: this section is NOT about moving your repository over to a new computer, to do that just follow the setup steps again but instead of generating a new ssh key pair, copy the keypair over to your new computer.
It can happen that you need to use/test your branch on another computer than where you do your development. The easiest example is e.g. a cluster, where you have access to other compilers and want to run Molcas in a parallel environment which is not available locally. What you should certainly avoid, is to duplicate your private key on many computers so you can clone the central repository there. There is a much safer and easier way to set up a repository on a remote computer that you can control from your main development computer.
In the following example setup, we will assume the remote computer is
alarik.lunarc.lu.se (one of the computer clusters in Lund).
- make sure you have SSH access to the remote computer (should already be the case)
- log into the remote computer and create an empty, bare repository:
git init --bare ~/molcas.git
- on your local computer, add the remote repository:
git remote add alarik alarik.lunarc.lu.se:~/molcas.git
- push your branch to the remote:
git push alarik
- log into the remote computer and clone a non-bare repository:
git clone -b nickname ~/molcas.git ~/molcas
If the branch does not exist yet on the remote side, git will propose the correct command to create it.
The typical steps that are needed to keep the remote in sync with your local development are:
- On the local side:
git push alarik
- On the remote side:
Basic git commands
Show what's going on
Showing the branch you are on:
To see the status (files changed, etc.) of the branch you are currently in:
This command is arguably the most useful command, as verifying the branch that you currently checked out as well as looking at the state of the work tree are frequently needed. You should use this command a lot!
To see an overview of changes:
git log git log --oneline --decorate --graph git log --oneline --decorate --graph --all # show graph of all branches git log --oneline --decorate --date-order --all # show chronological changes in all branches
Since these commands are very long, you can set git aliases for them as follows:
git config --global alias.lg 'log --oneline --decorate --graph' git config --global alias.la 'log --oneline --decorate --graph --all'
Then the following commands can be used:
git lg git la
The following command gives a complete Molcas log, i.e., a chronological ordered list of all changes in the master branch (the stable Molcas tree), both in default and a custom format:
git log --date-order origin/master git log --date-order --format="%h (%ai)%n Author: %an%n Title: %s" origin/master
To get the same for all changes eligible for inclusion into the master branch (all pushed changes), i.e. a way to represent the current "development" version of Molcas, just ask for the same for all valid remote branches (those not including a dash), both in default and a custom format:
git log --date-order --remotes=origin/[^-]* git log --date-order --format="%h (%ai)%n Author: %an%n Title: %s" --remotes=origin/[^-]*
To get information on patches you applied but haven't been included yet in the main Molcas branch, you can log just those changes by giving a range to the
log command. First, checkout your branch and fetch the latest changes, then:
git log --oneline --decorate --graph origin/master..nickname
where nickname is of course your own branch name. Commits with a
FAILED-... tag are snapshots that failed verification.
If you prefer using a GUI, you can install gitk if it's not already installed, and run:
The first command will display all commits leading to your current work tree. The second will display all commits available, including those in other branches which are not yet merged with the master.
Molcas police / pre-commit
Every time you commit something in your local git tree, a special script named
pre-commit located in
.git/hooks is run. This script currently runs the Molcas
police script, which checks for compliance with Molcas internal rules. If
police finds any problem it will be reported and, depending on its gravity, the commit may be cancelled until you fix it.
pre-commit script also prevents your committing changes to one of the "protected" branches (
daily-snapshot), which you would not be able to push to the server anyway. If you want to publish your changes, commit them to your personal
Note that the
.git/hooks/pre-commit script is overwritten when running
cmake, and any local changes you may have applied to it will be lost.
NOTE: If you do not configure or install Molcas, you have to manually copy the script to the hooks directory:
cp sbin/pre-commit .git/hooks/
Finding out when a bad change was introduced
In order to find out when a bad change was introduced, you can use git's bisect algorithm to close in on the bad change, if you know a previous version that was fine. To start the procedure, run:
git bisect start [bad commit] [good commit]
Then, you can start the automated procedure by providing a command that exits with returncode 1 in case of trouble. There is a small problem though: imagine that you are looking for a failing test, but some commits along the way were rubbish and they can't be tested. In that case you wish to skip that commit, and your script should give exit code 125. So, in summary, create a script that returns 0 for a good commit, 1 for a bad commit, and 125 for a commit you cannot reliably test. For example, we wish to skip commits that cannot be built, and we are looking for failing test 227:
git bisect run bash -c "make || exit 125; molcas verify 227 || exit 1"
after which you will then drop out in the last known good snapshot. The hash of the first bad commit will be printed on the command line.
Example 1: You checked out your branch and
molcas verify 046 fails. In the master branch however, the test passes. So, somewhere you have introduced a problem and you want to find out when.
You can now set up the bisect procedure with the knowledge that the bad commit is you current branch and the good commit is the master branch:
git bisect start nickname origin/master
The command to determine if there is a failure or not is in this case simply
molcas verify 046, but we also check that make doesn't fail, and if it does we mark the commit also as bad:
git bisect run bash -c "make || exit 1; molcas verify 046 || exit 1"
To end the procedure and go back to where you were just do
git bisect reset
Example 2: You checked out your branch and
molcas verify 227 fails in parallel. In the master branch however, the test passes. So, somewhere you have introduced a problem and you want to find out when. You are using CMake and your installation is in a different directory.
You can now set up the bisect procedure with the knowledge that the bad commit is you current branch and the good commit is the master branch:
git bisect start nickname origin/master
The command to determine if there is a failure or not is more complicated, so for convenience we will create a script that will handle it for us, called
#!/bin/bash export MOLCAS_NPROCS=1 cd /path/to/cmake/builddir make -j 6 || exit 125 export MOLCAS_NPROCS=2 molcas verify 227 || exit 1
The script of course assumes that you've already configured and built a Molcas installation at
/path/to/cmake/builddir. We skip failed
make by exiting with code 125, and we run the test in parallel. We then start the automated procedure with:
chmod +x test227-bisect.sh git bisect run test227-bisect.sh
To end the procedure and go back to where you were just do
git bisect reset
Resetting local changes
If you wish to abandon any (unrecorded) changes you made to tracked files and want to go back to the latest recorded state, use
git reset --hard
This will set all files in your work tree back to how they were before you modified them. This also means that any modifications will be lost!
You can go even further and reset your branch to a different commit, thereby abandoning even recorded changes. For example, you have committed a change but you decide that you want to get rid of it entirely. To reset your branch back to the previous commit, you would do:
git reset --hard HEAD^ git reset --hard <commit-hash-before-boo-boo>
Here we gave
HEAD^ as a revision specification. It is a relative revision meaning the parent of the commit that is currently checkout out. We could also provide an absolute revision specification such as e.g. the commit hash. For more information on providing relative revisions read
However, if you already pushed, even though you can reset your local branch, it's too late to reset your server-side branch since the server will reject any new push if it is not able to fast-forward your branch. If you really would like your branch fixed in that case, you'll have to contact one of the Molcas git administrators and ask them to hard-reset your branch to the same commit you reset to locally. After that, you will be able to push your (new) changes. In any case, it's a very good idea to not push too soon, as you'll be able to fix things more easily locally.
You can also soft reset a branch, which is actually the default behavior of
git reset. This will do the same as the hard reset above for the branch, but the files in the work-tree will not be touched. That means that any change you made will still be in the files and will show up as modifications. This is useful e.g. if you find out that you made a commit but forgot to include a change:
git reset HEAD^
Now, any files that were changed by the last commit will show up as modified. We can make further changes now until we are satisfied with our modifications. Don't forget to use
git add just as you would normally, and then with
git commit you make a new commit.
git reset command is very useful and powerful, especially since it is easy to commit e.g. unwanted debug print statements or mess up a merge. Be careful though, because once you have pushed a change, resetting and fixing is not possible any more since you cannot overwrite your branch. Therefore, it's a good idea to get your local branch into a nice shape before deciding to push it to origin.
Looking at one of the daily snapshots
If you need to look at the source code in one of the daily snapshots, you can either checkout the tagged version number directly, use one of the special branches
git checkout origin/daily-snapshot git checkout origin/master git checkout v8.1.160121-1800
(Note that older versions have a slightly different formatted tag, but otherwise the idea is the same.)
These command will checkout the respective snapshot version and will inform you that you are in a detached HEAD state. This is fine, as you shouldn't modify any of these branches directly anyway. If you really want a local branch for experimentation, it's better to create a new temporary branch, e.g.:
git checkout -b tmp origin/daily-snapshot ... do stuff ... git checkout nickname git branch -D tmp
After you're done with whatever is is you were doing, just checkout your own branch and then use
git branch -D tmp to delete the temporary branch.
Last changes to each line in a file
To see the last change that was made to each line of a file you currently have in your work tree, you can use the
git blame command, e.g.:
git blame src/rassi/prprop.f
will show the contents of the file
prprop.f with at the beginning of each line the name of the author of the change and the date that line was last changed. The commit hash is also included.
To show a log of commits only related to a specific file, you can use the
git log command with an optional file name after a double dash, e.g.:
git log --oneline -- src/rassi/prprop.f
To see the changes made to a file between two revisions, you can use the
git diff command:
git diff <hash> -- src/rassi/prprop.f
This will show you the difference since the commit with
<hash> till now.
For more options, please consult the
diff man pages.
Checking for changes in other branches not yet in master
If you are planning to make changes to a file, it can be good to check first if someone has already modified the file before it's included in the stable master branch. This way, you can avoid merge conflicts later on. Make sure you are in your branch, and that the latest master version has been merged, then use
git checkout nickname && git fetch && git merge origin/master git log --no-merges --remotes ^HEAD -- path/to/file
which means: look for all remote branches, but exclude merges and exclude any commits reachable from where I currently am. This will list all changes to that file that have not been applied in your work tree. If this command results in output, it means someone has been working on that file and you do not have those changes in your own branch, so if you change something a merge conflict is imminent. Best is to wait until the changes are applied in the master branch and then merge the master branch.
Checking if my latest changes made it into master
git checkout nickname && git fetch && git merge origin/master git log --no-merges origin/master..HEAD
Or, using the GUI:
git fetch gitk --all
and see if there is a line connecting
remotes/origin/master. If you pushed your changes a long time ago, this may be difficult to see. In this case, first update your branch:
git checkout nickname && git fetch && git merge origin/master && git push gitk --all
If your changes are not in the master, it may be that the tests have not finished yet or have failed. Check the tests page (possibly for previous days) and your mailbox (possibly in the spam folder).
Temporarily stashing away uncommitted changes
Any git command that modifies the work tree will fail if there are (uncommitted) modifications to files in the work tree that would be overwritten by the command. That is why most operations like checking out branches, merging branches, applying patches, or any other operation that affects the work tree should be carried out on a clean work tree, i.e. without any modified (tracked) files.
It can happen that you are working on something and you quickly need to apply some patch, merge origin/master, or make any other unrelated modification. In this case, you can temporarily stash away modification you made, do your stuff, and then put the modifications back. Suppose you wish to merge origin/master, but you already started to make some modifications:
... in the middle of lots of stuff ... git stash git merge origin/master git stash pop ... continue work ...
You can stash multiple times. Stashing works like a stack:
git stash puts modifications on top of the stack and removes them from the work tree,
git stash pop applies the last modifications on the stack back in the work tree and removes them from the stack. Read
man git-stash to find out more.
Creating a tarball with the source of a particular version
Sometimes you want to build several versions of Molcas, e.g. if you are trying to solve a bug and you want to compare two different versions, one which works and one which doesn't. In this example, we will assume that the version tagged with
master-13-12-05 works, and the one tagged
master-14-01-10 doesn't. You now wish to build those two particular version of Molcas. An easy way to do this is to extract the source of those two version from git with the archive command:
git archive --prefix=molcas-works/ -o molcas-works.tar.gz master-13-12-05 git archive --prefix=molcas-wrong/ -o molcas-wrong.tar.gz master-14-01-10
--prefix option lets you set a directory prefix to which the tarball will be unpacked, the
-o option lets you set the filename of the tarball. Git can infer from the extension that this is to be a compressed tar archive.
After running the above commands, you now have two archives which you can then unpack somewhere else and build Molcas from.
To be really fancy, you can streamline the above procedure using the same name for the tarball as you where using to indicate the commit (with either a tag or a hash), e.g.:
for commit in master-13-12-05 v8.1.14-08-20 1c6548a; do git archive --prefix=$commit/ -o $commit.tar.gz $commit; done
after which you have three tarballs of the snapshots
The above procedure is now automated and you can simply use:
molcas export master-13-12-05
to make an archive of the source of that version. Without any arguments, you get the source of the current HEAD.
Dealing with failed verification
The daily testing machines will notify you when your changes have introduced a verification failure. The e-mail contains a digest of the auto.log from the machines where the failure occurred. Also, the daily snapshot as well as any parent commit that introduced a problem will be tagged with a
FAILED-$date-$number tag. The only thing you need to do is fix the problem, commit the change and push it.
If the problem that was introduced is more complicated and you decide you need quite some time to solve it, you should NOT PUSH other stuff to your branch. If changes are pushed in a branch, it is merged again the next day. In this case this is meaningless, as you branch will continue to make the testing system fail. Of course, you can still locally commit changes without pushing, so be a good developer and keep stuff local until you have solved ALL your problematic tests.
If for some reason, you mistakenly pushed something experimental that you did not intend to have in the main developer version, you can just undo the changes by making a new commit and push it, or try
If you have made a lot of changes and you don't know which commit is responsible for the failures, you can use
git bisect to find the offending commit. Start by identifying the bad commit in your branch, this info will be in the email you receive. Suppose the hash starts with
8de6f12 (you don't need the rest normally), and there was one test that failed, test 074. The
git bisect start command takes a bad commit, and then a good commit as arguments. The last good commit is always origin/master:
git fetch git bisect start 8de6f12 origin/master git bisect run bash -c "make || exit 0; molcas verify 074 || exit 1"
The reason a failure with
make counts as pass is that you generally are not interested in intermediate commits that made the build fail, only commits that can actually be tested.
If you need to narrow down the search, e.g. you worked locally on some feature in caspt2 for months and then some tests fail in caspt2, you could specify you only want to look at files in
git fetch git bisect start <bad hash> <good hash> -- src/caspt2 git bisect run bash -c "make || exit 0; molcas verify ... || exit 1"
Dealing with merge conflicts
It can (will) happen that when merging
origin/master, there will be conflicts. This will leave your work tree in a state where there are modified files. This means that if you would want to abort the merge at this point, you'll have to run
git reset --hard to get your work tree back in the state before you started the merge.
In order to resolve the conflicts, you'll have to deal with the conflicting files, add them to the index, and then commit. The modified files that had a conflict will show up under Unmerged when you run
git status On branch tmp You have unmerged paths. (fix conflicts and run "git commit") Changes to be committed: renamed: Test/check/test823.check -> Test/check/test030.check renamed: Test/check/test832.check -> Test/check/test094.check deleted: Test/check/test103.check modified: Test/check/test184.check ... Unmerged paths: (use "git add <file>..." to mark resolution) both modified: Test/check/test052.check both modified: Test/check/test367.check
In the example above, there were two files with conflicts showing up under Unmerged. They are listed as both modified, meaning that the conflict is the result of diverging modifications on the common base. One can open the files with any editor and go looking for
>>>> indicating your changes and those present in the branch you are merging. You should edit the files to the point they look exactly as you want them. After that you can run
git add ... on those files, and finally
git commit. It can happen that the last step fails because there are (other) files that violate the rules, and you'll get a police log. If the problems are not caused by you, simply do
git commit --no-verify to avoid the cops.
Combining changes from different branches
It can be a bit daunting to interpret the special lines git has added to the unmerged files, and it is very easy to forget to remove the
>>>> lines. In order to help out with solving merge conflicts, I therefore recommend you to use a merge tool, i.e. a (graphical) diff program that can deal with merges. I tend to use the kdiff3 program, because it is very good at helping complex merges, but it can be a bit daunting to use. If you are a beginner, I recommend the diffuse program, which is explained below with an example. Other alternatives vary in the degree with which they support complicated merges, e.g. some popular tools are meld, xxdiff, or even
vimdiff. Please have a look at
git mergetool --tool-help to see if your favorite diff program is supported by git. If it is, you can probably find documentation online regarding merges.
So, in my case, I would launch:
git mergetool --tool=diffuse
which will go through the list of unmerged files, give you some rudimentary info about the conflict for that file, and ask you if you want to launch the external program. Just pressing enter will launch diffuse. You will be greeted by a window with four different panes. This can be different when using other diff programs, so this explanation only applies to diffuse!
Three of the panes are temporary files created by git to aid you in the merge. The actual file in your work tree is the only one that is relevant, and in diffuse this is the second pane from the left. You can verify this is your work-tree file by looking at the filename at the top of the pane. On the left of that pane, you see the version of the file as you had it in your branch, on the right you see the file as it is in the branch you are merging. If you look at the filenames of the temporary files, you'll see these are referred to as LOCAL and REMOTE respectively. On the far right you'll see the latest common version of the file, referred to as BASE in the temporary file name. That is the file as it was before changes started diverging.
The only thing you need to do now is to modify your work tree file (second pane from the left) until it looks like it is supposed to. Remember you also need to remove the special lines as well, there is not magic done later on the file, it will be committed as it is.
You can set diffuse as your default choice for merge tool with:
git config --global merge.tool diffuse
Then you only need to type
git mergetool and it will use diffuse or whatever merge tool you chose. It is a good idea to experiment extensively with it before you become comfortable with the whole merging process. Remember you can always abort the merge with
git reset --hard. If you want to play around locally, it's also a good idea to create a new branch:
git checkout stevenv git checkout -b tmp git merge origin/master git mergetool
In this case, I branched off
tmp from my own nickname branch, then merged
origin/master into it and then launched diffuse to deal with conflicts.
Selecting a version from a specific branch
In the example at the beginning of this section, two people had modified the check files, and we would like to select one of the versions instead of trying to merge the differences, as check files are autogenerated. To select our version we can do the following after we merged and looked at the status:
git checkout --ours Test/check/test052.check git checkout --ours Test/check/test367.check git add Test/check/test052.check git add Test/check/test367.check git commit
Alternatively, if you want to take the changes from the other branch that you are merging, use:
git checkout --theirs Test/check/test052.check git checkout --theirs Test/check/test367.check git add Test/check/test052.check git add Test/check/test367.check git commit
This will resolve the merge and commit the changes. After we have done this we need to generate new check files to match the code changes we introduced with the merge, and add those in a new commit.
NOTE: It is easy to mess up a merge, therefore it is a very good idea to get it right locally before pushing.
Totally clean a git repository
To totally clean your git repository, removing ALL changes you made and remove all files/directories that are not tracked by git, do:
git reset --hard git clean -x -f -d