Tag Archives: distributed version control system

How to split a git repository into two

First of all, as a disclaimer: I am not a git guru but. It seems this subject is somehow obscure and it required googling around almost one day. Well, let's get started. I assume that we have the following structure:

repo1.git/
                .git/
                directory1/
                directory2/
                directory3/

and we want:

repo2.git/
                .git/
                directory2/
repo3.git/
                .git/
                directory1/
                directory3/

Please note that only step one will be carried out on server and the rest of them on client.

Step 1. Create two new empty repositories

[codesyntax lang="bash"]

git init --bare repo2.git
git init --bare repo3.git

[/codesyntax]

Step 2. Clone the original repositories

[codesyntax lang="bash"]

git clone --no-hardlinks git@server:repo1.git repo2
git clone --no-hardlinks git@server:repo1.git repo3

[/codesyntax]

Step 3. Filter-branch and reset to exclude other files, so they can be pruned. This step is actually very time consuming. So, depending how large your repository is, you will have to go to buy a coffee or not :)

[codesyntax lang="bash"]

cd repo2/
git filter-branch --subdirectory-filter directory2 HEAD -- --all
git reset --hard
git gc --aggressive
git prune

[/codesyntax]

Step 4. Replace remote origin to point to new repo2.git and push the changes

[codesyntax lang="bash"]

git remote rm origin
git remote add origin git@server:repo2.git
git push origin master

[/codesyntax]

Step 5. Remove directory2/ from the repo3. You finished your coffee? If yes, go and buy another one :)

[codesyntax lang="bash"]

cd ../repo3
git filter-branch --index-filter "git rm -r -f --cached --ignore-unmatch directory2" --prune-empty HEAD
git reset --hard
git gc --aggressive
git prune

[/codesyntax]

Step 6. Replace remote origin to point to new repo3.git and push the changes.

[codesyntax lang="bash"]

git remote rm origin
git remote add origin git@server:repo3.git
git push origin master

[/codesyntax]

Setup Gitolite v3 and Email Notifications

The most common setting would be to configure the email hooks to send notifications on a push operation. This is not hard, however, I found the information of how to setup email notifications are pieces and pieces everywhere online, some of them are targeting Gitolite v2, and some of them are misleading. After tried some approaches, and here is what works for me.

1. Setup hook scripts

I tried

[codesyntax lang="bash"]

ln -s /usr/share/doc/git/contrib/hooks/post-receive-email /home/git/.gitolite/hooks/common/post-receive

[/codesyntax]

However, my git user doesn’t have the permission to execute this script. So, finally, I decided to simply copy the script to user folder. Don’t forget to setup Gitolite (again) after put hook scripts into place, so that Gitolite can deploy scripts to all bare repositories.

[codesyntax lang="bash"]

cp /usr/share/doc/git/contrib/hooks/post-receive-email /home/git/.gitolite/hooks/common/post-receive
chmod a+x /home/git/.gitolite/hooks/common/post-receive
$HOME/bin/gitolite setup --hooks-only

[/codesyntax]

2. Setup .gitolite.rc

Edit ~/.gitolite.rc file, and change GIT_CONFIG_KEYS as '.*',

[codesyntax lang="bash"]

vim /home/git/.gitolite.rc
:%s/'',/'.*',/
:wq

[/codesyntax]

If you skip this step, following errors will be presented when attempt to push repository specific configurations.

remote: FATAL: git config 'hooks.mailinglist' not allowed
remote: check GIT_CONFIG_KEYS in the rc file

If you installed git using my previous article about how to migrate from svn to git then you already have changed this.

3. Setup repositories

For the repositories desire to send email notifications, add

config hooks.mailinglist = 'user1@example.com, user2@example.com'
config hooks.emailprefix = '[repo_name] '

to local gitolite-admin repository as discribed in the Giolite README, then push.

Migrate from SVN to GIT - step by step tutorial

Lately I faced a new challenge. To fully migrate our SVN environment to GIT. After reading a little bit on internet I come up with this plan:

  1. Deploy a GIT server
  2. Create a new bare repository
  3. Make a SVN Clone on a different machine
  4. Push it on GIT server

Assumptions:

SVN Repository: https://192.168.0.5/svn/documentation

GIT Server: 192.168.0.22

Client: A Microsoft Windows machine with some IP from 192.168.0.1/24

Prerequisite packages for our client: TortoiseGit  and msysgit. For this tutorial I used: TortoiseGit-1.8.4.0-64bit.msi and msysGit-fullinstall-1.8.3-preview20130601.exe

Add C:\msysgit\msysgit\cmd to the %PATH% (I am not going to cover this here, but you can check this website: http://www.computerhope.com/issues/ch000549.htm)

1. Deploy GIT/GITOLINE/GITWEB

Server side - I assume we have a fresh debian/ubuntu installation

1.1. Install required software

[codesyntax lang="bash"]

apt-get install vim git

[/codesyntax]

1.2. Create user git with disabled password and login shell bash

[codesyntax lang="bash"]
adduser \
 --system \
 --shell /bin/bash \
 --gecos 'git version control' \
 --group \
 --disabled-password \
 --home /home/git \
 git

[/codesyntax]

1.3.  Now login with user git

[codesyntax lang="bash"]

su -l git

[/codesyntax]

1.4.  Install Gitolite

[codesyntax lang="bash"]

cd /home/git; git clone git://github.com/sitaramc/gitolite
mkdir $HOME/bin
gitolite/install -ln

[/codesyntax]

Client side

1.5. Generate RSA key (Create RSA key, after hiting the command just press enter when it ask for passphrase.). This key will be transferred to git server.

[codesyntax lang="bash"]

C:\Users\joe>C:\msysgit\msysgit\bin\ssh-keygen -t rsa -C "Git-Admin"
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa): c:\users\joe\.ssh\id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in c:\users\joe\.ssh\id_rsa.
Your public key has been saved in c:\users\joe\.ssh\id_rsa.pub.
The key fingerprint is:
aa:38:be:f9:3c:d0:87:fe:de:5a:a9:71:1d:d3:38:01 Git-Admin

[/codesyntax]

1.6. From client scp the key into git server.

[codesyntax lang="bash"]

C:\msysgit\msysgit\bin\scp C:\Users\joe\.ssh\id_rsa.pub root@192.168.0.22:~

[/codesyntax]

Server side

1.7. Now setup the Git Admin. Please note that we must be logged in with git user. To check this use whoami command

[codesyntax lang="bash"]

whoami
sudo mv /root/id_rsa.pub /home/git/Git-Admin.pub; sudo chown git:git /home/git/Git-Admin.pub
bin/gitolite setup -pk Git-Admin.pub

[/codesyntax]

1.8. Git user should be in the same group as your Apache user

[codesyntax lang="bash"]

sudo usermod -a -G www-data git
sudo chmod ug+rx /home/git/repositories

[/codesyntax]

Client side

1.9. Clone gitolite-admin repository. Now gitolite-admin directory will be present after cloneing from git server. The gitolite.conf is mainly use for User and group ACL for git server. Now from your system you can easily manage the user group ACL. But for this you have to do git push.

[codesyntax lang="bash"]

git clone git@192.168.0.22:gitolite-admin.git
cat gitolite-admin/conf/gitolite.conf

[/codesyntax]

repo gitolite-admin

    RW+     =   Git-Admin
 
repo testing
    RW+     =   @all

Server side

1.10. Edit the /home/git/.gitolite.rc and change the UMASK to 0002 and GIT_CONFIG_KEYS to '.*'

[codesyntax lang="bash"]

vim /home/git/.gitolite.rc
:%s/0077/0002/
:%s/'',/'.*',/
:wq

[/codesyntax]

1.11. Install gitweb

[codesyntax lang="bash"]

sudo apt-get install gitweb

[/codesyntax]

1.12. Configure gitweb. Edit /usr/lib/cgi-bin/gitweb.cgi and change the value of $projectroot as "/home/git/repositories" and $projects_list as "/home/git/projects.list"

[codesyntax lang="bash"]

sudo vim /usr/lib/cgi-bin/gitweb.cgi
:%s/our $projectroot = "\/pub\/git";/our $projectroot = "\/home\/git\/repositories\/";/
:%s/our $projects_list = "";/our $projects_list = "\/home\/git\/projects.list";/
:wq

[/codesyntax]

[codesyntax lang="bash"]

sudo vim /etc/gitweb.conf
:%s/$projectroot = "\/var\/cache\/git";/$projectroot = "\/home\/git\/repositories\/";/
:%s/$projects_list = $projectroot;/$projects_list = "\/home\/git\/projects.list";/
:wq

[/codesyntax]

Note: if the $projects_list is commented, uncomment it (remove the # from the begining of the line)

2. Migrate from SVN to GIT

Server side

2.1. Create a new git bare repository

[codesyntax lang="bash"]

cd /home/git/repositories/
git init --bare documentation.git

[/codesyntax]

Client side

2.2. Getting the author information.

[codesyntax lang="bash"]

svn log --xml | grep -P "^<author" | sort -u | perl -pe 's/<author>(.*?)<\/author>/$1 = /' > users.txt

[/codesyntax]

That gives you the log output in XML format - you can look for the authors, create a unique list, and then strip out the XML. (Obviously this only works on a machine with grep, sort, and perl installed.) Then, redirect that output into your users.txt file so you can add the equivalent Git user data next to each entry. The users.txt file should look like:

(no author) = Alex <alex@example.com>
bear = Bear <kong@example.com>
monika = Monika <cristina@example.com>
igor = Igor <igor@example.com>
joe = Joe <joe@example.com>

You must execute this command on SVN repository folder. If you are using Windows, then the previous command will not work, so I suggest the following approach.

* Generate the file (svn_users.txt), change it as you like and trasfer it to a linux box (I use GIT server for this)

* Connect to GIT server and process the file there using:

[codesyntax lang="bash"]

cat svn_users.txt | grep -P "^<author" | sort -u | perl -pe 's/<author>(.*?)<\/author>/$1 = /' > users.txt
rm svn_users.txt

[/codesyntax]

* Transfer the file users.txt to Windows machine.

2.3. Add the new repo that we want to migrate to gitolite-admin\conf\gitolite.conf. The file should look something like this:

repo gitolite-admin
    RW+     =   Git-Admin
 
repo testing
    RW+     =   @all
 
repo documentation
    RW+     =   @all

At this point all users will have RW access to our git repository.

2.4. Add new users to GIT. For each user you will need to have to a pair of public/private keys. Copy public keys to keydir folder in gitolite-admin clone, add and push the changes.

2.5. Import SVN repository. This is going to take a while, so go and grab a snack or something.

[codesyntax lang="bash"]

git svn clone https://192.168.0.5/svn/documentation --authors-file=users.txt --no-metadata

[/codesyntax]

2.6. The last thing to do is add the Git server as a remote and push to it.

[codesyntax lang="bash"]

git remote add origin git@192.168.0.22:documentation.git
git push origin --all
git push origin --tags

[/codesyntax]

3. Delete the imported SVN repository on client side and re-import it from GIT server

Server side

4. Securing Gitweb with htpasswd. At this point the Gitweb web page is accessible to all, which of course is not a good idea. So, if you would like to have a set of web pages that are protected, requiring a username/password to gain access, you will have to use htpasswd in apache and add users and passwords.

4.1. Configure htpasswd access in Gitweb

[codesyntax lang="bash"]

sudo vim /etc/apache2/conf.d/gitweb

[/codesyntax]

Alias /gitweb /usr/share/gitweb
 
<Directory /usr/share/gitweb>
    Options FollowSymLinks +ExecCGI
    AddHandler cgi-script .cgi
        AllowOverride None
    Order allow,deny
    Allow from all
    AuthType Basic
    AuthName "Git Access"
    Require valid-user
    AuthUserFile /etc/apache2/gitweb-htpasswd
</Directory>

4.2. Create htpasswd username and password.

[codesyntax lang="bash"]
sudo touch /etc/apache2/gitweb-htpasswd
sudo htpasswd -m /etc/apache2/gitweb-htpasswd user1
sudo htpasswd -m /etc/apache2/gitweb-htpasswd user2
sudo htpasswd -m /etc/apache2/gitweb-htpasswd user3

[/codesyntax]

4.3. Restart the apache server

[codesyntax lang="bash"]

sudo /etc/init.d/apache2 restart

[/codesyntax]

Note: You can use TortoiseGIT for doing all git operations on Windows Machine, of course. I was lazy and I didn't take screenshots with steps that you should do with TortoiseGIT :)

Links: