Create a dummy Fedora repository (for Dorrie)

I have been fiddling with Dorrie lately. Dorrie is a web interface to create Fedora spins, remixes and installation media. But working with Dorrie requires one to have a Fedora repository at disposal if one has to build Live ISOs. I rule the possibility of downloading packages from the internet for testing. You need to have a local repository for testing it in a nice and efficient way. I tried to create a dummy Fedora repository with 0-byte .rpm files and tried to mock livecd-creator, but I failed. I forgot that livecd-creator installs the packages during the process of creating a Live ISO. Nevertheless, this attempt was not without any result. I fixed and updated Dorrie’s code to handle offline repositories.

Now, we (I and sayan) are working on another feature for Dorrie: building installation media (just like Fedora installation DVD). We are using pungi for the purpose. If my calculations are correct, then building an installation media leaves an option to use 0-byte .rpm files. But, this is not the topic of this post ;). I will talk on how to create a dummy Fedora repository.

Fedora repositories mainly have two important components for a Fedora version: releases and updates. All files in the dummy repository can’t be fake, the metadata files have to be there for real, otherwise the packages names won’t be found. I have devised a shell script for creating such a dummy repository.


#!/bin/bash
# Destination directory
DESTDIR="dummy_repo/"

# RSYNC URL
URL="rsync://fedora.c3sl.ufpr.br/fedora/linux/releases/16/"

#file extensions to be touched
FILE_EXTENSIONS_PATTERN=".*\.rpm$\|.*\.iso$\|.*\.img$"

#patterns to exclude from rsync
EXCLUDE_PATTERN="--exclude=source/ --exclude=debug/ --exclude=repoview/
--exclude=x86_64 --exclude=Everything/ --exclude=Live/"

#This will create only the directories and 0-byte .iso and .rpm files
#Since the files created will have newer timestamp, when we run
#rsync for real, the .iso and .rpm files won't be downloaded
rsync -auvr $URL $DESTDIR --dry-run $EXCLUDE_PATTERN |
while read line; do
 if [ `expr match "$line" ".*/$"` -ne 0 ]; then
 echo $line
 mkdir -p $DESTDIR/$line
 elif [ `expr match "$line" "$FILE_EXTENSIONS_PATTERN"` -ne 0 ]; then
 echo $line
 touch $DESTDIR/$line
 fi
done

#Download metadata files
rsync -auvr $URL $DESTDIR $EXCLUDE_PATTERN --progress

This is just a sample code. The above code can be customized to do more versatile dummy cloning of Fedora repositories.

Advertisements

Update xnoise.spec

I got a nice and long review for my first package of xnoise :). You can read the review at

https://bugzilla.redhat.com/show_bug.cgi?id=586433

I made the suggested changes –

4c4
< Summary:        A media player for gtk+

> Summary:        A media player for GTK+
6c6
< License:        GPLv2

> License:        GPLv2+
12c12,14
< Source0:        xnoise-0.1.2.tar.gz

> #   make dist
> #to generate xnoise-0.1.2.tar.bz2
> Source0:        xnoise-%{version}.tar.bz2
15,16c17,20
< BuildArch:      i686
< BuildRequires:  vala, vala-devel, intltool, libtool, autogen, automake >= 1.11, gnome-common, gtk2-devel, sqlite-devel, taglib-devel, unique-devel, gstreamer-devel, gstreamer-plugins-base-devel, gettext, desktop-file-utils

> #BuildArch:      noarch
> BuildRequires:  intltool, libtool, autogen, automake >= 1.11
> BuildRequires:  gnome-common, gtk2-devel, sqlite-devel, taglib-devel
> BuildRequires:  unique-devel, gstreamer-devel, gstreamer-plugins-base-devel, gettext, desktop-file-utils
18c22
< Requires:   gstreamer, gstreamer-plugins-base, gtk2, gettext

> #Requires:   gstreamer, gstreamer-plugins-base, gtk2
21,32c25,27
< Xnoise is a media player for gtk+
< Xnoise is written in vala
< Xnoise can play every kind of audio/video data that gstreamer can handle
<
< #%package devel
< #Summary:    Development files for %{name}
< #Group:      Development/Libraries
< #License:    GPLv2
< #Requires:   %{name} = %{version}-%{release}
< #Requires:   pkgconfig
< #%description devel
< #Development packages for Xnoise

> Xnoise is a media player written in vala for GTK+.
> Xnoise can play every kind of audio/video data that gstreamer can handle.
> Xnoise uses a tracklist centric design and uses a hierarchical tree structure media browser along with plugin interface. Xnoise is always running in a single instance,so that music files that are associated with it, will always be added to the tracklist instead of starting a new instance.
67,70c62
< %{_libdir}/pkgconfig/xnoise-1.0.pc
< #%files devel
< #%{_libdir}/pkgconfig/xnoise-1.0.pc
< #%{_libdir}/xnoise

> %exclude %{_libdir}/pkgconfig/xnoise-1.0.pc
74,75c66,69
< * Tue Apr 27 2010 rtnpro <rtnpro@gmail.com> 0.1.2
< – RPM package xnoise

> * Sat May 1 2010 rtnpro <rtnpro@gmail.com> 0.1.2-2
> – xnoise.spec file cleanup and some edits
> * Tue Apr 27 2010 rtnpro <rtnpro@gmail.com> 0.1.2-1
> – initial RPM package xnoise

Submitted a scratch build at

https://koji.fedoraproject.org/koji/taskinfo?taskID=2152364

It got successfully packages this time for i686, ppc, x86_64 archs

Finally I updated the spec and SRPM file at rtnpro.dgplug.org via rsync.

Updated files at :

http://rtnpro.dgplug.org/Packages/xnoise/xnoise-0.1.2-1.fc12.src.rpm

http://rtnpro.dgplug.org/Packages/xnoise/xnoise.spec

A long night … fedX + get rpmfusion repo

I became desperate yesterday night to do some useful things, anything. Had a long offline Puja Holidays. I was struggling to understand “How to write Makefiles”, and yesterday it clicked and I ended up writing a Makefile for the ongoing fedX project. Though the makefile which I wrote is not that great, but I have a start now. Committed the changes to the fedX repository at

http://gitorious.org/~rtnpro/fedx/rtnpros-fedx-clone/

It was around 4:00 AM in the morning already. Made some changes in my gitorious project clones. Apart from that, I was trying to download the rpmfusion repositories for Fedora 11. I was using rsync on multiple screens for the purpose. But again and again, rsync used to stop downloading after working for sometime. Then I tried a Firefox addon “DownThemAll” and was very happy to see it work. All I needed was to browse the repository page listing all the packages, right click and select DownThemAll. And then I just needed to select where I want to save the downloaded files and start it. It downloads multiple files simultaneously and does not halt when one of the files cannot be downloaded, rather it pauses it and moves to others. It also supports resume, and has an option to skip the already downloaded files.

But I could not download the directories with DownThemAll. I guess it works only for files. Need to do some research on it. Any way, you can try to use DownThemAll. I think you will like it.

Finally, I went to sleep at 4:45 AM with my laptop running DownThemAll to download the rpmfusion repository. When I got up today morning, it was running fine. 😀

A very very long day @ Barasat University

On 14 July, 2009, me ( Ratnadeep Debnath aka rtnpro) and Meejanur Rahaman ( aka meejan) and Kishan Goyal were heading towards Barasat University. Kishan took a bus from Howrah at around 8:30 AM and we started from Ruby Hospital at around 9:30 AM to Barasat. We hoped that we’ll be reaching the university at most by 11:30 AM and will have enough time to work. Our bust took a turn from Daak Bunglow. We told the conductor that we need to go to KaziPara, so the conductor said that he’ll drop us a bit ahead. We got down, enquired about the university, and no one seemed to have heard of it. Then we enquired about KaziPara, and then took a van. Then we were amidst a big traffic congestion. We were taking to Taki Road KaziPara, where we came to know that we came off the path to the university and came to now from the folks that there was another KaziPara at Jasha Road. We took a van again to Jasha Road. When we got down there, we found that this was not also the place we wanted to go. We enquired about the university from the folks out there, they were all dumb about that.

We called Kishan, who had reached the college by 12:00 PM. I told him to start all the download processes running in different screens in the running computer in the server room at WBSU. He followed the instructions over the phone and successfully started the processes. Then we asked him to enquire about the address of the college. He said that we had to get down at KaziPara. In fact, he also said to the Bus conductor of D 81 from colony more that he had to get down at KaziPara :P. Then on further enquiry from the University folks, Kishan told us that we had to come to KaziBari, not KaziPara.

We again took a van from there, came to the Bus terminal, took a D-81 bus and got down at KaziBari. Then from there, we took a van to the university. Reached the university at around 1:30 PM. It was quite some “Getting lost” experience. A one and a half hour of wondering in Barasat without knowing the place we have to go. But, in return, came to know more about the Barasat locality.

In the server room at the University, I kept monitoring the downloads plus some self study. I started two more rsync processes to pull the rpmfusion-nonfree-releases and rpmfusion-free-updates. Now I saw that the download speed of the computer increased to around 500 KBps. Though we had more bandwidth, the downloads on 13 July 2009, were just taking place at most at 250 KBps. This was mainly because the servers limited the download speed. So when I turned on multiple download processes from different servers, the bandwidth at the university was being properly utilized.

With all the processes running nicely, we left the University at 5:15 PM, took a van to KaziBari, then a overloaded bus to terminal. Finally we found seats in a bus to Ruby Hospital. We reached Ruby Hospital at 7:45 PM. From there, meejan left for our mess, I went to Tonmoy’s house. He’s got a broadband connection. SSH Logged in to the computer at the university. Found some download completed. Started new rsync download of Fedora 11 updates, rpmfusion-nonfree-updates. Left his home at 9:15 PM, ran to my mess. Had dinner and went straight to bed.

What a day!

First visit to West Bengal State University, Barasat …

Today, 13th July, 2009, we ( me , i.e., Ratnadeep Debnath aka rtnpro, Kishan Goyal, Meejanur Rahaman, Harsh Verma, Chandana Boral) led by Indranil Das Gupta and Stephanie Das Gupta, went to West Bengal State University at Barasat, Kolkata. We started at around 10:30 AM from Ruby Hospital, and after a long journey ( changing two buses, then riding on a van, with  a few drizzles  on the way), we finally reached Barasat University at around 12:30 PM.

We were then taken to the server room of Barasat University. Indradg got us introduced to the wiring and connections in the server room,  the big UPS, the batteries being charged by the UPS, the mechanism for providing back up during power cuts, switching between the two ACs periodically and finally the HP Blade Server.

The UPS is a Liebert GXT-MT 6KVA UPS with a OpenComms Web Card. The OpenComms Web Card delivers SNMP (Simple Network Management Protocol) and Web support to the UPS in which it is installed. Then we were acquainted with the network configuration of the University Network ( wifi + lined connection).

Next job was to download the Fedora 11 repository. We brought in another computer, placed it in place of the server and installed Fedora 10 in it and started working. We were given the required bandwidth and my job was to start the rsync. But it didn’t go well with me. I was confused with how and where to start. I did rsync locally in earlier instances and not over the internet. I started with the man page. Then googled about “rsync”. Found some good documentation at http://fedoraproject.org/wiki/Infrastructure/Mirroring. It mainly dealt with setting up Fedora mirrors and how to enable rsync in them. But what I needed was just use rsync to pull Fedora 11 repository. It was quite some time, I was still stuck. Got some suggestions on configuring rsyncd.conf file, I was again redirected to the above link. Then we had some lunch and break.

I again sat working on it. Again went through the rsync man page. I found that,

$rsync [options]

gives a directory listing of the source directory. Chose rsync://ftp.riken.jp/fedora as the source. Found its directory structure using

$rsync -auvr rsync://ftp.riken.jp/fedora

I created the directory structure in my current folder ( here it was /home/$USER/f11_repo/). Now I started pulling Fedora 11 i386 release packages in the following way

$rsync -auvr rsync://ftp.riken.jp/linux/fedora/releases/11/Everything/i386/os/ /home/$USER/f11_repo/linux/fedora/releases/11/Everything/i386/os/

and it started syncing the source and destination directories.

Then we also started downloading the rpmfusion_free repositories for Fedora 11 using wget. Indradg wrote a shell script to automate the download of rpmfusion/free repository using wget. We also used screen to run the two processes in different screens so that they can be monitored remotely.

$screen -R [screen name]

In the mean time, others prepared some “Do not disturb, work in progress” labels and put them on the running computer, the server room door, etc. to make sure that no one messes with the running computer.

It was around 5:30 PM that we finally packed up. We got into the University Bus and reached Ruby Hospital at around 7:30 PM.

It was a long long day :).