Feeds:
Posts
Comments

Archive for the ‘p2p’ Category

finally … the online publication of my diploma thesis (DT) is here, it can be found online [1] including the source at [2].

i hope that the terminology introduced in this DT (chap 9) will be used. this is of course also true for concepts engineered in (chap 7).

candies can be found here:
 - chap 4: components used in a package manager
 - chap 4.6: different integration levels of a package manager
 - chap 5.13: ways to replicate a system
 - chap 5.15 ff
 - chap 6.2: evopedia deployment summary
 - chap 7 (you might want to read this a few times, it is quite complex)
 - chap 9: here i introduce some new terminology in package management
           (probably a _must read_)

see also the README for in [2] for further information.

links

[1] https://github.com/qknight/Multi-PlatformSoftwarePackageManagement/blob/master/Multi-PlatformSoftwarePackageManagement.pdf

[2] https://github.com/qknight/Multi-PlatformSoftwarePackageManagement

Advertisements

Read Full Post »

motivation

running mac os x from virtualbox helps to experiment with software deployment for that platform. i’m using gentoo linux as host system with 2.6.34-gentoo-r6 kernel on a 64 bit core 2 duo machine with virtualbox 3.2.10 OSE. there are various blog postings about how to get it running so i won’t repeat this here.

the most important thing is to use the snow leopard retail dvd. as the dvd coming with the mac (a mac book pro in my case) won’t work. you can get the retail dvd on ebay or various other sources. note: the retail dvd is not labeled ‘snow leopard retail dvd’ but something like ‘snow leopard update dvd’ in most auctions (don’t blame me if that is wrong).

how i use it

having snow leopard not only native, but also in a virtualbox image, makes testing of software easier.

hints

disk image size

i use about 40 gb here, and about 25 gb are actually in use with:

  1. latest patch (1 gb)
  2. xcode (6-8 gb)
  3. trolltechs qt sdk (2 gb)
  4. mac os x (8 gb)

i wasted a lot of disk space with a default installation. but i don’t want to spend too much time on that right now so i will leave it that way. i removed the printer drivers later on, saved me 600mb.

when installing software, map the image (dmg) using samba (running on the host machine)

import dmg images, which are used to install software from, using samba from the host. this way you do not have to copy the dmg onto the virtual machine using ssh or by mapping an external harddrive (note: the OSE version i am using, does not support USB host mapping anyway) into the virtual machine.

this also reduces the host image size by a few gigabyte of disk space. initially i was doing it wrong: i copied the file, installed from it and finally removed it. however, removing it did not have much of an effect, as virtualbox does not know about hfs+ partition scheme, therefore the file is only deleted inside the virtualmachine, still the host image uses the diskspace.

example:

my current disk image: mac os x [snow leopard].vdi is at 25Gb. but inside the virtual machine is is only 18gb.

smb.conf

[global]
  workgroup = MYGROUP
  server string = Samba Server
  security = share
   dns proxy = no
[rw-files]
   comment = Temporary rw files
   path = /home/user/smbshare/rw
   read only = no
   writable = yes
   public = yes
[ro-files]
   comment = Temporary ro files
   path = /home/user/smbshare/ro
   read only = no
   writable = no
   public = yes

don’t forget to:

  1. create the paths listed above
  2. secure the network as a rw path might be accessible from your lan

consider using a user/password for that.

downloading xcode

use wget to download it. it is about 3,6gb and it failed several times here. also resuming did not work well. after several attempts downloading it (must be about 12gb i’ve already downloaded) i had a, yet again, broken download at 99%. i thought: “not again”! and as there was no resuming (using a webbrowser) i downloaded a torrent.

next i used the torrent and started the download, which created a file with the same filename. i closed the torrent client (rtorrent) and moved the previously metioned 99% download file, to the place where the torrent expected it’s download. then i restarted the torrent download.

thanks i did not have an offset error (this can happen on wrong resumes, anyway i did not use resume).

finally the torrent checked all chunks and finnaly there was only 1mb left to download. so i repaired my download, using bittorrent.  all i had to do is wait 5minutes.

next thing was file integrity. it seems that apple does encode some individual information into the xcode download. if rumors are true, it is no wonder that users sue apple, see [1]. anyway, this means we can’t create a md5sum. it would be interesting to know what is going on.

what we do know already is that most of the file, apparently 99% of the download is generic. if not, then i would not be able to repair the xcode download with the torrent content of a different user. see discussion at [2] for more input on this.

create snapshots

virtualbox supports snapshots, use them! they are a great way to update various system components as xcode without having to uninstall any components. this results in a cleaner system, saves time & resources as deleted files are only deleted virtually inside the vm.

image growth

every virtually deleted file is not really deleted as virtualbox works on a block device layer. that means: if hfs+ decides to use deleted inode for a newly created file it will be good as vm disk and real disk are in sync. but i guess more often than not this is not the case and the real disk image will grow over time.

links

[1] http://www.heise.de/newsticker/meldung/Apple-wegen-angeblicher-Weitergabe-von-Nutzerdaten-verklagt-Update-1159403.html

[2] http://discussions.apple.com/thread.jspa?messageID=12770938

Read Full Post »

gentoo linux logo (copied from commons.wikipedia.org)

gentoo

evopedia [1], the offline wikipedia reader, has been ported to gentoo linux. the ebuild can be found at [2].

i would like to say thanks to all devs @ irc.freenode.net#gentoo-sunrise! especially to those who reviewd my ebuild.

hope this ebuild will be moved into the official distribution anytime soon.

source

/usr/local/portage/sci-misc/evopedia/evopedia-0.4.2.ebuild

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=3

inherit qt4-r2

DESCRIPTION="Offline Wikipedia Viewer"
HOMEPAGE="http://evopedia.info/"
SRC_URI="http://evopedia.info/src/${P}.tar.gz"

LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

RDEPEND="
        >=x11-libs/qt-gui-4.4:4
        >=app-arch/bzip2-1.0.6
"
DEPEND="${RDEPEND}"

src_configure() {
        eqmake4 evopedia.pro PREFIX="${EPREFIX}"/usr
}

links

[1] http://evopedia.info

[2] http://bugs.gentoo.org/show_bug.cgi?id=349210

update: 2010-12-24: added the source (contents of evopedia-0.4.2.ebuild)

Read Full Post »

see [2] for licenseadding torrent support for a qt project should be as easy as using QHttp (IMHO). as there is no QTorrent base class in qt network yet one can use the qt torrent example [1].

i’ve modified [1] to create a gui-less torrent downloader as i was searching for an issue with that example. my problem was that [1] worked nicely but my adaption of the torrent code did not.

it connected to the tracker and it reported that there was a peer but it did never find a seed.

a few hours later i found the solution to that problem: one has to initialize the upload/download rates (as the default value is 0).

anyway: [2] features a very minimal codebase which is very easy to adapt to your qt project.

links

[1] http://doc.qt.nokia.com/4.2/network-torrent.html

[2] http://github.com/qknight/torrent

update: i’ve already reported this issues at http://bugreports.qt.nokia.com

Read Full Post »

tcp

weil der artikel [1] in deutsch verfasst ist,  verfasse ich dieses blog posting auch mal in deutsch.

ich habe wegen einem seminar eine hausarbeit schreiben muessen. das thema war freigestellt, d.h. habe ich mich fuer ‘tcp‘ entschieden. ich hoffe ich kann damit das verhalten von tcp etwas klarer machen.

eigentlich sollte die arbeit nur 5-7 seiten umfassen, es ist aber etwas mehr geworden weil man hausarbeiten mit 12pt verfasst und nicht mit 10pt wie ich es urspruenglich geschrieben hatte.

ich habe zum pdf [1] auch noch das latex tex file [2] und das bibtex [3] ins verzeichnis [4] gelegt.

ich freue mich ueber feedback jeglicher art.

links

[1] http://www.lastlog.de/misc/tcp/tcp.pdf

[2] http://www.lastlog.de/misc/tcp/tcp.tex

[3] http://www.lastlog.de/misc/tcp/tcp.bib

[4] http://www.lastlog.de/misc/tcp/

Read Full Post »

source: wikipedia, see 'The BitTorrent - Logo'

BitTorrent logo, source: wikipedia

if you ever wanted to publish a file via bittorrent, here is a small guide how to do it with linux (using a shell). this guide – in contrast to most other guides found in the net – is NOT about downloading files using bitTorrent, instead is is about PUBLISHING FILES! first of, our tools:

  • bittornado [1] for tracker/seeder (that is the central part of your hosting service)
    BitTornado-0.3.17.tar.gz worked, while net-p2p/bittornado-0.3.18-r2 did fail
  • mktorrent [7] to create a .torrent on the shell
  • ktorrent [2] used as ‘normal’ client (that would be a user of your service)

bittorrent is basically nothing more than a simple protocol how to copy files and how to enforce integrity by hashing a big file into chunks of smaller files of equivalent 1Mib blocks (size may vary from .torrent to .torrent). a mixture of all related computers hosting chunks are called the swarm. if in doubt just read the wikipedia article [3] to get the terms & concepts. a very nice matrix about bitTorrent clients and tracker capabilities can be found at [4].

the basic idea about hosting downloads with bitTorrent

open source projects often offer files via bitTorrent. to offer a file in bitTorrent one has to do this:

  1. create a .torrent of the file or directory one wants to offer
  2. host a tracker (which manages the swarm, usually a tracker is a central server)
  3. connect aseed’ to the tracker, so that others can download the file and spread it further
  4. test & monitor the tracker or swarm
  5. security

in general this is nothing new as this technique is used already for years. i would like to create a very lightweight documentation about how these 4 points are achieved with ease, so here we go:

(1) create a torrent

let’s use one of my 23Mib screencasts to create a .torrent for (just use whatever you want) using mktorrent [7]

# wget ‘http://lastlog.de/misc/wordpress/libnoise-viewer.ogv’
# mktorrent -a http://myregisteredname.dyndns.org:6969/announce -c ‘a screencast showing what libnoise-viewer does’ libnoise-viewer.ogv

mktorrent 0.4 (c) 2007 Emil Renner Berthing
Hashed 92/92 pieces.
Writing metainfo file… done.

Now we created the libnoise-viewer.ogv.torrent as it can be found in the directory after we used mktorrent. for my private computers i use a dyndns.org DDNS service which is updated on every reconnect of my firtzbox.

(2) host a tracker

in my case i host this tracker at a dialup connection using dsl. this might not be ideal (but redundancy might get better if clients use a dht, more on that later). since the computer is behind a nat (network address translation) we have to forward some ports (sockets) to the server hosting the tracker behind the nat. this can be done by using your fritzbox configuration dialog (or equivalent, whatever you use – i will just pick that example with a fritzbox) .

# mkdir torrent;
# mv *.torrent torrent/
# bttrack.py –port 6969 –dfile dstate –logfile tracker.log –allowed_dir torrent/
# netstat -tulpen
tcp        0      0 0.0.0.0:6969            0.0.0.0:*               LISTEN      0          234712     21631/python2.5

we could check for html output with curl as well (curl should basically installed on every linux installation)

curl localhost:6969
the output should be some html code with tracker version and a list of torrents (tracked files)

so we see that the bttrack.py service using tcp with port 6969 is up and running. if you are behind nat – as i am – also check from a remote machine, if the service is working (can also be done using curl).
if you need to configure a fritzbox or other socket/html based router which has no direct internet configuration enabled you can also use a ssh redirect. in my case i can access my server behind the nat with ssh. so if you want to configure your fritzbox as if you had your computer plugged in the local lan swith see a different documentation of mine at [8] (german only).
i used the tcp portrange from 6881 to 6999 for the tracker as well as for the seeder connections, so forward these ports.

(3) connect a ‘seed’ to the tracker

without doing this nobody will be able to download anything, since nothing is there! so let’s add some data. be aware that if that seed is behind nat as well it might not work at all.
# btdownloadheadless.py –responsefile /root/torrent/libnoise-viewer.ogv.torrent –minport 6881 –maxport 6999 –max_upload_rate 20 –saveas /root/libnoise-viewer.ogv
saving:         libnoise-viewer.ogv (22.8 MB)
percent done:   0.0
time left:      Download Succeeded!
download to:    /root/libnoise-viewer.ogv
download rate:
upload rate:    12.4 kB/s
share rating:   oo  (0.4 MB up / 0.0 MB down)
seed status:    0 seen recently, plus 0.076 distributed copies
peer status:    1 seen now, 7.6% done at 6.7 kB/s
so now you have to upload the libnoise-viewer.ogv.torrent to somewhere, or copy it to your client directly.
you might want to have a look at the other parameters as for instance:
  • –max_upload_rate <arg>
    maximum kB/s to upload at (0 = no limit, -1 = automatic) (defaults to 0)
  • –minport <port>
  • –maxport <port>

you can also use several of these clients from different machines to ensure redundancy here as well. it will scale pretty well! you can also use any other torrent program as ktorrent or rtorrent for instance.

(4) test & monitor the tracker or swarm

use rtorrent or ktorrent and import the libnoise-viewer.ogv.torrent and see if the download works. if it works you are done!
monitoring is basically done by visiting the tracker, as:
  • there must be at least one seed (that is your seed)
  • see the logs created by the tracker, that is: tracker.log
  • you can also do bandwidth monitoring, ktorrent has a nice monitor included (in case you are seeding) but you most likely don’t need bandwidth monitoring for the tracker as it won’t use much bandwidth
  • both services might open many connections and some firewalls can’t handle this well since the resources needed for ‘connection tracking’ in statefull firewalls will be exceeded soon resulting in strange effects.

ktorrent downloading the torrent

(5) security

i did run the script as root but if you want to establish a service which should be secure and reliable DO NOT RUN AS USER ROOT! just create a new user and run it as this new user. this is very easy since bittorrent does NOT use any privileged ports (1-1024) but bitTorrent uses unprivileged ports instead (all others as 1025-65535). in my example even less.

if you have frequent issues with the tracker you could use minit [5] to restart the service automatically.

summary

i hope this helps you to offer big downloads easier. just keep in mind that you have to follow two rules:

  1. for every new download you need to create a new .torrent file
  2. you need to inject a seed for every .torrent via a tracker

using bittorrent will show it’s full potential when we migrated to ipv6 since nat is (asipv6 mentioned in about every 5th posting on my blog) is a real problem.

errata

  • since the tracker seems not to be stable for both versions of bittornado we now use bittorrent instead. the concepts apply 1:1 so the adaption isn’t very complicated
  • be aware that NAT will most likely kill a lot of upload potential of the clients since most clients will not have a direct valid ipv4 address but will be behind NAT all the time with improper port forwarding. this will then result in much less throughput and clients will have small download rates in general. one exception to this rule will be two clients which can utilize the full potential if one has configured proper port forwarding while the other is behind nat.

links

[1] http://en.wikipedia.org/wiki/BitTornado

[2] http://ktorrent.org/

[3] http://en.wikipedia.org/wiki/BitTorrent_(protocol)

[4] http://en.wikipedia.org/wiki/Comparison_of_BitTorrent_clients

[5] http://www.fefe.de/minit/

[6] http://www.linux.com/news/software/applications/8230-bittorrent-for-linux

[7] http://torrentfreak.com/how-to-create-a-torrent/

[8] http://lastlog.de/wiki/index.php/Ssh

Read Full Post »