Feeds:
Posts
Comments

Archive for the ‘windows’ Category

finally … the online publication of my diploma thesis (DT) is here, it can be found online [1] including the source at [2].

i hope that the terminology introduced in this DT (chap 9) will be used. this is of course also true for concepts engineered in (chap 7).

candies can be found here:
 - chap 4: components used in a package manager
 - chap 4.6: different integration levels of a package manager
 - chap 5.13: ways to replicate a system
 - chap 5.15 ff
 - chap 6.2: evopedia deployment summary
 - chap 7 (you might want to read this a few times, it is quite complex)
 - chap 9: here i introduce some new terminology in package management
           (probably a _must read_)

see also the README for in [2] for further information.

links

[1] https://github.com/qknight/Multi-PlatformSoftwarePackageManagement/blob/master/Multi-PlatformSoftwarePackageManagement.pdf

[2] https://github.com/qknight/Multi-PlatformSoftwarePackageManagement

Advertisements

Read Full Post »

motivation

searching for a ‘cross platform‘, ‘cross distribution‘ package manager, one question that arises in regards to nix is:

can nix be used for ‘source deployment’ and ‘binary deployment‘ on windows?

in the last post i discussed how to install nix 0.16 [1] on windows but back then i could not find any working gui tool. i was not able to compile such a tool either with the tools found in nixpkgs [2]. the problem is that the used toolchain is optimized for posix environments (linux and others) with dependencies to X11.

how to use nix to build software for windows then? the solution to this problem is easy: we have to extend the current toolchain (used in nixpkgs) by a new one.

first, let’s see what others do: other distributions already support ‘cross compilers’ to compile program in linux (but for windows). the drawback of such a solution is that it binds the developer to a certain distribution. so what concepts do exist, in order to build software, for windows:

  • windows+cygwin (running nix inside cygwin) using the posix toolchain (nixpkgs)
    this is already working and supported
  • windows+cygwin (running nix inside cygwin) using a different toolchain: MinGW
    this is what this blog posting is about
  • using nix on any posix conform distrubtion with a cross compiler setup using the MinGW toolchain
    there are some references to MinGW found in the nixpkgs already, according to developers on irc.freenode.org#nixos [10] this is used for cross compiler setups. still i could not find any mingw expression to experiment with such a cross compiler setup in nix.

so why would i want to have windows+cygwin+nix with a native windows mingw toolchain?

  • nix can be used as a package manager for windows
  • nix does support ‘source deployment’ and ‘binary deployment’
    so it is a important target for developers on all platforms/distributions
  • nix is not bound to any distribution in particular
    which also makes it an excellent tool for ‘cross distribution’ packaging (when using nix in a prefix installation in contrast to using NIX OS)
  • nix can be used to create abstract targets of all kind
    examples: creating NSIS installers, automated software builds/tests
  • nix can be used to build  ‘cross compiler chains’
    this would actually be interesting for me as well (but not covered here)

creating a new mingw based nix toolchain

this is not that easy, but we somehow have to replace/extend the linux based toolchain with a MinGW toolchain optimized for windows.

first: what should be supported?

on windows, that is:

  • direct hardware access (using windows api and device drivers) / directX
  • using Qt natively (without X11)
  • things like ‘start menu’ shortcuts

we will use qt 4.7 as an example how to do that.

how to build software for windows natively?

first let’s build software for windows, how it would be done using windows:

  • download the Automated MinGW installer (should contain gcc 4.x) at [3]
  • download the qt 4.7.1 library (i used the zip distribution for this experiment) [4]

to build the libraries (&later the software)

making mingw work (native installation)

  1. as already mentioned, download the ‘Automated MinGW installer’
  2. install MinGW

NOTE: do not add anything to the global PATH, as we are going to setup multiple toolchains, this would be a source of horrible side-effects.

so instead of a global PATH, we could alter the PATH used in cmd.exe on the fly (every time we start a new cmd.exe) with:

PATH = %PATH%;c:\mingw\bin;

NOTE: windows does not have something like RPATH [5] (rpath is not cmake specific but that linked page should give you an basic idea what rpath is about), that means a windows program is lost if the library (something like mingw.dll) is not found in the same directory as the binary is in. as a quick fix to the problem one could expand the PATH variable to include also the place, where the respective library is in.

as we’ll with the new nix-toolchain, this simple fact is very problematic.

[7] provides an interesting (maybe outdated) table about how such things are handled in other distributions.

making qt work (using mingw, still native installation)

so as the compiler setup is done, we need to add the qt library. download a file called qt-everywhere-opensource-src-4.7.1.zip at [5].

  1. using the cmd.exe from above (with the PATH adaption)
  2. unzip qt-everywhere-opensource-src-4.7.1.zip
  3. cd qt-everywhere-opensource-src-4.7.1
  4. configure.exe
  5. make install (will install into the same directory, but we don’t care right now)

afterwards we can execute qt programs from the cmd.exe shell. to make the examples executable from everywhere either:

  • (DON’T DO THAT!): add the MinGW path and the Qt-library path to the global PATH variable
  • (DON’T DO THAT!): copy the needed dll files into EACH qt example directory to make the examples work from everywhere (either from  cmd.exe shell or when starting from explorer.exe)
  • add another PATH, namely: c:\qt-everywhere-opensource-src-4.7.1\bin in the cmd.exe shell

as a final step we test if the qt examples work (this is kind of a module test), run any qt example from cmd.exe with the extended PATH settings applied. as we now know how to get this toolchain working in a native environment, we can try to automate this process using nix expressions.

adapting this buildchain into nix expressions

to get a working toolchain inside nix i packaged the installation of MinGW from c:\MinGW and created something like mingw.tar.bz2 for redistribution. see the nix expressions in [6] and consult the README in order to get the nix expression to work in your cygwin environment (this time not cmd.exe anymore, instead use the cygwin shell).

  • winmingw.nix downloads my re-release of mingw and installs it
  • it also makes the tools like mingw32-make available for use inside your cygwin shell

NOTE: make sure your cygwin (not cmd.exe) has ~/.nix-profile/bin in your PATH environment variable (example: export PATH=~/.nix-profile/bin:$PATH).

how to install cygwin/nix and my nix expressions on windows:

  1. install a full cygwin environment (several gb of files, use the setup.exe installer)
    or see my last post about this
  2. make nix 0.16 work
    again: see my last post about this
  3. use git clone with my local repo in [6]
    cd ~; mkdir .nixpkgs; cd .nixpkgs; git clone …
  4. install winmingw: nix-env.exe -i winmingw
  5. check that the mingw tools are there, type: mingw32-make.exe or mingw32-make

NOTE: after installing winmingw notice that there are a lot of new tools symlinked from ‘~/.nix-profile/bin’. interestingly this also includes symlinks to libraries (read dll files) files. so why symlink dll files?

installing qt using the new mingw toolchain

there is another anomaly using cygwin in comparison to all other posix systems. in windows all executables do have the .exe suffix while on linux they don’t. as the cygwin shell must be compatible to both worlds, the designer of cygwin made this decision:

  • programs can be executed by typing: ‘curl.exe’ but also by typing ‘curl’, both times it is the same program which gets executed
  • there is an exception to the rule above, if a script called ‘curl’ also exists (which is executable), it is executed instead of curl.exe
  • if the script ‘curl’ and the binary ‘curl.exe’ are in the same directory, then the ‘curl’ script has precedence, when typing ‘curl’ in a shell

at least [8] states it that way. i also experimented with these commands:

  1. touch AA; touch AA.exe (both files get created)
  2. touch AA.exe; touch AA (only AA.exe is created and later the timestamp is updated)

i guess (1) is true on all posix systems (at least for those i’ve been working with so far). but (2) is cygwin specific and implies some issues on would not think of.

to track down an issue which is caused by the fromer rule it cost me at least 1-2 hours. the problem was that my winqt.nix expression was fine but always exited with ‘error 1’ after the unzip operation (without any error message, except the exit 1 code). i always thought that i did something wrong with the nix language. but then i decided to run the command (which would normally be executed by nix) in a shell and found out that the cygwin shell bahaves differently than the cmd.exe shell for the exact same command.

to my surprise i got prompted by unzip (in the cygwin shell) with the question if i want to [o]verwrite or [r]ename ‘configure.exe’ (cause by a file configure). i wondered quite some time what problem could cause this?! so i extracted it again, but this time in cmd.exe with no prompt.

important conclusions

Edit: further discussion should be redirected to the nix wiki [11].

problem 1: symlinks and dll files

  • all programs in nix are installed into a unique prefix but are available as a symlink from ~/.nix-profile/bin
  • symlinking the same program twice does therefore not work but happens rarely anyway
  • however: symlinking two different dll files sharing the same name is  not possible, this is the biggest issue right now

a solution would be static linking. therefore:

  • two programs having the same binary name (‘firefox’ for example)
  • two programs sharing the same library name (‘mingw32.dll’ for example)

the former point is easy to accomplish but the later is not. ‘viric’ (a developer from irc.freenode.org#nixos) pointed out that in order to fix this issue we could use ‘side-by-side sharing’ [9].

NOTE: it would be very interesting to see, if such a thing would actually work.

problem 2: directory and file handling

the other important problem is how file handling is implemented in cygwin. the current behaviour seems to be convenient for users but is a no-go for packaging as it might cause a lot of unforeseen consequences (and we all have played halfile and know where this could be going).

links

[1] http://nixos.org/nix/

[2] http://nixos.org/nixpkgs/

[3] http://mingw.org

[4] http://qt.nokia.com/products/

[5] http://www.cmake.org/Wiki/CMake_RPATH_handling

[6] https://github.com/qknight/mingw-nix-on-windows

[7] http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html

[8] http://hem.bredband.net/richardc/Cygwin.html

[9] http://msdn.microsoft.com/en-us/library/ms995843.aspx

[10] http://webchat.freenode.net/?channels=nixos

[11] http://wiki.nixos.org/wiki/Nix_on_Windows

Read Full Post »

motivation

nix (see [2])  is a package manager like apt (used in debian/ubuntu) or portage (used in gentoo). windows does have a package manager which is typical among linux distributions. so here some recent findings and how to use nix (and nixpkgs) on windows.

reproduction of my nix system

setup of my system:

  • core 2 duo processor
  • windows xp sp3
  • cygwin (setup.exe version 2.738) with a full installation [1] (some 2-7 GB, 1h download)
    using setup.exe, do use ‘default parameters’ everywhere, expect for the software selection: use all (do not install the source, binaries are enough)
  • nix 0.16 release from [2]
  • install (as mentioned in the docs) to a ntfs partition

components i used (get installed using cygwin::setup.exe:

  • gcc version 4.3.4 20090804 (release1) (gcc), target i686-pc-cygwin
  • libtoolize (GNU libtool) 2.4
  • aclocal (GNU automake) 1.11.1
  • autoheader (GNU Autoconf) 2.68
  • automake (GNU automake) 1.11.1
  • autoconf (GNU Autoconf) 2.68

how to build

as there is no windows installer (for example using NSIS) you have to bootstrap nix from source.

  1. extract nix-0.16 to c:\nix
  2. start the cygwin shell (desktop icon)
  3. in the shell, type: ‘cd /cygdrive/c/nix/nix-0.16
  4. type: ‘./bootstrap.sh‘ (this is very important)
    that is why several of my attempts to get nix running failed ;P
  5. type: ‘./configure
  6. type: ‘make install
  7. afterwards you can use it with: ‘nix-env -qa ‘*’‘ for example

how to use nix

this is quite easy but it does not work as mentioned in the documentation (which worked only for unix systems so far). the full cygwin installation installes many tools.

  1. check your environmen for nix, type ‘nix-env.exe’ and test if that program works (should be in the path)
  2. if ‘nix-env.exe’ does not work, check /usr/local/bin for a propper nix installation
  3. check if there is a unrar tool already (not included in full-cygwin installation with: ‘unrar.exe
  4. you have to subscribe to a nix channel, see the nix documentation: chapter 2. quick start [4]
  5. if ‘nix-env.exe’ works, install ‘unrar’ using: ‘nix-env.exe -i unrar
    you should see compiler stuff and it installs a million *.drv files…
  6. now type: ‘unrar.exe‘, should say: “bash: unrar.exe: command not found”
  7. now type: “export PATH=~./nix-profile/bin:$PATH
  8. and try the command from step 5 again, now you should get a working ‘unrar.exe’ installation

what worked using nix on cygwin-windows

most of the software related to X did not work, as windows and mesa don’t go along nicely. i’ve tried a few programs:

  • wget.exe worked (checked with ‘which wget’) and a download
  • hello.exe worked (GNU hello)
  • unrar.exe worked (compiled, installed, probably to work, didn’t extract anything)
  • qt (nix throws a nix exception): unsupported platform for Mesa
  • irssi (nix tries to compile the needed software but fails at glib-2.24.1)
  • firefox (nix throws a nix exception): flashplayer is not supported on this platform

summary

great to have nix on windows. no it’s time to use it, here are a bunch of scenarios:

  • use it to create windows software
  • use it to create installers (using NSIS)

but both points would be  independent of nix, so this results in a very nice build system. maybe combined with hyrdra this is a great way to build software.

links

[1] http://www.cygwin.com/cygwin-ug-net/setup-net.html

[2] http://nixos.org/nix/

[3] http://hydra.nixos.org/build/858291/download/1/manual/

[4] http://hydra.nixos.org/build/858291/download/1/manual/#chap-quick-start

Read Full Post »

evopedia 0.4.2 on windows

there is now a official 0.4.2 release and this obsoletes the zip-file based git (git, the version control system) build, which i released a few weeks ago. as a surprise for me, my own wiki posting, my manual for building evopedia on windows, didn’t work. so i had to hack the Makefile manually as  ‘-lbz2’ had to be removed.

in the windows build i’ve applied a patch, so that the network setting is ‘enabled’ by default (was set to ‘auto’ previously). the ‘auto’ setting does only make sense for the maemo platform, on the nokia n900 device, anyway.

NSIS – Nullsoft installer

finally i experimented with the nullsoft installer, read some example scripts and gave up. then i thought why not copy the nullsoft installer script of my favourite project: springrts.com. altough i needed a few minutes to find the right file, i soon realized that the spring setup script was generated by a wizzard…. sounded exactly what i needed.

so i discoveded [2]: HM NIS EDIT: A Free NSIS Editor/IDE

HM NIS EDIT

it took me about 2minutes to create a experimental installer and i have to point out that ‘HM NIS EDIT’ is excellent software. i like it very much because i hate to spent time in my virtualbox windows xp image as the fonts are very small and all the windows tools feel so wrong.

i needed abou 30minutes to create this nice installer. hope it works for everyone – please report back to me.

the evopedia installer

so here are the files:

  • you can find the installer evopedia-0.4.2-installer.exe at [3]
  • the nsi source for generating the installer can be found at [4]
  • a md5 sum of the two files can be found at [5].

how to install

execute the installer, select the ‘small’ evopedia. wait for about 3minutes, until the download is complete. then experiment with it.

to get an idea what dumps we have without installing evopedia, see [6]. there is however a nice p2p torrent downloader integrated into evopedia 0.4.2, so you don’t have to do downloads/updates manually.

how to uninstall

if you don’t like evopedia you can:

  1. use the uninstaller (to save approx. 6mb)
  2. don’t forget to remove the small evopedia (and maybe other evopedia archives you have downloaded) manually.

i doubt that i will add an automation for the second step to the uninstaller, so this must be done manually, even on futher releases.

links

[1] http://nsis.sourceforge.net/Main_Page

[2] http://hmne.sourceforge.net/index.php

[3] http://lastlog.de/misc/evopedia-0.4.2-installer.exe

[4] http://lastlog.de/misc/evopedia-0.4.2-installer.nsi

[5] http://lastlog.de/misc/evopedia-0.4.2-installer.md5

[6] http://evopedia.info/dumps/

Read Full Post »

evopedia, the offline wikipedia reader, is now running on windows xp and probably also on all later windows versions.

evopedia on windows xp

how i did it

i’ve created a build which uses dll files from qt. i actually wanted a static qt build but that did not work. so here is how i built the evopedia

downloading the needed software

visit [1] and download the Qt SDK for Windows (~322MB). i also downloaded git for windows at [2].

using git i then downloaded evopedia:

git clone https://qknight@github.com/qknight/torrent.git

finally i downloaded bzip2-1.0.5-bin.zip at [4]. we are going to use this as we need the:

  • bzlib.h (for compiling)
  • bzip2.lib (for linking)
  • bzip2.dll (for runtime)

i’ve just created:

  • c:\evopedia
  • c:\evopedia\bzip2 (where i moved all files from the bzip2-1.0.5-bin.zip)

modifying evopedia for windows

it’s basically about making some changes to src.pri (which is used in evopedia.pro)

  • INCLUDEPATH += bzip2/include
  • LIBS += $$quote($$_PRO_FILE_PWD_)/bzip2/lib/bzip2.lib

afterwards the evopedia project can be opened with qtcreator (or directly used with the qt shell and qmake). just build the software in ‘release mode’ and after the buildstep is done, copy the bzip2.dll into the same directory as the evopedia.exe is in. now the program is ready to be used.

please note that a relative LIBS (see LIBS below) path did not work as qtcreator sometimes does out-of-source builds.

i had this problem:

  • evopedia/ (project dir)
  • evopedia/debug (debug build dir)
  • evopedia-build/release (release build dir)

don’t ask me why qtcreator decided to build there. using a relative LIBS path did not work, as:

  • LIBS += bzip2/lib/bzip2.lib

resulted in this linker error:

g++ -enable-stdcall-fixup -Wl,-enable-auto-import -Wl,-enable-runtime-pseudo-reloc -Wl,-s -mthreads -Wl -Wl,-subsystem,windows -o release\evopedia.exe object_script.evopedia.Release  -L”c:\Qt\2010.05\qt\lib” -lmingw32 -lqtmain bzip2/lib/bzip2.lib -lQtGui4 -lQtNetwork4 -lQtCore4
mingw32-make[1]: Leaving directory `C:/evopedia-build-desktop’
g++: bzip2/lib/bzip2.lib: No such file or directory
mingw32-make[1]: *** [release\evopedia.exe] Error 1
mingw32-make: Leaving directory `C:/evopedia-build-desktop’
mingw32-make: *** [release] Error 2
The process “C:/Qt/2010.05/mingw/bin/mingw32-make.exe” exited with code %2.Error while building project evopedia (target: Desktop)When executing build step ‘Make’

testing the software

i’ve used wine 1.1.34 for testing and downloading an torrent using the evopedia dumpSettings dialog worked. also adding the archive to the list of active archives worked (see step 7 below). but i could not open any webpage, does wine not have a standard browser already set?

upcomming

  • pictures don’t work with this release, so be patient
  • i’m planning on using a cross compiler from linux to create the windows version of evopedia, maybe also an installer.

if you want to check out my windows xp build, download [5].

how to use the test build?

  1. download [5]
  2. use winzip or winrar to extract all files to a folder. example: c:\evopedia
  3. start evopedia.exe and in the menu there is an entry called: dumpSettings, click it
  4. the dumpSettings dialog opens. it has a menu as well, click “Refresh Archive List” to query online archives. ensure that you have a working internet connection.
  5. now the list in the dumpSettings dialog should contain a list of several lanauges as “de,el,en,fr,it,sl,small”.
  6. use small to test the evopedia application, click ‘small’ and then ‘download’
  7. if your download completet use: ‘menu’ – ‘manually add archive’ and select the directory in c:\tmp

c:\tmp is probably no good default location but we did not change that yet. also note that crei removed the automatic usage of ‘just downloaded’ archives. this makes step 7 necessary.

links

[1] http://qt.nokia.com/downloads

[2] http://msysgit.googlecode.com/files/Git-1.7.3.1-preview20101002.exe

[3] http://github.com/qknight/torrent

[4] http://gnuwin32.sourceforge.net/packages/bzip2.htm

[5] http://www.lastlog.de/misc/wordpress/evopedia-git-windows-build.zip

update

23.10.2010 pictures work in windows with this build, just select ‘allow’ in the network settings instead of ‘auto’. this is done using the menu: menu – network – allow

Read Full Post »

i have a windows xp server on which i use ‘acronis true image home 2009’ for periodical backups.

my problem:

my 1TB backup storage was filled with incremental/full backups although i’ve used incremental backups with consolidation enabled.

to investigate the issue i’ve installed a windows xp into a VirtualBox on my linux machine – to experiment – in the hope to find some easy solution. previously i’d been searching in forums for a fix to the issue but it seems a lot of ppl have the same issues, too, without providing a fix.

the acronis documentation does no explain what ‘acronis’ actually does or it is far to complicated. i’ve read the manual (not the printed version but instead the acronis help when pressing the ‘?’ button on the ‘backups – incremental’ dialog) a few times but i did not understand it as it is quite complex and i don’t like how they explain the single steps.

what i want:

first let’s see what i want:

  1. a incremental backup should be made every day (the first backup is a full backup of course)
  2. the main archive (the first full backup) should be validated on every backup
  3. if more than 6 backups are there, delete the oldest one
  4. the old backup may only be deleted if the new backup is 100% consistent
  5. the backup must always be in a consistent state, no merging of a full+diff when not having another full backup around

i’m not sure if the backup merge (merging a full and a incremental backup) is atomic, which would mean that if the merge fails the old files are not lost. after some experiments i’m not sure but i doubt that it is atomic. i have a bad feeling about this. so i think (4) and (5) can’t be done directly. maybe with two backup jobs, one every ‘odd day’, the other every ‘even day’ would be a solution. but there are so many indications which i can’t take into account – so i currently think the best is to go with (1),(2) and (3) only while checking the backup from time to time manually (which i would do anyway).

so here is the configuration of how i achieve (1),(2) and(3) but NOT (4) and (5) :

backup-method

backup-options

automatic consolidation

problem to exceed the harddrive capacity

to recall: my primary problem was that the backups did exceed the harddrive capacity as no old archives were consolidated:

that resulted from the ‘backup method’ where i also checked the last point ‘create a new backup after x incremental or differential backups’ which then would create a new backup. that means: consolidation was never done since the counter for consolidation was 6 (shown in the screenshots from above) but a new backup was created after 3 successive backups (disabled in the first screenshot).

it seems to me that the acronis is executing like this:

if (sum(backups) > consolidation_threshold) -> consolidate backups

where sum(backups) is interpreted: accumulate all different types as full|incremental|differential

example: sum(one full backup and 3 differential backups) = 4

however: if the checkbox ‘create a new backup on every x’th backup’ is checked the algorithm is never executed when x is smaller than the consolidation_threshold leaving old backups undeleted!

fix

currently i disable the ‘create a new backup after x incremental or differential backups’. that means all options are set as done in the 3 screenshots.

summary

this is a very typical flaw in gui design as intuition misleads based on the facts presented. a very clear and intuitive approach would be a schedule visualization for the current setup. i’m not very happy with acronis currently.

i’ve also purchased the more recent version of ‘true image home 2010’ but it still is the same issue. maybe someone understands this better than me, then please give me a hint.

Read Full Post »