Feeds:
Posts
Comments

Archive for the ‘package manager’ Category

what does ‘cloud’ stand for?

simply speaking, a cloud consists similar computers (homogeneous hardware). usually every single cloud computer runs the same OS (host system), each controlling various guests. the main technical motivations are:

  • load balancing of cpu load (move the VM to a machine with more CPU power)
  • load balancing of input/ouput load (RAM increase; faster storage raid; in memory databases)
  • load balancing of bandwidth usage (move the VM to the most demanding users)
  • increase redundancy (reduce hardware failures; reduce power loss issues)
the main non-technical motivations seem to be:
  • marketing – ‘cloud’ sounds cool, although a ‘cloud’ is basically just a subset of the internet
  • vendor lock-in – probably no surprise to anyone
  • centralization – cheaper to manage; grants more control over the platform

if you are a customer to a service which is hosted in the cloud you usually don’t see the cloud at all, hence the term ‘cloud’:

  • amazon: when buying books or other things
  • google: using google search; reading email; google maps
  • microsoft azure: whatever that platform is good for, is actually anybody using this?
if you are using one or more machines from a cloud, there are basically two interesting patterns:
  • actively maintain each computer: implement distributed file systems and distributed services
  • using an abstraction: someone implemented the handling of nodes and services you are going to develop thus it is running on top of an abstraction
to sum up, if you want to use cloud computers, you have to decide between:
  • SaaS – Software as a Service (something like google mail)
  • PaaS – Platform as a Service (something like ms azure / amazon e2)
as the trend in hardware design is going towards multicore along with NUMA it seems the cloud is undergoing similar changes. as a rule of thumb i’d say that ‘cloud computing can be seen as an approach to build a distributed operating system‘.

cloud problems

not too long ago you would have maintained your own infrastructure with access to the hardware and software used. but times have changed and the americanization of things, that is by building ‘super services’, is about to change the internet yet again.

i see this issues (no special order):

loss of control

this is probably the strongest argument against using third party proprietary services as you can’t fix it when it is broken. but cloud computing usually means a loss of privacy as well. the article [2] mentioning various points from richard stallman and larry ellison probably makes this point clear. it is interesting to see this SaaS wikipedia article [3] which reads like a campaign for SaaS – probably written by someone with a marketing background. there is the dangers to loose your data to foreign countries, as mentioned in [7].

loss of own infrastructure

you don’t have your own infrastructure anymore, thus you don’t have physical control over your devices. additionally you then depend on working internet connections. it is likely that the infrastructure you rely on runs in one or several different countries.

loss of software not designed for the cloud

the various versions of the GPL had a great influence on how software could be used and distributed but with the advent of the cloud this changes drastically. the way programs, especially webservices, are designed makes the GPL concept useless as it does not affect you at all. however, there is a new license, the ‘Affero General Public License’ [4] which fills that gap.

why is wordpress is not licensed AGPL i wonder? my first guess is laziness as every author of every single patch would have to be asked for license change persmission. but the wordpress hosters could be using the GPL to greenwash their software as they would not have to hand out proprietary extension which might not be released. but who knows?!

loss of knowledge how to setup services similar to today’s cloud servcies

think about email – who operates his own mailserver nowadays? most friends of mine use google mail and this implies: once you are familiar with a service and its workflow you usually do not want to change. especially if the service seems to be free as in google mail for example (but most of my friends seem not to care that google replaced ‘currency’ by ‘privacy’ which is used as payment instead).

as a consequence the knowledge about how to run your own mail server gets lost. if you understand german, listen to alternativlos 18 – ‘Peak Oil, den Weltuntergang, und wie man sich vorbereiten kann’ [5] minute 74 ff – they discuss this issue.

my personal experience

i have a strong tendency to use devices which are capable of bringing me certain services offline. this is why i put a lot of effort into the evopedia application for instance. the nokia n900 is probably another good example where i try to maintain an offline infrastructure – i didn’t even have mobile internet on the n900 for a complete year and yet i was able to do most things using sip/mappero/evopedia and others.

here are some thoughts about online services i use:

wordpress

i use wordpress.com right now and i really hate it for these points:

  • you can’t write offline
  • initial uploading images or updating them is a frustrating process
  • i sometimes loose parts of articles while writing
  • there is no good backup process for offline backups
  • i hate the WYSIWYG editor as it does not work very well
  • wordpress is inconsitent in producing a good web 2.0 workflow, it feels like reloading the page all the time instead of doing so for single dom-tree elements only, as it would be done with web 2.0; if you don’t trust me, have a look at [6] – how the upcoming wikipedia editor works
of course i could host wordpress on my own webserver and i wanted to do that for a long time. the problem is that wordpress is optimized to be run on wordpress.com thus i think it might be too much work for me to support it with proper security updates and plugin management. instead i search for a blog system which uses markdown in combination to git but i didn’t find yet what i am searching for.
don’t get me wrong, i really like wordpress but i don’t like this dependency and lack of flexibility using their software.

google mail/docs

i really love ‘google docs’ as it is a wonderful collaborative platform but i can’t use it as i have to disclose all documents to google i’d be working on.

google android

like google mail and google docs, android has a very good cloud integration. but if you want to use services other than google’s, it is a horrible platform. for instance i keep installing xabber [12] although google uses jabber but intentionally made you require to install third party software in order to use non google jabber. same goes for most other services. if i had to use an android phone i would buy one with proper CyanogenMod [13] support.

github.com

great service for source code hosting using git. still the platform itself is not available like for http://gitorious.org/ or  http://gitlabhq.com/. github.com uses a wiki which is bound to the platform and not contained in the git repo.

note: although i never used http://www.fossil-scm.org i like the idea that it contains a wiki in the repository as well

i use github.com only for free and open source projects.

better without clouds

the conventional use of the term ‘cloud’ simply indicates a buzzword or business term for vendor lock-in and centralized infrastructure you don’t have control of. that is good to know as it helps to recognize and avoid such services. what one should use instead is decentralized infrastructure located near the user, connected to the internet where needed, giving the user the control over the platform.

arguably this concept is implemented as a new trend called ‘personal cloud‘ or ‘private cloud server‘. but these terms are limiting the trend to personal or private matters, yet i would like to see it in businesses as well.

hardware

following the concept of decentralization users can host their own files and other things as address books / calenders on their own home devices.

a list of interesting devices to give you an idea:

  • sheevaplug [9] – there is even a nixos version for this device (by viric)!
  • pogoplug [10]
  • tonidoplug [11]
  • fritz!box (with myfritz and fritznas) [14]

software

software implementing services

a list of software i find interesting:

  • despora [15] – decentralized facebook
  • owncloud [16] – dropbox like service
  • sparkleshare [21] – is a collaboration and sharing tool that is designed to keep things simple and to stay out of your way.
  • tomahawk [19] – a nice music streaming service
  • various p2p / torrent like services:
    • mldonkey [17]

still most ‘personal or private clouds’ scale differently compared to the big 3 mentioned in the beginning of this article. for instance, most of these services are configured in the client/server way and they usually do not implement concepts as failover, backups or load balancing. for that to happen it requires a new set of tools and decentralized frameworks based on p2p technologies – which has just not happened yet.

there is also a political issue: most internet users do not have a decent upload channel, which basically means that their internet connection is not very good.

software for managing services

  • openshift [20] – is a cloud computing platform as a service product from red hat
  • openstack [21] – is a global collaboration of developers and cloud computing technologists producing the ubiquitous open source cloud computing platform for public and private clouds.
  • disnix [8] – is a distributed deployment extension for Nix, a purely functional package manager.
i’ve used neither but i like to point out that there is ongoing open source involvement and interestingly non of these technologies are used in private clouds. private clouds seem to implement the classical client/server paradigm at the moment. there is a remarkable exception, that is filesharing using p2p/kademlia which implements a basically read only storage which scales pretty well already.

a matter of design

to make the private cloud or a decentralized cloud a success we need:

  • a standardized package manager with proper software life-cycle management
  • symmetrical internet connections with decent upload/download speeds
  • transparent support for scalability/reliability/redundancy (the points mentioned in the beginning of the article)
  • powerful hardware with low power usage but capable of high loads
  • encryption and certificates or a chain of trust
  • ipv6 – we need good endpoint communication capabilities
  • a clear understanding of where we want to put our personal data and how we can protect it
i think each requirement on its own is already implemented somewhere but not in combination to each. there is not yet a library providing the software/protocol requirements and the hardware is either not powerful enough or is not intended to be used in that way required.

conclusion

still it is a long way for the private clouds to have the same level of features/quality as the big clouds already have. for the time being it seems to be complicated for the average internet user to use the internet without loosing too much of his individuality, thus the freedom of expression.

links

[1] http://www.google.de/search?sourceid=chrome&ie=UTF-8&q=richard+stallman+cloud

[2] http://www.guardian.co.uk/technology/2008/sep/29/cloud.computing.richard.stallman

[3] http://de.wikipedia.org/wiki/Software_as_a_Service

[4] http://en.wikipedia.org/wiki/Affero_General_Public_License

[5] http://alternativlos.org/18/

[6] https://www.mediawiki.org/wiki/VisualEditor:InezSandbox

[7] http://www.engadget.com/2011/06/30/microsoft-european-cloud-data-may-not-be-immune-to-the-patriot/

[8] http://nixos.org/disnix/

[9] http://de.wikipedia.org/wiki/SheevaPlug

[10] http://pogoplug.com/

[11] http://en.wikipedia.org/wiki/Tonido

[12] http://www.xabber.com/

[13] http://www.cyanogenmod.com/

[14] https://www.myfritz.net/was_ist_myfritz.xhtml

[15] http://de.wikipedia.org/wiki/Diaspora_(Software)

[16] http://de.wikipedia.org/wiki/Owncloud

[17] http://de.wikipedia.org/wiki/Mldonkey

[18] http://trac.edgewall.org/

[19] http://www.tomahawk-player.org/

[20] http://en.wikipedia.org/wiki/OpenShift

[21] http://openstack.org/

[22] http://sparkleshare.org/

Read Full Post »

the problem

lately nix-channel stopped working with this error message:

# nix-channel --update
fetching list of Nix archives at `http://nixos.org/releases/nixos/channels/nixos-unstable/MANIFEST.bz2'...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 421 100 421 0 0 5212 0 --:--:-- --:--:-- --:--:-- 10023
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
caching /nix/store/2n6ifd0gyw3cg4v780yxdl307sfi4mmv-MANIFEST.bz2...
bzip2: (stdin) is not a bzip2 file.
you have an old-style or corrupt manifest `/nix/var/nix/manifests/nixos-unstable nixos-d8b23fa7b687df69ca99c50a659cb9b7.nixmanifest'; please delete it at /nix/store/gq4mqkg155jbpdzs44s6nc6ys1q3aqmh-nix-1.0pre2614_005d1e4/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/Nix/Manifest.pm line 346.
cannot pull cache manifest from `http://nixos.org/releases/nixos/channels/nixos-unstable' at /var/run/current-system/sw/bin/nix-channel line 88.

the fix

# ls -la /nix/var/nix/manifests/

total 28
drwxr-xr-x 2 root root 4096 Jun 6 20:19 ./
drwxr-xr-x 11 root root 4096 Jun 4 12:36 ../
-rw-r--r-- 1 root root 0 Jun 6 19:44 cache.lock
-rw-r--r-- 1 root root 10240 Jun 6 19:44 cache.sqlite
-rw-r--r-- 1 root root 0 Jun 6 20:19 cache.sqlite-journal
lrwxrwxrwx 1 root root 56 Jun 6 20:19 nixos-unstable-1b76fe40b585f44b15266540562d24b5.nixmanifest -> /nix/store/xrl4xmx6jz3zvcqwfy3k2qfycdh2454n-MANIFEST.bz2
-rw-r--r-- 1 root root 64 Jun 6 20:19 nixos-unstable-1b76fe40b585f44b15266540562d24b5.url
lrwxrwxrwx 1 root root 56 Jun 6 19:44 nixos-unstable\ nixos-d8b23fa7b687df69ca99c50a659cb9b7.nixmanifest -> /nix/store/2n6ifd0gyw3cg4v780yxdl307sfi4mmv-MANIFEST.bz2
-rw-r--r-- 1 root root 70 Jun 6 19:44 nixos-unstable\ nixos-d8b23fa7b687df69ca99c50a659cb9b7.url

first, remove all the manifests

# rm /nix/var/nix/manifests/*

then we can remove the store entry:

# nix-store --delete /nix/store/xrl4xmx6jz3zvcqwfy3k2qfycdh2454n-MANIFEST.bz2 1 888 3 pts/17 /nix/var/nix/manifests root@eliteBook-8530w 12-06-06 21:35:59
finding garbage collector roots...
removing stale link from `/nix/var/nix/gcroots/auto/v6aa8ai44dd10ypv8vg6shf8b5mzkhfn' to `/nix/var/nix/gcroots/per-user/root/channels.tmp'
deleting `/nix/store/xrl4xmx6jz3zvcqwfy3k2qfycdh2454n-MANIFEST.bz2'
1 store paths deleted, 6557759 bytes (6.25 MiB, 12840 blocks) freed

afterwards nix-channel –update works

# nix-channel --update
fetching list of Nix archives at `http://nixos.org/releases/nixos/channels/nixos-unstable/MANIFEST.bz2'...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 421 100 421 0 0 5191 0 --:--:-- --:--:-- --:--:-- 10023
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
cached contents of ‘http://nixos.org/releases/nixos/channels/nixos-unstable/MANIFEST.bz2’ disappeared, redownloading...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 421 100 421 0 0 5221 0 --:--:-- --:--:-- --:--:-- 10268
100 6404k 100 6404k 0 0 1499k 0 0:00:04 0:00:04 --:--:-- 1705k
caching /nix/store/xrl4xmx6jz3zvcqwfy3k2qfycdh2454n-MANIFEST.bz2...
downloading Nix expressions from `http://nixos.org/releases/nixos/channels/nixos-unstable/nixexprs.tar.bz2'...
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 425 100 425 0 0 5417 0 --:--:-- --:--:-- --:--:-- 10365
100 3701k 100 3701k 0 0 1404k 0 0:00:02 0:00:02 --:--:-- 1627k
unpacking channel Nix expressions...
these derivations will be built:
 /nix/store/4yd4l79h2nifm56rjpxq7hx10q57yz48-channels.drv
building path(s) `/nix/store/r3i2m2by6hcsa3ic99nyhjdb7p551izl-channels'
unpacking channel nixos-unstable

summary

i don’t know why it stopped working. maybe i did checkout the wrong channel by mistake? but maybe someone else might find himself in the same situation and this posting might help to get things going quickly.

thanks to niksnut@freenode#nixos

Read Full Post »

what is this?

i recently upgraded my hetzner root server and therefore had a system with 2x3tb disks. as fdisk can’t be used to partition disks > 2tb i had to use gpt instead which was quite tricky until it was working. so here is my installation guide. parts of it applies also to other distributions.

this guide uses concepts from the hetzner wiki OpenBSD installation guide [1].

note:

  • gpt is used for both disks
  • there is no extra /boot partition (the system will directly boot from the lvm which is on top of the mdadm); this works since grub2
  • this setup is pretty similar to using fdisk (MBR) partitions
  • this guide still uses BIOS to boot (no EFI/UEFI)
  • /dev/sda1 and /dev/sdb1 are very small partitions (2Mib); they are used to store the grub2 boot stages, see [5]

disk layout

update: 26.5.2012: updated the image according to the swap comment from nbp. swap should not be in the lvm as it might degrade the performance.

the installation

first remove old partitions/mdadm setups

uninstall:

lvremove /dev/myvolgrp/home
lvremove /dev/myvolgrp/system
lvremove /dev/myvolgrp/swap
vgremove myvolgrp
pvremote /dev/md0
mdadm --stop /dev/md0
# to remove the md0 permanently
mdadm --zero-superblock /dev/sda1
mdadm --zero-superblock /dev/sdb1

creating the partitions

update 26.5.2012: also add the swap partition here (not done below!).

parted /dev/sda
mklabel gpt
mkpart non-fs 0 2
mkpart primary 2 3001G
p
Number Start End Size File system Name Flags
1 17.4kB 2000kB 1983kB non-fs
2 2097kB 3001GB 3001GB primary

set 1 bios_grub on
p
Number Start End Size File system Name Flags
1 17.4kB 2000kB 1983kB non-fs bios_grub
2 2097kB 3001GB 3001GB primary

creating the new mdadm softraid device

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

LVM+filesystems

pvcreate /dev/md0
Physical volume "/dev/md0" successfully created

vgcreate myVolGrp /dev/md0
Volume group "myVolGrp" successfully created

lvcreate -n system -L50G myVolGrp
lvcreate -n swap -L8G myVolGrp

mkfs.ext4 -O dir_index -j -L system /dev/myVolGrp/system
mkswap -L swap /dev/myVolGrp/swap

note: the disk layout diagram mentiones a tmp partition which happended to be added later 😉

using a virtual machine + vnc to boot the iso image

preparing the host system:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

on the hostsystem

#download latest console only 64bit nixos installer
nixos-minimal-0.1pre33860-33874-x86_64-linux.iso

make sure /dev/myVolGrp/system and /dev/myVolGrp/swap are not in use:

apt-get install sudo
qemu-system-x86_64 -enable-kvm -m 1024 -hda /dev/md0 -net nic -net tap -cdrom nixos-minimal-0.1pre33860-33874-x86_64-linux.iso -boot d -vnc localhost:0

note: in contrast to original article [1] i use ‘-enable-kvm’ which speeds things up!

from your homecomputer

execute this two commands (in two different shells):

ssh -L 5900:localhost:5900 root@176.9.99.117
vncviewer localhost

inside the qemu/kvm system via vncviewer

how we have to prepare install the system on the devices we had preparted in the steps before:

inside do:
login as root
mount -L system /mnt

cd /mnt
nixos-option --install
vi /etc/nixos/configuration.nix

stop dhcpcd
ip a add 172.2.0.2/16 dev eth0
ip r add via 172.2.0.1
echo "nameserver 8.8.8.8" > /etc/resolv.conf
# use ping www.google.de to verfy that the routing is working

# example url, configuration.nix is appended to this article
curl http://lastlog.de/configuration.nix
mv configuration.nix /mnt/etc/nixos/configuration.nix
# now the installation, make sure you read the nixos installation guide as well, but in short:
nixos-install
# only the grub2 installation should have failed (as there is no /dev/sda1 in the virtual machine!)
#finally we halt the system
halt

im hostsystem we need to install grub2:

apt-get install grub2
grub-install --no-floppy --root-directory=/mnt --recheck /dev/sda
Installation finished. No error reported.

grub-install --no-floppy --root-directory=/mnt --recheck /dev/sdb
Installation finished. No error reported.

# now we add a ssh key so we can login into this system later on
cd /mnt
mkdir root
cd root
mkdir .ssh
chown 0700 .ssh/
cd .ssh
echo "ssh-rsa AAAAB3Nz.....aU79sGVhyOPRz joachim@ebooK" > authorized_keys

from your homecomputer login into the installed system (reboot the host) and then issue this command:

ssh root@176.9.99.117 -i ~/.ssh/myprivatekey

after the first login, nixos-rebuild switch might fail with this error message:

nixos-rebuild switch --fast
building the system configuration...
updating GRUB 2 menu...
installing the GRUB bootloader on /dev/sda...
/nix/store/iaypdz5mm1qk8izs9412cb28v9vwwcn4-grub-1.99/sbin/grub-probe: error: no such disk.
Auto-detection of a filesystem of /dev/mapper/myVolGrp-system failed.
Try with --recheck.
If the problem persists please report this together with the output of "/nix/store/iaypdz5mm1qk8izs9412cb28v9vwwcn4-grub-1.99/sbin/grub-probe --device-map="/boot/grub/device.map" --target=fs -v /boot/grub" to
grub-probe --device-map="/boot/grub/device.map" --target=fs -v /boot/grub
grub-probe: info: Cannot stat `/dev/disk/by-id/scsi-35000c5003f556643', skipping.
grub-probe: info: Cannot stat `/dev/disk/by-id/scsi-35000c5003f5363a6', skipping.
grub-probe: info: changing current directory to /dev.
grub-probe: info: changing current directory to pts.
grub-probe: info: changing current directory to shm.
grub-probe: info: changing current directory to myVolGrp.
grub-probe: info: changing current directory to md.
grub-probe: info: changing current directory to disk.
grub-probe: info: changing current directory to by-label.
grub-probe: info: changing current directory to by-uuid.
grub-probe: info: changing current directory to by-partlabel.
grub-probe: info: changing current directory to by-partuuid.
grub-probe: info: changing current directory to by-path.
grub-probe: info: changing current directory to by-id.
grub-probe: info: changing current directory to snd.
grub-probe: info: changing current directory to mapper.
grub-probe: info: opening myVolGrp-system.
grub-probe: error: no such disk.

so what is inside this device.map anyway?

cd /boot/grub
cat device.map
(hd0) /dev/disk/by-id/scsi-35000c5003f556643
(hd1) /dev/disk/by-id/scsi-35000c5003f5363a6

Jordan_U#grub@irc.freenode.net recommended to remove the device.map. that made it work:

rm /boot/grub/device.map

summary

took quite some time to figure all this out so i guess someone else might have interested in this guide as well. i also tried to install, using EFI, but soon discovered that this might be a very complicated road to go and therefore skipped that.
it is cool to see that there is a very helpful community surrounding key projects required to get this installation done. i would have had to spend much more time if i wouldn’t have had someone to ask from time to time.

links

[1] http://wiki.hetzner.de/index.php/OpenBSD
[2] https://wiki.archlinux.de/title/Gpt
[3] https://wiki.archlinux.org/index.php/GRUB2#GPT_specific_instructions
[4] http://www.wensley.org.uk/gpt
[5] http://en.wikipedia.org/wiki/GNU_GRUB#GRUB_version_2

configuration.nix

# Edit this configuration file which defines what would be installed on the
# system. To Help while choosing option value, you can watch at the manual
# page of configuration.nix or at the last chapter of the manual available
# on the virtual console 8 (Alt+F8).

{config, pkgs, ...}:

{
require = [
# Include the configuration for part of your system which have been
# detected automatically.
./hardware-configuration.nix
];

boot.initrd.kernelModules = [
# Specify all kernel modules that are necessary for mounting the root
# file system.
#
# "ext4" "ata_piix"
"af_packet" "snd_pcm_oss" "snd_mixer_oss" "rtc_cmos" "rtc_core" "rtc_lib" "snd_hda_codec_via" "i915" "joydev" "drm_kms_helper" "snd_hda_intel" "rng_core" "drm" "snd_hda_codec" "thermal" "i2c_algo_bit" "button" "snd_hwdep" "intel_agp" "psmouse" "i2c_i801" "evdev" "snd_pcm" "video" "agpgart" "pcspkr" "serio_raw" "iTCO_wdt" "i2c_core" "snd_timer" "output" "e1000e" "snd" "soundcore" "snd_page_alloc" "sg" "loop" "ipv6" "kvm" "freq_table" "processor" "thermal_sys" "hwmon" "ext4" "mbcache" "jbd2" "crc16" "raid456" "async_pq" "async_xor" "xor" "async_memcpy" "async_raid6_recov" "raid6_pq" "async_tx" "md_mod" "sd_mod" "crc_t10dif" "sata_sil" "ata_piix" "dm_mod" "usb_storage" "usb_libusual" "usbhid" "hid" "ohci1394" "ieee1394" "ahci" "libata" "scsi_mod" "ehci_hcd" "uhci_hcd" "usbcore" "nls_base" "scsi_wait_scan" "unix"
];

boot.loader.grub = {
# Use grub 2 as boot loader.
enable = true;
version = 2;

# Define on which hard drive you want to install Grub.
devices = [ "/dev/sda" "/dev/sdb" ];
};
boot.extraKernelParams = [ "vga=normal" "nomodeset" ];

networking = {
hostName = "nix9000"; # Define your hostname.
# wireless.enable = true; # Enables Wireless.
};

# Add file system entries for each partition that you want to see mounted
# at boot time. You can add filesystems which are not mounted at boot by
# adding the noauto option.
fileSystems = [
# Mount the root file system
#
{ mountPoint = "/";
#device = "/dev/sda2";
label = "system";
}
#{ mountPoint = "/boot";
# label = "boot";
#}

# Copy & Paste & Uncomment & Modify to add any other file system.
#
# { mountPoint = "/data"; # where you want to mount the device
# device = "/dev/sdb"; # the device or the label of the device
# # label = "data";
# fsType = "ext3"; # the type of the partition.
# options = "data=journal";
# }
];

swapDevices = [
# List swap partitions that are mounted at boot time.
#
{ label = "swap"; }
];

# Select internationalisation properties.
# i18n = {
# consoleFont = "lat9w-16";
# consoleKeyMap = "us";
# defaultLocale = "en_US.UTF-8";
# };

# List services that you want to enable:

# Add an OpenSSH daemon.
services.openssh.enable = true;

# Add CUPS to print documents.
# services.printing.enable = true;

# Add XServer (default if you have used a graphical iso)
# services.xserver = {
# enable = true;
# layout = "us";
# xkbOptions = "eurosign:e";
# };

environment.systemPackages = with pkgs; [
zsh wget wgetpaste vimprobable2
];

# Add the NixOS Manual on virtual console 8
#services.nixosManual.showManual = true;
}

Read Full Post »

what is this?

i bought an UPS with two ports: serial and usb. and because i did not know much about the UPS (AEG – PROTECT HOME VA 600) i started to look at the communication protocol. turns out there are lots of good tools for serial line interception but nearly none for the usb stuff. sadly the driver i wrote isn’t needed at all as an email to the nut-ML revealed that this UPS uses the Q1 protocol which is already supported pretty well using the blazer_usb and blazer_ser module.

anyway it was pretty interesting to hack on NUT using debian and later nixos. so here is a guide how to log/analyze serial traffic and how to write a simulator for either side.

and not to forget: thanks to Arnaud Quette for his ups/nut support. there is also a brief nut setup introduction, see [1].

sniff serial port data between UPS and PC

  1. set serial settings to:
    – enable serial port
    – port number com 1 irq 4 io port 0x3f8
    – port mode: host device
    – port/file path: /tmp/interceptty
  2. maybe correct the permissions to /tmp/interceptty
  3. interceptty -s ‘ispeed 2400 ospeed 2400’ -l /dev/ttyS0 | tee mylog | interceptty-nicedump
  4. on the linux host:
    tail -F mylog | grep “<“
  5. start virtualbox vm with a windows xp installed
    note:
     ignore this virtualbox warning: “Ioctl failed for serial device ‘/tmp/interceptty’ (VERR_INVALID_PARAMETER). The device will not work properly.”. it works anyway, at least on my system (using ubuntu 10.10 with standard virtualbox).

using the virtual python UPS

  1. on the server side open /dev/remserialVM
    remserial -d -p 23000 -s “2400 raw” -l /dev/remserialVM /dev/ptmx 
  2. on the client side (same host), do:
    remserial -d -r 127.0.0.1 -p 23000 -s “2400 raw” -l /dev/remserialPY /dev/ptmx
  3. chmod 0777 /dev/remser*
  4. change the virtualbox serial settings:
    – port mode: host device
    – port/file path: /dev/remserialVM
  5. then format a ‘message’ with a hexeditor also called “hexeditor”
  6. start the vm
  7. then send the formated message:
    cat message > /dev/remserialPY
  8. if the message was received by the windows ups monitoring software (it will think that the message it received originated from the UPS and not that it was crafted manually)
note: instead of manually sending messages, i also used the script: ./simulate-ups.py which does that automatically.
note: simulate-ups-monitor.py can be used in an analog way but simply using the ups with a real serial port. i should mention btw, that i was using both a usb2serial adapter and an old computer which still contains one of those ancient serial ports.

simulate-ups.py

#!/usr/bin/python
import serial
ser = serial.Serial('/dev/remserialPY', 2400)

line = ''
count=0

def process_command(cmd):
        print " < incomming: " + cmd
        if cmd == "Q1":
                print "REQUEST FOR DATA FROM USV"
                n = ("20").decode("hex")
                d = ("0d").decode("hex")
                a = ("28").decode("hex") + \
                    "000.0" + n + \
                    "000.0" + n + \
                    "000.5" + n + \
                    "005" + n + \
                    "00.0" + n + \
                    "00.6" + n + \
                    "25.0" + n + \
                    "00000001" + d
                ser.write(a)

while True:
        ch = ser.read(1)
        if ch == "\x0d":
                process_command(line)
                line = ''
        else:
                line = line + ch

simulate-ups-monitor.py

#!/usr/bin/python
import serial
import re
import time
import sys

ser = serial.Serial('/dev/ttyS0', 2400)
#, serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE, 0)

line = ''
count=0
status="unknown"

def write(cmd):
        #print "sending " + cmd;
        ser.write(cmd)

def print_status(status):
                print "status is: Unknown|LostCom|Normal|ScheduledShutdown|60SecsShutdown|ActiveShutdown|CriticalPowerFail: " + status

def process_command(cmd):
        valid = re.compile(r"\([0-9][0-9][0-9].[0-9] [0-9][0-9][0-9].[0-9] [0-9][0-9][0-9].[0-9] [0-9][0-9][0-9] [0-9][0-9].[0-9] [0-9][0-9].[0-9] [0-9][0-9].[0-9] [01][01][01][01][01][01][01][01]")
        if valid.match(cmd):
                #print status + " : VALID REPLY FROM USV   ->    " + cmd
                #(239.5 239.5 235.6 000 49.9 13.6 25.0 00001001
                netz_eingang=cmd.split(' ')[0].lstrip('(')
                netz_unknown=cmd.split(' ')[1]
                netz_ausgang=cmd.split(' ')[2]
                percent=cmd.split(' ')[3]
                hz=cmd.split(' ')[4]
                bat_voltage=cmd.split(' ')[5]
                temperature=cmd.split(' ')[6]
                bits=cmd.split(' ')[7]
                bit1=bits[0]
                bit2=bits[1]
                bit3=bits[2]
                bit4=bits[3]
                bit5=bits[4]
                bit6=bits[5]
                bit7=bits[6]
                bit8=bits[7]
                print status + " " + cmd
        else:
                print "invalid reply detected: " + cmd
                sys.exit(1)

write( ("51310d").decode("hex"))

while True:
        ch = ser.read(1)
        if ch == "\x0d":
                process_command(line)
                time.sleep(1)
                write( ("51310d").decode("hex"))
                line = ''
        else:
                line = line + ch

simulate the UPS monitor

# ./simulate-ups-monitor.py
VALID REPLY FROM USV -> (241.5 241.4 237.5 000 49.9 13.5 25.0 00001001
VALID REPLY FROM USV -> (241.5 241.4 237.5 000 49.9 13.5 25.0 00001001
VALID REPLY FROM USV -> (241.4 241.4 237.5 000 49.9 13.5 25.0 00001001

summary

so would i buy a AEG Protect Home VA 600 again? currently there is no ‘time left’ estimation and therefore i shutdown the system either after 25 seconds or on LB (low battery) but after reloading the batteries the shutdown usually is triggered by the 25 seconds rule after a state change to OB (on battery). i think this is a decent setup and therefore i would probably buy that UPS again. but i don’t really have a clue about UPS devices so there might be much better ones in the same price range, maybe someone on the NUT/UPS ML can make a better recommendation.

what i really dislike is that this product ships with linux support BUT not with NUT support. i later realized that they created their own linux software. what a waste of time, i would rather love to get the specification and then use NUT instead – probably this is the case for nearly all the users seeing that this devices has linux support. but my request to get the specification was simply ignored, so i think there are better vendors out there.

another interesting aspect of nut is how complex the integration in the system is.

links

[1] https://nixos.org/wiki/How_to_setup_UPS/NUT

Read Full Post »

i’ve just finished a wiki page on how to develop arbitrary software on nixos [1] (thanks to viric!). as this is fundamentally different to all other linux and non linux operating systems i think this is worth a posting about this subject in my blog.

the interesting aspect is that nix/nixos provides such a development environment per project so one is not forced to pollute the system environment with the ongoing changes which always lead to horrible side effects as regression (you know when old habits stop working as a tiny update of libX breaks tool Z).

the way it is used is covered by [1] already.

a slightly more complex example

config.nix

     1  {
     2    packageOverrides = pkgs : with pkgs; rec {
     3      # example environment from viric
     4      sdlEnv = pkgs.myEnvFun {
     5        name = "sdl";
     6        buildInputs = [ stdenv doxygen SDL SDL_image SDL_ttf SDL_gfx cmake SDL_net pkgconfig ];
     7      };
     8
     9      # a custom library NOT included in nixpkgs (maybe later it is but assume for this example it is not)
    10      libnoise = callPackage ./libnoise.nix {};
    11
    12      # this is the needed environment for development of my spring random map generator
    13      # type 'load-srmg-env' to load it after installing it using 'nix-env -i env-srmg'
    14      srmgEnv = pkgs.myEnvFun {
    15        name = "srmg";
    16        buildInputs = [ stdenv doxygen cmake libnoise qt4 ];
    17      };
    18    };
    19  }

in the ~/.nixpkgs/config.nix expression i added a custom library which is then available with nix-env, this way it can be installed using (nix-env -i libnoise).

the interesting point is that line 2 contains the rec keyword indicating that all 3 attributes in the attribute set (line 2 to 18) may recursively reference each other. this is required as the the srmgEnv on line 14 where the buildInputs lists libnoise.

the libnoise expression is outsourced (line 10) into the file libnoise.nix (listed below).

libnoise.nix

     1  {stdenv, doxygen, fetchgit, cmake}:
     2
     3  stdenv.mkDerivation rec {
     4    name = "libnoise-1.0.0";
     5
     6    # i also change bits in the library and therefore i like to have it local
     7    # in case i change anything this needs to be done to reflect the change
     8    # 1. make the change 
     9    # 2. use 'git add file_which_has_changed'
    10    # 3. use 'git commit'
    11    # 4. use 'git log' to find the most recent rev
    12    # 5. paste the copied rev in the rev field below
    13    # 6. reinstall the libnoise 
    14    src = fetchgit {
    15      url = /home/joachim/Desktop/projects/libnoise;
    16      rev = "8b5b89b7241a112dfe0b387f7589ea9a2df00b02";
    17      sha256 = "";
    18    };
    19
    20    buildInputs = [ cmake doxygen ];
    21
    22    meta = {
    23      description = "libnoise";
    24      homepage = "http://www.github.com/qknight/libnoise";
    25      license = "LGPL2";
    26      maintainers = with stdenv.lib.maintainers; [qknight];
    27    };
    28  }
    29

the libnoise.nix file is interesting as it references a local git repository. it also lists what to do in order to alter the package.

once the srmg-env is installed (nix-env -i env-srmg) it can be used using: load-srmg-env. as mentioned in [1] this environment will then behave as if one had used ubuntu linux and then installed all the required libraryies.

future

as i noted in [1] nix will soon get a toggle (nix-build –run-env ‘<nixpkgs>’ -A xterm, see [2]) which will clone the environment of virtually any sourceScription on the system. this means one can hack on any software easily by injecting code into the build chain on an arbitrary position – still, this changes won’t be persistent, meaning:

  • after reinstallation of the sourceScription the former version will be installed
  • the environment will not last a reboot of the system (not 100% sure about this)

still it is one step towards the concept of the midstream platform (mentioned in my diploma thesis) and is a great way to test a quick hack.

update: 23.5.2012

another interesting potential property is that tools like kdevelop could be patched to automatically see all the include paths of a complete project and therefore are able to provide automatic code completion without having too much manual effort.

 

kdevelop can do that already! when importing the project’s ‘CMakeLists.txt’, kdevelop reads the ‘found’ entries and therefore collects all the library paths!

links

[1] https://nixos.org/wiki/Howto_develop_software_on_nixos

[2] https://github.com/NixOS/nix/commit/7f38087f35e6f74a73bfdb28da8acd8930565d51

Read Full Post »

n900 findings

315px-Nokia_N900-1repairing the n900 camera

i own a nokia n900 and i’m very happy with it. some time ago it fell of my pocket and then the 5mega pixel camera didn’t work anymore as well as the flashlight. i’m a big fan of QR codes and i was often using the flashlight so this was a problem.

the funny thing is that reflashing the device [2] made it work again. now i wonder why the camera is working again. the camera module is plugged to the n900 as can be seen in the videos on youtube, just search for it [3]. maybe while hitting the ground the camera was disconnected from the device while the device was running and that screwed the firmware? i don’t have a clue but i think it is worth mentioning.

0. symptoms

once the camera was not working i had problems:

  • starting the n900 (coldstart) if still connected to any charger, it simply would not boot
  • starting the camera application; it would report something like: camera device not ready or something similar and instantly close the widget
  • flashlight did not work anymore

1. the backup

1.1 create a backup using the backup tool coming with the n900

this is a really great tool, don’t forget to check all the 4 points in the list ‘what to backup’.

also consider to read [4] (i didn’t do that but on this page there are some screenshots of the backup util i’m referring to).

1.2 dpkg and a list of all files in ‘opt’

maybe that is of use later so i note down all files and installed packages:

ls -la opt > opt_fils

dpkg -l > dpkg_l

1.3 now make a backup of all files

as i also develop for the n900 i had already a working ssh root login:

rsync -av –progress –delete n900:/home/user .

2. eMMC/VANILLA

RX-51_2009SE_10.2010.13-2.VANILLA_PR_EMMC_MR0_ARM.bin

./flasher-3.5 -F ../RX-51_2009SE_10.2010.13-2.VANILLA_PR_EMMC_MR0_ARM.bin -f

3. FIASCO/Rootfs

RX-51_2009SE_20.2010.36-2_PR_COMBINED_MR0_ARM.bin

./flasher-3.5 -F ../RX-51_2009SE_20.2010.36-2_PR_COMBINED_MR0_ARM.bin -f -R

4. restoring the backup

once the device was booted after the udpate the camera was working. what a relief! but i didn’t know where to copy the backup as the folder using the ‘mass storage usb’ feature was nearly empty. so i created a new one and connected it again using usb later. after the backup directory was migrated to the device i disconnected the usb cable and restored my 50mb backup. took quite some time and after the final reboot the device also installs the software which was removed in step 2 and 3. i decided to skip this and to install only needed software manually.

note: after the backup is restored and the n900 restarted the wlan starts working again.

5. restoring the applications

a list of software i installed:

  • gainroot
  • openssh client and server
one these applications are installed plug in the usb adapter and set the device into ‘pc suite mode’ and use gainroot to type:
ifconfig usb0 10.0.0.1/24 up
on the host computer type:
ip a add 10.0.0.2/24 dev usb0
ip l set usb0 up
ssh root@10.0.0.1
to find applications use the app catalog on the n900 but to find the associated package i did:
apt-cache search adblock | grep plus
(and similar commands)
all other applications can be installed using apt-get via usb or wlan:
  • rsync
  • adblock plus
  • adflashblock-css
  • agenda timer
  • battery-eye
  • countdown timer
  • datetoday home desktop widget
  • evopedia
  • flashlight
  • convert
  • gpsjinni
  • headphone daemon
  • healthcheck
  • htop
  • iptraf
  • irssi
  • mad-developer
  • mappero
  • mbarcode
  • mbarcode qr code plugin
  • mbarcode webrequest plugin
  • mbarcode wifi plugin
  • mclock
  • mstardict
  • nmap
  • ogg support
  • orrery
  • password safe
  • personal ip address
  • quicknote
  • recaller
  • tuner
  • view contact info with bigger font
  • wifieye
  • wifi switcher
  • foreca weather applet
using the ssh shell one can type this on the device as root:

apt-get install adblock-plus-1.0
apt-get install adflashblock-css agenda rsync
apt-get install battery-eye wifi-switcher countdowntimer datetoday-home-widget evopedia flashlight-applet
apt-get install gpsjinni headphoned healthcheck htop network-monitor irssi mad-developer
apt-get install maemo-mapper mbarcode-plugin-qrcode mbarcode-plugin-wifi mbarcode-plugin-webrequest mbarcode mbarcode-core
apt-get install mclock mstardict nmap ogg-support decoders-support orrery pwsafe
apt-get install personal-ip-address quicknote recaller tuner contact-zoom wifieye foreca-weather-applet

other software i newly installed
  • monav
  • snuggle
  • wallet
  • mdbus2
again – using the ssh shell one can type this on the device as root:

apt-get install monav-routing-daemon snuggle qtwallet mdbus2

6. remove the photo click sound and the intro animation

ssh into the n900 and type:

rm ./opt/usr/share/hildon-welcome/media/Hands-v32-h264.avi ./home/opt/usr/share/hildon-welcome/media/Hands-v32-h264.avi

rm /usr/share/sounds/camera_snd_title_*wav

reboot

random notes

  • after the reinstallation everything is much faster now. the settings dialog loads twice as fast and the desktop feels much smoother.
  • application load times are similar to before
  • for some programs i had to copy files from my backup to the device manually:
    • my documents
    • evopedia dumps
    • monav map data
    • my camera pictures
    • osm2go
    • mstardict
    • music
    • quicknote
  • monav is probably one of the coolest programs for the n900, thanks very much to Christian Vetter and Christoph Eckert

links

[1] http://maemo.cloud-7.de/Gallery-N900-exploded/

[2] https://wiki.maemo.org/Updating_the_tablet_firmware#Updating_.26_Flashing_your_device

[3] http://www.google.de/search?q=Nokia+N900+Disassemble&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:de:official&client=firefox-a

[4] http://prokonsul.blogspot.com/2011/01/make-backup-and-restore-from-it-on-n900.html

[5] http://natisbad.org/N900/n900-commented-hardware-specs.html

Read Full Post »

kdevelop on nixos

kdevelop on nixos

issue 1

i’ve adapted to a nixos only desktop and i’m facing new problems every day, lately i wanted to use kdevelop but it wasn’t able to import a normal ‘Makefile’based project. next i installed lots of packages somehow mentioned at [1]. still that alone didn’t help so i ran:

kbuildsycoca4

kbuildsycoca4 is a utility which generates a cache of .desktop files and such to be used by programs. PovAddict#kde-devel@irc.freenode.net recommended to set KDEDIRS correctly because when doing so running kbuildsycoca4 manually wouldn’t be neccessary.

not sure how to solve this problem correctly yet but in case i find out i will add it to this post.

update: 30.06.2012

in /etc/nixos/configuration.nix write:

environment.systemPackages = with pkgs; [
  kde4.kdevelop
];

after nixos-rebuild kdevelop will have makefile and cmake support. however, at least for proper cmake support, needed when kdevelop should build the project, you will need to prepare a decent environment where all the needed programs and dependencies are listed, see [2]

issue 2

another point which bugs me is that auto-completion doesn’t work correctly as there is no such thing as /usr/include on nixos as one might expect when coming from a different linux distribution. didn’t find a solution here yet either…

update: 30.06.2012 – found a nice solution to this problem, just have a look at [2] as well

links

[1] http://techbase.kde.org/Projects/KDevelop4/HowToCompile

[2] https://invalidmagic.wordpress.com/2012/03/23/developing-software-using-nixos/

Read Full Post »

Older Posts »