Author: Flayol Frederic

Olinuxino goes to hadoop

One application I would like to test is to know how Olinuxino goes if you using it in an hadoop environment.

At first sight it might not be a very good solution. Indeed it doesn’t have a very big storage habilitity and you can not think of using a smartmedia card as a normal filesystem.

The solution I propose to test is to use a laptop (or another machine) as a master (and hdfs filesystem). I would like to know if filesystem is a bottleneck or if it is the network.

Step 1:Install an hadoop server

In my case it is my laptop and I use a 30Go partion on my sd disk. The IP I will use for my server is 192.168.1.7. 

This step will install a « normal » hadoop distribution on a PC and will be used as

To simplify the best thing is to add alias in /etc/hosts

Step 2: setting up the board

Download standard image fot olinuxino. It can be found here: https://www.olimex.com/wiki/images/2/29/Debian_FS_34_90_camera_A20-olimex.torrent taken from the official olinuxino github: (https://github.com/OLIMEX/OLINUXINO/tree/master/SOFTWARE/A20/A20-build).

First problem is that by default network is not enabled. Change the file /etc/network/interfaces and add

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

Then type:

sudo dhclient eth0
/etc/init.d/networking restart

Get your inetaddress on the board by typing

ifconfig:
    eth0      Link encap:Ethernet  HWaddr 02:cf:07:01:5a:b7

    inet addr:192.168.1.254  Bcast:192.168.1.255  Mask:255.255.255.248
    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get install ssh

edit /etc/ssh/sshd_config

and ensure that the line « PermitRootLogin yes » and « StrictModes no » line has theses values

/etc/init.d/ssh restart

To test if everything is correctly setup go to your server computer and type (password by defaut for root is olimex)

ssh 192.168.1.254 -l root

Try also to connect to your server (in my case adress is 192.168.1.75 and I create an account names local)

ssh 192.168.1.75 -l local

Step 1: Adding a User

sudo addgroup hadoop_group
sudo adduser --ingroup hadoop_group hduser1
sudo adduser hduser1 sudo
su – hduser1
vi ~/.bashrc
# add the folowing line 
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf/jre/
export HADOOP_HOME=/home/hduser1/hadoop
export MAHOUT_HOME=/home/hduser1/hadoop/mahout

ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Step 2: installing hadoop

 

sudo aptitude install openjdk-7-jre

wget http://apache.mirrors.ovh.net/ftp.apache.org/dist/hadoop/common/hadoop-2.7.0/hadoop-2.7.0.tar.gz

tar zxvf hadoop-2.7.0.tar.gz

mv hadoop-2.7.0/hadoop hadoop

Now we have to modify the configuration file to acces to the masternode and the hdfs

Edit hadoop-env.sh

set line export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-armhf/jre

edit the file core-site.xml
and add

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/tmp</value>
    <description>A base for other temporary directories.</description>
    </property>

    <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.1.75:54310</value>
    <description>The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
    </property>

Now you can tst if evrything works:
hadoop fs -ls

Step 3: installing mahout

One easy way to test hadoop is to install mahout. This project includes several hadoop jobs to classify data. So we will using it for test purpose.

Download it from apache:

cd hadopp
wget http://apache.mirrors.ovh.net/ftp.apache.org/dist/mahout/0.10.0/mahout-distribution-0.10.0.tar.gz
tar zxvf mahout-distribution-0.10.0.tar.gz
mv mahout-distribution-0.10.0 mahout

Now everything should be corectly set.

Step 4: Benchmarking

Go into hadoop/mahout

do: sudo apt-get install curl examples/bin/classify-wikipedia.sh

Now to bench do

time examples/bin/classify-wikipedia.sh

How to correctly set up a toolchain

Creating a tool-chain on you PC to compile for ARM, can a good idea and so for several reasons:

  • Your PC is far faster than your Box, and then you can compile all the system in several minutes whereas you would need more than half an hour.
  • You can easily automatize a night build with the latest source to test

But it is not so easy:

You have to compile for a processor than is not necessary yours (from Intel to ARM or GPU for instance) and you would also need to install native library. In this case you have to have your native libraries ans those for the ARM. In this case you cannot use package to install library, you have to do it « by hand »

To easily set up a toolchain you have to:

Install a gcc compiler for ARM. There’s a good version embedded in Ubuntu, and this is the easier part:

sudo apt-get update

sudo apt-get install arm-linux-gnueabi-cpp

For cross library like libc (which is the most important library for compilation)

How to compile the Kernel

Compiling a kernel for A20 is a little more difficult that the way is done on linux.
We need to do three important things:

  • Creating a bootstrap binary
  • Compiling a kernel
  • Compiling specific modules
  • Creating a SD Card

To simplify the process I created a script that automatizes this process,from the start to the copy to the SD card.
To use it:

git clone git@github.com:fflayol/olinuxino-lfs.git
cd system/kernel
./generate.sh

But for a pedagogic purpose I will detailed what this script do.

Creating a bootstrap binary

To begin you have to create a boostrap binary. This binary will be used to initialize and start the box when power on.
After that this binary will start the linux kernel.

sudo apt-get install gcc-4.6-arm-linux-gnueabihf ncurses-dev uboot-mkimage build-essential
git clone -b sunxi https://github.com/linux-sunxi/u-boot-sunxi.git
cd u-boot-sunxi/
make A20-OLinuXino_MICRO_config
make CROSS_COMPILE=arm-linux-gnueabihf-

Compiling the Kernel

Now the next is to compile a kernel to have a specific arm kernel, and support all features we need.
It is an official-modified linux kernel by linux-sunxi.

git clone https://github.com/linux-sunxi/linux-sunxi -b stage/sunxi-3.4
cd linux-sunxi/
cp ../olinuxinoa20_defconfig arch/arm/configs/.config
cp ../olinuxinoa20_defconfig arch/arm/configs/
make ARCH=arm olinuxinoa20_defconfig
patch -p0 < ../sunxi-i2c.patch
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j4 uImage

Compiling modules

Modules are small programs that handles different service in your box. As the kernel is « generalistic » it doesn’t includes all available. If you do not put your specific modules, you won’t be able to use « graphic acceleration » of your box. As a result you will have poor performance.
To do so:

make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j4 INSTALL_MOD_PATH=out modules

This command compiles all modules and put them into out directory.

Creating a SD Card

Now as we have to compile all things,to put them altogether.
The solution is to use a SD Card of at least class 10 4GO.
We split this SDCard in two partition. The first part will include th bootstrap and the second partition will be looks like what we used to have. As this part is quite tricky I didn’t included in the script.

Put you SDCard in the reader (we assume that it can be accessed under /dev/mmcblk )

sudo fdisk /dev/mmcblk
n p 1
start: 2048 end: 34815
n p 2
leave the start and end value to default
w

Be careful when doing that you erase everything you have on your card !
Now format you partition:

sudo mkfs.vfat /dev/mmcblk0p1
sudo mkfs.ext3 /dev/mmcblk0p2

Now add files to the partition

cd ../u-boot-sunxi/
#to simply path
export card=/dev/mmcblk0
#add boot part on partition 1
sudo dd if=/dev/zero of=${card} bs=1M count=1
sudo dd if=u-boot-sunxi-with-spl.bin of=${card} bs=1024 seek=8

Now add the kernel the second partition.

cd ../linux-sunxi/
sudo mkdir -p /mnt/sdmedia
sudo mount /dev/mmcblk0p1 /mnt/sdmedia
sudo cp arch/arm/boot/uImage /mnt/sdmedia
sudo cp script.bin /mnt/sdmedia
sudo sync
sudo umount /mnt/sdmedia
sudo rmdir /mnt/sdmedia 

Now the last part is to add the rootfs and erase nominal driver by the one we’ve just compiled.

Just do the rootfs part, already described in a previous article.

Copy all files into /dev/mmcblk0p1

How to easily started with Mali GPU -1-

The Mali-400 is a very powerful GPU but not very easy to use.

This series of article will help you understand and use it.

First of all download « lite edition » of Sourcery from Mentor: GCC Toolchain

 

This includes a gcc compiler and OpenCl (support): Mali OpenCl

Be careful that the OpenCL and Gcc toolchain are 32 bits. If you use a 64bits version, be sure to install 32bits library.

To verify is everything is correctly installed type
cd Mali_OpenCL_SDK_v1.1.0/samples/sobel and type make

Depending on your environment it can be difficult to install a cross compilation, if you can not succeed you can compile directly on the Olinuxino A20, but compilation time will be longer.

How to easily create a rootfs for A20

git clone https://github.com/fflayol/olinuxino-lfs or wget https://github.com/fflayol/olinuxino-lfs/archive/master.zip 
unzip master.zip
cd olinuxino-lfs//system/rootfs/
./generate.sh

One of the most difficult problem is to create a linux rootfs. Indeed, as we use a specific target board, it can be difficult to have an up to date system.

Nevertheless I’ve found a command that can automaticly creates a  debian rootfs.

Easy way to create:
In a Xterm do:

cd system/rootfs/
second_stage.sh

Wait a while and after that a full debian wheezy rootfs will be created.

This script uses a verify useful command named debootstrap. This command will download the debian filesyem for wheezy with armhf (arm hard float, that means we have a float processor).

Problem is that this root filesystem is an « original version » that can be older than one year.

That the reason why I created a second script named second_stage.sh.

This script is called by generate in a chroot (it means that we executes the command in closed environment, the one we created with deboostrap). The ails of this script is to add deb repository, updates packages with apt and then configures statically networks interface.

Now the next step to have a fully functional environment is to :

  • add a specific kernel binary for A20.
  • configures and adds specific modules and library for A20 to enable and enhances performance.