Remotely accessible Raspberry Pi webserver with dynamic dns

raspberryThis tutorial will show you how make a webserver running on your Raspberry Pi accessible through a domain.

I assume the following prerequisites:

  • Raspberry Pi with Raspbian installed
  • Router with NAT and portforwarding capability
  • Nginx or other web server installed and running @ port 80 on your Raspberry Pi

If you haven’t installed your Raspberry Pi yet, please refer to the official documentation.

 Dynamic DNS

Two problems arise when trying to access a Raspberry Pi externally:

  1. In a common home network setup, the Raspberry Pi is connected via LAN or Wlan to a router, which accesses the internet. The router hereby maintains the IP adresse provided by the ISP and can be accessed externally. With private home internet connections, this IP adresse is not constant and changes every 24 hours. Ergo, a service is needed which maps the router’s IP adresse to a domain and keeps track of IP adresse changes.
  2. Furthermore, in order to protect the network clients, i.e. the raspberry pi, the router uses NAT (network adresse translation) to assign internal IP adresses to the network clients. Thus, only the router is accessible remotely, while the other network members remain hidden. As the webser running on the Raspberry PI must be available, port forwarding must be used. Within the router administration, port forwarding has to be activated for port 80  (HTTP) assigned to the ip adresse of the Raspberry Pi.  This means that all TCP packets sent to the ip adresse of the router on port 80 are forwarded to the Raspberry Pi, which now is externally reachable on port 80. Please consult your router documenation on how to change these settings.

Registering a free dynamic dns account

twodns.de offers free dynamic dns service and does not require a monthly login by the user. You can register by providing an email adresse. After confirming your email adresse, you can create up to five domains and map them to your IP adresse. Choose a low TTL (time to live) to ensure that your site is available even with frequent IP adresse changes of your router.

Creating a cron job for automatic ip adresse update

If your router supports custom dyndns services, you can set your router to update your IP adresse. Follow the instructions provided by twodns. If this is not the case, you  can programme your Raspberry PI to update the adresse in constant intervals using a cron job.

twodns provides an API to update your IP adresse. For this purpose, you will require the token which is linked to your account. You can retrieve your token by logging into your twodns account. Look for API in your dashboard.

Once you have received your toke, e.g. 12345abcdefg, create a shell script called update-dns.sh with your preferred editor.

vim update-dns.sh

Add the following lines to your script.

curl -v -X PUT -u "email@email.de:12345abcdefg" -d '{"ip_address": "auto"}' https://api.twodns.de/hosts/test.dd-dns.de

The script uses curl to send a http PUT request to twodns server. email@email.de is your account email adresse, which is followed by a colon and your token. The IP adresse is retrieved automatically from the current IP adresse of the Raspberry PI. ‘test.dd-dns.de’ is your dyndns domain and must be replaced by your domain name.

Save and exit the editor. Next, make your script executable:

chmod u+x update-dns.sh

Edit your crontab.

sudo crontab -e

Finally add the following lines to your crontab.

@reboot /home/pi/update-dns.sh
5,20,35,50 * * * * /home/pi/update-dns.sh

The first line runs the update script on reboot, while the second line lets it run every 15 minutes (namely: 5 minutes past the hours, 20 minutes past the hours etc.). If you want to minimize the time where your router’s IP adresse has changed but has not been logged to your dyndns domain, choose smaller time intercals here. Save and quit the editor and reboot your Raspberry Pi. If all went well, you should  be able to access your webserver using your dyndns domain.

Plesk – Maximum upload size in phpMyAdmin

Plesk runs on a separate Apache instance which uses its own configuration. In order to change the maximum file size, edit the file /usr/local/psa/admin/conf/php.ini.

Add or update the values for upload_max_filesize and post_max_size, e.g. 25M for 25 megabytes. 

As last step, restart the Apache instance responsible for Plesk user interface.

/etc/init.d/psa stop
/etc/init.d/psa.start

Read-Only VNC using x11vnc

Viewing Ubuntu Desktop machine via VNCHere’s how to remotely log into a Linux machine desktop session from a windows machine. With x11vnc you can log into a running desktop machine session connected to a physical screen. I am assuming a Debian or Ubuntu-based host machine with a running X environment.

 

  1. Download and install x11vnc.
    sudo apt-get install x11vnc
  2. Now let’s create a password.
    x11vnc -storepasswd

    The password is written to:

    ~/.vnc/passwd

    (Note: The vnc connection is unencrypted and therefore should only be used within trusted network. See the x11vnc documentation for secure connection via SSH tunneling)

  3. Once you have created your password, start the server with
    x11vnc -usepw -forever -viewonly
  4. You can now connect to your machine via a VNC client and are promted for the password by doing so. For Windows, I recommend ultravnc viewer.

Autostarting a browser on the Raspberry Pi

In this tutorial, I’ll show you how to start a browser after booting on the raspberry pi.

  1. Disable login
    In case you want the browser to start without having to log in, apply the following steps:

    sudo vi /etc/inittab

    Comment this line …

    1:2345:respawn:/sbin/getty 115200 tty

    … like this.

    #1:2345:respawn:/sbin/getty 115200 tty1

    Add the following line:

    1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

    Save and exit vi with :wq.

  2. Start X server after boot
    Edit this file:

    sudo vi /etc/rc.local

    And add the following line before the line ‘exit 0’.

    su -l pi -c startx

    Save and quit vi.

  3. Autostarting bash script
    Let’s create a bash script that should be executed after startup. Here we shall place the command to execute the browser. Of course, you can extend this script as you like.

    vi /home/pi/start.sh

    Now add the following code to start the browser midori in fullscreen mode pointing at the adress www.bbc.co.uk.

    #!/usr/bin/sh
    midori -e Fullscreen -a www.bbc.co.uk &

    Make your bash script executable.

    sudo chmod +x /home/pi/start.sh

    Add the script execution to your LXDE autostart file.

    sudo vi /etc/xdg/lxsession/LXDE/autostart

    Add the line

    @sh /home/pi/start.sh

    That’s it! You’re good to go. Try rebooting.

  4. Optional steps
    You may notice your screen going black or a screensaver starting which might not be what you want for your application. Luckily, you can change this in a few easy steps.
    Install the package x11-xserver-utils.

    sudo apt-get install x11-xserver-utils

    Now edit the autostart file again. Uncomment the screensaver and add a few lines. Your file should look something like this.

    @lxpanel --profile LXDE
    @pcmanfm --desktop --profile LXDE
    #@xscreensaver -no-splash
    @xset s off
    @xset -dpms
    @xset s noblank
    @sh /home/pi/start.sh
    

    Your screen should stay on permanently.
    If you would like to hide the mouse, install the tool unclutter and add the following line to the top of the start.sh:

    unclutter -idle 5 &

 

 

Installing Cloudera’s Hadoop Distribution CDH4 to a Virtual Machine

In this post I shall explain how to install Cloudera’s Hadoop distribution CDH4 to a virtual machine running Ubuntu 12.04 LTS from scratch. The second part can be used to install Hadoop natively to you Ubuntu 12.04 installation. However, this tutorial was written with a development or experimental environment in mind.

Requirements:

  1. A Computer with at least 4GB of RAM (8GB+ recommended), as the virtualization of Hadoop consumes a lot of memory.
  2. 64 bit processor with activated Hardware virtualisation in BIOS (If you don’t have a 64 bit processor, you won’t be able to use Ubuntu with CDH 4 and have to choose another OS)

 Virtual machine installation:

  1. Download the .iso-file for Ubuntu 12.04 LTS 64 bit from www.ubuntu.com.
  2. Next, download Virtualbox for your host operation system, e.g. Windows 7. The host operating system is the system running Virtualbox, whereas the virtual operating system is called guest system.
    You will find the binaries for your specific operating system with further installation instructions at https://www.virtualbox.org/wiki/Downloads.
  3. After installation, open Virtualbox and create a new virtual machine. Call it Ubuntu Hadoop or any other name and select Linux, Ubuntu 64bit as operating system.
  4. In the next step, you will have to create a new virtual hard disk. I’d recommend a minimum of 20GB in VDI format with dynamic allocation. Dynamic allocation means that the file on your host disc won’t be a fixed size but will grow with the hard disc of the guest system. However, it will be limited to a fixed size, i.e. 20GB. Initially this will be slower than a fixed size disc, while consuming less space on the host system.
  5. Once you have created the virtual machine, start your virtual machine by double clicking on the name. You can now choose the downloaded Ubuntu .iso-file as start disk.
  6. This will open the Ubuntu installer. Follow the on screen instructions to install Ubuntu to the virtual machine. Visit the Ubuntu Homepage for further help with the installation.

Prerequesites to installing CDH4

  1. Install open ssh server package.
    sudo apt-get install openssh-server
  2. Set a password for root. WARNING: This is only set for this virtual machine. In production environments, this could present a security risk.
    sudo passwd
  3. Edit /etc/hosts in a editor with root privileges. Uncomment the second line. It should look something like this.
    127.0.0.1    antony-VirtualBox localhost
    #127.0.1.1    antony-VirtualBox
    
    # The following lines are desirable for IPv6 capable hosts
    #::1     ip6-localhost ip6-loopback
    #fe00::0 ip6-localnet
    #ff00::0 ip6-mcastprefix
    #ff02::1 ip6-allnodes 
    #ff02::2 ip6-allrouters

Installing Cloudera Hadoop Distribution (CDH4)

  1.  Go to www.cloudera.com and locate Products – CDH. Click ‘Download and Install CDH 4’. On the next page, click ‘Download and Install CDH 4 automatically’. On the following page, under Cloudera Manager 4.1.1, click ‘download‘. Save the .bin-file to disk.
    Cloudera Homepage
  2. Make the installer executable and run it as root.
    chmod u+x ./cloudera-manager-installer.bin
    sudo ./cloudera-manager-installer.bin
  3. Accept the licenses and follow the on-screen instructions. It is important to be patient! The installer may seem to have crashed at times, however, it simply takes its time to install. At the end of the installation a browser should open.
  4. Log in with the credentials ‘admin’, ‘admin’.
  5. You can now add hosts to your cluster by clicking ‘Hosts’ and then add hosts. Enter localhost or 127.0.0.1 as IP. In the process you can choose whether to install YARN or MRv1. Be sure to select the latter.
  6. After the installation and configuration of your cluster, you can access your running services under ‘services’.
  7. You have successfully installed a one-node cluster on the virtual Ubuntu machine.

Recovering deleted photos from memory card

The other day, I received a friend’s camera where no pictures were readable anymore. After examining the memory card, it was clear that the file system was corrupted, however some photos might still be available.

Photorec is a great tool for this job. It can be found in the ubuntu repositories and is pretty straightforward to use. It basically reads all bits and bytes on the memory card and looks for the beginning and end of image files.

In case you are worried about overwriting files on the memory card with the tool, you can first create an image of the disk.

sudo dd if=/dev/sdb1 of=image.dd

/dev/sb1 is the path to your mounted memory card. If you are unsure which device it is, check your mount table with the ‘mount’ command. Keep in mind that this step can take quite some time to finish, depending on the size of the memory card.

Then simply run :

photorec image.dd

Follow the on-screen instructions to finish recovering the files. In my case, photorec was able to recover all lost pictures.

Using counters in Hadoop MapReduce

Sometimes when running MapReduce jobs, you want to know whether or how often a certain event has occured during execution. Imagine an iterative algorithm that should run until no changes were made to the data during execution. Let’s assume that the change happens in the map function.

A common mistake would be to use the context object and set a value in the configuration object.

context.getConfiguration().set("event", "hasOccured");

This approach only works when executing the job on a single machine. When running on a cluster of computers each mapper or reducer will have its own configuration object. Therefore, global reading and writing is not possible. It can only be used to store information before executing the job and passing this information to the mappers and reducers, e.g. filenames of auxiliary files, etc.

In order to know how often a mapper has changed a data item on a global level, we will use a so-called counter. First you have to define an Enum which represents a group of counters. In this example we will only have one counter.

public enum MyCounters {
Counter
}

From within the map method of our mapper, we can access the counter and increment it when we change a dataset.The counter is identified by the enum value.

context.getCounter(MyCounters.Counter).increment(1);

Finally, we can read the counter after job execution and see whether the data has changed.

job.getCounters().findCounter(MyCounters.Counter).getValue();

All counters are displayed during job execution.

Summary: Counters are a useful feature provided by the hadoop framework to globally certain values during job execution. They can also be analyzed to count how many damaged or malformed datasets were in the input data.

Phonegap: Building Android apps without eclipse

In this tutorial I will explain how to get a simple Android project up and running using phonegap in a Linux environment without Eclipse. I will assume basic knowledge of Android app development. If you need help with installing the Android SDK, head over to developer.android.com. You should know how to operate a terminal.

Prerequisites

  1. You will need the Android SDK downloaded and unzipped. Also you have to add the tools from the SDK to your PATH environment, e.g. by adding it to your .bashrc file.
  2. Install a target API with the android command. In my case, I will use Android 2.2.3. You can check which targets are installed by executing the command
    android list targets
  3. Install a virtual device or activate debugging in your Android phone
  4. Download and unzip phonegap from www.phonegap.com. (Version 1.4.1 used in tutorial)
  5. Install ant by executing
    sudo apt-get install ant

Creating the project

  1. Open a console and start by creating a new folder for your project.
  2. Locate the folder containing the unzipped phonegap archive. Copy the folders phonegap and .phonegap from /*your-phonegap-folder*/lib/android/example/ to your project folder.
  3. List android targets and note the id of the target you’d like to use for your project. In this case only one target is installed.
    Next, you have to change the config file ./phonegap/config. Choose a name for the package and the main activity. Set the Target to the noted value. Your config file should look something like this:VERSION=1.4.1
    PROJECT_PATH=./project
    PACKAGE=com.antony_neu.TestApp
    ACTIVITY=TestApp
    TARGET=1
  4. Execute ./phonegap/create. This will create your project according to your config file.
  5. The html and script files are located in ./assets/www. An example phonegap file can be found here by default.
  6. Once you are ready to build, fire up an emulator with emulator -avd AVD_NAME and execute ant debug install . The app will be installed to your emulator device and is located in the Apps menu.

Troubleshooting

Q: I update my script files but there are no changes on the device. What is wrong?
A: Try deleting the ./bin folder and run ant again.

Benchmark your hard disk in Ubuntu

If you want to compare your hard disk speeds in Ubuntu 11.10, you can do this in three easy steps:

  1. Start the programme ‘Disk Utility’.
  2. Select the desired hard disk you want to test and hit ‘benchmark’.
  3. Compare the results. Here I’m comparing an old conventional hard disk drive to my new SSD drive. As expected, you notice a huge difference in speed.