• About
  • Home

Big is Small

~ APIs, ML Engineering at scale, and Cloud is making it all small, connected and intelligent.

Big is Small

Tag Archives: Virtualization

Setting up a local Development Environment for Playing with Big Data: Part 1

23 Monday Jun 2014

Posted by santanu77 in How-to

≈ 2 Comments

Tags

Big Data, LXC, VirtualBox, Virtualization

Creating a VirtualBox Environment

I wanted to have a local environment with more than one virtual nodes so that I use that environment to simulate a cluster of servers distributed systems need. For example if I need a hadoop cluster for development work it would be so neat to have everything within my MacBookPro itself. I decided to use LXC because it would be lightweight to run so many nodes on a single physical system. But I did not want to install the LXC software directly my MacBookPro. I decided to use a VirtualBox guest on my MacOS host. This guest can be used to host the Linux containers. CentOS was my preferred OS for the guest. But I soon realized installing LXC on CentOS 6.3 is a bit challenging. I had read something about CentOS natively supporting LXC. Probably that was another future version. The version I had of CentOS also had Security Enabled Linux (SELinux) that makes it even more cumbersome to run LXC on CentOS. So I got a VirtualBox image of ubuntu instead from the following link http://virtualboxes.org/images/ubuntu-server/

Ubuntu Linux Server Edition 14.04 x86
Size (compressed/uncompressed): 430 MB/1.4 GB
MD5SUM of ova image: 7afed719e42e59f870509b6ffe53c442
Link: https://s3-eu-west-1.amazonaws.com/virtualboxes.org/ubuntu-14.04-server-i386.ova.torrent
Active user account(s)(username/password): ubuntu/reverse
Notes: US keyboard, Guest Additions NOT installed. Additional packages: OpenSSH server.

Configured a NAT and a host only network on it. Gave it about 3 GB or RAM considering that my MacBook only has about 8GB.

screenshot

Then start it up!

 
ubuntu@ubuntu-i386:~$ uname -a Linux ubuntu-i386 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:08:14 UTC 2014 i686 i686 i686 GNU/Linux 

NAT helps accessing internet from the guest box. Host only network on the guest allows it to be accessed by a static IP from the host. I could now ssh into the box from my iTerm running on the host Mac OS.

Installing LXC on Ubuntu

Used the following simple steps to install LXC on ubuntu. It was very straightforward. Went through without any issues

sudo apt-get install lxc

Note that after the installation one more bridge network interface is added to the ubuntu OS

ubuntu@ubuntu-i386:~$ ifconfig

eth0 Link encap:Ethernet HWaddr 08:00:27:60:64:8b
 inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
 inet6 addr: fe80::a00:27ff:fe60:648b/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:43555 errors:0 dropped:0 overruns:0 frame:0
 TX packets:19870 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:62172162 (62.1 MB) TX bytes:1239397 (1.2 MB)

eth1 Link encap:Ethernet HWaddr 08:00:27:ea:17:80
 inet addr:192.168.100.102 Bcast:192.168.100.255 Mask:255.255.255.0
 inet6 addr: fe80::a00:27ff:feea:1780/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:3867 errors:0 dropped:0 overruns:0 frame:0
 TX packets:3112 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:389120 (389.1 KB) TX bytes:900839 (900.8 KB)

lo Link encap:Local Loopback
 inet addr:127.0.0.1 Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING MTU:65536 Metric:1
 RX packets:20 errors:0 dropped:0 overruns:0 frame:0
 TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:1560 (1.5 KB) TX bytes:1560 (1.5 KB)

lxcbr0 Link encap:Ethernet HWaddr 82:b0:dc:e5:4e:a5
 inet addr:10.0.3.1 Bcast:10.0.3.255 Mask:255.255.255.0
 inet6 addr: fe80::80b0:dcff:fee5:4ea5/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

Testing a Container

I needed to check that the installation worked by creating a container and accessing it

ubuntu@ubuntu-i386:~$ sudo su
root@ubuntu-i386:/home/ubuntu# lxc-create -n test1
root@ubuntu-i386:/home/ubuntu# lxc-start -d -n test1 -o output.log
root@ubuntu-i386:/home/ubuntu# lxc-ls --fancy

NAME STATE IPV4 IPV6 AUTOSTART
------------------------------------------
test1 RUNNING 10.0.3.92 - NO

In order to login to the container use:


root@ubuntu-i386:/home/ubuntu# lxc-attach -n test1

Check that network is working:


root@test1:/home/ubuntu# ping google.com
PING google.com (74.125.236.162) 56(84) bytes of data.
64 bytes from maa03s16-in-f2.1e100.net (74.125.236.162): icmp_seq=1 ttl=61 time=30.2 ms
64 bytes from maa03s16-in-f2.1e100.net (74.125.236.162): icmp_seq=2 ttl=61 time=22.8 ms
64 bytes from maa03s16-in-f2.1e100.net (74.125.236.162): icmp_seq=3 ttl=61 time=45.3 ms

In order to stop and destroy the container :

root@ubuntu-i386:/home/ubuntu# lxc-stop -n test1
root@ubuntu-i386:/home/ubuntu# lxc-destroy -n test1

Now that the containers are available within the VM these can be used to install hadoop or any other software.

In Part 2 of this blog write-up I will explore using Docker instead of linux containers. Advantage of docker is that various pre-installed containers are easy to pull and run in the containers, without having to start the installation from the scratch.

Topics

AI analytics API Big Data container Device docker IoT java kubernetes logging LXC Machine-Learning ML Oauth Oauthv2.0 performance Protocol security Sensor VirtualBox Virtualization

Recent Posts

  • ML Certifications to pursue in 2022
  • Operational Challenges of Data Science
  • Data Science Platform Capabilities
  • Data Management Capabilities Needed for Real-time Predictive Maintenance Use Cases
  • Analytics Platform Assessment Questionnaire Download

Blog Posts

  • January 2022 (1)
  • July 2019 (1)
  • January 2019 (1)
  • December 2018 (1)
  • September 2017 (1)
  • September 2016 (1)
  • July 2016 (1)
  • March 2016 (1)
  • February 2016 (2)
  • October 2015 (1)
  • September 2015 (1)
  • May 2015 (1)
  • April 2015 (2)
  • September 2014 (1)
  • June 2014 (2)

Categories

Follow @Santanu_Dey on Twitter

My Tweets

Blog at WordPress.com.

  • Follow Following
    • Big is Small
    • Already have a WordPress.com account? Log in now.
    • Big is Small
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...