Wednesday, February 19, 2014

Eucalyptus Snapshot issues

A little background:
Company and location of the Cloud were removed intentionally.
Version:            3.2.2
Environment:     Full HA
Data Storage: SAN (Dell Equallogic)

There are two issues related to Snapshots as elaborated below with resolutions. Please note that there could be multiple reasons but in this case only the resolution which were tried to fix the issues were only discussed.

Issue(s):

A.    Snapshot goes to pending state when it is deleted immediately after its creation i.e. while it was still in 
        pending state and was in the process to completion.

B.    Creating new snapshots fails.

Resolution to issue A:
1.      Stop SC service on the disabled one and wait till it comes to NOTREADY state.
# service eucalyptus-cloud stop
From the CLC, check the status using;
# euca-describe-services
or,
# euca_conf --list-sc

2.      Restart SC service on the enabled one and wait till it comes to ENABLED state.
# service eucalyptus-cloud restart
From the CLC, check the status using;
# euca-describe-services
or,
# euca_conf --list-sc

Note: The above action would clear the pending snapshot from the DB.

Resolution to issue B:

1.      Modify /usr/share/eucalyptus/connect_iscsitarget_main.pl on SC with the following entry;
        $ip = "sandatahost";

2.      Modify /usr/share/eucalyptus/disconnect_iscsitarget_main.pl with the following entry;
        $ip = "sandatahost"

3.   Add the "sandatahost" info in /etc/hosts on SC as;
        <IP of sandatahost> sandatahost

4.      Remove the IPTABLE rule on SC for SAN management port as;
        iptables -t nat -D OUTPUT -p tcp -m tcp --dport 3260 -j DNAT --to-destination <IP of
        sandatahost>:3260

Note: The above actions re-established the connection between SC and Database (SAN). And, this resolved the 
           issue of Snapshot Creation.

Migrating Windows-2008 image from Eucalyptus-3.3.1 Cloud To Eucalyptus-3.3.0

Environment:
Please note that IP addresses used here are for example only

Cloud A: 10.10.104.77
Eucalyptus Version: 3.3.1
1. Has a Windows-2008 image

Cloud       B: 10.10.104.12
Eucalyptus Version: 3.3.0

Objective:
To Migrate the Windows-2008 image from .77 to .12

Steps:
1. Copy the admin.zip from 10.10.104.77 to 10.10.104.12

2. source the eucarc file (obtained from 10.10.104.77) on 10.10.104.12

3. euca-download-bundle -b Windows-2008 -p windows_2008 -d ./bundle
Bundle downloaded to './bundle'

4. cd ./bundle
euca-unbundle -m windows_2008.img.manifest.xml -d /var/lib/eucalyptus/bukkits/win-08/

(Where, -d denotes the destination folder to unbundle the image to get the original one)

********************************************************************
Actual output:
# euca-unbundle -m windows_2008.img.manifest.xml -d /var/lib/eucalyptus/bukkits/win-08/
100% |=======================================================================================================================================|   2.34 GB  11.22 MB/s Time: 0:03:43
Wrote /var/lib/eucalyptus/bukkits/win-08/windows_2008.img
********************************************************************

5. VERY IMPORTANT:

Now, source the eucarc of the 10.10.104.12 system.

6. Bundle image:
euca-bundle-image -i windows_2008.img -r x86_64
Wrote manifest /var/tmp/bundle-cX7fFL/windows_2008.img.manifest.xml

7. Upload image:
euca-upload-bundle -b Windows-2008 -m /var/tmp/bundle-cX7fFL/windows_2008.img.manifest.xml
Uploaded Windows-2008/windows_2008.img.manifest.xml


8. Register image:
euca-register -n windows-2008 Windows-2008/windows_2008.img.manifest.xml IMAGE   emi-
        A8F13B8B

9. Now, run an instance (Windows-2008) using the image [Specify keypair (-k), and also type (-t) of
        resource to be used];
euca-run-instances -k bijitkey -t m2.2xlarge emi-A8F13B8B

10. At this point you may wish to check /var/log/eucalyptus/nc.log on the (NC), also keep an eye on the
/var/lib/eucalyptus/instances/work/ULXIPFPAQCOSBKCMXOAMJ/i-4C313CAB/console.log
       (where; ULXIPFPAQCOSBKCMXOAMJ/i-4C313CAB could be different in your case; "i-
        4C313CAB" is the instance ID);

Please remember; the console.log would only get generated once NC downloads the image and 
        initiates the instance;

***************************************************************************
2013-09-23 06:10:12 DEBUG 000029470 doDescribeResource       | returning status=enabled        cores=1/4 mem=3440/7792 disk=58/93 iqn=iqn.1994-05.com.redhat:14bdc74c272
2013-09-23 06:10:12 DEBUG 000029470 doDescribeInstances      | invoked      
        userId=eucalyptus correlationId=UNSET epoch=164 services[0]{.name=cc00           .type=cluster .uris[0]=http://10.104.10.14:8774/axis2/services/EucalyptusCC}
2013-09-23 06:10:12 DEBUG 000029470 doDescribeInstances      | [i-25C63CF1] Extant 
        (not migrating) pub=10.104.3.101 vols=vol-A90F3F43:D
2013-09-23 06:10:12 DEBUG 000029470 doDescribeInstances      | [i-4C313CAB] Pending 
        (not migrating) pub=10.104.3.102 vols=
2013-09-23 06:10:13 DEBUG 000028647 walrus_request_timeout   | wrote 17825792000 
        byte(s) in 1147508 write(s)
2013-09-23 06:10:13  INFO 000028647 walrus_request_timeout   | downloaded 
        /var/lib/eucalyptus/instances/cache/emi-A8F13B8B-ce88f9dc/blocks
2013-09-23 06:10:13 DEBUG 000028647 art_implement_tree       | [i-4C313CAB] 
        implemented artifact 019|emi-A8F13B8B-ce88f9dc on try 1
2013-09-23 06:10:13 DEBUG 000028647 find_or_create_artifact  | [i-4C313CAB] checking 
        work blobstore for 020|emi-A8F13B8B-ce88f9dc (do_create=1 ret=1)
***************************************************************************

11. Once the instance is in running state (below output);
# euca-describe-instances
RESERVATION     r-E6B83FD8      135487945746    default
INSTANCE        i-4C313CAB      emi-A8F13B8B    10.104.3.102    172.16.230.221  running                 bijitkey        0               m2.2xlarge       2013-09-23T13:06:50.899Z        cluster1        
        monitoring-disabled      10.104.3.102    172.16.230.221                  instance-store
TAG     instance        i-4C313CAB      euca:node       10.105.10.21

12. Generate the password (specific to windows instance);
# euca-get-password -k bijitkey.priv i-4C313CAB
fL7k3aua

13. Try login using the RDP with Pub IP and the Administrator username and password shown above;

14. If RDP fails to work; enable RDP port in the Default firewall group (since "default" was used to run the
         instance)
# euca-authorize -P tcp -p 3389 -s 0.0.0.0/0 default

15. Try login to instance again, this should log you in !!