4

I am trying to autoscale my present instance, I am running a medium instance right now and auto scale with small instance. I used command line tool to config the settings, this is the configurations I have used to scale and I am running minimum of one instance apart from my regular instance, that means one more instance and I have attached to the load balancer.

s-create-auto-scaling-group groupname --launch-configuration launchconfig --availability-zones ap-southeast-1a --min-size 1 --max-size 5 --load-balancers prod

But when I have checked the load balancer it says "Out of Service" reason "Instance has failed at least the Unhealthy Threshold number of health checks consecutively". How can I solve this, using its public DNS, I am not able to get any response from the instance and also not able to ssh to it since the key value pair is not attached to the newly created instance.

what is the problem.How do I solve this.

Please help me out its little urgent, since I am struck up at this point from almost 2 days.

as-describe-launch-configs --show-long --headers



testLC,ami-e8c4bdba,t1.micro,(nil),(nil),(nil),(nil),default,2012-02-03T07:14:54.461Z,true,arn:aws:autoscaling:ap-southeast-1:346266270015:launchConfiguration:175a16db-1f6a-4514-9233-ac7cb34bca90:launchConfigurationName/testLC

as-describe-auto-scaling-groups --show-long --headers

testASG,testLC,ap-southeast-1a,2012-02-03T07:19:10.706Z,prod,EC2,1,5,1,300,0,(nil),(nil),arn:aws:autoscaling:ap-southeast-1:346266270015:autoScalingGroup:c4b584d0-bac4-4507-b972-4fc2b1bc53ac:autoScalingGroupName/testASG,(nil)

as-describe-auto-scaling-instances
i-43796716  testASG  ap-southeast-1a  InService  HEALTHY  testLC

elb-describe-lbs --headers --show-long

prod,prod-11719395.ap-southeast-1.elb.amazonaws.com,prod-11719395.ap-southeast-1.elb.amazonaws.com,Z1WI8VXHPB1R38,"{interval=120,target=HTTP:80/user/sign_in/,timeout=30,healthy-threshold=5,unhealthy-threshold=3}",ap-southeast-1a,(nil),(nil),"i-495dda1c, i-43796716","{protocol=HTTP,lb-port=80,instance-protocol=HTTP,instance-port=80,policies=AWSConsolePolicy-1}",(nil),"{policy-name=AWSConsolePolicy-1,expiration-period=180}","{owner-alias=amazon-elb,group-name=amazon-elb-sg}",(nil),2012-02-01T10:36:08.810Z

elb-describe-instance-health loadbalancername --headers --show-long

INSTANCE_ID,i-495dda1c,InService,N/A,N/A
INSTANCE_ID,i-43796716,OutOfService,Instance has failed at least the UnhealthyThreshold number of health checks consecutively.,Instance
Jeevan Dongre
  • 741
  • Evidently EC2 isn't as popular here as I thought. I would have expected at least some suggestions by now. Let's see if I understand your problem: you want one medium instance running permanently (either not autoscaled, or part of group1), and a small instance that is autoscaled (part of a different group than the medium intsance - e.g. group2), behind a load balancer. (You can't mix different instance types in an autoscaling group, since downscaling terminates at random). Regarding the SSH issue - did you pass --key key_name and --group security_group to as-create-launch-config? – cyberx86 Feb 03 '12 at 01:08
  • 1
    Please provide relevant portions of the output from the following: as-describe-launch-configs --show-long --headers, as-describe-auto-scaling-groups --show-long --headers, as-describe-auto-scaling-instances,elb-describe-lbs --headers --show-long, elb-describe-instance-health loadbalancername --headers --show-long. Also (to confirm that the necessary ports are open on your autoscaled instances): ec2-describe-group. (Without more info - I'd guess a security group issue on your autoscaled instance; but there are quite a few other possibilities). – cyberx86 Feb 03 '12 at 01:12
  • Hi @cyberx86 I am running a medium instance and attached a small instance, both the instance belong to the same group, but the small instance which I am creating is not passing the health check up and also when I hit the public DNS its not responding. I have not passed the security certificates, but I have passed the security group. I was expecting you to respond to this thread, thank you very much I will paste the configs in my next comment. – Jeevan Dongre Feb 03 '12 at 04:58
  • 1
    I am still not quite clear on the 'both instances belong to the same group' part - is the small instance the only instance being created by autoscaling? Meaning that the medium instance is independent (although, behind the same load balancer). How are you verifying that DNS is not responding? I presume that DNS specific tasks (e.g. dig should work (if not, please add that to your question) - but the security group controls everything else (including ping - so if incorrectly set, the instance will be completely inaccessible - which sounds like your scenario). Does it work without the LB? – cyberx86 Feb 03 '12 at 05:21
  • Okay let me be clear. I am already running a medium instance, and I added that instance to the load balancer say "ST" while configuring the auto scaling using CLI I specified min 1 which is a small instance. so now medium instance which was already der is working fine under the LB "ST" but the other instance which is small booted which is also under LB "ST" is not responding I am not able to ssh and when I hit the url my chrome says. site not found?? – Jeevan Dongre Feb 03 '12 at 05:31
  • 1
    Also - the requested configs, above, are not going to fit in a comment - so just add them to your question. (Given that it is 12:30 am here, I won't be able to look at them for quite a few hours - but they will help anyone else who looks at the question.) If you are testing with ping - see this to permit it through your security group - and check for other restrictions (e.g. only permitting traffic from a particular source). Finally see, if you can relaunch your m1.small with a key so you can ssh in. – cyberx86 Feb 03 '12 at 05:32
  • sure, I will reconfigure every thing for the zero, I will pass the keys, in my previous configurations I have not added the keys, I guess thats the problem. Let me check, good nite and take care !!!! – Jeevan Dongre Feb 03 '12 at 05:39
  • 1
    One more point of mention - check your console log on the m1.small if you still can't reach it - it is possible your instance didn't start correctly for some reason (e.g. unable to mount a disk - shouldn't be a problem with a c1.medium to m1.small, but would be a problem for a m1.large to m1.small); (Also, you will, of course, need to setup the security group on the autoscaled instance to allow SSH, even once you have the key in place - out of curiosity, what happened when you tried to SSH into the non-functional m1.small that didn't have a key? Did it connect before giving an error? – cyberx86 Feb 03 '12 at 06:29
  • permission denied (public key) I am configuring the auto scaling stuffs for the zero, I just took the elb-sescribe-lbs, I figured out Security=(nil), i deleted the my previous done settings. – Jeevan Dongre Feb 03 '12 at 06:56
  • @cyberx86 hi, I have added the description/ command of the output which u asked, in the mean while I tried to add the keys, but I am not able to figure out how to do it and also, I tried some thing but it did not turn out. Kindly help me out – Jeevan Dongre Feb 03 '12 at 12:09
  • 1
    both security group and keys need to be set as part of the launch config. Using as-create-launch-config, pass the --key YOUR_KEY_NAME and --group YOUR_SECURITY_GROUP. The permission denied, public key error suggests that you are able to connect to the instance, but not authenticate (since, no key). Will take a look at the configs in a few hours - have a class to teach in a couple minutes. – cyberx86 Feb 03 '12 at 16:43
  • @cyberx86 sure I will it and let u know in the couple of hours. Thanks for responding back, have a great day – Jeevan Dongre Feb 03 '12 at 18:01
  • @cyberx86 after adding security group and key name I am not to ssh to the newly created instance but when I hit the public DNS in the browser the site doesnt load and also the load balancer status says the name thing which is title of this post. – Jeevan Dongre Feb 03 '12 at 18:43
  • @cyberx86 after logging into newly created instance, I did not find my app folder it was completely empty – Jeevan Dongre Feb 03 '12 at 19:21
  • Sounds like you have managed to get SSH access, and figured out why the health check was failing (the file didn't exist). Now you just need to determine why the files don't exist. Try launching an instance of the same AMI you are using for autoscaling as an independent, on-demand instance and see if your files are there (are they on a separate EBS volume, or automatically pulled from somewhere?). Let me know if you still need some help with this problem. – cyberx86 Feb 03 '12 at 21:46
  • I tried launching more instances but even those instances are empty they do have any files, they are on separate EBS volume. Can u help me out to solve this issues please?? – Jeevan Dongre Feb 04 '12 at 02:51
  • Sure - I have some time - let's give chat a try and see if we can figure out your problem – cyberx86 Feb 04 '12 at 03:23

1 Answers1

5

There are a number of considerations to take into account here. Firstly, to resolve the most limiting problem - the lack of SSH access.

Since your previous launch config did not specify a keypair, you will have no valid credentials with which to access the instance. Unfortunately, the initial keypair cannot be added after the instance is launched.

To remedy this, you must create a new launch config, passing the --key and --group parameters, in addition to all the parameters you passed previously. --key takes the name of the keypair you want to use, while --group takes the security group name (if not in VPC) or ID.

In cases where you can't access your instance, the console log can help you to verify the instance has in fact successfully booted. A common issue is boot failure due to missing volumes (especially trying to mount ephemeral volumes that exist only on the larger instance types, when booting a smaller instance type).

An important point of mention is that an AMI is not updated if you change a running instance. You must explicitly create the new image. As such, if you try to launch a new instance using the same AMI you are currently using on a customized instance, there is a good chance that you will simply be launching one of the default AMIs, and not one with your customizations on it.

Use ec2-describe-images to determine the block device mapping of your image - and the snapshot the volume is based on - this will verify that you will be mounting an EBS volume that has your customizations built into it.

If you do not have an up to date AMI to use for autoscaling:

  • Create a snapshot of your EBS volume
  • Create an AMI with ec2-register -n IMAGE_NAME -s SNAPSHOT_ID
    • If you have additional EBS volumes to attach, specify those by adding the --block-device-mapping (-b) parameter (e.g. -b /dev/xvdf=SNAP_ID)
  • Verify that you have the correct block device mappings with ec2-describe-images

Once you have an up to date AMI, you need to create a new launch config that will use this AMI. If desired, you can pass additional block device mappings to the command. Use as-create-launch-config, passing it your new AMI and all the parameters you used previously.

Finally, you must update your autoscaling group. This group is associated with a particular launch config - the new launch config will not be automatically detected and has no effect on the autoscaling group until you explicitly associate it. Use as-update-auto-scaling-group GROUP_NAME --launch-configuration CONFIG_NAME to make this change.

Once the changes have been made, you can simulate an autoscaling event using the as-execute-policy command.

Remember to give your instances a few minutes to boot up - if your ELB is showing instances as unhealthy, you may want to increase the --grace-period of your autoscaling group.

cyberx86
  • 20,915