AWS EC2 instance unreachable after reboot ([Errno 101])

Written by Claudio Kuenzler - 4 comments

Published on June 16th 2017 - Listed in Linux Internet AWS

---

Already two weeks ago I had a problem with a newly installed EC2 instance launched from an AMI image on the AWS cloud (Amazon Web Services). After I rebooted the instance, an updated Ubuntu 16.04 Xenial machine, the system didn't come up anymore. Even the ping from another instance in the same subnet didn't work. 

Today I came across the same problem. Different region (this time Europe-Ireland, the one from two weeks ago was Europe-Frankfurt), different VPC, different network segments, but the exact same problem. The instance didn't come up anymore after a reboot. Just prior to this instance I have set up two other instances the exact same way (except a different Availability Zone was used) and neither of these had a problem after a reboot.

By using Instance "Settings -> Get System Log" I was able to get some information whats going on:

 [[0;32m  OK  [0m] Reached target Network.
[    7.511380] cloud-init[770]: Cloud-init v. 0.7.9 running 'init' at Fri, 16 Jun 2017 11:46:18 +0000. Up 7.41 seconds.
[    7.515990] cloud-init[770]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[    7.520087] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.524021] cloud-init[770]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[    7.528206] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.532219] cloud-init[770]: ci-info: |  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |
[    7.536186] cloud-init[770]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
[    7.540244] cloud-init[770]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
[    7.544243] cloud-init[770]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[    7.548428] cloud-init[770]: 2017-06-16 13:46:18,547 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    8.514438] cloud-init[770]: 2017-06-16 13:46:19,550 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[    9.517358] cloud-init[770]: 2017-06-16 13:46:20,553 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[   10.520266] cloud-init[770]: 2017-06-16 13:46:21,555 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',))]
[... messages repeating ...]

For some reason, the network interface ens3 didn't come up:

| Device |   Up  |  Address  |    Mask   | Scope | Hw-Address        |  
|  ens3  | False |     .     |     .     |   .   | 0a:76:72:96:c7:52 |

No reason whatsoever in the log file.

Even in the EC2 administration UI the status check was at 1/2, so something's definitely not working correctly:

As there is no console terminal access in AWS, I had no other choices than to:

• reboot the instance in the EC2 administration -> didn't help
  
• stop the instance, then start the instance -> didn't help
• stop the instance, change the instance type -> didn't help

All the tools given by AWS didn't help. As the instance is not usable anymore, I terminated it and created a new instance from the same AMI image. Same instance type, same network settings, everything exactly the same. This time, the instance was able to reboot with no problem whatsoever. I came across a question on serverfault (amazon ec2 - Unable to connect to EC2 instance after "reboot") which describes the same problem as I experienced (already twice now in two weeks), but unfortunately without solution. For now the most plausible explanation to me is a bug in the AWS infrastructure which hits randomly (?). I didn't find a way to reproduce this problem.

Summary: You rebooted your EC2 instance (from within the guest OS)? It doesn't come up afterwards? Even pings from the same subnet don't work? You get the same message as above in the system log? Recreate the instance from scratch. :-(

Add a comment

Show form to leave a comment

Comments (newest first)

Sergio from wrote on May 30th, 2022:

Julio's workaround worked great, thanks!.

---

Julio Lemus from GT wrote on Apr 20th, 2022:

How about I hope it helps you, I had to restart my instance and I had the same error, and I couldn't access it either by ssh, or by serial console, I didn't have any access to the instance and I did the following:
I tell you that I had to restart my instance and I had the same error, I restarted several times and it did not pass that check, what I did was stop the instance and then add an interface with private IPv4, turned it on it did not work, I stopped it again and I disassociated the interface, I stopped the instance again for 5 minutes and then started it and it gave me a successful result, it worked for me and passed the verification of the two states

---

ck from St. Gallen wrote on Sep 2nd, 2019:

Thanks, Sushant, for that hint. Your suggestion might help for systems using netplan, but in my case it was a Ubuntu 16.04 which didn't have netplan as network management then.

---

Sushant from wrote on Sep 2nd, 2019:

I was facing the same issue. I solved it by editing /etc/netplan/50-cloud-init.yaml

And pasting following code:

network:
ethernets:
ens3:
dhcp4: true
version: 2

This will give dhcp server to set IPv4 settings.

---