Cannot boot Linux Mint 18.1 after running apt-get upgrade

Written by Claudio Kuenzler - 0 comments

Published on June 19th 2017 - Listed in Linux

---

Last Friday (June 16th 2017) I ran apt-get upgrade on a Linux Mint 18.1 machine and then rebooted the machine. To my big disbelieve the machine didn't come up anymore. After the grub2 menu, the following error appeared:

Loading Linux 4.4.0-53-generic ...
error: attempt to read or write outside of disk 'hd0'.
Loading initial ramdisk ...
error: you need to load the kernel first.

Press any key to continue...

Erm...Noooo! T_T

I booted into the Linux Mint 18.1 Live CD, mounted the drive /dev/sda1 as /mnt, bind mounted sys dev and proc and recreated the initramfs inside the chroot jail of the installed Linux Mint:

mount /dev/sda1 /mnt
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
chroot /mnt /bin/bash
mkinitramfs -o /boot/initrd.img-4.4.0-53-generic
update-grub
grub-install /dev/sda
exit
reboot

But I still got the same error after the reboot... I also manually selected the correct partition in grub in order to rule out a grub2 error:

grub> ls
(hd0) (hd0,msdos5) (hd0,msdos1)
grub> ls (md0,msdos5)
        Partition hd0,msdos5: No known filesystem detected...
grub> ls (md0,msdos1)
        Partition hd0,msdos1: Filesystem type ext* - last modification time 2017-06-19 05:44:26....
Partition start at 1024KiB - Total size 486333440KiB
grub> set root=(hd0,msdos1)
grub> set prefix=(hd0,msdos1)/boot/grub
grub> insmod normal
grub> normal

This got me back to the (updated) grub menu, but once I selected the entry for Linux Mint I got the same error again.

I booted again with the Live CD, this time with the goal to reinstall the Linux Kernel.

mount /dev/sda1 /mnt
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
chroot /mnt /bin/bash
apt-get install linux-image-generic

To my big surprise a new Linux Kernel 4.4.0-79 was installed. I expected apt to return "already installed" or something like this.

The installer returned some errors complaining about missing Linux headers but still finished the installation of the new linux-image package. I installed the header package afterwards:

apt-get install linux-headers-4.4.0-79-generic

The installation of the new Kernel packages created a new initramfs in /boot/initrd.img-4.4.0-79-generic and also a new/updated entry in /boot/grub/grub.cfg with the new available Kernel version.

I exited the chroot environment and rebooted the machine:

exit
reboot

And this time Linux Mint 18.1 was booting again.

Update (the reason): A dying disk

Update July 4th 2017: I experienced an extremely laggy OS on this very same Linux mint machine so I checked the current status of the disk. And found this:

# smartctl /dev/sda -a
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-79-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Blue (SATA)
Device Model:     WDC WD5000AAKS-00V1A0
Serial Number:    WD-WMAWF2141256
LU WWN Device Id: 5 0014ee 0575fb08f
Firmware Version: 05.01D05
User Capacity:    500.107.862.016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Tue Jul  4 08:43:57 2017 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)    Offline data collection activity
                    was suspended by an interrupting command from host.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         ( 7380) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  88) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x303f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       698
  3 Spin_Up_Time            0x0027   144   141   021    Pre-fail  Always       -       3800
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       104
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   024   024   000    Old_age   Always       -       55700
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       102
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       37
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       66
194 Temperature_Celsius     0x0022   097   093   000    Old_age   Always       -       46
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   189   189   000    Old_age   Always       -       899
198 Offline_Uncorrectable   0x0030   197   197   000    Old_age   Offline      -       277
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   198   193   000    Old_age   Offline      -       464

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     53425         579131664
# 2  Extended offline    Completed without error       00%     50463         -
# 3  Extended offline    Completed without error       00%     40082         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Yes, you see correctly: 899 sectors pending re-allocation and 277 sectors definitely defect. In the SMART self-test log there's even a read failure mentioned. Given the amount of hours this disk has lived so far (55700) this is not at all a surprise.

At least now I understand why apt-get upgrade resulted in such a broken OS before: The disk was already dying then.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.