Thursday, March 23, 2017

Muse failed to start this morning

Muse is a very noisy machine. I purchased new fans to solve the problem, but they have not been shipped yet. In the meantime, I have to shutdown the machine at night to preserve the sleep of my son Vincent.

This morning, I had an unpleasant (and then a semi-pleasant) surprise. The machine's fans started kicking up, then down, again, three times. And then the boot beep and the BIOS screen. And then the machine shuts down, and it does it once more. Fans up, down, up, down.

And then an ominous message on screen (that's the bad surprise) telling me that there was a BIOS checksum error. You get a bit frightened when you see this

The pleasant surprise is that I then saw a message about flashing some backup BIOS. And after a minute or so, the machine booted again, correctly it seems.

You have to give kudos to the engineers who implemented this.

Returning Nano to macOS

I now have enough machines and virtual machines that I don't need to run Linux on Nano anymore, so I'm returning it to macOS. That will be the only macOS machine on the "main bus" (the wired network backbone). I can use it as a file server for disks I use occasionally. That machine has 5 USB ports, that's useful to connect disks. But many of these disks are in HFS format, so Linux is not that good dealing with them.

And it has never been über-stable under Linux anyways. What finally convinced me was what happened when I ran dnf update this morning. During the update, the video became "a bit glitchy":

Filed Red Hat Bugzilla 1435184.

First Windows 10 guest crash

Just crashed and rebooted. I'll probably never know what happened.

Restoring VMs on muse

Instructions on how to import VMs from existing qcow2 images. In my case:

virt-install -n fedora25-clone -r 2048 --os-type=linux   --os-variant=fedora25 --disk ~/.local/share/libvirt/images/fedora25-clone.qcow2,device=disk,bus=virtio -w bridge=virbr0,model=virtio --spice --import

For the fedora25-qxl image, it was apparently corrupt beyond repair, though I have no real idea why. Fortunately, since I was actually trying to restore VMs from another disk, I still have the original, so I could copy the file back. Unfortunately, the same thing happens on that copy:

The annoying thing is that this corrupts the guest after the fact:

Error starting domain: internal error: process exited while connecting to monitor: 2017-03-23T13:13:53.997667Z qemu-system-x86_64: -drive file=/home/ddd/.local/share/libvirt/images/fedora25-qxl.qcow2,format=qcow2,if=none,id=drive-virtio-disk0: qcow2: Image is corrupt; cannot be opened read/write

I'm pretty sure about the disk corruption because it happened twice, exactly the same behavior. Filed Red Hat Bugzilla 1435315. Some comments here on things to run. I will run them tomorrow, it's too late now and I had to shutdown the noisy muse.

Bridge network

Followed these instructions:

• Added the following to /etc/sysconfig/network-scripts/ifcfg-enp4s0:

  BRIDGE=br0

• Create /etc/sysconfig/network-scripts/ifcfg-br0 containing:

  DEVICE=br0
  TYPE=Bridge
  BOOTPROTO=dhcp
  ONBOOT=yes
  DELAY=0
  

• Run the following commands:

  iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
  dnf install iptables-services
  service iptables save
  service iptables restart

Note that the dnf install is not mentioned in the original documentation, I grabbed it from this answer on ServerFault

Well, that did not go as planned. Restarting the network did not work:

[root@muse ddd]# service network restart
Restarting network (via systemctl):  Job for network.service failed because the control process exited with error code.
See "systemctl status network.service" and "journalctl -xe" for details.
                                                           [FAILED]

The status does not give me much info:

[root@muse ddd]# systemctl status network.service
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; generated; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-03-23 17:06:48 CET; 6s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 10908 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)


Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org network[10908]: RTNETLINK answers: File exists
Mar 23 17:06:48 muse.dinechin.org systemd[1]: network.service: Control process exited, code=exited status=1
Mar 23 17:06:48 muse.dinechin.org systemd[1]: Failed to start LSB: Bring up/down networking.
Mar 23 17:06:48 muse.dinechin.org systemd[1]: network.service: Unit entered failed state.
Mar 23 17:06:48 muse.dinechin.org systemd[1]: network.service: Failed with result 'exit-code'.

Will see if a reboot helps.

BTRFS bad tree block

Looking at journalctl output, I did not see much interesting in terms of network. But I get tons of messages like this:

Mar 23 16:53:27 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:27 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:41 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:41 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:58 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:58 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:58 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:53:58 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:08 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:08 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:08 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:08 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:29 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:29 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896
Mar 23 16:54:38 muse.dinechin.org kernel: BTRFS error (device sdb4): bad tree block start 11022913622198796681 52432896

Does not look too good. This is a brand new disk.

Restarted on the original disk (/dev/sdc). Ran btrfsck /dev/sdb4. This ended in a core dump. Oh well, I tried. I'm starting to think that either I'm really unlucky with disks, or that BTRFS is not quite ready for prime-time yet.

Detecting qxl calls

A small systemtap script from Frediano:

//probe process("/usr/lib64/xorg/modules/drivers/qxl_drv.so").function("qxl_image_create")
probe process("/usr/lib64/xorg/modules/drivers/qxl_drv.so").function("uxa_put_image")
{
        printf("%s:%d: %sn", probefunc(), tid(), $$parms);
//        printf("info: %sn", $info->$$);
//        printf("finfo: %sn", $info->finfo->$$);
        print_usyms(ubacktrace());
        println();
}