I was about to create a new Galera Cluster where a third node is simply being used as Arbitrator, not as data node.
As this is a simple process starting up, I usually just added a line in /etc/rc.local to start the process. But in the new Ubuntu 18.04 (and newer) this file does not exist anymore:
root@arbitrator:~# file /etc/rc.local
/etc/rc.local: cannot open `/etc/rc.local' (No such file or directory)
Sure, I could write a SystemD unit file for a garbd.service now, but lazy, you know.
So I wondered if and how /etc/rc.local is still working in Ubuntu 18.04 and I stumbled on the manpage of systemd-rc-local-generator:
"systemd-rc-local-generator is a generator that checks whether /etc/rc.local exists and is executable, and if it is pulls the rc-local.service unit into the boot process[...]"
Cool, let's give it a shot.
root@arbitrator:~# echo "/usr/bin/garbd --cfg /etc/garbd.conf &" > /etc/rc.local
root@arbitrator:~# echo "exit 0" >> /etc/rc.local
root@arbitrator:~# chmod 755 /etc/rc.local
After a reboot, the process didn't show up. What happened?
root@arbitrator:~# systemctl status rc-local
? rc-local.service - /etc/rc.local Compatibility
Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
Drop-In: /lib/systemd/system/rc-local.service.d
??debian.conf
Active: failed (Result: exit-code) since Mon 2018-06-11 16:53:47 CEST; 1min 53s ago
Docs: man:systemd-rc-local-generator(8)
Process: 1182 ExecStart=/etc/rc.local start (code=exited, status=203/EXEC)
Jun 11 16:53:46 arbitrator systemd[1]: Starting /etc/rc.local Compatibility...
Jun 11 16:53:47 arbitrator systemd[1182]: rc-local.service: Failed to execute command: Exec format error
Jun 11 16:53:47 arbitrator systemd[1182]: rc-local.service: Failed at step EXEC spawning /etc/rc.local: Exec format error
Jun 11 16:53:47 arbitrator systemd[1]: rc-local.service: Control process exited, code=exited status=203
Jun 11 16:53:47 arbitrator systemd[1]: rc-local.service: Failed with result 'exit-code'.
Jun 11 16:53:47 arbitrator systemd[1]: Failed to start /etc/rc.local Compatibility.
Exec format error? Huh?
I came across an older article from SUSE (SLES12), which looked kind of similar: The after.local.service fails to start with exec format error. The resolution on that knowledge base article:
"Add a hashpling to the begining [...]"
D'oh! I compared with a /etc/rc.local from another system (Xenial) and indeed it starts with the shell, like a normal shell script. I changed my /etc/rc.local and added #!/bin/bash:
root@arbitrator:~# cat /etc/rc.local
#!/bin/bash
/usr/bin/garbd --cfg /etc/garbd.conf &
exit 0
The rc-local service could now be restarted with success:
root@arbitrator:~# systemctl restart rc-local
root@arbitrator:~# systemctl status rc-local
? rc-local.service - /etc/rc.local Compatibility
Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled)
Drop-In: /lib/systemd/system/rc-local.service.d
??debian.conf
Active: active (running) since Mon 2018-06-11 16:59:31 CEST; 11s ago
Docs: man:systemd-rc-local-generator(8)
Process: 1948 ExecStart=/etc/rc.local start (code=exited, status=0/SUCCESS)
Tasks: 3 (limit: 4664)
CGroup: /system.slice/rc-local.service
??1949 /usr/bin/garbd --cfg /etc/garbd.conf
Jun 11 16:59:31 arbitrator systemd[1]: Starting /etc/rc.local Compatibility...
Jun 11 16:59:31 arbitrator systemd[1]: Started /etc/rc.local Compatibility.
And garbd is started:
root@arbitrator:~# ps auxf | grep garb | grep -v grep
root 1949 0.0 0.2 181000 9664 ? Sl 16:59 0:00 /usr/bin/garbd --cfg /etc/garbd.conf
/etc/rc.local still works in Ubuntu 18.04, when
1) it exists
2) is executable
3) Starts with a valid shell hashpling (is that really the name for it?)
Polo from wrote on Aug 6th, 2021:
Thanks a lot, very clear and short explanation !
Thomas from Los Angeles wrote on May 1st, 2021:
Just what I needed. Works like a charm. Thanks.
Jason from United States wrote on Dec 19th, 2020:
To answer your question, the correct term for the two-character string "#!" is "shebang" vice "hashpling."
Ken from wrote on Jan 29th, 2020:
Hashbang, Poundbang, Shebang...Banger & Hash?...idk, all different ways of saying the same thing I guess. Though admittedly, hashpling is a new one for me haha
Ingo Baab from Deutschland wrote on Jul 28th, 2019:
shebang: https://bash.cyberciti.biz/guide/Shebang
dbkeys from wrote on Jun 20th, 2019:
I always heard it (#!) was called Pound Bang
Makes sense, because when you pound you hear (or feel) a bang.
Thanks for the help !
Frank from wrote on Apr 3rd, 2019:
Thanks! This is exactly what I needed, very nicely formatted. I never knew about the "systemctl restart rc-local" command. I would always reboot the machine to test.
Oliver R. from wrote on Nov 29th, 2018:
hashpling ... just call it "shebang", this is probably the most common and correct term for it.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office OpenSearch PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder