The kernel receives system control from the bootloader. After a while the kernel starts the init daemon. The init daemon (/sbin/init) is the first daemon that is started and receives process id 1 (PID 1). Init never dies.

When /sbin/init is started, it will first read its configuration file /etc/inittab. In that file, it will look for the value of initdefault (3 in the screenshot below).

[paul@rhel4 ~]$ grep ^id /etc/inittab 
id:3:initdefault:

The value found in initdefault indicates the default runlevel. Some Linux distributions have a brief description of runlevels in /etc/inittab, like here on Red Hat Enterprise Linux 4.

# Default runlevel. The runlevels used by RHS are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you don't have network)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)

Runlevel 0 means the system is shutting down. Runlevel 1 is used for troubleshooting, only the root user can log on, and only at the console. Runlevel 3 is typical for servers, whereas runlevel 5 is typical for desktops (graphical logon). Besides runlevels 0, 1 and 6, the use may vary depending on the distribution. Debian and derived Linux systems have full network and GUI logon on runlevels 2 to 5. So always verify the proper meaning of runlevels on your system.

Init will continue to read /etc/inittab and meets this section on Debian Linux.

l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1
l2:2:wait:/etc/init.d/rc 2
l3:3:wait:/etc/init.d/rc 3
l4:4:wait:/etc/init.d/rc 4
l5:5:wait:/etc/init.d/rc 5
l6:6:wait:/etc/init.d/rc 6

On Red Hat Enterprise Linux it is identical except init.d is rc.d.

l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

In both cases, this means that init will start the rc script with the runlevel as the only parameter. Actually /etc/inittab has fields separated by colons. The second field determines the runlevel in which this line should be executed. So in both cases, only one line of the seven will be executed, depending on the runlevel set by initdefault.

When you take a look any of the /etc/rcX.d/ directories, then you will see a lot of (links to) scripts who's name start with either uppercase K or uppercase S.

[root@RHEL52 rc3.d]# ls -l | tail -4
lrwxrwxrwx 1 root root 19 Oct 11  2008 S98haldaemon -> ../init.d/haldaemon
lrwxrwxrwx 1 root root 19 Oct 11  2008 S99firstboot -> ../init.d/firstboot
lrwxrwxrwx 1 root root 11 Jan 21 04:16 S99local -> ../rc.local
lrwxrwxrwx 1 root root 16 Jan 21 04:17 S99smartd -> ../init.d/smartd

The /etc/rcX.d/ directories only contain links to scripts in /etc/init.d/. Links allow for the script to have a different name. When entering a runlevel, all scripts that start with uppercase K or uppercase S will be started in alphabetical order. Those that start with K will be started first, with stop as the only parameter. The remaining scripts with S will be started with start as the only parameter.

All this is done by the /etc/rc.d/rc script on Red Hat and by the /etc/init.d/rc script on Debian.

Here we use chkconfig to list the status of a service in the different runlevels. You can see that the crond daemon (or service) is only activated in runlevels 2 to 5.

[root@RHEL52 ~]# chkconfig --list crond
crond          	0:off	1:off	2:on	3:on	4:on	5:on	6:off

When you compare the screenshot above with the one below, you can see that off equals to a K link to the script, whereas on equals to an S link.

[root@RHEL52 etc]# find ./rc?.d/ -name \*crond -exec ls -l {} \;|cut -b40-
./rc0.d/K60crond -> ../init.d/crond
./rc1.d/K60crond -> ../init.d/crond
./rc2.d/S90crond -> ../init.d/crond
./rc3.d/S90crond -> ../init.d/crond
./rc4.d/S90crond -> ../init.d/crond
./rc5.d/S90crond -> ../init.d/crond
./rc6.d/K60crond -> ../init.d/crond

Here you see how to use chkconfig to disable (or enable) a service in a certain runlevel.

This screenshot shows how to disable crond in runlevel 3.

[root@RHEL52 ~]# chkconfig --level 3 crond off
[root@RHEL52 ~]# chkconfig --list crond
crond          	0:off	1:off	2:on	3:off	4:on	5:on	6:off

This screenshot shows how to enable crond in runlevels 3 and 4.

[root@RHEL52 ~]# chkconfig --level 34 crond on
[root@RHEL52 ~]# chkconfig --list crond
crond          	0:off	1:off	2:on	3:on	4:on	5:on	6:off

Every script in /etc/init.d/ can have (comment) lines to tell chkconfig what to do with the service. The line with # chkconfig: contains the runlevels in which the service should be started (2345), followed by the priority for start (90) and stop (60).

[root@RHEL52 ~]# head -9 /etc/init.d/crond | tail -5
# chkconfig: 2345 90 60
# description: cron is a standard UNIX program that runs user-specified
#              programs at periodic scheduled times. vixie cron adds a
#              number of features to the basic UNIX cron, including better
#              security and more powerful configuration options.

Services can be enabled or disabled in all runlevels with one command. Runlevels 0, 1 and 6 are always stopping services (or calling the scripts with stop) even when their name starts with uppercase S.

[root@RHEL52 ~]# chkconfig crond off
[root@RHEL52 ~]# chkconfig --list crond
crond           0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@RHEL52 ~]# chkconfig crond on
[root@RHEL52 ~]# chkconfig --list crond 
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off

The Debian equivalent of chkconfig is called update-rc.d. This tool is designed for use in scripts, if you prefer a graphical tool then look at bum.

When there are existing links in /etc/rcX.d/ then update-rc.d does not do anything. This is to avoid that post installation scripts using update-rc.d are overwriting changes made by a system administrator.

root@barry:~# update-rc.d cron remove
update-rc.d: /etc/init.d/cron exists during rc.d purge (use -f to force)

As you can see in the next screenshot, nothing changed for the cron daemon.

root@barry:~# find /etc/rc?.d/ -name '*cron' -exec ls -l {} \;|cut -b44-
/etc/rc0.d/K11cron -> ../init.d/cron
/etc/rc1.d/K11cron -> ../init.d/cron
/etc/rc2.d/S89cron -> ../init.d/cron
/etc/rc3.d/S89cron -> ../init.d/cron
/etc/rc4.d/S89cron -> ../init.d/cron
/etc/rc5.d/S89cron -> ../init.d/cron
/etc/rc6.d/K11cron -> ../init.d/cron

Here we remove cron from all runlevels. Remember that the proper way to disable a service is to put K scripts oin all runlevels!

root@barry:~# update-rc.d -f cron remove
 Removing any system startup links for /etc/init.d/cron ...
   /etc/rc0.d/K11cron
   /etc/rc1.d/K11cron
   /etc/rc2.d/S89cron
   /etc/rc3.d/S89cron
   /etc/rc4.d/S89cron
   /etc/rc5.d/S89cron
   /etc/rc6.d/K11cron
root@barry:~# find /etc/rc?.d/ -name '*cron' -exec ls -l {} \;|cut -b44-
root@barry:~#

This screenshot shows how to use update-rc.d to enable a service in runlevels 2, 3, 4 and 5 and disable the service in runlevels 0, 1 and 6.

root@barry:~# update-rc.d cron defaults
 Adding system startup for /etc/init.d/cron ...
   /etc/rc0.d/K20cron -> ../init.d/cron
   /etc/rc1.d/K20cron -> ../init.d/cron
   /etc/rc6.d/K20cron -> ../init.d/cron
   /etc/rc2.d/S20cron -> ../init.d/cron
   /etc/rc3.d/S20cron -> ../init.d/cron
   /etc/rc4.d/S20cron -> ../init.d/cron
   /etc/rc5.d/S20cron -> ../init.d/cron

And here is an example on how to set your custom configuration for the cron daemon.

root@barry:~# update-rc.d -n cron start 11 2 3 4 5 . stop 89 0 1 6 .
 Adding system startup for /etc/init.d/cron ...
   /etc/rc0.d/K89cron -> ../init.d/cron
   /etc/rc1.d/K89cron -> ../init.d/cron
   /etc/rc6.d/K89cron -> ../init.d/cron
   /etc/rc2.d/S11cron -> ../init.d/cron
   /etc/rc3.d/S11cron -> ../init.d/cron
   /etc/rc4.d/S11cron -> ../init.d/cron
   /etc/rc5.d/S11cron -> ../init.d/cron

You can see your current runlevel with the runlevel or who -r commands.

The runlevel command is typical Linux and will output the previous and the current runlevel. If there was no previous runlevel, then it will mark it with the letter N.

[root@RHEL4b ~]# runlevel 
N 3

The history of who -r dates back to Seventies Unix, it still works on Linux.

[root@RHEL4b ~]# who -r
         run-level 3  Jul 28 09:15                   last=S

You can switch to another runlevel with the telinit command. On Linux /sbin/telinit is usually a (hard) link to /sbin/init.

This screenshot shows how to switch from runlevel 2 to runlevel 3 without reboot.

root@barry:~# runlevel 
N 2
root@barry:~# init 3
root@barry:~# runlevel 
2 3

The shutdown command is used to properly shut down a system.

Common switches used with shutdown are -a, -t, -h and -r.

The -a switch forces /sbin/shutdown to use /etc/shutdown.allow. The -t switch is used to define the number of seconds between the sending of the TERM signal and the KILL signal. The -h switch halts the system instead of changing to runlevel 1. The -r switch tells /sbin/shutdown to reboot after shutting down.

This screenshot shows how to use shutdown with five seconds between TERM and KILL signals.

root@barry:~# shutdown -t5 -h now

The now is the time argument. This can be +m for the number of minutes to wait before shutting down (with now as an alias for +0. The command will also accept hh:mm instead of +m.

The binary /sbin/reboot is the same as /sbin/halt and /sbin/poweroff. Depending on the name we use to call the command, it can behave differently.

When in runlevel 0 or 6 halt, reboot and poweroff will tell the kernel to halt, reboot or poweroff the system.

When not in runlevel 0 or 6, typing reboot as root actually calls the shutdown command with the -r switch and typing poweroff will switch off the power when halting the system.

halt, reboot and poweroff all write to /var/log/wtmp. To look at /var/log/wtmp, we need to use th last.

[root@RHEL52 ~]# last | grep reboot
reboot   system boot  2.6.18-128.el5   Fri May 29 11:44   (192+05:01)
reboot   system boot  2.6.18-128.el5   Wed May 27 12:10    (06:49)
reboot   system boot  2.6.18-128.el5   Mon May 25 19:34   (1+15:59)
reboot   system boot  2.6.18-128.el5   Mon Feb  9 13:20   (106+21:13)

When rc is finished starting all those scripts, init will continue to read /etc/inittab. The next line is about what to do when the user hits Ctrl-Alt-Delete on the keyboard.

Here is what Debian 4.0 does.

root@barry:~# grep -i ctrl /etc/inittab 
# What to do when CTRL-ALT-DEL is pressed.
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

Which is very similar to the default Red Hat Enterprise Linux 5.2 action.

[root@RHEL52 ~]# grep -i ctrl /etc/inittab 
# Trap CTRL-ALT-DELETE
ca::ctrlaltdel:/sbin/shutdown -t3 -r now

One noticable difference is that Debian forces shutdown to use /etc/shutdown.allow, where Red Hat allows everyone to invoke shutdown pressing Ctrl-Alt-Delete.

[root@RHEL52 ~]# grep ^p /etc/inittab 
pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"
pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled"

It will read commands on what to execute in case of powerfailure, powerok and Ctrl-Alt-Delete. The init process never stops keeping an eye on power failures and that triple key combo.

root@barry:~# grep ^p /etc/inittab 
pf::powerwait:/etc/init.d/powerfail start
pn::powerfailnow:/etc/init.d/powerfail now
po::powerokwait:/etc/init.d/powerfail stop

The first command to learn is systemctl list-units --type=target (or the shorter version systemctl -t target). It will show you the different targets on the system.

[root@rhel7 ~]# systemctl list-units --type=target
UNIT                LOAD   ACTIVE SUB    DESCRIPTION
basic.target        loaded active active Basic System
cryptsetup.target   loaded active active Encrypted Volumes
getty.target        loaded active active Login Prompts
graphical.target    loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target     loaded active active Local File Systems
multi-user.target   loaded active active Multi-User System
network.target      loaded active active Network
nfs.target          loaded active active Network File System Server
paths.target        loaded active active Paths
remote-fs.target    loaded active active Remote File Systems
slices.target       loaded active active Slices
sockets.target      loaded active active Sockets
swap.target         loaded active active Swap
sysinit.target      loaded active active System Initialization
timers.target       loaded active active Timers

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

16 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[root@rhel7 ~]#

Targets are the replacement of runlevels and define specific points to reach when booting the system. For example the graphical.target is reached when you get a graphical interface, and the nfs.target requires a running nfs server.

To switch to a target (for example multi-user.target), we now use systemctl isolate (instead of the equivalent init 3 to change the runlevel).

[root@rhel7 ~]# ps fax | wc -l
169
[root@rhel7 ~]# systemctl isolate multi-user.target
[root@rhel7 ~]# ps fax | wc -l
129
[root@rhel7 ~]#

To change the default target, we again use this systemctl command (instead of editing the /etc/inittab file).

[root@rhel7 ~]# systemctl enable multi-user.target --force
rm '/etc/systemd/system/default.target'
ln -s '/usr/lib/systemd/system/multi-user.target' '/etc/systemd/system/default\
.target'
[root@rhel7 ~]#

This command removed the file /etc/systemd/system/default.target and replaced it with a symbolic link to the multi-user-.target target.

Dependencies are no longer defined by alfabetical order of running scripts, but by configuration in /etc/systemd/system/. For example here are the required services for the multi-user.target on Red Hat Enterprise 7.

[root@rhel7 ~]# ls /etc/systemd/system/multi-user.target.wants/
abrt-ccpp.service     hypervkvpd.service      postfix.service
abrtd.service         hypervvssd.service      remote-fs.target
abrt-oops.service     irqbalance.service      rhsmcertd.service
abrt-vmcore.service   ksm.service             rngd.service
abrt-xorg.service     ksmtuned.service        rpcbind.service
atd.service           libstoragemgmt.service  rsyslog.service
auditd.service        libvirtd.service        smartd.service
avahi-daemon.service  mdmonitor.service       sshd.service
chronyd.service       ModemManager.service    sysstat.service
crond.service         NetworkManager.service  tuned.service
cups.path             nfs.target              vmtoolsd.service
[root@rhel7 ~]#

Debian8 is not fully migrated yet.

root@debian8:~# ls /etc/systemd/system/multi-user.target.wants/ 
anacron.service       binfmt-support.service  pppd-dns.service  ssh.service
atd.service           fancontrol.service      remote-fs.target
avahi-daemon.service  lm-sensors.service      rsyslog.service

Typical rc scripts are replaced with services. Issue the systemctl list-units -t service --all (or systemctl -at service) to get a list of all services on your system.

[root@rhel7 ~]# systemctl -at service | head -5 | column -t | cut -c1-78
UNIT                 LOAD    ACTIVE    SUB      DESCRIPTION
abrt-ccpp.service    loaded  active    exited   Install      ABRT     coredump
abrt-oops.service    loaded  active    running  ABRT         kernel   log
abrt-vmcore.service  loaded  inactive  dead     Harvest      vmcores  for
abrt-xorg.service    loaded  active    running  ABRT         Xorg     log
[root@rhel7 ~]#

And here an example on how to see the status of the sshd service.

[root@rhel7 ~]# systemctl status sshd.service
sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
   Active: active (running) since Wed 2014-09-10 13:42:21 CEST; 55min ago
 Main PID: 1400 (sshd)
   CGroup: /system.slice/sshd.service
           --1400 /usr/sbin/sshd -D

Sep 10 13:42:21 rhel7 systemd[1]: Started OpenSSH server daemon.
Sep 10 13:42:21 rhel7 sshd[1400]: Server listening on 0.0.0.0 port 22.
Sep 10 13:42:21 rhel7 sshd[1400]: Server listening on :: port 22.
[root@rhel7 ~]#

The chkconfig and service commands are considered 'legacy'. They are replaced with systemctl.

This screenshot shows the new way to start and stop a service.

[root@rhel7 ~]# systemctl start crond.service
[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=active
SubState=running
UnitFileState=enabled
[root@rhel7 ~]# systemctl stop crond.service
[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=inactive
SubState=dead
UnitFileState=enabled
[root@rhel7 ~]#

And here is the new way to stop and disable a service.

[root@rhel7 ~]# systemctl stop crond.service
[root@rhel7 ~]# systemctl disable crond.service
rm '/etc/systemd/system/multi-user.target.wants/crond.service'
[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=inactive
SubState=dead
UnitFileState=disabled
[root@rhel7 ~]#

This screenshot shows how to enable and start the service again.

[root@rhel7 ~]# systemctl enable crond.service
ln -s '/usr/lib/systemd/system/crond.service' '/etc/systemd/system/multi-user.\
target.wants/crond.service'
[root@rhel7 ~]# systemctl start crond.service
[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=active
SubState=running
UnitFileState=enabled
[root@rhel7 ~]#

You can also use systemd to kill problematic services.

[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=active
SubState=running
UnitFileState=enabled
[root@rhel7 ~]# systemctl kill -s SIGKILL crond.service
[root@rhel7 ~]# systemctl show crond.service | grep State
LoadState=loaded
ActiveState=failed
SubState=failed
UnitFileState=enabled
[root@rhel7 ~]#

The poweroff, halt and reboot commands are considered legacy now and are handeld by systemctl. The table below shows the legacy commands on the left and their new systemd equivalent on the right.

The systemctl utility has a buil-in remote control providing there is an ssh daemon running on the remote system.

This screenshot shows how to use systemctl to verify a service on an other RHEL server.

[root@rhel7 ~]# systemctl -H root@192.168.1.65 status sshd
root@192.168.1.65's password:
sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
   Active: active (running) since Thu 2014-09-11 13:04:10 CEST; 16min ago
  Process: 1328 ExecStartPre=/usr/sbin/sshd-keygen (code=exited, status=0/SUCCE\
SS)
 Main PID: 1363 (sshd)
   CGroup: /system.slice/sshd.service
[root@rhel7 ~]#

There are other tools...

systemd-analyze                 systemd-loginctl
systemd-ask-password            systemd-machine-id-setup
systemd-cat                     systemd-notify
systemd-cgls                    systemd-nspawn
systemd-cgtop                   systemd-run
systemd-coredumpctl             systemd-stdio-bridge
systemd-delta                   systemd-sysv-convert
systemd-detect-virt             systemd-tmpfiles
systemd-inhibit                 systemd-tty-ask-password-agent

For example systemd-analyze blame will give you an overview of the time it took for each service to boot.

[root@rhel7 ~]# systemd-analyze blame | head
          1.977s firewalld.service
          1.096s tuned.service
           993ms postfix.service
           939ms iprinit.service
           925ms vboxadd-x11.service
           880ms firstboot-graphical.service
           839ms accounts-daemon.service
           829ms network.service
           822ms iprupdate.service
           795ms boot.mount
[root@rhel7 ~]#