Month: May 2023

Memories

While working at Amcut – traveling to the near west side of Chicago with a field support rep (darned if I can remember his name, but not Darren). We went to service one of the machines at Ryerson Steel. At lunch time went to this little hole-in-the-wall chicken place. I was really naive, having moved to the city from the UP just months before. This was 1979, and different times. We were the only two white guys in the place (and probably in a 2-mile radius).

The year was 1974. I was attending a photography camp at Michigan Tech summer camp for kids. The photographer teaching the course was a world-class photographer, and the class taught composition in outdoor photography. We were shooting at McClain State Park, on the western coast of the Keeweenaw Peninsula. The day was sunny and 70ish degrees, with a few white cumulus clouds dotting the sky. Her name was Kim. What a beautiful day.

It was winter. We were on a ski trip to the Porcupine Mountains, and the day was cold and clear. A north wind off of Lake Superior met us at the top of the hill as we came off the lift. The snow squeaked under our skis as we traversed across the hill to the top of the run. You can see the blowing snow coming in off the lake and up the hill. Adjust the goggles and zip the top of the jacket all the way up – trying to turtle your face down behind the up-turned collar. Take off and hit that mogul field! Get down that hill, and hit the lodge for some warm up, then do it – again and again. At the end of the day, the bus ride home with 30 other kids, all exhausted from a full day on the slopes.

Systemd Parallel Execution Fallout

The Linux boot manager, systemd, speeds up the boot time of a computer running Linux by executing startup tasks in parallel. This is great, for the most part, but does introduce some potential issues depending on how your system is configured. For a stand-alone workstation or server, chances are you may never run into problems. Introduce some network-based services, however, and the odds start to turn the other way.

Allow me to introduce the following configuration, and discuss how systemd needed to be tuned to provide an operational environment:

  1. Workstations and servers use automount and NFS to mount file systems of shared data and user home directories.
  2. To manage user logins openLDAP (or AD) is used and the sssd daemon manages this configuration.
  3. The system clock is synced using the chrony daemon.
  4. Users rely on cron to startup their services at boot time using @restart in their crontab files, which point to startup scripts in their NFS-mounted home directories.

Now, let’s consider how the system boots using vanilla systemd – with no pre-configured dependencies on the network being active, automount, nfs, sssd, cron, and chrony all start in parallel. While some of these daemons are tolerant of missing network activity, many are not.

Startup scripts in the user’s cron files may not execute, because the system has no ownership data for the cron file, as sssd has not started or is not able to query AD/LDAP. In addition, the NFS-mounted home directory for the user is not present, so the cron entry points to an invalid location. If your NFS mounts are not done using automount, they may fail entirely if not made using the background option.

You get the idea… We must configure network-aware services correctly to ensure we have dependable and consistent services start at system boot. To do this, we modify files in the /etc/systemd/service directory. If you are using configuration management (puppet, etc..) be sure to do this using that facility. Provided below are the configuration files used to stabilize the environment described above.