Memories

While working at Amcut – traveling to the near west side of Chicago with a field support rep (darned if I can remember his name, but not Darren). We went to service one of the machines at Ryerson Steel. At lunch time went to this little hole-in-the-wall chicken place. I was really naive, having moved to the city from the UP just months before. This was 1979, and different times. We were the only two white guys in the place (and probably in a 2-mile radius).

The year was 1974. I was attending a photography camp at Michigan Tech summer camp for kids. The photographer teaching the course was a world-class photographer, and the class taught composition in outdoor photography. We were shooting at McClain State Park, on the western coast of the Keeweenaw Peninsula. The day was sunny and 70ish degrees, with a few white cumulus clouds dotting the sky. Her name was Kim. What a beautiful day.

It was winter. We were on a ski trip to the Porcupine Mountains, and the day was cold and clear. A north wind off of Lake Superior met us at the top of the hill as we came off the lift. The snow squeaked under our skis as we traversed across the hill to the top of the run. You can see the blowing snow coming in off the lake and up the hill. Adjust the goggles and zip the top of the jacket all the way up – trying to turtle your face down behind the up-turned collar. Take off and hit that mogul field! Get down that hill, and hit the lodge for some warm up, then do it – again and again. At the end of the day, the bus ride home with 30 other kids, all exhausted from a full day on the slopes.

Systemd Parallel Execution Fallout

The Linux boot manager, systemd, speeds up the boot time of a computer running Linux by executing startup tasks in parallel. This is great, for the most part, but does introduce some potential issues depending on how your system is configured. For a stand-alone workstation or server, chances are you may never run into problems. Introduce some network-based services, however, and the odds start to turn the other way.

Allow me to introduce the following configuration, and discuss how systemd needed to be tuned to provide an operational environment:

  1. Workstations and servers use automount and NFS to mount file systems of shared data and user home directories.
  2. To manage user logins openLDAP (or AD) is used and the sssd daemon manages this configuration.
  3. The system clock is synced using the chrony daemon.
  4. Users rely on cron to startup their services at boot time using @restart in their crontab files, which point to startup scripts in their NFS-mounted home directories.

Now, let’s consider how the system boots using vanilla systemd – with no pre-configured dependencies on the network being active, automount, nfs, sssd, cron, and chrony all start in parallel. While some of these daemons are tolerant of missing network activity, many are not.

Startup scripts in the user’s cron files may not execute, because the system has no ownership data for the cron file, as sssd has not started or is not able to query AD/LDAP. In addition, the NFS-mounted home directory for the user is not present, so the cron entry points to an invalid location. If your NFS mounts are not done using automount, they may fail entirely if not made using the background option.

You get the idea… We must configure network-aware services correctly to ensure we have dependable and consistent services start at system boot. To do this, we modify files in the /etc/systemd/service directory. If you are using configuration management (puppet, etc..) be sure to do this using that facility. Provided below are the configuration files used to stabilize the environment described above.

9/11 – 20 years later

On September 11, 2001, I was in an elevator heading to our “turnover” meeting at the Chicago Mercantile Exchange. (Every morning, the IT group got together to discuss any issues encountered overnight or not resolved from the previous day. Today we would call this meeting a morning standup.) On the way down to the interstitial, someone in the elevator said they heard something about a plane hitting one of the twin towers in NYC. At the time, we all assumed it was a Piper or small plane, and continued over to the meeting.

Once in turnover, the meeting began as usual, but shortly after all eyes were on the TV in the corner of the room. (All conference rooms in the CME had TV screens tuned to one of the financial news channels.) Once he saw what was happening, Jim Krause, who was running the meeting, spoke up and said, “Everyone over to ops, NOW!” and the entire room emptied out and headed back through the interstitial to the operations room (the NOC).

The NOC at the CME in those days consisted of a dozen operator stations lined up in front of a wall of monitors displaying various system performance meters, and a few big screen TV’s displaying the news. There was also a “squawk box” – a party-line type phone that was connected to all the other exchanges.

I was standing near the center of the room, directly across from two of the big screen TV’s, which were displaying the events in NY. One tower was engulfed in flames, but the second had not yet been hit. The squawk box was buzzing with other exchanges trying to ascertain what was happening at the NY Merc, which was near Tower 1. At the time, before the first tower fell, trading on all exchanges had halted, and the NY Merc was evacuating. While upper management at the CME (Maz Chadid and Jim Krause) were talking with them, we saw the second plane hit. We could hear the impact and screaming through the squawk box. There was a lot of commotion and screams in the room. At that moment, we all realized this was a planned attack, not some random issue with a single plane. Some folks in the room pointed out the people that were leaping to their death out of the upper floors of the tower. Reports started coming in about other planes in the air with similar intentions. Jim made an announcement that we should start evacuations of our building. We were located one block from Sears Tower (now Willis Tower), and there were some strong concerns that might be a target.

Around this time, the first tower came down. I remember Maz, who was standing next to me screamed out, “Oh my GOD!” Nobody was expecting that, and the crumbling of the tower seemed to take forever. I can see that in my mind to this day. Again, there was focus on getting the building evacuated, but reports of the train stations being like cattle pens left little opportunity to make it out of the city. News of the Pentagon impact was coming in, and there was talk of another plane over Pennsylvannia that jet fighters were scrambling to intercept. (I heard this on the radio.)

The second tower fell, to more screams and sobs of anguish in the room. There was little that could have happened to increase the sadness, shock, and sorrow in that room. The realization that thousands had just perished violently in the collapse of those two towers swept over the room. Then there was just the sound of sobbing, tears, and those folks trying to console each other. We watched and listened as Bush spoke, then as Congress sang “God Bless America” on the steps of Congress. https://www.c-span.org/video/?c4816002/user-clip-congress-spontaneous-sings-god-bless-america-91101

Maz and Jim asked a few folks to stick around and help shut things down, and told everyone else to try to find a way home to our loved ones. I was asked to help out, which I did. I had no way home, anyway, and knew that I wouldn’t be able to get in touch with anyone as all the cell circuits were jammed.

While trying to get things shut down, Maz, Jim, and my boss, Joe Panfil, were on the phone with the CEO of the NYMEX. They were reaching out to offer CME support for their loss, and to inquire what CME could do to help. Over the next several hours, several things were decided:

  1. The CME would build and infrastructure out to trade the NYMEX contracts – not to steal the contracts, but to allow the NYMEX to open within two days. (Yes, essentially we were building the NYMEX at the CME in less than two days.)
  2. The current CME disaster recovery plans was not realistic (called “tower redundancy” because our DR site was in the north tower, at 10 S. Wacker, while the primary data center was in the south tower, at 30 S. Wacker). A new “Remote Data Center” was needed to provide true disaster recovery.
  3. Our existing NOC was insufficient to allow operations to effectively monitor and track world events. The NOC would need to be included in the revised DR plans.

The events of that day and the focus on these three decisions turned out to consume my life over the next three years. In the short term, I was called upon to lead the team in building out the NYMEX infrastructure. By 9/13, we had accomplished that task, and I was on my way to California with my family, as our daughter was scheduled to start classes at Cal Poly the following week.

Originally scheduled to fly, we had traded our flight for a cross-country road trip. As we headed west, it was amazing to see the traffic. There were no flights in the air, and it was clear that there was a sense of patriotism awoken in everyone on the road. Prior to 9/11, it was rare to see an American flag on a car. Every time we stopped for food or gas, I would search for a flag I could put on our van. I felt it was important to find one, but everywhere I looked, there were none to be found. It’s not that the store was out of stock, it was just something that wasn’t available. I finally found some red, white, and blue streamers that were meant to be inserted into a bicycle handlebar, and attached them to my radio antenna. It was the best I could do. Folks would see those streamers when passing me and honk and give me a thumbs-up.

The patriotism fueled by the events of 9/11 was amazing. I think every American felt the same, we were in sorrow for the loss, and united in anger for bringing those responsible to justice. We saw trains moving military gear toward the ports in California, and knew what was coming. By this time, it was well known that Osama Bin Laden was behind the attacks, and we were going to hunt him down. News reports came in with details on how the terrorists had infiltrated our flight training schools, and used the aircraft as weapons of mass destruction against us. The country was united in the need for some form of justice, but little did we know at the time how long it would take to finally make that happen.

Now, twenty years later, and the divide in this country is deeper than I’ve experienced in my life. It’s so sad that it took such a devastating act of terrorism to unite the country, and twenty years for it to all fall apart. Twenty years ago, the flag flew as a sign we were all united against the acts of terrorism that we had just lived through. Over just a few weeks following the events, you could walk into any convenience store or gas station and find an array of American flags, where there had been none before. Today, if you display the flag on your car or home, you are labeled a “white supremacist”, a “racist”, or even “domestic terrorist”. Are we in Oz? In what state of reality does this make sense?

I fly my flag with pride in being an American, in memory of the day terror struck deep on our soil, and stand with those that believe that the freedom granted to us by the U.S. Constitution is still the law of this land, protected by those that have served to defend it, to whom I cannot show enough respect and thanks.

I will never forget.

Immigrant Day

Apparently it’s “Immigrant Day” today.  

Interesting.  

My great-grandparents immigrated to the US aboard a sailing ship from Sweden, Norway, and The Isle of Man.  They came to the USA for many of the same reasons folks immigrate here today, work, freedom, a better way of life.  Times were tough for them.  More difficult than anything most of us know or experience today.  When they landed here, there was no government assistance.  You were expected to learn English, to assimilate, to find a way of supporting yourself and your family.  They succeeded in doing that against some incredible odds.  There was little in the way of medical care – antibiotics had not yet been discovered, so I lost relatives to the Black Dyptheria plague of 1917.  Settling in the UP of Michigan, there were long, cold winters, with only rudimentary insulation in the home they had built, relying on wood they had cut the summer before for heat.  No electricity, no indoor plumbing, and food preserved from the summer before to take them until the next harvest.  

Service in the First World War was met with pride and patriotism. The heyday of the roaring 20’s offered a brief respite from the decades of struggle and war, only to welcome  The Great Depression.  Wood lots and potato fields sustained them.  Patriotism and pride of the American Way maintained faith that they would see better days ahead.  Both of my parents grew up during this time, and told me stories of how difficult this time really was. I have photos that show my mom, as a teenager, weighing no more that 80 lbs, and looking on the brink of starvation.  She told stories of having potato soup one night, and the next night having only potato peeling soup.  They survived. They didn’t ask or expect the government to come to their rescue.  They found work, made do, mended socks, wore hand-me-down clothes and shoes two sizes too big because that’s all they had.  Very few alive today lived through this, but any one of them will tell you that nothing since has been so devastating as these times.

So, today is “Immigrant Day”, huh?  My “celebration” is to reflect on the struggles of my immigrant ancestors and thank them for their morals, perseverance, assimilation, patriotism, hard work, and strength.

Time

Technologist, electrical engineer, carpenter, machinist, draftsman, welder, mechanic, photographer, genealogist, chauffer, chef, husband, dad, and grandpa.  How many hats do I wear?  What would life be like if I had no interest in most of these things?  Would I feel better, knowing that I didn’t have all these projects to do, or would I just get bored?  

I have family photos to catalog, tag, scan, and archive.  My nature and outdoor photography is setup on  http://www.thedigitaledge.net/Nature/Favorite-Scenery/n-vt3x6w, and  a collection of genealogical and historical photos at http://www.thedigitaledge.net/History.  I also have three large totes full of additional photos that need to be scanned, tagged, and cataloged to my site. I also have a collection of roughly 5000 negatives from the early 1900’s through roughly 1950 taken by a close family friend from my home town.  I am doing archival scans on these – creating high-resolution TIFF and JPG images and one lower-resolution JPG of each image, then burning the TIFFs onto high density archives quality Blu-ray disc.  I’ve completed 2100 of these so far, and just need to find the time…

To finish cleaning and setting up my shop.  I’ve got a 2500 square foot shop, with space dedicated as an auto/machine/welding shop, woodworking shop, an electronics/office/drafting area, and a finishing room. I’ve spent four years thing to get the shop setup in a way that would allow me to start or complete countless woodworking projects.  The entire place needs to be organized, cleaned, and rearranged.  The woodshops need cleaning, dust collection setup, new electrical and lighting run, and tool and supplies storage built. All I really need to get this place fully functional is to find the time…

To remodel the house…  I’d like to  gut the kitchens, family room, dining room, living room, hall bath, and in-law suite.  The in-law needs a complete remodel, and I’ve drawn up plans to add a new entrance.  The existing footprint is laid out poorly, and it could be so much nicer than what it currently is.  The same goes for the main house – the center of the house has been carved up into four rooms.  I’d like to open that space up, and create a huge open floor plane for family entertaining, using a rustic, lodge-like decor to compliment the ranch layout.  I have the tools and talent to do this, all I need is time…

To help my son restore his 1942 Willis MB jeep.

Yes, the list goes on, and on…  this is just a few.

Can I just prioritize this list?  Not likely.  Can I just pick “the important things”?  Um, what would those be?  Overwhelmed?  You bet.

Sigh.

Plan for the future

As we move through life, we plan.  We plan on how things are going to be, what to do, where to go, how much we save, what to buy.  Decisions are made based on our plans, and we do our  best to make those decisions with regard to how we have everything planned out.  

Then….  chaos!  

Life tosses you a curve ball, and those decisions you’ve made don’t look so great now.  Depending on the depth of that curve, these could be some devastating change-the-course-of-your-life events.  Your well-laid plans are now out the window.  Even the contingencies you put in place are gone, and all you have left are the scraps left behind, to pick up those pieces and try to move on. To try to make some new plans that take you to a better place.

I’ve always thrived on challenges.  Every time I’ve been knocked down, I come back swinging – I meet the challenge head on, with an optimistic belief that I can and will rise above, making new plans to put myself in a better place.  We say we learn from our mistakes, but all that learning doesn’t make the crystal ball work any better.  There are always surprises. Always unexpected events that keep us on our toes.  That’s life.  Anyone that tries to tell you all their plans have worked out exactly as they’ve planned are lying, or have very poor memory.  The best we can hope for is attaining success in the long term.  Keep trying, keep planning, and keep your sights set high.

Destiny versus chance

As I was growing up, mechanics and building things seemed very natural and made sense.  Bolts and nuts, screws, nails, glue and twine were my friends.  Drafting, designing, and building seemed to be my direction in life.  By the 6th grade, I knew I was going to be an architect.

Plans change, and my career didn’t involve designing or drawing buildings, but I still love to do floorplan design, and my drafting work put food on our table for many years.  It also provided a path to what turned out to be a lifetime career doing Unix system administration.
While in pursuit of a degree in Electronics Engineering, I answered a help wanted ad for a position of typist.  I had learned touch typing in junior high, and I thought this might be a good part time job to hold as I was going to school.  Answering the ad, they had me come in to take a typing test.  As I was chatting with the receptionist in the front office, a doorway to the back room opened, and inside the room I spotted rows and rows of drafting tables!  I mentioned to the receptionist that I knew drafting, and the person that had come out of the back room heard my remark.  As it turns out, this was the contract drafting supervisor, and the typing job I was looking for turned into a contract drafting position that eventually sent me onsite to AT&T Bell Laboratories, my future employer, and the introduction to my career.

Never take for granted the possibility that a chance encounter can change your life.

Spreading it too thin

Every day I think about how many projects and tasks I have to do, and a long time ago I came to the sad realization that I will never have the time to complete them all before I’m gone.  This is the first time I’ve actually written that down, and reading it makes me realize that had I not spread my self so thin, I might have completed some extraordinary accomplishments.  It’s the guy spinning plates on the top of sticks.  He can keep only so many of them spinning before one crashes to the floor.  He has a stack of them that he never even is able to get to – like those projects of mine that are numbers 97 through 120…

Prioritize!  Yes, that’s the ticket!  All I need to do is sift through the stack, and decide which item is on the top of the queue.  I think, for now, that would be some sleep.