I carried a pager for a short while, pager, remember those? On-call weeks were hell, even when nothing happened. It's the expectation of getting the call, and the expectation that a system outside your primary domain of expertise will be down. I wasn't paid enough to weather the stress, and I don't think there's a hard number to suit every one. If I ever had input to how an on-call rotation should be handled, I'd make it a pair of on-call persons. They'd switch off as they please, but there will always be 2 persons coordinating the response. One person could come in, with the other person on the phone, or both.
It's the loneliness. Somewhere in this vast organization connected with pipes that no one single person has put together, some element would break. Mercifully the impact ripples and you kinda know about the subsystems to track down the leak and patch it up and alert the domain owners for the next working day. You're the person they're going to call, there's no cavalry, it's you looking at incomprehensible gibberish while the rest of the world is asleep. Even the desk who has called you doesn't get involved, they're done. For me, the idea that it's just me until it's fixed (or degrades further), was the hardest thing to cope with.
I'd pair up my ideal on-call team. It'll cost more, but I'll pay it.
Incidentally pagers are still being carried in SCIFs, where there are no mobiles and sometimes no outside line.
I was also on pager duty with physical pagers many years ago.
The perspective I had was that it was a rite of passage for a systems engineer. I noticed that only the more junior engineers were on front-line duty. I believe the thinking was that by forcing junior engineers to go through pager duty, they would be much more diligent when they propose, create or approve changes down the road. Whether or not this is a perverse incentive which ultimately ruins productivity & team morale is a separate conversation...
We had a primary & backup who got paged at the same time (due to the financial implications of timing), so there wasn't a total sense of isolation. But, there was definitely that sense of helplessness when it turned out to be a problem in a totally different system (and country). Worrying about something going down definitely kept you on a tight leash with regard to WiFi and laptop accessibility. We did get a little wild with the wording on the policy. Some bars downtown had excellent WiFi, so when some of us were on call during Friday/Saturday we would just make sure we had a laptop in the car and pager on body. Definitely not the best situation to be in, but for some teams the amount of time you were on pager duty could be measured in years.
The usual way to handle the "different system" issue is to have on-calls in each team, so you could loop them in for their stuff or even transfer the ticket and go back to sleep; and on-call-focused documentation within teams so you can have some notion of every component that they are not intimately familiar with.
I hate on-call; it doesn't stress me out but I'm generally very protective of my personal time, especially weekends.
However, I actually found my greatest on-call highs were from figuring out issues in other people's stuff that was 100% unfamiliar to me, sometime at 4am. In the most memorable case of Rogue Night-Time CPU Spike I didn't even have source code access :)
I have found in my career that being too good at on-call/L3 escalations only makes them come to you more. Teams that have slopey shoulders and phones that seem to always go to voicemail.. don’t get reprimanded, and your team gets to keep playing hero.
This is an observable truth. Be good, but don't show that you're too good. You'll get called all the time, even for stuff you're not good at. You'll be the janitor for all kinds of idiotic mess, and the perverse incentive will make them call and rely on you more, and you get less of your work and what's important to you done. Here's a joke that'll provoke cynical laughter: What do you get for winning a pie eating contest? More pies.
I suppose that could be true in small organizations... not a problem in a more regimented on-call setup. My phone is on alarms-only at night when I'm not on-call, so they cannot physically call me :) I suppose people could still ask during the day, but generally in a good on-call culture it's frowned upon (unless you own the component or it's an all-hands-on-deck emergency where extra hands are actually helpful).
We have our "ops" guys that monitors the environments, and a list of people for each system that is more or less always on call. Each service/job that is being monitored has a procedure/description for how to respond to failures, and if nothing else works, the people on the list gets a call.
You're not required to take the call, but you get a fairly large sum of money if you do, meaning people usually take the call despite not being required to.
Because of laws here, you're required to have 11 hours of uninterrupted "free time" after leaving work (no laws on how long a work day is), so taking a call usually means you violate that, which in turn restarts the clock on your "free time", meaning you're entitled to "sleep in" on company time the next day. Doing so of course wreaks havoc on all planning, so very few people actually take the full 11 hours off, and instead just get in "late" around 9am.
Edit: I should probably have explained that the calls happen on average once per system every 2-4 weeks, and the amount of people on call is probably limited to <5%-10% of the engineers.
When I worked in Austria and Germany, most employees started between 6:00 and 8:00 AM at the latest. Although there was nothing against coming in after 9:00 AM due to flex-time, it would guarantee you'd miss all the morning coffee chit-chat with internal company gossip and by the time you'd leave work and head to the city, the shops would already be close to closing.
Yeah that's the other thing. A lot of shops and agencies close at 5pm, so if your come in to work at 10am and work until 6pm? Everything is closed once you leave, good luck getting anything done after work.
Nowadays absolutely, at least for groceries. Then again I don't live there any longer :) Back when I was a kid? Not so much.
Just looked it up. They were actually allowed to open from 7a.m. but at least where we lived, the regular shops in the city would only open at 10a.m. Shops were allowed to open until 6:30p.m. and not sure how long they actually did.
We only had one car and lived too far from a large grocery store that would carry everything and at proper prices to walk/bike for larger volumes. So we only did them when my dad was home and could drive us (mostly - I do remember having to bike there from time to time).
But usually it meant doing it on Saturdays. Shops were allowed to open until 2p.m. on Saturdays and closed on Sundays. We definitely made use of the "long Saturday" - one Saturday per month, they were allowed to open to 6p.m.
I sometimes wonder where the hell most HNers work where 9am is not considered late start. At a warehouse I worked at the shift was 6am to 2pm. And it was great that way as you avoided most of the summer heat. The software company where I work now, 8am is standard, that way you're nicely done at 4pm and have the entire evening to yourself. My wife is also a programmer and usually starts at 7:30am because that's their first scrum.
> starts at 7:30am because that's their first scrum
We have scrum inside mandatory hours. There's a fairly strict company policy that you cannot schedule _recurring_ meetings outside mandatory hours, and if you do, you cannot expect people to attend, so scrum here is 9am. Even scheduling non recurring meetings outside mandatory hours is frowned upon.
As for the schedule itself, it fits nicely with (european) family life. My kids start school at 8am, so i usually drop them off at school on my way to work, and get in around 8:30am (before COVID anyway). I get off work at 4pm, and have the entire evening with my family.
I'm aware that at least some parts of the US has (or had ?) a much more fluid line between work and free time, where people will leave work to have lunch with their family, return to work, leave to watch a soccer game (or whatever), and return to work, and leave work around 7pm.
To me that sounds much more hellish than just working 8am to 4pm :)
This isn’t super common where I am (although you can do it). I had a coworker who practiced something like this so they could go to the gym in the afternoon when it would be less crowded. Or for the soccer example, not much of a choice if you want to watch live European sports.
Also at least in Seattle it seems like every doctor’s office keeps bankers hours (M-F 9-5) so you would have to disrupt your own workday to go to an appointment, though in practice I haven’t known people to bother putting in extra work if their projects are reasonably getting done on time.
Back in ancient times, my first software job was with an crusty old fashioned company. We were required to be in before 8am, many people arrived as early as 6am. Dress code was also very strict (but more lenient than it was historically)
My second software job was with a company that was just as old but much more modern (not startup-ish). the boss required the team to be in before 9am, daily team meeting was at 9.
My job now has flextime with core hours between either 10-3 or 10-2 (can't recall which) but very few people arrive after 9am. All other jobs I've worked have flextime with similar core hours, there was only one job that had a true "no rules" approach.
My preferred hours used to be 730-330. The last place I was at the majority turned up around 930, and then milled about drinking coffee and chatting for the first hour or so.
I'm working for a small(er) (30ish people) software company and 9am is the general start of the work day, but it's not regulated or enforced. Some people start at 8am, but I for myself start mostly around 10am, except there are some meetings before.
Every single software team I've worked in or heard of in the USA? ;) Usually it's tied to the first meeting, I have a mind-destroying train wreck of a "scrum" 3 times a week at 11am so it doesn't make any sense to start before 11.
When I used to wake up early consistently in a different team I tried to move the morning scrum to 7am, or maybe 8am, or maybe come on 9:30am?! But people weren't buying it. Meetings before 10am are only scheduled in extreme circumstance.
In the UK I'm 9am to 5pm, however my contract allows for unpaid overtime. Usually management are pretty good with this though, and give some unofficial time off in lieu if I have to work OT.
Unless i was in serious financial trouble, or in a learning situation, i would never accept a contract that "allows for unpaid overtime".
Most of the companies i know don't allow for "paid free time", so why should i give away my time for free to increase profits for the company ?
The exception to this is of course "job salary", but i never accept those either unless i have a say in what exactly makes up the "job" part, and i usually tell the employer that "job salary" goes both ways, so if i'm done with my job Wednesday afternoon, i'll say "enjoy your weekend" and be back next monday. That's when we usually agree on paid overtime :)
>>Unless i was in serious financial trouble, or in a learning situation, i would never accept a contract that "allows for unpaid overtime".
At least in the UK the law specifically forbids working more than 48 hours a week, so it's pretty common to see contracts where you opt out of this regulation. Not because the company even intends to make you work overtime, but because if you happened to work more, even entirely out of your own free will, you could still sue the company for working more than 48 hours in a week. So it's a very common thing to add in contracts here.
There's no company policy mandating me to start that early yet I start between 7am and 7:30am, by 8am the office is mostly full.
The earlier you start the more daylight left to enjoy at the end of the day. 9am really sounds awfully late. People who start at 9 don't enjoy their morning any more than I do. They just wake up later. And it forces them to stay longer in the evening while I'm out taking advantage of the extra day light to do outdoor activities.
>People who start at 9 don't enjoy their morning any more than I do.
I am a morning person which means I really treasure my morning coffee and jazz music and reading time. I wake up at 7am and enjoy two sweet hours of a cold, dreamy morning before starting work fresh at 9am.
I am a night person which means I really treasure my nightly herbal tea and hip hop music and comic book time. I take a hot shower at 10pm before wandering into bed sleepy at 10:45pm.
None of this is rigid but I just feel like you are making super subjective arguments about a person's choice in time to work and sleep and live their life.
I've seen some companies that clock the hours you're "in the building" and allow coming up early to leave early.
What happens in practice is that some people will come in early, start working early, and leave early. Some other people will check-in early, enjoy the office-provided amenities in the empty office, and actually start working when the rest of the team gets in the office.
Some smarter company just ask you to check-in before, say, 10 am and don't care when you leave (either early or late -- and don't require you to check-out explicitly by passing the badge) -- it's up to you and your boss/manager: as long as you meet the deadline, leave as soon or as late as you see fit.
I would usually be able to make it to work by 9:00am. That's coz of when I could drop the kids off at school and then catch the train. Since covid they don't have morning supervision any longer. Thanks to covid the office is closed anyway but if it was open and they'd require even 9am never mind earlier, then my next words are "Bye bye".
That said even once Covid restrictions are over forget me coming to the office every day of the week. I enjoy my life way too much to let that be taken away again. Yes I moved out here by choice and chose to take the longer commute over city life, I do realize that. But I really don't get what the problem would be with just meeting at the office like once a week or whenever 'needed' with your team instead of being forced to go and partake in all the social banter that kills concentration.
I'm with you 100%. I'm usually the first one in the office (well, pre-COVID anyway) and the first one to leave. Anecdotally, it's same since we started working from home - I usually try to start around 7-7:30am, and I generally don't start getting replies from coworkers until 8:30am or later.
I see a lot of threads and comments about starting at 9-10am, and I don't understand it. I'm usually eating lunch at 11am and done with work by 3:30pm. It's so nice! The few times I've had to work until 5-6pm have been awful for me. I feel like I don't really have any time to really DO anything before bed.
Those who like to start their day mid morning and those who like to start their day early morning. To accommodate everyone a lot of companies use a standard "core time" of 10-2 where everyone needs to be in the office.
It's sad that different chronotypes aren't taken into account for work schedules. There are many an individual contributor, and manager, who naturally outperform outside of the usual 9-5 framework. Formally it's called Delayed Sleep Phase Syndrome (DSPS). Practically these are the people whose ancestors were guarding the tribe keeping it safe at night.
We have the same policy. Mandatory hours so there is common time where everyone is at work, but if you do your other hours after or before mandatory hours is up to you. Works really well. There is plenty of time when everyone is at work and can communicate but late and early people can set they schedule as they like. Plenty of tasks to do when you don't need other people.
I’m in Sweden, so Scandinavia. Every company I know has 40h work weeks. Lunch break is not included so if you start working at 8, take a 1 hour lunch break around 12, then you can log off at 17.
The normal (office) working hours here are 37.5 hours per week, excluding lunch break, which is usually 30 minutes. That breaks down to 7.5 hours of work + 30 minutes of lunch break, meaning you get in at 8am and leave at 4pm.
Last gig I had in Germany: it was 40h/week. I never seen a 36h/week schedule in France either... (I'm talking about standard software engineer positions).
In France base is 35h/week, 37/39 are frequent with the extra 2-4h hours then available 1:1 as extra vacation time ( RTT, you get hours on top of 35 worked over the year / 7 ( standard work day) of them over the year).
I'm currently on 35, might move to 37 ( majority at my company).
If we are counting actual working hours, it is also 7 hours a day in Poland. (8hr/day inclusive of 30 minutes lunch plus cumulative 30 minute break from staring at a screen).
It is similar in Poland.
Here most colleagues can start between 6-9am and finish between 14-17. This is inclusive of a formal 30 minute lunch break and 10 minute breaks every 1 hour. (In practice most people just take 1 hour lunch instead). I enjoy this because I have much more time outside of the office to enjoy my life.
What is the point of such a crazy schedule? If someone has to do a double-graveyard shift and is basically useless the rest of the week, why not just split that shift into two so you don't destroy all of the people who try to cover such shifts.
People can generally work 4-12 without much trouble or needing to recover after it other than weekends. Doing 12-8 might be an every other night kind of thing. Having a couple people handing those off might just cover it.
The author is in the military - I've read a fair few things by current and former service members and two of the most common themes are:
- the disregard for health and wellbeing of each soldier
- being required to do nonsensical things that are counterproductive and likely compromise medium-to-long term goals of the outfit you're attached to
So I agree with what you said - there is very likely a way to schedule these shifts so that these problems do not occur. The Army has little interest in this though, their requirement is that the shifts need to be covered and they currently are so there is no problem.
One particular experience that stands out when I was in the Air Force was some Sergeant changing our physical training time from after work to 5:30AM. As in everyone commutes to the base and is in formation at that time. It was just sadistic torture essentially lol
This is a good example of why I didn't want to join the army. I understand the importance of instilling order and readiness,but it can be done in many ways, not necessarily with 'just because I can'. I've heard countless examples like this, the worst being where some poor guys had to carry a cigarette butt on stretches for a few miles, then dig a massive pit and bury it.
The United States Navy had serious problems with work demands causing sleep deprivation. Here’s a news story about a policy update they made:
“The “Comprehensive Crew Endurance Management Policy,” signed off Dec. 11 by Naval Surface Force Pacific and Naval Surface Force Atlantic, is the first update to the joint instruction issued just months after two 2017 fatal at-sea collisions rocked the Navy.”
Basically the crews were too tired to think clearly and crashed into things, causing damage and deaths on their ships.
It's like that quote about how the market can stay irrational longer than you can stay solvent - the organization can stay dysfunctional longer than you can stay sane. For any truly big system, inertia is a more powerful force than you think.
The isn't a strategic or tactical purpose that makes this kind of situation an intentional or desired outcome. Any sufficiently large organization, and this especially holds true by militaries, are typified by the following two characteristics. First, turnover at scale necessarily prevents the rank-and-file, in aggregate, from learning and developing over time. Every year, X number of people will leave the organization for reasons the organization can't control (retirement, spouses moving, legal mandates, whatever) and they will be replaced by people who are completely unfamiliar with the organization and any lessons the people leaving may have learned. Not replacing the people leaving will leave positions unfilled, which is untenable, so they will be filled by whoever is around to fill the chair. This is inevitable and unavoidable. Second, there are only so many issues that upper management can focus on. By the time people rise up through the ranks and get into a position where they have the power to do anything, the vast majority of what it means to exercise that power is to act like a hospital at a mass casualty event - triage for the most critical issues that need your attention and trust that a solid reliance on process will prevent anything else from collapsing.
If you're looking for logic then you don't understand how these organizations work.
If you're looking for logic then you don't understand how these organizations work
The logic is that everyone must be replaceable on a moments notice because they might be shot and someone else will need to pick up that job. Having roles that you need to be intimately familiar with in order to be effective is the exact opposite of what the military needs. If a job can be done by someone with minimal training while desperately sleep-deprived then that is perfect for their needs.
Even in offices, the myth is that everyone is replaceable at a moment's notice, because that is a comfortable thought for those in charge that makes it easier to play "4D chess" with the corporate pawns and integrate seamlessly with quarterly and yearly budgets. It's based on the law of averages and cements ending up with mediocre workers and products at best.
Exceptional workers in key (non-management) positions will take whatever they've learned with them, never to return. That leads to failed excavation projects looking for the original reasoning and business logic. Then, things need to be recreated from scratch because there is no alternative besides hiring the former worker as an expensive consultant. All because people failed to plan ahead and guarantee actual continuity, focusing exclusively on the small subset of elementary tasks. In the end, the work becomes more disorganised and product/service quality suffers because of gaps in domain knowledge, context and competence.
That myth might be relatively harmless in the primary and secondary sectors to a large degree, but is actively harmful in the tertiary, quaternary and quinary sectors. If you factor in that most of the work - both in terms of quality and volume - is done by a small minority of employees, the prevailing management dogma is costing companies lots of money to entertain the notion that worker retention is a secondary consideration. It's especially myopic because it usually doesn't take a whole lot to keep people happy in their jobs, leveraged as they are by their own obligations.
Yes, a lot of this kind of "everyone is replaceable" 4D chess mentality of management leads to a fat middle.
I've seen this both at large, stodgy organizations that have frequent layoffs and at smaller organizations with Frank Underwood CTOs.
Bottom performers get managed out on the firms schedule, but top performers find their way out on their own schedule. This can lead to great chaos.. people getting un-fired, throwing money at the key man risks you accidentally created, and over-correcting with too-fast over-hiring which then leads to inevitable future firings.
You are 100% correct. It costs a lot of $ and time (6 months or longer) to replace a good software developer. So if you do it all the time, your company will be operating way below what it needs to be competitive. The moment your company management starts saying “everybody is replaceable” you know the company is going downhill and it is time for you to find a better job.
That's an ideal, not the reality. The reality is actually pretty frequently similar to the private sector - someone is the only person who knows how to do something, they leave, and they're replaced by someone who doesn't know how to do it and picks things up as they go along. It's one of the reasons why militaries can feel pretty dysfunctional. The only way of ensuring that people are adequately replaceable is to have formal training programs, and the only way of ensuring that those training programs improve over time is to have a feedback loop to take in lessons from the field and to keep abreast of updates to the state-of-the-art. Ultimately this is too costly, even for militaries, with possible rare exceptions in small special-ops type units where people do rotate between operations and training and rarely de-facto leave the unit and as such, do not suffer from the same manpower pressures of most military units while enjoying attention of upper management when desired.
Welcome to Hell Korea. The country is infamous for poor working conditions, particularly in the military, where the author was apparently serving out their mandatory military service.
Military organizations tend to detach personal for long shifts so the remaining unit can stay somewhat functional in their normal schedule.
When I was on guard duty it was usually a 24h shift followed by a 24h break for a group of 8 or 10 privates and 2 NCOs. That means that roughly a quarter of the company is permanently detached. If you switch to 12hour shifts, you have to consider that normal operations take place during daytime. At any day you then have three groups unavailable: One on duty, one resting from the night shift, and one preparing for the night shift.
I'm not military, but worked with them for two years, sometimes with similar schedules.
My view about being a soldier, a serviceman has changed drastically during that projects. The cause is noble, and the thing they're doing is very hard. Make no mistake here.
On the other hand, they're stripped of their emotions and humanity to an extent. They become something else, an hardened being after some time.
Because, they need to do what has to be done and they shouldn't think through it, but just act.
I'm a grandson of a commander. I love soldiers, but this is how they're trained, and this is how they should be to be able to perform this duty.
One could argue the qualities of a soldier or officer (individuals) transcend their organisational context (groups) and time. In other words: politics, technology or best practices change, not the responsabilities. The ability (necessity, even) to adapt to that fact defines a fit-for-service soldier/officer.
So, I contend you find remarkable soldiers even in terrible groups, and that they (individuals) should not be judged on their group but on their own qualities of service.
Whether you find those parameters noble is entirely subjective and not mine to question but I do believe anyone who sincerely understands and accepts these obligations deserves respect, as an individual.
> So, I contend you find remarkable soldiers even in terrible groups, and that they (individuals) should not be judged on their group but on their own qualities of service.
So, we should praise ISIS fighters? I am intentionally picking up example of extreme, but very real organization. Same question can be asked about SS members. They knew what they joined. Does them doing good job in that organization is really praise worthy?
The better quality their service was, the more victims they have.
You're distorting it. I was trying to point out that, they have to deform themselves to somewhat non-human states.
It's not valid only for servicemen. Firefighters are same. Police are same. Long haul truckers, cargo ship crew are same.
I think everybody making sacrifices for doing what they find worthy are eligible for some respect. They accept to deform themselves to do something they find value in.
Do not add ethnicity, politics, geography, race into this. It's about being human, and other parts are neither part of my comment nor has place in this specific thread.
I am not distorting. I am using unambiguous example of what I have in mind.
> soldiers even in terrible groups, and that they (individuals) should not be judged on their group but on their own qualities of service.
Unless you was forced into that group, which was not case of SS but in some cases could be ISIS, you know what you were joining.
I don't think sacrifice itself is worth respect, if the goal of that sacrifice is destructive and wrong. If you sacrifice yourself in service of group that does genocide or tortures to get power and so on, you did genocide and tortured people so that your group have power. The ideology of "sacrifice is worth respect no matter what" empowers exactly these groups first. Believing this makes people more likely to join these.
The victims, especially those who stood against that group should be celebrated. Those who said "no" deserve respect. Those who were avoiding still deserve more respect.
> I don't think sacrifice itself is worth respect, if the goal of that sacrifice is destructive and wrong.
Wrong is a subjective term and relative to viewing point. This is exactly what I'm avoiding here. I'm not discussing whether it's right or wrong from one particular perspective. Moreover, I'm trying to direct discussion away from war and other sensitive matters with all the power I have with my words, but you want to fixate the subject to ISIS and other terrorist organizations, that's fine.
As a citizen of a country which has to defend against terrorism in point-blank distances, I understand you, but forcing the subject to this perspective is not correct, at least in this context. Especially me, as the OP, trying to clarify it further.
My family is a victim of a genocide, but world won't acknowledge it. With your reasoning my family was on the wrong side, in a war which they weren't actively participating and was not killing anyone.
When a TV channel renames the people killing our citizens, they become heroes. They become right and we become wrong. Hah.
So, don't pull this into politics, war, genocides and other stuff. That tar pit is not a part of this conversation. It shouldn't be.
I intentionally uses also example of SS which was state army. ISIS see themselves as army for state too. For that matter, genocide in China and Rwanda were all helped by regular armies.
It is simply not true that all armies would primary defend. You make the "state run army therefore good" implication, not me. You are not leaving politics out when you claim that sacrifice of joining army is always wortht of respect. It is very political statement that prevents criticism of armies and empowers abusers in uniform.
What you are doing is trying to divorce consequences of joining those organizations and paint their members in unfairly good lights.
Sacrifice is also subjective. Some of those people like the organizations they are in.
> It is simply not true that all armies would primary defend. You make the "state run army therefore good" implication, not me.
Maybe the armies I've seen are only were defending and wasn't causing any harm, and my view is distorted on that one. I can accept that. OTOH, the core of my comment is "Accepting a hard duty and dedicating yourself to it at the level of deforming yourself is worthy of respect".
This is not a weapon/army thing. Healthcare workers are doing this for more than a year now and most people respond with "So, what?". This is just ridiculous.
> What you are doing is trying to divorce consequences of joining those organizations and paint their members in unfairly good lights.
That's a brave accusation considering the information I've disclosed in the comment you're replying to.
> Sacrifice is also subjective. Some of those people like the organizations they are in.
Liking an organization doesn't exclude sacrifices. A lot of people are sacrificing their personal and family lives because they like the corporations they work in.
The aforementioned healthcare workers also fit into this definition. This is not a job which you can do without liking/loving. It also needs some sacrifices. Both personal and interpersonal.
Being a soldier is in itself not honourable or noble. Most atrocities are committed by armed soldiers killing unarmed civilians (or killing civilians using drones because they happen to stand near a wanted target). It is what you do as a soldier (or civilian) that makes you honourable or not.
I remember having to nurse the batch jobs in the morning when I was working in app support. I had trained my spouse to look out for the yellow "failed" line and let me know when I was away from the laptop getting freshened up; failures weren't super common but they did happen and such incidents basically took the best part of the day towards investigation/clean-up/root-cause-analysis summary and such activities. That experience helped me swear off any kind of support role and instilled a lot of empathy towards support staff.
One place I worked had six operators on site 24/7 in a restricted access floor of the building. At any time five of them would be asleep and the sixth would watch the cameras and wake everyone up whenever someone arrived in the elevator lobby. They had there own rotation of who would be woke and who got to sleep. Even during the weekday no one could gain access to their room without being buzzed in, so their manager was none the wiser. He found a pillow one day but it was explained away as belonging to one of the operator’s kids and that answer satisfied him.
I see the parallels, but I don't do those either, fuck that, there's no reward for that. It's a subculture that a small percentage of people are able to follow, and often only while they're young.
This seems counter-productive. Humans with sleep deprivation are prone to mistakes (as the author admits).
I know "we do what our bosses tell us"- but honestly if it was me I would be informing my boss that this is counter-productive and will cost the company (or in this case the country and it's countrymen). Much better to have staggered shifts or even a small night crew.
> if it was me I would be informing my boss that this is counter-productive
I suspect you’re fairly new to the workforce. Try that for a few years, see how little effect it has but how much energy it sucks away from you, and then resign to simply doing “what the bosses want” even if it means setting piles of cash on fire on a regular basis.
I’m not sure what fairly new qualifies as. But I’ve been a sysadmin in various companies and countries for 12 years now. (and a person who is on-call for nearly all of that.)
Maybe it’s a European thing, but my bosses have never shied away from the ugly truth that extracting value from humans requires that the humans are able to function properly. Which includes days off, good work life balance and proper rest.
I know it's not a popular idea, but here it goes anyway: senior software engineers, please say NO to being on call. It just puts some much pressure on engineers (like me) who don't want to be on call in exchange of more money: managers then think of you as someone who is not very commited to the company, bla bla bla...
I'm commited to the company I work at any given time, from 9am to 5pm. That's all the time a company can get from me.
So once both senior and non-senior engineers opt out of on-call, who do you expect to take care of issues that happen out of hours? (In case of companies who deal with 24h traffic and transactions)
That's realistically out of reach for non-enterprise companies. If they're not primarily standard senior engineers doing typical day work, you need at least 4 people to cover the on-call week + redundancies for leave. You're talking about employing a whole extra team just to handle an occasional callout.
In my opinion, it’s almost mandatory for a well functioning software development environment to have developers on call (next to ops people).
Bugs that get you woken up in the middle of the night are fixed in no time. Documentation that is lacking, or spaghetti code, both preventing you from quickly fixing an issue instead of spending hours on it? Fixed the next day.
If the stakes of writing bad code are low, code quality will suffer. No better fix for those low stakes than being on call.
There are absolutely some industries where this makes sense (life-saving health care, national defense, things like Amazon where you literally stop making money when things go down).
90% of developers don't work in these industries or on these types of projects. Hell, I work in healthcare, for a private company, and there is zero reason for any of us to be on call. Patient quality doesn't suffer one iota if our software is down, which it rarely is.
It really sounds like you're advocating that developers be on call to "fix" the "problem" that there's no reason for them to be on call?
What they're saying is that if someone else is oncall, you tend not to deal with the quality issues of the system (it's a negative externality). If the system you own doesn't require any on-call at all, then great. But for those that do (let's say your a web developer for nytimes.com), the code and system owners should be on-call, not a separate ops team. Ideally being on-call is something you barely notice because nothing happens.
> No better fix for those low stakes than being on call.
I prefer a company with a strong tech culture: code review, thinkg long-term (marathons instead of sprints) and design systems with architecture-first on mind (instead of feature-first), automated testing, QA people, don't deploy on Fridays...
The point is "Friday deployments" are not needed (unless we are talking about hotfixes). What's the use case? Can't it wait until Monday? Why are all in such a rush?
I would say that if things are tested appropriately (it's a big if, and not an easy thing to do honestly), when you commit a new feature it should go through the testing process and right into production. If that makes you nervous then there's more testing that needs to be added. Canary deployments with extensive monitoring and automated rollbacks and make this almost entirely hands-off.
For most software systems the goal should be to abstract away the actual act of "deploying" and let the infrastructure handle that on its own.
Because the company makes your rating/pay dependent on BS metrics that drive the wrong behavior, such as merging on Friday so that your cycle time isn't too high. Might mean a paycut. Friday deploy gone wrong? Fix the issue, issue another PR that is approved and also deployed in no time? You're the hero and your metrics are also awesome. Cycle time baby!
Because then you have double the changes to deploy on Monday. And if you push that logic further (no deploys on Thursday, no deploys after 3pm, no deploys +-3 days around holidays, etc.) you end up with very high density deploys, which are riskier.
I don't understand the article on many levels... Is it supposed to be hyperbole?
- Flatulence is not "you rotting from the inside"
- I have friends that have (had) night-shifts back-to-back to a regular shift, and were on-call for for several nights a month. They never failed to answer the phone. They worked as the on-call doctor for a whole station in a clinic, not answering or not listening would have been disastrous! They admitted to be "zombies" but they were functional.
I'm not saying that this doesn't sound like hell, and I seriously am glad I never accepted a job with on-call duties, but this form of hyperbole rubs me the wrong way.
If you've actually never been on call, it's hard to understand the way the body responds without experiencing it really.
Having every other weekend cut short as you always need to be near a PC from 4pm Sunday...
The sheer terror of being woken up from a deep sleep after a long night of work just as you've finally fallen asleep...
Resigning yourself to not sleeping as you finally close out the call, full of adrenaline as you see the first glimmers of light in the sky...
Knowing the problem is the same problem as last week, last month, and that the causes are things outside your control (the 1 staffer in Asia being sick or on holiday, insufficient hardware CTO won't upgrade, bugs in orphaned upstream systems, political turf wars)...
It depends on the environment. I have been in both types.
I have been in an environment for about 5 years where we Devs were on call for 7 days every month and a half-ish. This was with a business that saw traffic 24/7 with weekends and off hours being higher actually. I was called at 2am once and a couple times on a weekend and I didn't even have to open my computer. I was able to instruct the NOC in what a safe reboot procedure (that wasn't in their SOP) would be and it solved the issue. We had automated testing including performance testing with signoff required from the architect if we were too far off the baseline. This was an awesome environment to be working in and being on call was both easy money and if you ever did get a call you knew it was because something really went wrong.
Contrast that with my current environment where you couldn't pay me enough money to make me sign up for the on call rotation. Monitors that fire Pager Duty alerts multie times a day including nights. An absolute nightmare and many if not most of these self heal too.
Yes, 2 weekend & 1 night call in 5 years is absolutely tolerable.
I was at a shop with clever management that fired faster than they backfilled and left the team in such disarray that the replacements started quitting as well. Ended up from a team of 6 down to effectively me & another guy rotating support.
I’m talking about getting woken up once or twice a month at 2-3am to join a live incident bridge & debug systems for 2 hours, while being asked to give an update & ETA every 15 minutes, trying not to wake your wife.. and finishing just in time to see the light in the sky. To then head into the office and deal with more fire drills all day and then get asked why feature releases were behind schedule.
Current org has auto-generating incident channels in slack which seems clever until you are on rotation and there are 5 different incident channels live at any given time with messaging popping in throughout the day as some are resolved in minutes, hours or days.
A lot of this is just management failing the marshmallow test.
Oncall is where you go in, then go home, then go in, then go home, but as you touch your back door, you are called back. You say something regrettable to the caller and then apologise. You’re so tired your almost crying and can’t get your laces done up. You drive in, can’t recall the drive and the you are due back for your day shift in 2 hours.
I know what you mean, how small do they make violins anyway?
In a 24/7/365 cargo operation for a hundred years the port chemists were on call to open their lab after hours and on weekends & holidays each time our ships came in, to operate the office no differently than when you get a vessel during the regular workday. But during regular office hours you definitely need to be there too. These ships are big and they don't always come in every day or on time very often. You get an ETA but very few things can make a vessel get there early, and many different things can delay a vessel, either case being unpredictable. For these call-outs excessive after-hours fees would be added to the nominal invoice, and good incentive pay would trickle down the food chain.
Statistically since there are only 45 office-hours in a 168-hour week, most of the vessels had always incurred call-outs.
After being in my own private practice for a while one day there was a little flood and lost everything I had worked for for 20 years.
I could recover better financially with a business than a home, and I couldn't keep both so I gave up domestic life and started staying in the lab.
Sure I was "homeless" but I had a roof over my head and I worked with what I had remaining, to survive attending to the vessels where I still had capability.
Since I was always there, I made a deal with a small contractor to get right to work whenever they needed it, charging only decent basic fees per vessel regardless of after-hours. They were expected to benefit by not passing on this arrangement to the ultimate clients who would still be charged after-hours premiums, which was most of the time.
It made survival possible and he was only revealing to clients that things were now covered 24/7 without having to wait for people to be called in. Within a few weeks all of the much bigger international operators in our ports had changed over to full 24 hr shift coverage along these lines just to compete with little old me, the homeless CEO.
Thanks for the kind words, but be careful what you ask for :)
Actually still in the same business, it's a profession that will always be with me even if I do something completely different next.
After the flood it was clear I would have to close eventually, my lab and my life as I knew it were ruined, I just kept working in the ruins.
Closed my company seven years ago after my surviving equipment finally wore out, and went to an employer where I had already served them as a contractor since the 90's. Like any other place they needed help so I went right in and added an extra $million in capabilites the first year with just the gear they already had. Haven't stopped yet.
Of course there's staff there 24/7 now and I'm a senior scientist so it's a straight day job for me but compared to what I am known to do it's been like retirement to have all nights, weekends, and holidays off, so I've kept up 50 to 60 hr weeks the whole time, just to keep up the pace I've always had. Get more than twice as much accomplished this way, not declining performance which many people understandably find occurs to them.
Didn't stop overcoming operational problems and preparing for exponential growth, but with a few disasters hitting this company, that was not able to be deployed.
So with 12 more years progress in only the last 6 years that totals more than 80 years of progressive accomplishments in only the last 40 years.
Now with covid, economy is down and a bit of our equipment is idle too often, I'll be "retiring" and selling these enhanced solutions better than my employer can do. So I'll be an entrepreneur again since I've reached the point where their shareholders and myself will be better off with me on the outside after I've built about all the foundation that can be done for now.
Some of the people I've been working with in the labs have said I must work like Elon Musk or something.
Definitely not I said, he's much younger, I've been doing it much longer, he works like me.
That's pretty funny.
In IT it would be more like just getting phonecalls in the middle of the night and operating Service Now tickets, "Mr John Jackson, it is from mainframe services. We need you to file a ticket for team Z to restart service Bob and server Alice." Then go back to bed, and get well paid for it.
Contrast that with my dad. NOC calls at 2am and he has to drive I to work for half an hour. Do a 5 minute fix and drive back.
Since they knew him and that he did good work sometimes they would call him even if he wasn't actually on call. He'd ignore those calls but you still wake up... And one time they actually sent a frickin taxi to our house. So half an hour after we ignore the phone, someone rang the door bell. Taxi from company X.
He never quit that job but he signed up next union representative election and actually got elected too :)
I mean, I get military discipline; in war, you need to be able to rely on your people, they need to do what they are told and not question things because we'll be here for a while, etc.
But this is a guy at a desk on a night shift who seems to be mentally dead most of the time anyway, to the point where he's not able to do his duty (sleep through phones), AND he handles ammunition in this (self described) half dead state.
The military knows the risks of sleep deprivation and half awake personnel.
As far as oncall go, this is an anti-pattern. You should have metrics, alarms and tickets, and oncall should get alerted out of working hours only of high impact issues that require human intervention.
If it is about the requirement to pull all nighters or crazy schedules, some jobs do it in a regular basis (I would know, my sister is a nurse in a hospital).
Do you mean what is the point of being on call? If you really mean the point of the article it's simply for the author to share his experience. From what I know of Korea and the military in general, he has no choice.
Sounds like an absolutely lazy and incompetent person. People need to really think about their job situations in the broader context that sometimes you just have to be there and be ready to work and be available. Not every job is a "creative" 9-5 job that you can work remotely, there are many types of applications where it is required that someone can be there to resolve issues as they come up. And theres nothing wrong with that.
It's the loneliness. Somewhere in this vast organization connected with pipes that no one single person has put together, some element would break. Mercifully the impact ripples and you kinda know about the subsystems to track down the leak and patch it up and alert the domain owners for the next working day. You're the person they're going to call, there's no cavalry, it's you looking at incomprehensible gibberish while the rest of the world is asleep. Even the desk who has called you doesn't get involved, they're done. For me, the idea that it's just me until it's fixed (or degrades further), was the hardest thing to cope with.
I'd pair up my ideal on-call team. It'll cost more, but I'll pay it.
Incidentally pagers are still being carried in SCIFs, where there are no mobiles and sometimes no outside line.