This is the first article on the subject and the idea is to take different point of views. Today the inspiration comes from Bradford Cross
Bradford Cross has a very strong experience and expertise of leading engineering teams. He wrote this article in July this year based on practices in companies like Google, Facebook or Twitter. There is also a video version of this article.
It is not presented like this but after trying to do a summary, there is some kind of “team lifecycle” based on his ideas.
1) “3 keys to happiness (Mission, Winning and Growth)”
Always start your journey with the why ! And as a manager, it is normal to explain how your teammates will add points to the scoreboard. And of course, every individual needs to grow and builds a career path.
It is not that obvious. Many engineering teams are in their routine, not so sure why they are doing it. There are indicators but mostly it explains more why the team is failing (number of incidents for example) and talents are in “prison break” mode.
For data engineering teams and because data is a cool subject, it looks like you do not need it. It is really not and because of the high turnover in the data engineering field, this is the difference between why do talents stay or not in your team.
2) “Scale with talent magnitude not headcount”… “A well balanced senior, mid, junior team creates culture of growth and fun”
To quote him “I see a lot is hiring too fast based on headcount targets instead of productivity targets… Too bottom heavy and your senior engineers spend all their time mentoring instead of building… Too top heavy and there’s not enough space for leaders and people get frustrated with one another or check out. »
Again, any manager would agree on that but in the real life, it more looks like having a lot of Headcount is a measure of your success and we do not want juniors because we will loose time. Be Peter Graves in Mission : impossible who is selecting carefully a limited number of agents.
If you are looking Data engineers with 10 years experience in everything, you are sure to fail. Talent in Data is not linked to the number of years experience and the « newbie » who has never heard of data pipelines before the interview could become your top gun in just 6 months (true experience).
3) Key roles of the modern engineering management
« For the past decade, there’s been a lot of work on separating the engineering manager from the tech lead role as opposed to what has previously been called the ‘TLM’ or tech lead manager, which holds both responsibilities and usually does one or both of them poorly as a result.”
There is no perfect world and having two roles could help but is not the ultimate solution. At the beginning of your lifecycle, you need mainly a technical lead and then the Engineering management part will become more important because people in the team have now different needs.
In 2020, in the middle of the cloud revolution for data, tech lead is without no doubt the most important role.
4) Why scrum is one of the worst mistakes in the history of tech
It is more explicit in the video but for Bradford Cross, “Basic agile planning is good but this was about self promotion to sell trivial certification… Oversold as a management methodology… confusion of roles, org design and management practices… Product owner = PM… Scrum master = tiny part of EM responsibilities”. He believes in “Goals and empowerment vs working bees and centralized queue”.
He is right that Engineering can’t be just here to take the next ticket and just do it without taking the chance to do something bigger and that could be re-used. On the other side, Scrum and agile methodology is the right way to work with the business. A compromise has to be found between business priorities and building the tech asset for the company.
There is a high probability that the data use case you are working on will be delivered using scrum way of working. It is important to valorise that we have to deliver the use case but also a data platform that will make us more productive for the next use case.
5) Technical aggressiveness in a practical way
“not seeing and investing enough where the core IP really is, or in overengineering something that should have been a quick hack. It’s very common that teams burn a lot of calories overengineering things very badly, apply technical aggressiveness to trying new framework flavors of the day, or laying out a complex architecture with many layers of useless indirection, but of course that’s ‘future proof’ in some evil ways”
Especially in the data engineering field, it is very fast to be « quick and dirty » (most of the time not so quick but very dirty) or « building a cathedral mode, we will come back to you next year ». Being a Practicing « Pragmatic » will help to take decisions and in world where managers are said to be useless, this is clearly when it helps to have good ones.
There is new « Data tech » nearly every day : the new database, the new dataviz tool, the new AI software. It is clearly a jungle and there is no map or compass. You have to build your own ones.
6) Engineering teams function well with less than 10 people
« Teams can function well at less than 5-10 people. There’s no need for subteams or team splits, clear roles and responsibilities like engineering manager and tech lead, and it’s easy to have team alignment on the right way to do things… Engineering teams fall apart as they scale past 10 people.. Engineering teams also often lower the bar as they scale instead of raising it »
Most of the managers will disagree and tell you they are good enough to manage much more than 10 people. I do see this limit too and it has something to do with the « double pizza team » idea. And it is also why every slot in the team is so important. Having met great data engineering teams, there all have in common that the number of people is very small compare to what they deliver. Having 100 FTE working on data engineering as a goal or KPI is a mistake (true story).
7) A culture of distributed ownership sets up winning and growth (forget about the hero mode).
« Managers seek to distribute work as widely as possible and aligned with individual skills for highest impact and focus on areas of growth — optimizing team results based on individual impact and growth. They empower teams with clear goals, and separate direction from execution to allow for ‘managing to the level’ — enough direction for the right level of each person to own their own execution and as much of the direction as they can while delivering success. »
Too often, you have engineering teams that look like the Avengers or the Justice League. You have so-called super heroes but with very different super powers and if Superman is not here, you are immediately in trouble because there is only one Superman. And real life is not a movie. You will have people in the team as the team becomes bigger with very very limited impact because they do not have super power and they can’t fly. This is where you are really « building the team » because you have to think about the impact and growth of every individual in your team.
In data engineering team, if you have only one person who is able to fix your data pipeline when it is broken and 19 others who cannot do it, this is the sign it is time you have missed something.
Conclusion : why it is a loop ?
I see it as a loop because when you have reached #7 step, it is very important to come back to Mission, Winning and growth and what makes a team happy. It means sometimes that you have to totally reboot the team and start a new cycle.
Yet Another Blog on Data 