So last week I wrote a quick cheat-sheet for how to conduct an engineering manager interview. I used the words “engineering leader” interchangeably with “engineering manager” in the article and I did it on purpose. There is another kind of engineering leader to consider and today I am going to write about it.
Imagine having a cool startup idea—not like your regular Thursday “this is a cool idea” idea—that makes other people nod and stare into the distance thinking “Wow! That is a cool startup idea!”. You take your idea and you have enough social capital to get a meeting with a serious investor. The meeting is going really well. He actually puts down his phone to listen to you. He is engaged and has interesting questions. He is leaning in as he can see a new product or service that can scale and the two of you really like each other. The excitement is palpable. At the end of the presentation he asks two really solid questions that lead you to believe that this is going to be discussed on partner-Monday at his firm next week.
Then he wants you to flip back to the team slide for a moment. He leans back ever so slightly and asks “Who is going to be your CTO?”. There is a momentary pause before you say “I do not have one yet” and before you can add anything else he has reached for his phone and is getting ready for his next meeting.
I hope this never happens to you. If it was about to happen to you, count yourself lucky you are here. Today I am going to give you a primer on how to hire your CTO, or architect.
A CTO means many different things to many different people, especially to investors and to company leaders at various points in a company’s life cycle.
For most startups, a CTO is “the most senior coder we could afford at the time who is willing to create interfaces and write documentation”.
For companies at a later stage, a CTO is “the person who can negotiate the product roadmap between keeping legacy code working and growing, and adding new features at a rate that keeps forward momentum going”.
For a public company, a CTO is “a talented engineering leader who can communicate with the board, the rest of the C-staff, set technical direction, and respond to technical and governmental policies that impact the day-to-day operations of the business”.
Finally for struggling companies, a CTO is “the engineer with the most amount of tribal knowledge who we promoted to CTO so they don’t quit, leaving us with a site that no one understands”.
When you are hiring your CTO you should understand what you need and why.
Here are some of the common responsibilities for a CTO.
APIs and Interfaces
Creating systems that are usable, re-usable, extensible, and maintainable is really hard. There are so many shortcuts you can take to get in front of customers that each introduce their own flavors of pain, from increased cost to decreased performance at scale. Navigating those waters and making a system that you can grow with and give to other people to use is a challenge.
Documentation
Whether it is something for the board, the customer, the team, or even yourself, you should get a CTO who can present things in writing. Bonus points if they are good speakers too but at the very least they should give you a usable diagram with the cloud bubble and the database cluster in it that represents what it is that you are using and/or building.
Technology Platform Choices
Are you on AWS? Are you using Go, Java, or PHP? What operating system is needed for development? What tools are needed for deployment? There are many factors that play into choosing your technology stacks and this should be one of the duties of your CTO. There are many people who will advocate for a particular language because they are familiar with it. Sometimes it is the wrong choice for the business.
Buy vs Build Decisions
Similar to the languages and technologies being used, your CTO should be able to look at a product in the marketplace and the size and budget for the engineering team and make a decision whether or not to spend X dollars or Y months of time to build something. At the very least they should be able to present realistic arguments for why it is important to the business or what risks are entailed by going one way or the other.
Operations and Infrastructure
Choosing where to host your applications or what devices to target comes with risks. If the marketplace being addressed is all android phones, or all mobile phones, what features are not supported on older handsets? What are the different problems created by using AWS vs Google Cloud? What is the cost of your partnership there? What kinds of lock-in to your partner are you accepting by using them and their various flavors of exclusive services?
Research and Development
This is the best and the worst part of a CTO role—managing the business risk. Are there new technologies being created or developed? Does some complicated math need to be used to create a new library? Are there new hardware or software packages to be looked at to help find the product-market fit for your business? Tinkering with technology is a wonderful pastime. It is easy to get lost in solving academic problems and coming up with cool new systems and tools.
Technical Roadmap
As you build your business, what are some of the legacy decisions that need to be revisited? Are there old systems you have purchased that are past their end-of-life? Do you have older systems that are slowing down new feature development? Did something change in the usage pattern for your business such that previous assumptions need to be reconsidered for what storage to use?
Due Diligence
If you are looking at using technology, acquiring talent, or flat out buying whole businesses, who is looking under the hood? Does the data they use to facilitate a business transaction hold up under scrutiny? Is their software homegrown or cobbled together from other systems? Does it have security risks or is it functioning poorly under heavy loads? Usually half of the people who are open to selling their business have peaked in some fashion and are trying to make the prospect of acquiring their business or assets look more attractive than it really is. Finding that inflection point where things are about to slow down is key in order to make sure that you do not overspend on an asset that is not growing or scaling as projected.
Intellectual Property
What happens if your data warehouse gets hit by a meteor? How fast can you get back to servicing customers if the servers are suddenly turned off? How are you protecting your intellectual property from theft or replication? Are any of your ideas patentable? What security measures are put in place to safeguard your customer’s trust?
Compliance
Do you have GDPR compliance for Europe? Do you conform to California’s new data privacy laws? If you are giving away chances to win things, are you falling into gambling or lottery laws? Are you collecting PII from your customers? Are you looking to increase your PCI compliance for the purposes of deepening your payment relationship with your customers?
It seems like a large list of things to cover and it is hard to find people who have done all of these things. Here are some questions you could ask while you are hiring your CTO.
What is the difference to you between a CTO and a VP of Engineering?
I am deliberately not going to answer this question. You absolutely should ask the candidate this question. I have seen a VPE act as a CTO and a CTO act as a VPE. You will likely want to hire one and assume they are going to do the work of both because those gosh darned engineering leaders are expensive. It is rough to hire one who is also going to be doing the job of the other. They have very distinct responsibilities in the long term and you should be comfortable with that. If you do not know the difference between the two, this will be an educational process for you too. If this fills you with an overwhelming existential dread then reach out to me privately and I will give you a more direct answer.
What language do you like to use when starting a new project and why?
This gets back to the personal choice question above. There are times when you choose a language you love and there are times you choose a language that other people will love more. Do you want to hire lots of people fresh out of school? You might want to look at what modern languages they learn. Are you building mobile games? You are likely to want to use Unity or Unreal Engine for your client to make the cost of supporting Android and iOS more bearable. Pay close attention to what languages they love and why. I have built stuff in C#, Java, PHP, Go, C++, and a myriad of other languages—each for different reasons at different points in my career.
When you are joining a team with a project in flight what are the first things you want to know about it?
This is a little bit of a softball question. Ultimately you want to know how many times the candidate has joined a team mid-flight. If they do not have questions here it means they may not have had to join an existing project, and usually not one in crisis. I have my own battery of questions that I love to ask people when I join an organization.
Why would you use document storage vs relational database storage? Which flavors of each do you prefer?
Oh the joy of discussing NoSQL and SQL! It is such a fun conversation to have with people and everyone has their own opinion on it. There are no right or wrong answers here. Okay that is not entirely true. If you are dealing with customer payment transactions, you probably want to store stuff in a relational database. There are whole companies and communities that exist to argue the pros and cons of what kind of storage to use and when.
Design one of the following connected systems for me at a high level using a whiteboard: Authentication, Payment Flow, Chat, or Presence.
If you are building online services, your CTO had better have built one of these in the past. If you are building client only technologies there are similar systems you can whiteboard out. You should not necessarily expect someone to get a perfectly functional solution in twenty minutes on a whiteboard, but they should cover all the relevant storage pieces, data transactions, and also speak to what the requirements are. I have been asked in broad terms how to set up all of these things at one point or another in my career.
What is the hardest technical problem you have ever had to solve?
This is another softball question. The hardest problem I ever had to solve happened so early in my career and is so arcane that is essentially irrelevant in today’s world. However it was an interesting business problem and an impossible hardware problem, which is why it is the story I tell people when asked. A lot of people really struggle with this question because it is so vague. It is by design because I want to look at the shape of the answer. Did it scratch their curiosity itch? Did it save a company money? Did they unlock a new marketplace? What was the outcome of their solution?
What is something you wish you knew when you started your career?
Most engineering projects are a collection of hastily made decisions and huge enablers of massive hindsight. It is good to listen to things people have learned along the way. I often want to know this from someone so I can figure out what their biggest regrettable moment was and how they have grown from it. I know the two or three things I would tell younger John Szeder. I suppose it is actually a pretty big list but a couple things really stand out for me.
How do you learn a new language?
This is an important one for me. I struggle with recruiting leadership that does not possess professional curiosity and a love of learning. I hope everyone is trying new things all the time— that is how growth and abundance happen. If you are not looking at new technologies or trying out new ideas or tools, you are at risk of suffering skillset obsolescence in a decade or possibly less.
Tell me about your current successor.
Finally, and most importantly to me, is the need to know you are preparing for the day that something happens—either a promotion, or a change in career direction. Can you leave the organization in capable hands? If you are grooming a successor and comfortable talking about it, that is a measure of confidence and ability I respect. I always try to find one person I am grooming to fill my shoes at any given moment, sometimes multiple people. There are risks and challenges with having multiple successors—they all want their moment in the sun. It is possible you can set a group of talented peers at each other’s throats if somehow they think that succession is a zero-sum game.
That is a lot of questions and each tells me something interesting about a candidate.
One of the biggest challenges with this whole process is that you probably do not have all of the tools to properly validate all of the things your candidate is telling you. Here is a set of terms to look for when you are having conversations with a potential CTO.
Trade-offs
This is the biggest thing I want to know about a CTO or an architect. Are they looking at the Total Cost of Ownership (TCO) of something? Did they look at all the vendors? Did they measure the time versus the cost of different partners and tools? Every choice has pluses and minuses. The ability to measure the trade-offs before making an informed decision is really valuable to me.
Technical debt
Everyone hates this word. Your CTO should have some experience with managing technical debt. Every large scale business has it and everyone has a method for mitigating or reducing it. If you have a CTO candidate who cannot speak to their experience in managing technical debt, I would be very worried.
Refactoring
Another dirty word. Almost every software system is built out of a design that existed before a product-market fit was found. When the system is live it often requires changes that do not fit within the existing structure.
Refactoring is an often maligned but very healthy part of maintaining software. If you are not updating your systems regularly then you risk having a bloated pile of unmanageable software within the next four months to ten years.
Deprecating
Turning off a system is sometimes impossible, especially in a SAAS world. How are you managing old systems? How are you mitigating the risk of old deployments? Managing the lifecycle of old systems and sunsetting older APIs and servers is important. The more you maintain full backwards compatibility on systems, the more drag you are creating on forward momentum. There is always a moment where you sit down and say “this is where we draw the line”, whether it is every week, every quarter, or once annually. Making a software system deprecated means it is still live and not fully supported. It is a key step to deleting or obsoleting a system. I consider it a deep-orange flag if someone turns off a system and says “it is deprecated”. It is an important thing to know and I have caught people saying this incorrectly.
Recovery
How long does it take you to relaunch your service after a catastrophic event? How long does it take a developer to get a fresh laptop after theirs gets filled with coffee and rendered unusable? The ability to recover things and get going quickly is very important in the modern day SAAS world. Whether it is time to push a build from a fresh laptop, to how long it takes to service a customer on a freshly spun-up cloud instance, this is an important thing to talk about.
Learnings
Another thing I want to hear in response to my questions is what they learned along the way. “Profit from your successes, learn from your failures” is a nice mantra. If you are not learning things from your mistakes, you are likely to repeat them. I need to find evidence of learning from failure through the interview conversation.
Compliance
This is an important one for legislated industries or public companies. Being able to navigate the compliance processes for your business is hard and takes a specialized skill set. You need to be able to gather requirements, negotiate with stakeholders, demonstrate functionality, and triage failures. Every part of that process requires care, especially as you add zeros to the size of the business.
That should be a good framework to help you evaluate potential technical leaders, whether CTO or senior architect, for your company.
My final thought on this matter is that this is something you can get outside help with. There are lots of people out there who can assist in reviewing the qualifications of a potential CTO and give you the upside and downside of a respective candidate. You should always balance getting a formal assistant in reviewing their qualifications against your own gut feeling. If you get a strong signal from an external candidate review and you have reservations or you get a lukewarm assessment but you really like the candidate, you should understand that you will be living with this person day in and day out as you build your company together. Ultimately that should be the filter that you apply to this whole process.
And here we are at the very end of another article! I hope you enjoyed it, or learned something from it. Maybe both? I value your responses and questions every time I write one of these. Please keep the comments and feedback coming!
