Introduction by Seba and Jeevan

My last international flight before COVID was to RSA in San Francisco in February 2020.
There I met Jeevan Singh again. Jeevan attended one of our first Black Hat trainings and has introduced and improved the threat modeling practice within Segment  a lot.

Jeevan wrote one of the best blog posts on practical threat modeling in 2021 on how to have the engineers take the lead in threat modeling.
I am very happy that Jeevan agreed to include it in our newsletter.

Seba, 18 January 2022

So here it is:

Hi all,

I am Jeevan Singh, I run the Application Security program at Segment.
I am very honored to be featured in this iterations of Toreon’s Threat Modeling Insider newsletter.

A year ago we at Segment decided to do something radically different, we wanted the Engineering organization to take over threat modeling responsibilities. We wanted to go down this route for many reasons:

1. Engineers will find more vulnerabilities than Security Engineers because they know their systems better
2. Security won’t be a bottleneck for the Engineering team if the Engineering team does the threat model themselves
3. Since every engineer knows how to threat model, that means every feature we build will have security built into it Before going down this route, I reached out to security experts to see if they agreed with the potential outcomes.

Before COVID hit, I even had an a coffee with Sebastien, to talk more about my plans.
We both agreed that this would be the next evolution of threat modeling.
Segment is very big on giving back to the community, if you have any questions about the article or the open sourced training, please feel free to reach out to me at sstm@segment.com.

I would be happy to answer any of your questions.
Links:
– https://segment.com/blog/redefining-threat-modeling
– https://github.com/segmentio/threat-modeling-training
– sstm@segment.com

Jeevan Saini on March 29th 2021

It was a cold and dark mid-December day in Vancouver; another Staff Security Engineer and I were a part of a developer-led threat modeling session. We couldn’t find a single additional security concern for the developer-led team. The developer leading the project had considered all the options!

When the session was over, the Staff Security Engineer asked: “What have we done? What purpose do we even have any more?” We joked about losing our jobs, but we knew that this is the best kind of problem to have!

Record scratch, freeze frame: You may be wondering, “How did we get here?”

If you’ve never heard of Threat Modeling before, you’re not alone. Threat Modeling can mean very different things to different people. At Segment, threat modeling is the process of systematically identifying assets within a system or feature, brainstorming ways to compromise those assets, and collaboratively designing ways to defend them.

Threat modeling is a critical step in Segment’s Security Development Lifecycle. Before we make a feature from Segment available to the public, the Security Engineering (SecEng) team reviews it. Our job is to reduce the risk of a bad actor exploiting a vulnerability in that feature, which ultimately ensures the safety of our clients’ data.

We love to do threat modeling because we get to fix vulnerabilities before they are coded into the system. It is much cheaper to fix vulnerabilities like this during the design phase. Fixing vulnerabilities in production might mean that a developer has to learn about the vulnerable system from scratch, on-the-fly, and spend valuable time context switching. We also save money because we reduce the number of vulnerabilities discovered in our bug bounty program. And since developers spend less time fixing vulnerabilities, it means they get to focus on building features, not fighting fires.

From a security perspective, it also means fewer security incidents where we have to activate our SIRT and possibly tell our customers we’ve messed up. Everyone wins, as we’ve essentially created a Utopia for developers and Security engineers!

Threat modeling sounds great, but it isn’t without its challenges. As much fun as threat modeling is, there are usually only a handful of folks at an organization that are good at threat modeling. Security teams can often become bottlenecks on threat modeling and reviews due to the ratio between Security Engineers and Engineers in an organization. Some people think threat modeling is too expensive when counted in engineering-hours, given the number of people involved. But given the engineering-hours you save in responding to incidents, this isn’t true.

With Segment growing at a very rapid pace and our feature set growing even quicker, it was time to try something different. Self-serve threat modeling checks all of the right boxes: it removes bottlenecks, and helps deal with an unbalanced ratio of security engineers to developers. If we could meet those requirements, we’d have built ourselves a Segment Threat Modeling Utopia.

In our security Utopia, the Engineering teams perform the threat modeling exercises, and the Security team are optional attendees. Security can then pick and choose which sessions they attend based on interests, priorities, and size of attack surfaces.

The security “goal state” is important to model, because it solves many problems. In our current world, the Security team’s availability becomes a bottleneck, but in our goal state, no engineering team is blocked by a lack of Security personnel, because the engineering teams can run their own threat modeling sessions and the Security team are optional attendees. In our current world, threat modeling can take up additional engineering hours to review, where in the goal state, threat modeling activities are built into the design review, which is already a standard meeting in the development cycle. And while threat modeling is hard to scale when only a handful of people are responsible for it, it scales beautifully when everyone can do it.

Developers know their features better than anyone: they understand what works well, where the gaps are, and where they might run into trouble. In contrast, Security Engineers can only spend a few hours on a specific feature when they assess it, but are expected to provide in-depth security concerns about that feature. It makes a lot more sense for the feature experts to discuss potential security concerns, rather than the folks that are looking at the problem from the outside.

Size matters. The Engineering teams are considerably larger than the Security Engineering team. As a Security Engineer, I know when more folks look at a problem, we discover more potential issues. We believe that with training and practice, coupled with their intimate knowledge of their features’ architectures and codebases, our Engineering teams will be better at threat modeling than our Security team. With this outcome on the horizon, it just makes sense to invest in self-service threat modeling training.

Self-serve threat modeling is an Engineering-wide initiative and it has taken considerable developer time. We have a comprehensive curriculum that goes through several stages to make sure this project is a success.

There are five distinct phases to have Engineering self-serve threat model:

1. Traditional Phase
2. Training Phase
3. Observation Phase
4. Review Phase
5. Security Optional Phase

1. Traditional Phase

This is less a stage, and more “where we were before we started the self-serve threat modeling project.”

All features need a security review during the design phase. Security is responsible for the review and will build the threat model for the feature when appropriate.

2. Training Phase

In order to shift to a self-service model, the first and most important step is to train developers how to perform a threat-modeling exercise. During this stage, you teach developers the threat modeling workflow, strategies for how to find threats, how to prioritize them, and how to decide which threats should be addressed.

(We’ll cover the actual training in more detail below.)

3. Observation Phase

This is where Segment is currently in our journey.

The vast majority of developers have been fully trained, and now teams lead their own threat modeling sessions. The Security team is still (virtually) in the room, and are there to help guide engineers to a successful session.

The people who are leading the session create all of the necessary pre-read material before the threat modeling session begins. We use an established threat-modeling section as part of our standard design document template. The design document provides the appropriate context, but developers are asked to create architectural diagrams, explain what is in-scope and out-of-scope for the meeting, and catalog the assets involved in the project. The assets could be infrastructure, the application system itself, data in the system; anything that’s worth protecting gets listed out as an “asset.”

Security’s role during this phase is to attend the meeting and help guide the conversation whenever required. Security does NOT tell the developers what the vulnerabilities are, but rather guides the team to discover the vulnerabilities themselves. Although a threat model is occurring, this is still a great time to teach developers how to discover vulnerabilities themselves.

There are still situations where developers do not discover important vulnerabilities, even with hints, and that is fine. When they observe this, Security gets involved and helps guide the thought process so the development team learns to discover those vulnerabilities. Again, this is a teaching moment, and everyone involved goes away feeling better about the threat model.

We’ve had many developer-led self-service threat modeling sessions so far, and we know that things are going well when a Staff Security Engineer says “Well, I feel completely useless in this meeting.” 

Threat modeling is not an easy skill to learn, and it takes practice to become good at it. It is our responsibility as Security Engineers to create a culture of learning. As our teams start to discover critical- and high-severity vulnerabilities more consistently, they graduate on to the next phase.

4. Review Phase

In this phase, the Engineering teams are so good at threat modeling that the Security team no longer needs to be in the meeting. Our job is to review the artifacts from the threat modeling session, and validate that we don’t anticipate additional critical- or high-severity vulnerabilities.

You might keep finding medium- and low-priority vulnerabilities. That’s great! You’re encouraged to share them with the team, and explain how you discovered those vulnerabilities.

To go on to the next phase, the Engineering teams must discover critical-, high- and medium-severity vulnerabilities consistently. As an aside, after a few cycles of this, the Security team should start seeing a noticeable decline in bug-bounty and penetration test findings for those teams’ features.

5. Security-Optional Phase

This is the ideal world that we Security Engineers dream about. Once teams are at this stage, it is optional for Security to even review threat models.

We believe it may take us a year or two to get to this phase, but we know that in the end, it will have been worth the effort.

The team at Segment reviewed many options for training. We ultimately decided that the best way to proceed would be to build the training from scratch. The training itself had to be Segment-relevant and specific to our cultural norms.

The training was delivered to the entire Engineering organization, and anyone else who wanted to sign up: this included product managers, tech writers, and members of the general IT staff. This meant that the training had to be engaging for all levels of skill and experience. Classes were made up of a mix of folks that were junior, mid-career, and very senior. The training had to also be relevant to people who have never done threat modeling before, as well as to others who are security champions.

We broke the training down into three parts, which we delivered in three-week intervals:

• Part 1 – Introduction to Threat Modeling
• Part 2 – Foundations of Threat Modeling
• Part 3 – Live Threat Modeling

The goal of the first training session is to introduce the concept of threat modeling, teach the workflow, and show developers the tools Security uses to find vulnerabilities.

We introduced the concept of threat modeling to the developers by making parallels with how you might think about personal safety. This is much more relatable than diving right into software vulnerability concepts.

Because we started this process in April of 2020, in the very first exercise we talked about what our grocery shopping habits were like before, and then after the pandemic. This exercise got us talking about risks, and how they evaluate and decide their tolerance for risks. We work through several non-software risk exercises to help make threat modeling relatable.

Later in the session we introduce STRIDE to the class, and do a deep-dive and talk about each vulnerability class. We then talk about real-life vulnerabilities and how they impact applications. We do some light exercises to help the class get used to asking the right security questions.

The goal of the second training is to get the developer’s hands dirty with exercises to solidify their knowledge.

We introduce the concepts of “assets,” and the value of a diversity of viewpoints in the threat modeling sessions. It’s important that as we go through a threat modeling exercise, everyone in the class understands the assets in the system we’re assessing. It’s also important that all members of the Engineering teams get involved in the threat modeling exercises. Each member of the team has a slightly different background and experience, and so they each see the system in a different way. When finding potential vulnerabilities, everyone’s voice is important.

As a part of this training, we have a lengthy discussion about a fake Segment feature: a credit-card billing system. The class is broken up into small teams, and each team reads a design document and talks through it to discover potential vulnerabilities in the system. At the end of the evaluation, everyone comes back together, and we discuss and compare the top three threats from each group. The class learns how to read a design document with a focus on security, how to identify security vulnerabilities, and how to prioritize risks.

Fake Segment application we learned upon

Performing an end-to-end threat modeling session with the team cements the concepts that we discussed.

In the final training, we do an actual threat modeling exercise. We discuss the feature we are threat modeling, and review the architectural diagrams. We then list the assets in the system, and discuss what is in, and out of scope for the exercise. Next, we spend some time finding potential vulnerabilities, and discuss how to prioritize them. Finally, we discuss when the vulnerabilities we discovered should be remediated, based on severity and priority.

We’ve learned a LOT while rolling out our self-serve threat modeling program. Here’s some advice if you’re considering doing something similar.

Fail fast

One of our initial learnings was to fail fast and iterate on the training contents. There were two teams willing to sit through the very first training iterations.

Immediately after the initial sessions, I held focus groups to get attendees’ feedback and better understand how to improve the training. It had an immediate impact – we adjusted the length of our sessions, and changed what concepts we introduced and when. With feedback from the first team, we modified our approach, and ran the training for the second team. With the feedback and suggestions from those two initial teams, we solidified the training content and approach. We were then ready to roll it out for the rest of the organization.

Trust your counterparts

I was initially hesitant, but I have been totally impressed by our developers. They learn quickly and are interested and engaged in the training. When you create a safe space for them to learn and make mistakes, you’ll get to your self-service goals much faster. Trust that they can figure out how to uncover vulnerabilities, and coach the team on how to find them.

Security culture improvements

As I rolled out the threat modeling training, I noticed that people in the classes really enjoyed the training. This led to more people reaching out to me about the concerns that they have about vulnerabilities within their system, allowing me to give them proactive advice and support. In the past, developers only contacted me sometimes, but now they reach out far more often, and in greater numbers.

The simplicity in the training made the scary topic of “security” accessible to everyone, which makes them less afraid to ask us security-related questions. Many folks have even asked about more security learning resources, and how to join application security related meetups.

With this project, we identified and created security champions at Segment. There are folks that are interested in learning about, security, and who often impress us with their security knowledge. We have only seen benefits from this program.

Every company is unique! I recommend that you start somewhere, and adjust the training to fit your culture.

We at Segment don’t want you to start from scratch! Today, we’re open-sourcing Segment’s Threat Modeling training content for you to adapt and modify. You can find the links below.

• Introduction to Threat Modeling
• Foundations of Threat Modeling
• Live Threat Modeling

We hope that you can use these training modules, give feedback on how we can make it better, and share what you find with the community.

We have a strong sense of community here at Segment and in addition to sharing our training slides and this blog post, we have an email address so you can submit your questions and thoughts about the program. I am always interested in hearing feedback, good or bad!

Email address: sstm@segment.com

So, we’re on our way to Utopia, and things couldn’t look brighter. Time to rest on our laurels, right? Wrong. We’re constantly thinking about how we could improve our threat modeling training, and build sustainable systems. We’ve trained all of our current developers, but what about new Segmenters in the future? We’re thinking about how best to incorporate this training into onboarding for new hires.

We’re also thinking about how to keep this knowledge fresh. Maybe you took the training months ago, but haven’t had to use it since, and now you have this huge project – yipee! Threat modeling refresher courses to the rescue! Finally, for the people who really enjoyed the training and want to learn more, or who really want to deep dive on advanced strategies for large and complex systems, we’re working on a Threat Modeling “201” class.

If creating or updating a threat model is essential to your line of work, then this course is for you. Our threat modeling practitioner training targets software developers, architects, product managers, incident responders, and security professionals.