I oversee the Shared Services team that provides engineering services to our 2000+ person Engineering organization. Of the many services that we provide, one of the most important is our security service, which consists of a centralized team of security experts that are tasked with ensuring Citrix development teams follow industry standard security best practices for every product under development. We apply these practices consistently across our organization through the framework of a Secure Development Lifecycle (SDL), which defines the security phases and activities for each stage of the development process and include such things as static analysis, threat modeling, design reviews, tooling, fuzzing automation, and pen testing.
We must treat our activities as if they were a business, and that means delivering services to our internal customers (SW development teams), with a quality and timeliness that is competitive with alternative approaches. The benefits of centralizing a service can be completely negated if the shared service in question, becomes a bottleneck that reduces an engineering team’s product velocity. Nowhere is this truer today, than in cloud computing environments, where DevOps and CI/CD development practices are driving shorter and shorter development cycles.
This post shares some of the approaches we have taken at Citrix to design and develop products that meet the highest standards of security while delivering those products and features with great quality and velocity.
We started with breaking the typical “right-shift” model that security teams traditionally faced. In waterfall development models, security teams were often siloed from the rest of the organization. Large projects might go on for many months and as the release date approached, the security team would finally be asked to step in and do the appropriate security reviews. This would never work effectively. More importantly, delayed design reviews might expose serious architectural flaws leading to security vulnerabilities that might only be solved through a complete product redesign.
Organizationally, we took the approach of deploying our security team in a manner that enabled us to support our development colleagues as efficiently as possible. Like many large companies today, we have a distributed product development team spread across multiple geos, including the USA, UK, India, and China. We deliberately chose an organizational model that provides both local and tightly integrated security team members at each site, so that every development team wherever they are situated, has immediate access to high quality security expertise in the same time zone and location. This collaborative partnership helps engineering teams go faster, and our security teams become enablers, rather than blockers that get in the way of work being done.
Next, we ensured that security is integral to every step in the product development process. We embody the idea that to deliver an effective security function at DevOps speeds, we must start designing and validating security as early in the development life-cycle as possible. At Citrix, we refer to this process as “left-shifting”. The secret to this success is a strong partnership with our developer colleagues that’s built around empowering every developer to take greater responsibility over the security of the software they are writing. If this is done well, adoption of security best practices will scale to any size organization, even if the centralized team of core security experts remains relatively small.
At the core of “left-shifting” is training. At Citrix, comprehensive on-line security courses are offered to every engineer through our Security Ninja Black Belt training program. As engineers advance in their training, they reach different levels of expertise, recognized by receiving a white belt, yellow belt, green belt, blue belt, purple belt, red belt and finally, after at least 80 hours of training, a black belt. Further encouragement for completing the training is backed up by offering a monetary award for all engineers on reaching their black belt status!
Supplementing this online Ninja training, our security team offers deep-dive classroom training at each of our development sites on such topics as web application security, threat modelling, fuzzing, applied cryptography, and mobile security.
Even though engineers are now armed with the knowledge to write rock-solid, secure code, the code still needs to be validated and like all successful cloud companies, the heart of testing at Citrix is accomplished through extensive use of test automation practices. While we’ve always expected development engineers to write automated unit and component level tests to validate the software that they create, we are now enabling them to include tests specifically focused on validating the security of the code they are writing. Development guides have been created for each type of environment that our development teams worked in. Whether it is Web, Android or iOS, each guide covers the many types of security vulnerabilities common to each environment and how to detect them through software test automation. Code snippets are included to accelerate test creation and the result is a continually expanding battery of automated tests written by developers that are executed on every code commit and product build cycle.
We further supplement our security practices through the consistent use of automated static analysis tools that are designed to analyze an application’s source, bytecode, or binary code to find security vulnerabilities. These tools run daily and can find security flaws in source code such as potential buffer overflow and SQL injection conditions automatically.
Our security team structure coupled with additional practices such as threat modeling, pen testing, automated vulnerability scanning and fuzzing, ensure that security practices at Citrix are front and center of all development activities while also supporting our development teams to deliver product at DevOps speeds.
Please look out for my next blog post, which will share best practices on setting up effective bug bounty programs.
