Build Vs. Buy: Strategic Decisions in Procuring Tools for ADAS and AV Development
November 8, 2021
Many autonomous vehicle (AV) engineering organizations face the dilemma of building versus buying when it comes to developer tools. While in-house solutions have the advantage of being purpose-built to address specific business challenges, these solutions require teams to invest significant time and resources for development and maintenance. When working with a proven third-party tool, engineering teams face an upfront purchasing cost. However, this approach allows engineering teams to start building immediately, accelerate productivity, and reduce time to market.
This blog post lays out key considerations when choosing the right tools for advanced driver-assistance systems (ADAS) and AV testing and development. It also shows how third-party commercial tools can enable teams to deploy safe autonomous systems faster. We will address the following questions:
There are six criteria that ADAS and AV development teams typically evaluate when deciding whether to build their own infrastructure tools or buy proven third-party tools instead (Table 1). We will discuss each criterion in detail below.
ADAS and AV companies are racing to bring the world’s safest, fully-autonomous systems to market. For these companies, their competitive advantage lies in their algorithms, and not the tools that enable the development process. In fact, third-party tools can help speed up an ADAS or AV team’s time to market and development cycle. Vendors often assist with the initial integration process. They make a tool available for immediate use upon integration and have a large, dedicated team to manage it full-time. Building an in-house tool of similar quality may take development teams a much longer time because they would need to plan, design, build, test, and integrate a new solution from scratch. On top of that, the development and maintenance of internal tools often risk getting deprioritized in favor of external-facing, revenue-generating products.
As simulation and other ADAS and AV infrastructure tools are difficult to build, expertise is essential to achieving product depth and quality. While ADAS and AV teams may see simulation, data processing, testing, and verification and validation (V&V) as important development processes, they often have limited resources to build and maintain tools that enable each of these areas. In addition, hiring and building out a team of experienced technical experts in each of these domains is difficult in today’s competitive labor market. Third-party vendors are subject-matter experts in building these tools, as they have the time and resources to continuously conduct product research, collect feedback from customers, expand product features, and detect and solve errors.
Proponents of building simulation tools may argue that an in-house team can better determine which features are needed and build exactly those features. Third-party tools, on the other hand, offer a variety of pre-built features that cover a range of different use cases. In reality, third-party tools can also tailor their features to specific customer needs. For example, the Applied Intuition team can ship advanced features like Object Sim’s* intelligent lane-changing actors (Figure 1) in just a matter of weeks.
The decision of building vs. buying a tool doesn’t end at the time of deployment. The ongoing maintenance of a tool is required for sustaining product quality but is often ignored. When purchasing a third-party tool, ADAS and AV teams have a chance to transfer resource-intensive maintenance work to a vendor instead of needing to continually take care of the maintenance work in-house.
No tool is more customizable than an in-house tool built from scratch. However, this flexibility comes with speed and cost trade-offs. An off-the-shelf, high-quality third-party tool is usually sufficiently customizable to successfully accommodate an AV program’s unique ODDs (e.g., construction or autonomous mobile robots) and algorithm development needs.
Beyond customizing their simulation and infrastructure tools to meet their use cases and development needs, ADAS and AV teams need to extend and build their own workflows on top of these tools. While in-house tools are always extensible by the nature of their development, many third-party tools also offer solutions such as plugins to support different workflows. For instance, a plugin in a simulation tool may allow development teams to add their own actor behavior to the simulator.
In-house teams often have concerns about a third-party tool’s ability to integrate with the tools they already use. Luckily, some trusted vendors offer API solutions that make third-party tools compatible with other products in the ecosystem. This allows development teams to integrate a third-party tool with their existing vehicle dynamics models, drive data analysis platform, HIL infrastructure, and requirements management tool.
Although third-party tools have a significant upfront purchasing cost, their overall cost of ownership is often lower than for in-house tools of similar quality. As with every product, a software tool needs to be built and then maintained continuously (e.g., monitoring and fixing bugs, adding or deprecating features). The latter requires significant engineering resources. Development teams are often required to take significant resources away from their core business to keep their in-house tools in a good state.
Many of the concerns that in-house teams might have regarding third-party tools can be alleviated by choosing high-quality, customizable tools that support integrations with other products. For many use cases and development needs, the right third-party tools can benefit ADAS and AV teams by providing them with best-in-class product depth and feature quality while saving them time and engineering resources.
Within the simulation software category, commercial and open-source tools are available. Why would an engineering team choose a paid, commercial tool when they can access an open-source simulator for free? Some common considerations include:
Open-source projects allow development teams to iterate quickly, as teams can directly use and modify the project’s source code to suit their needs. However, since commercial vendors work directly with different customers every day, they are often able to better understand industry-specific problems and build accurate and robust product features such as sensor models (Figure 3).
Commercial tools are easier to certify than open-source tools. Getting their systems certified in compliance with safety standards such as ISO 26262 and SOTIF is a crucial step toward production deployment for all ADAS and AV teams. The certification of open-source tools is more complicated than that of commercial tools since anyone can incorporate and make changes to open-source software.
Commercial simulators and infrastructure tools are often better suited for production deployment. Commercial vendors develop and refine their software in iterative development cycles to solve real-world problems. They constantly receive feedback as customers encounter issues or identify new business needs during real-world testing and development. Open-source software is often built in university research labs and used by general research communities without an industry-driven feedback loop.
An ADAS or AV end-to-end toolchain is a collection of multiple tools that work together to support different steps in the ADAS or AV development cycle. Instead of working with multiple vendors for simulation, data processing, testing, and V&V, an end-to-end toolchain enables development teams to use a collection of tools from the same vendor in an integrated manner.
Working with a provider that offers an end-to-end toolchain does not mean that development teams are unable to also use other vendors’ tools. Here are some reasons why:
Many ADAS and AV teams are hesitant to use an end-to-end toolchain because they already use different tools for different parts of their development cycle and are concerned that an end-to-end toolchain would require them to replace all of their existing tools. Quite the contrary, components that make up end-to-end toolchains are usually modular and compatible with other tools, allowing teams to pick and choose the products they need. Our V&V platform Validation Toolset*, for instance, can run simulations with any simulator, allowing development teams to create abstract scenarios and generate logical scenarios for any ODD (Figure 4). When Validation Toolset is used together with our map editing tool Map Toolset*, it can also leverage best-in-class proprietary features like Applied Intuition map sweeps.
.png)
Another potential concern is the ability to share different types of data across tools and make this data accessible from anywhere. A modular end-to-end toolchain is actually ideal for this purpose, as all of its parts communicate with each other and support common data formats such as OpenX standards. This way, end-to-end toolchains enable development teams to share and access scenarios, requirements, and test cases, whether they are stored on-premise or in the cloud. No matter how many different tools a team is already using, it can be beneficial to utilize an end-to-end toolchain as connective tissue among all other tools.
We hope that this post answered some of the common questions associated with buying third-party tools for ADAS and AV development. At Applied Intuition, we build state-of-the-art, production-ready tools that support a variety of development use cases for ADAS and AV programs of all sizes. To learn more about AApplied Intuition’s end-to-end toolchain and how our products can adapt to your tooling needs, contact our team of ADAS and AV engineers.
*Note: Object Sim was previously known as Simian, Validation Toolset was previously known as Basis, and Map Toolset was previously known as Meridian.
.webp)
.webp)
