The Need for Test Data

Last week at the RSA Conference, I spoke to several vendors about their challenges offering products and services in the security arena. One mentioned a problem I had not heard before, but which made sense to me. The same topic will likely resonate with security researchers, academics, and developers.

The vendor said that his company needed access to large amounts of realistic computing evidence to test and refine their product and service. For example, if a vendor develops software that inspects network traffic, it’s important to have realistic network traffic on hand. The same is true of software that works on the endpoint, or on application logs.

Nothing in the lab is quite the same as what one finds in the wild. If vendors create products that work well in the lab but fail in production, no one wins. The same is true for those who conduct research, either as coders or academics.

When I asked vendors about their challenges, I was looking for issues that might meet the criteria of Allan Friedman’s new project, as reported in the Federal Register: Stakeholder Engagement on Cybersecurity in the Digital Ecosystem. Allan’s work at the Department of Commerce seeks “substantive cybersecurity issues that affect the digital ecosystem and digital economic growth where broad consensus, coordinated action, and the development of best practices could substantially improve security for organizations and consumers.”

I don’t know if “realistic computing evidence” counts, but perhaps others have ideas that are helpful?