A friend forwarded me an interesting link last week. Apparently the State of New York has taken a legal stand on the prohibition of publishing benchmarks in some End User License Agreements (EULA).

Attorney General Andrew Cuomo today announced that the State has reached a settlement with Blue Coat Systems, Inc. over the company’s use of end user license agreements that claimed to prevent consumers from testing and criticizing Blue Coat products. Blue Coat is a leading developer of enterprise-level security hardware and software products.

“Companies don’t have the right to pick and choose who can test their products, nor can they legally silence criticism,” Cuomo said. “Companies that believe in their products should welcome the free exchange of information and opinions — not threaten legal action against critics. In this case, many customers did not even have an opportunity to soldier through a lengthy license agreement — which threatened their rights — until buying the product. That is unacceptable.”

A column in PC World last week commented on this decision and another one against T-Mobile –

Another decision that should be of particular concern to PC Magazine readers was handed down recently in New York State. Blue Coat Systems had decided it could contractually forbid customers from criticizing its products. It did so by inserting an “antibenchmarking” clause into its EULA. Seems it didn’t want any customers comparing its proxy servers with the competition’s. The state attorney general’s office filed suit, at which point Blue Coat quickly settled, paid a small fine, and removed the clause from its contracts. Amen.

I wonder what kind of EULAs we have violated by benchmarking products that come into PC Magazine Labs. Turns out we published a review of VMware Workstation without the company’s approval, as required in its EULA. Yeah, sorry about that, guys.

To be fair, VMWare does permit the publishing of benchmarks. You just need their permission first -

“You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review.”

According to the benchmarking PDF on VMWare’s website, you have to get your test plan approved by VMWare prior to testing. Your results must also be approved by VMWare prior to publishing.

It will be interesting to see if these rulings against T-Mobile and Blue Coat have an effect on the policy by VMWare.