Seek proof of a product’s value

Posted on September 18, 2014 in Dairy Performance
By Dr. Al Schultz
Whether it is a feed additive, a management practice, an equipment purchase or a whole myriad of inputs for your dairy, you are faced with choices.  But which of the choices is the best investment?  How do you know?

It is jokingly said that if you feed multiple products that all promise to improve milk production by 5 pounds, your cows should average 150 pounds of milk per day.  Of course, we know this isn’t true, but how can we make decisions that offer us the best chance of financial return?  How do you know if two products used together are additive in their response or if each negates the effect of the other?

Nothing is guaranteed.  Even well-researched products may not perform to expectations every single time, but you can improve your bottom line by purchasing those inputs that have the best record of success.

Let’s discuss the five levels of proof that a product will perform to expectations.  For purposes of this discussion, we will talk in terms of a feed product choice, but remember this same approach applies more broadly to all purchased inputs and management practices.

The coffee shop story
Anecdotal evidence is the easiest evidence to obtain and the most common driver of purchasing decisions.  Usually, this evidence is in the form of a testimonial.  Your neighbor who uses a product and likes it is inclined to tell others about the experience.  This may be a simple recommendation to a friend or an advertisement with a quote that reaches a much bigger audience.  To make it even better, a company may use someone highly esteemed in the industry as its spokesperson.

In the case of anecdotal evidence, no data is collected, but a relationship seems to exist between the product and desired results.  Often, this is “practical experience” speaking and could be some of the best advice that you get.  At other times, it may be just timing or misinterpretation of the results that made it look successful.  Your circumstances and your farm may be completely different.  Don’t ignore this “advice,” but don’t bet the farm on it either.

Before/after comparison
The second level of the proof pyramid is an uncontrolled study.  In an uncontrolled study, data is collected showing a before-and-after scenario.

Often, this is in the form of commonly kept records.  A change was made, and the parameter in question also changed.  It is assumed that the change and the result have a “cause-and-effect” relationship.  In reality, other factors may have changed at the same time and affected the outcome.

For example, your fat test is depressed to near all-time low levels and you try a new product.  Suppose that the fat test improved.  Is it because of the product addition?  Or did other feed adjustments or a change in the weather have an impact?

Quite often the challenge with before-and-after comparisons is other parameters change at the same time and mask and effect of the product in question.  This doesn’t mean that you should totally discard any data that was collected, but understand the limitations.

Remove outside influence
This third level of proof tries to take out the influence of other factors.  Unlike the uncontrolled study, in this case all variables are held constant, and only the item in question changes.  For example, in a controlled study, the product may be tested on one pen of cows while another pen of similar cows receiving all the same other inputs does not receive the product.  In other words, the only difference between the two pens is the addition of this one product.

But a controlled study is not foolproof.  Is the difference in results a true difference or just due to chance?  Without statistics, you aren’t sure.  A controlled study does provide better evidence than anecdotal evidence or an uncontrolled study, but without statistics it still may be misleading.

Real or random
This fourth level of proof adds statistics to the controlled study.  Thus, you can judge whether the observed differences in a trial are real or just due to random chance.  Your confidence level in the results goes up with a statistical difference.  Often, results are shown with a measure of probability.  For example, a statistical difference with 95 percent probability means that 95 out of 100 you would expect to see a difference due to the effects of the product under consideration.  That is reassuring!  Nothing is absolutely 100 percent, but if you can be right on a product selection 95 percent of the time, that is pretty good.  If the decision is a high-dollar decision, you may want several trials with significant differences to be more positive about the rewards.

Removing some bias
Peer-reviewed published studies are at the pinnacle of the pyramid of proof.  Peer review is a process used in publication of manuscripts and research results.  In our industry, the most common source of published peer-reviewed studies is the Journal of Dairy Science.

The author’s research is scrutinized by other experts knowledgeable about the subject matter.  Showing the work to others before publication improves the likelihood that any weakness in the design of the experiment or interpretation of the results is corrected before it becomes accessible to a wider audience.  This is the highest quality of information upon which to make a decision.

Peer-reviewed scientific information is least affected by the bias (or agenda) of the person collecting and summarizing the data.  It is also the most expensive.  Hence, often you will not have the luxury of this level of investigation.

Ask for data
A conclusion based on personal observation and practical experience may be just as correct as the results published in a peer-reviewed scientific journal.  But, over the long haul, you will be more successful if the basis of decisions is supported by the higher levels of proof.

When presented with a purchase decision, ask to see the data behind the product.  If none is provided, be a little cautious about the purchase decision.  Don’t pass up a great product just because it doesn’t have peer-reviewed data behind it; but, on the other hand, be a little more cautious because the results are less defined for you.

Without the assurance of higher levels of proof, you will want a higher expected return on your investment because of the elevated uncertainty.

This artice was originally written for and published by Hoard’s Dairyman in June 2014.

About the author:  Dr. Al Schultz is vice president of technical services at Vita Plus and has been an employee owner for more than 35 years.  Schultz grew up in eastern Wisconsin on a registered Holstein dairy farm. One of his “claims to fame” is the family showing of the Grand Champion Holstein cow at the 1968 World Dairy Expo. He has earned all of his degrees in dairy science from the University of Wisconsin–Madison. Currently, Schultz oversees activities related to product formulation, quality control and regulatory issues. In addition, he works closely with the dairy team and has corporate responsibilities. He lives with his wife in Verona, Wis. His two grown children and their spouses include a diverse mix of a teacher, lawyer, airline pilot and doctor.

Category: Dairy Performance
Feed additives