Abstract

Advances in software development tools have made it feasible for the relatively rapid construction of visually complex multimedia tutorial and expert systems with relatively little computer expertise. These significant advances allow true experts in a discipline or clinical technique to generate useful materials with less time investment. However, this ease of development requires that end users carefully evaluate programs for content and data management algorithms when they consider using them in their practice.

Tutorial programs are designed to convey information. Evaluation of the quality of the content should be the most important evaluation consideration. Who is compiling the information? What are their qualifications? How much direct input into the program have they had? This last question can be difficult to assess just from promotional materials or title screens in the program. Look for the level of referencing and degree of internal qualification of information. Can you identify original sources with reasonable effort? Are appropriate caveats presented with speculative information?

A sophisticated interface with high quality visual and audio effects is attractive and can enhance the learning experience. However, not only can it disguise weak content, it can obscure the availability of excellent content. Occasional use of images for purely esthetic purposes is reasonable and to be expected, but the majority of images and sounds should contribute to the understanding of the user. Images should be easily related to textual materials and labels. The interface itself should not be exceptionally complex and should, ideally, allow for alternate ways of navigating the program. A key requirement for programs destined to be used more than once is the ability to rapidly search the information available and move to selected points. Although there are applications for software that is used only once and requires that you move linearly through a single path to each piece of information in sequence, most users prefer to be able to customize their path. When testing new software, try searching on various subjects of interest to you. Not only will this give you a better idea of the content quality, but it will allow you to evaluate the speed and usefulness of the search engines embedded in the program.

Expert systems take the user a step further than tutorial systems. Expert programs can include tutorial subroutines. This may be a major benefit, but this is not a mandatory consideration. Expert systems are designed to take input from the user and help in decision parsing. All of the concerns related to tutorial programs apply to expert systems. The quality of data within the program determines the quality of the decisions it can make and graphic displays can enhance or obscure the understanding of the basis of those decisions. For veterinarians, most of these programs are focused on disease diagnosis or the generation of differential diagnoses. It is critical that the user be aware of the algorithms being used to generate these decisions. Levels of confidence for the decisions should be readily apparent when using the program. High quality programs should allow the user to select among decision making algorithms and help menus or manuals should clearly explain the mathematic basis of the parsing occurring with each option.

Several of the commonly available algorithms include factor inventory, weighted factors, Baysean methods, and combination analysis. Each of these can be done in several different ways but basically each represents an increase in sophistication. Factor inventory simply tallies the number of attributes entered by the user that match those included in the database. It is the least sophisticated approach to ranking diagnoses and has obvious limitations. However, when the data for frequency of occurrence of diagnostic factors or symptoms is weak, this method has the advantage of not being affected. Weighted factors greatly improve the likelihood of accurate parsing of a differential as they take into account the fact that all attributes of a disease are not seen with equal frequency in all cases. Of course, accuracy of these methods is affected by the match between the programmed expected frequencies and reality. Baysean methods are based on Baye’s Theorem which assigns a quantitative change to the probability of a diagnosis based on absence as well as presence of a factor. The major advantage in these methods is that they incorporate the impact of the absence of a finding in the decision parsing. Combination analysis considers the possibility that multiple diseases may be present at one time and may better explain a set of findings. The complexities of the algorithms involved in assessing combination potentials vary greatly from simple disease attribute addition methods, which are most commonly used to very sophisticated modifications of probability theory.