from Leadership Medica n. 9/2001

“Comparison” and the “benchmarking”

Differently from a usage, sometimes incorrect, of the term in Public Health environments, the “Benchmarking” can not be taken as a mere comparison of activities or processes between two health services firms. It is necessary to premise that we don’t want to refuse the absolute importance of a mere “comparison” between activities and processes, which also is of remarkable managerial utility, but yet different from the benchmarking.
Besides, it is useful to notice that the “comparison” is not necessarily realized between two structures of different health services firms; it is possible to draw profit even from a comparison, in time, of the production processes of the same service. In this sense, the comparison is meant essentially in two typologies (see table 1).

First of all a “cross section” comparison, which is necessarily synchronic, between two services, or departments, or hospitals (of the same, or of different health services firms), whereas the processes are comparable. The “cross section” comparison does not have to be referred to a “best in class” structure: remember the regional reports in Italy, where data from the various health services firms are compared. Instead, the most important problem is the comparability of the data. About it, Casati reminds the need that the necessary homogeneity for a correct comparison has to be: structural (similar dimension); functional (comparable case – mix); countable (same survey methods and elaboration of the costs).
The second comparison type, on the contrary, is diachronic, “over time”, that is a monitoring (such has to be considered this case) of the modifications that happen in time, in terms of output and of outcome. In that case the choice of a unity of time is imposed: month, year, etc. Even if it seems pleonastic to remind it, it is opportune to underline that, except for different indications imposed by the circumstances, the comparisons over time happen regularly between similar periods of time; for example: the first quarter of two or more following years, or the second semester of two or more following years, the annual data for a certain number of years and so on. (Table 1) The diachronic comparison suggests the trend of the results, within the limits where suitable attention to the homogeneity of the absorbed resources and of the operational conditions was paid. The synchronic comparison involves, on the contrary, above all the < relationship between the human, financial and instrumental resources used and the quantitative, qualitative and economic level > of performances, even though we must underline that it is < very difficult in public administration to have comparable data > (Macrina and Ravaioli), since often the method of collecting data is not properly shared, in order to obtain a reliable limit of homogeneity, that is even before the elaboration and interpretation of the resulting numerical values. It is clear that the comparison can concern: quantitative aspects; qualitative aspects; economic – financial aspects. If the comparison is set with the best structures that operate in a similar range of activities (“best in class” in Anglo-Saxon literature) it is possible to evoke the term “benchmarking”. The benchmark to which we report, therefore, is the optimal standard of performance.

Definition and classification

The number of definitions of “benchmarking” in literature is uncountable. It is necessary, first of all, to remember the definition given by the man to whom the paternity of the benchmarking belongs to, Camp. He defined such tool of management as a continuous process of product measuring, services and business routine, through the comparison with stronger competitors, or with leading enterprises of a sector. Within the plentiful literature on the matter, even Italian contributions are numerous and authoritative, some of which derive from, or however find position in the specific field of Public Health. In particular, Zanetti reminded that in business management, benchmarking < means a management technique, that consists in activities of structured and permanent evaluation, with the purpose of comparing the products / processes of one’s own health services firm with the market leaders >, in order to underline also that < the differences found considering the standard of the health services firms of reference are analysed to define strategies, aims, improvement plans and courses for tending to excellence >. In any case, if on one hand, mainly in business management, the benchmarking is the process of analysing the indicators of success to be used to make the health services firm grow and to improve the products’ quality, on the other hand, more properly in health areas, the benchmarking represents the critical point of view, or an “instant photo” (Roediger) of the practices adopted, that lets us determine which are the preferable and the “best in class” to which we should refer. Yet, because of the market peculiarity, the benchmarking can assume different connotations in Public Administration, in comparison with the private, where it was born and improved with the passing of years.
Besides, we also believe that < in Public Administration the benchmarking should become of collaborative type, not only for the scarce competitive mechanism that links public health services firms, but also for the necessity to raise the performance standard of all health services firms that offer public services > (Compagno and Cagnina).
Nevertheless, from the impelling necessity to align with the best Public Services in the European Union, it turned out the demand that the Italian Public Administrations recover in efficiency terms, effectiveness and management inexpensiveness, also in compliance to the norms emanated by the government in the last ten years (Marchitto). Even about classification criteria, the literature about benchmarking is wide. The division offered by Masoni is certainly original. He distinguishes a normality benchmark (where the results of others are assumed as a norm for the project), from the excellence benchmark (set – in relation to the results obtained from others – as a sight to overcome). Azzone, instead, distinguishes a performance oriented benchmarking from a processes oriented benchmarking: in the performance oriented benchmarking the “best practice” is a < point of reference for other organizations > and it constitutes, therefore, a < stimulating aim (opposite to the historical data) >; in the processes oriented benchmarking, on the contrary, the comparison concerns (if the routines can pacifically coexist) < management modalities of the different organizations >. Agreeing with Compagno and Cagnina, our opinion is that it is needed to distinguish first of all: I. the environment of the comparison (internal or external); II. the object of the comparison (institutional finality).

Classification in accordance with the environment of the comparison

We identify: the internal benchmarking, when performed in the context of the organization, usually among services that have the same institutional duties, but are located in different geographical areas of the same firm. The internal benchmarking is easier to realise, even because it evokes less resistances, it is feasible in acceptable times and it is a useful tool for a first knowledge and verification of the business processes. In public administration, the reasons that can make the difference in the services performance, in different territorial realities, have been stated by Marchitto; we remind the most useful: the socio-cultural model and, most of all, behavioural, possibilities of interpretation of the norms, modalities of distribution of the service, attitude toward the consumers, sense of duty, of affiliation and of the hierarchy, ability to assume decisions. For what was quoted previously, he notices that such routine, often, is not a real benchmarking, but a mere comparison with a structure that offers good quality, even if rarely meaningful for a best practice. The external benchmarking, when carried out with external firms, sometimes competing.
The external benchmarking implies the comparison of some business processes with those used by accredited organizations in the field. It may be (but not necessarily) competitive; in this case, some difficulties may arise to establish a constructive dialogue with the partners, with the risk to compare only production factors, rather than processes (Trivellini and Caliendo). The functional benchmarking or process oriented, when performed with firms that act in other sectors. The compared object is a definite function, or a determined process, independently from the fact that the products / services of the two compared firms are different. Usually these are support activities, such as personnel management, economic service, computer service, etc. The generic or pure benchmarking, when performed with firms that are considered absolute leaders in the sector (the best in class), that < contains the basic philosophy of the benchmarking > (still Trivellini and Caliendo). Such form of benchmarking implies a remarkable and consolidate experience, since as a rule absolutely different procedures are analysed.

Classification in accordance with the object

We distinguish: an operational benchmarking, that involves the comparison among the services offered; the processes that bring directly to supply / distribution of products / performances are verified; a managerial benchmarking, or the comparison among the procedures of support to the production line; a strategic benchmarking, with which the determinants in the competitive advantage are analysed, through the observation of the strategies that have brought other organizations to success.
About the choice of the benchmarking partner, we recall the main elements, often underlined in literature; in order of importance (still Compagno and Cagnina):
I. localization;
II. affiliation to the same branch of activity;
III. potential credibility or reputation;
IV. will to participate;
V. performance or supposed leadership.
About this, Marchitto underlines that < hardly a single firm or a single administration realizes, contemporarily, excellence situations in all the process phases >, for which it is necessary to figure out the partner in relation to the process to be submit to benchmarking, after you have verified that such partner is the “best in class” in that process and not in general.
Azzone, after reminding that the organizations have to be separate, first of all, on the performances level, or in the quality of the managerial solutions, he thinks that the organizations can be identified in relation to four typologies:
I. ”star”, leader organizations with very high performance levels;
II. emerging organizations, in course of improvement;
III. declining organizations, characterized by obsolete processes and obsolete technologies;
IV. “dog” … whose term (no offence to the man’s best friend) appears clearly and explains sufficiently.

The Benchmarking path

The planning of an effective system of benchmarking involves a correct methodological articulation, that is a really effective path in order to obtain a concrete process of improvement in the quality of performances: that also is the subject of many scientific contributions (see table 2).

A careful analysis of the operational models, mainly of the courses used to carry out the benchmarking, allows to notice that, in fact, they can be referred to the known cycle of Deming (see table 3).

Marchitto, considering the need of having, before the benchmarking planning, a sort of auto-diagnosis about < the weak points and the errors of the preceding cycle >, he suggests a chronological change, precisely from PDCA to CAPD, in accordance with the sequence shown in table 4.

In the path sequence, independently from the clearness of the ideas on what is to be compared, the greatest difficulty stays however in the choice of the partner to compare to, even for the often unsolved matter of the identification of the best in class in Public Health.
About the process to be verified, on the contrary, it is good rule to focus the attention at first on the “critical factors of success” (Cevolani), which means to perform first of all a mapping of the processes (for example by a flow-table) and then rearrange these in accordance with priority, to be submit to benchmarking.

Problems of the Benchmarking in Public Health

The benchmarking derives from the experience of great industrial firms, even if some principles can be transferred in the field of Public Health.
Yet, certainly, the results in sanitary field are difficult to be identified and interpreted, for rather a lot of reasons: first of all it is difficult to test the results directly; whereas the measures are somehow identified, they are not always homogeneous and/or the same used among benchmarking partners; the results can be measured in different dimension contexts, so that a good result, obtained in accordance with a specific term of evaluation, could compromise the result in accordance with another parameter (this can be observed more clearly in the perspective of the trade-off between costs and effectiveness); the results of a benchmarking are not necessarily to be referred to one of the treatment components; rather, they are a consequence of different phases in the search and in the evaluation; not all these phases are verified during the permanence in the hospital, so that the deductions about the hospital performance and the results could be even absolutely fictitious; it is possible to observe still the persisting of political, psychological and sociological factors for which an organization doesn’t succeed in reaching the performance level of the predetermined benchmark; in the end, the limits previously remembered, often, in sanitary field, make the identification of the “best practices” problematic.
Nevertheless, though it is difficult to identify proper indicators of outcome for medical treatments, it is yet possible to test the processes and the output of treatments, that contribute to produce outcome.
Again, even if a series of proper indicators are towards the performance of the wholly considered system, it is opportune that the benchmarking turns (and in this sense has remarkable utility) also and above all to the different levels of the organization, that is to say where the strategic decisions express, or should express, a real change of the managerial behaviour. Yet, since the benchmarking needs information about the performance of a certain moment of the organization, the exchange of information with those who held the “best practice” and the following implementation of changes happen generally at the individual level, or in the context of a specific service giving performances.
From it comes that, the information on the performance and on the necessary changes, are available also for high management, but they revert only in a limited field of activity. The exchange of information is in fact certainly of main importance to build an effective program of benchmarking. The distributors of performances are required to communicate with the organizations that hold the best practice, sharing information about the processes and the routines that lead to a higher performance; but this doesn’t always happen.
Another occasion of “communication” are congresses, where it could be possible to exchange information effectively and directly, though it is common to observe that such information are often spoil with a certain opportunism, marked from the fact that not infrequently only good results are underlined, while specific gaps are disguised, by an interpretation of the data which is opportunely manipulated.
Actually, inaccurate or incomplete information are not necessarily always to be linked to the need to confer a positive image to one’s own activity, if not, more expressly, to hide criticality cues; sometimes that happens even for a kind of jealousy of one’s own work method, out of envy, etc.

The advantages of benchmarking

The advantages of benchmarking, when effected in accordance with correct methodological applications, are various (Cocconi): advantages in terms of costs (therefore, efficiency, effectiveness, economics); advantages in terms of quality (continuous improvement); advantages in terms of business culture (growth of the organization). Benchmarking experiences in the sphere of competence of Health are reported in rather a lot of scientific contributions.
About it, we remind the cost analysis of hospital management in the program Medicare (Medicare Cost Report: see Magnus and Smith); or the benchmarking for evaluation and the support of programs about pharmaceutical costs of the “MeritCare Medical Center”, outfit from the Millard Fillmore Hospital (see Murphy; Nelson). Jones underlined the importance and the contribution of the activity of benchmarking aiming to a correct redistribution of the resources in the British Health Service.
Daniels and coll. remind the “Benchmark of fairness” (impartiality) in the context of the evaluation of the Health System reform in the United States; mainly, it is underlined that < Fairness is a wide term that includes exposure to risk factors, access to all forms of care, and to financing. It also includes efficiency of management and resource allocation, accountability, and patient and provider autonomy >; Daniels and coll. remind also that < The benchmarks standardize the criteria for fairness >, with the possibility to be used at all the levels, either national or local. Yurk and coll. recalled instead the applications of the benchmarking, among which, beside the improvement of the quality and the consequent satisfaction of the customer, there is also a full participation in the business strategic planning. Dove and Greene, let alone Homa-Lowry, underlined the role of the benchmarking in the context of quality evaluation of the treatments, mainly its links to the increase of costs and to the reduction of resources. John considers extremely relevant that the benchmarking is used in Public Health in the evaluation of the treatments’ outcome (“Therapy Outcome Measure”), but also in the diffusion and sharing of information. Ellis look out upon a punctual and wider application of the benchmarking process to “Evidence Based Medicine”. Weissman and coll. imported the concept of ABC (“Achievable Benchmark of Care”), or the feasible benchmarking in health treatments, characterised by the following three main connotations: the benchmarking expresses a measurable level of efficiency; the benchmarking can be achieved in accordance with provable procedures; the benchmarking is the result of objective data, reproducible and predetermined. Nevertheless, Weissman and coll. underline that suppliers of high performance are selected in a way to define beforehand a level of excellence; yet, the same suppliers – even offering an elevated performance – are not able to influence appreciably the levels of a specific benchmark, if they have modest case histories available. In the wide bibliography on the theme, it finally deserves to be mentioned the “2001 Benchmarking Guide” (Hoppszallern), manual that summarizes evaluation methods of financial, technical-professional and marketing order for any hospital structure. In particular, it highlights positive and negative aspects, in the comparison among hospitals with absolutely different management level.
We finally recall our attention (see Beretta) on ethically wrong behaviours, that is when the comparison of the data is actually used to try to hide the aims of costs reduction (“downsizing”), rather than to regain the efficiency, effectiveness and economics of the health care performances.

Conclusions

In spite of the negative aspects recalled, it is our belief that a progressive introduction of the benchmarking in Public Health, though demanding an initial and intense effort to implement the methodology, can give very interesting results, with remarkable advantages aimed to a real process of marketing change (see table 5).

Mainly, even to overcome concrete problems, previous to the benchmarking procedure and usually tightly related to a certain resistance to reveal the real productivity data and the habitually adopted procedures, it is desirable the introduction of forms of anonymous association, as, for example, the “benchmarking network”, managed by an organization (benchmarking club), that shares information and, if necessary, also the elaboration and the general interpretation of the data, without yet revealing the origin, suggesting indicators, standards and quality improvement procedures. Perhaps, to some the anonymity appears a little weird … but probably the effectiveness may broadly justify it. (traduzione Interpres- Giussano)

Mauro Martini
Direttore del Nucleo di Valutazione dell’Azienda Sanitaria di Ferrara (Italia)

Link consigliati

Benchmarking Club (Italia): http://www.business-italy.it/benchclub

Association for Benchmarking Health Care (USA): http://www.abhc.org/

Best Practice Network (USA): http://ww1.best4health.org/startbp.cfm

The Benchmarking Network (USA):
http://www.well.com/user/benchmar/tbnhome.html

The Benchmarking Exchange (USA):
http://www.benchmarking.org/

Benchmarking in Australia: http://www.benchmarkingplus.com.au/

Welcome to Benchmarking (UK): http://www.benchmarking.co.uk/

The Essence of Care (UK): http://www.doh.gov.uk/essenceofcare/

Le Benchmarking Club de Paris (Francia): http://www.bench-club-paris.asso.fr/

Benchmarking (Germania): http://www.benchmarking.de/

Bibliografia

-Azzone G. Innovare il sistema di controllo di gestione. Economic Value Added, Benchmarking, E-economy: le nuove metodologie di valutazione. ETAS Ed., 2000.

-Beretta S. Il benchmarking delle attività amministrative: la ricerca di standard di prestazione. In: -Beretta S. e Coll. Il benchmarking dei processi amministrativi. EGEA Ed., 1999.

-Bocchino U. Il benchmarking. Giuffrè Ed., 1994. Camp R.C. Business Process Benchmarking: Finding and Implementing Best Practices. Milwaukee: ASQC Quality Press, 1995.

-Carter L.R., Lankford S.S. Physician’s Compensation: Measurement, Benchmarking, and Implementation. John Wiley & Son, 2000.

-Casati G. Programmazione e controllo di gestione nelle aziende sanitarie. McGraw-Hill Ed., 2000.

-Cevolani A. Il processo come punto di azione del benchmarking. In: Quaderni Qualità Agenzia Sanitaria Regione Emilia

– Romagna. Fare benchmarking in Sanità. Clueb Ed., 1998.

-Cocconi A. Il benchmarking nelle Aziende Sanitarie. In: Quaderni Qualità Agenzia Sanitaria Regione Emilia

– Romagna. Fare benchmarking in Sanità. Clueb Ed., 1998.

-Compagno C., Cagnina M.R. Il benchmarking nei processi di qualità. In: Gori E., Vittadini G. Qualità e valutazione nei servizi di pubblica utilità. ETAS Ed., 1999.

-Czarnecki M.T. Benchmarking Strategies for Health Care Management. Aspen Publishers, Inc., 1994.

-Daniels N. e Coll. Benchmarks of fairness for health care reform: a policy tool for developing countries. Bull. World Health Organ., 78(6): 740-50, 2000.

-Deming W.E. In: JCHAO. Introduzione ai principi del miglioramento della qualità. Centro Scientifico Ed., 1998.

-Dove H.G., Greene B.R. Benchmarking medical group practices using claims data: methodological and practical problems. J. Ambulatory Care Manage Oct, 23(4): 67-77, 2000.

-Ellis J. Sharing the evidence: clinical practice benchmarking to improve continuously the quality of care. J. Adv. Nurs., 32(1): 215-25, 2000.

-Homa-Lowry J. Measuring and benchmarking quality in hospitals. Mich. Health Hosp., 37(2): 12-3, 2001.

-Hoppszallern S. 2001 Benchmarking Guide. Hosp. Health Netw., 75(1): 43-9, 2001. -John A. e Coll. Benchmarking can facilitate the sharing of information on outcomes of care. Int. J.

-Lang Commun. Disord., 36, suppl., 385-90, 2001.

-Jones C.S. Towards benchmarking in British acute hospitals. Health Serv. Manage Res., 14(2): 125-38, 2001.

-Macrina A., Ravaioli P. Controllo di gestione e Nuclei di valutazione. Giuffrè Ed., 1998.

-Magnus S.A., Smith D.G. Better Medicare Cost Report data are needed to help hospitals benchmark costs and performance. Health Care Manage Rev., 25(4): 65-76, 2000.

-Marchitto F. Benchmarking nella Pubblica Amministrazione. FrancoAngeli Ed., 2001.

-Masoni V. Monitoraggio e valutazione dei progetti. FrancoAngeli Ed., 1997.

-Murphy J.E. Using benchmarking data to evaluate and support pharmacy programs in health systems. Am. J. Health Syst. Pharm., 15; 57 suppl., 2: S28-31, 2000.

-Nelson R.E. Benchmarking in health-system pharmacy: experience at MeritCare Medical Center. Am. J. Health Syst. Pharm., 15; 57 suppl., 2: S25-7, 2000.

-OECD. OECD Science, Technology and Industry Scoreboard 1999: Benchmarking Knowledge-Based Economies. OECD, 1999.

-Patterson D.J. Indexing Managed Care: Benchmarking Strategies for Assessing Managed Care Penetration in Your Market. McGraw Hill, 1997.

-Roediger J.M., Medicine & Business Formally Benchmarking Your Medical Practice. Physician’s News Digest, August 1999.

-Spendolini M.J. The Benchmarking Book. AMACOM, 2001.

-Trivellini M., Caliendo G. Cenni storici e tipologie di benchmarking. In: Quaderni Qualità Agenzia Sanitaria Regione Emilia–Romagna. Fare benchmarking in Sanità. Clueb Ed., 1998.

-Weissman N.W. e Coll. Achievable benchmarks of care: the ABCs of benchmarking. J. Eval. Clin. Pract., 5(3): 269-81, 1999.

-Yurk R. e Coll. Benchmarking applications: linking state strategic planning, quality improvement, and consumer reporting. J. Public Health Manag. Pract., 7(3): 47-58, 2001.

-Zanetti M. e Coll. Il medico e il management (Glossario). Accademia Nazionale di Medicina, 1996.