Head of quality department of the «TDSK» JSC,
Corresponding member of the Russian Academy for Quality,
Ph.D. student, assistant of the Department of Automation and Robotics
in Mechanical Engineering of the Institute
of Cybernetics of Tomsk Polytechnic University,
Quality manager of the «STI TDSK» Ltd,
master's degree student of the Department
of Management and Technology in Higher Professional
Education of the Institute of Humanities,
Social Sciences & Technologies of Tomsk Polytechnic University,
One of the requirements which standard ISO 9001 includes is measuring the Quality Management System effectiveness. However, there is no uniform method for complex estimation of effectiveness.
Currently there are many problems concerning the measurement of QMS effectiveness and each separate process in the QMS of an organization faces the kind of a problem being of both theoretical and practical importance. In this article the author offers the method for analysis of effectiveness with implementation of the Balanced Score Card (BSC), with the process "Getting the construction ready for settling" as an example of this method. .
Keywords: Quality management system, ISO 9001, effectiveness and efficiency, BSC(1) (Balanced Scorecard), KPI(2) (key performance indicator)
Quality management can be considered to have three main components: quality control, quality assurance and quality improvement. Quality management focuses not only on the product quality, but it aims at its perfection as well. Quality management, therefore, uses quality assurance and control of products to achieve more consistent quality. Quality management system (QMS) can be defined as: “A set of coordinated activities to direct and control an organization in order to continually improve the effectiveness and efficiency of organization activity.” 
Standard ISO 9001:2000 «QMS. Requirements» is one of the basic tools to improve the organization activity in the area of quality. One of the requirements which this standard includes is measuring the Quality Management System effectiveness. However, there is no uniform method for complex estimation of effectiveness. In management, effectiveness relates to getting the right things done. 
Peter Drucker reminds us that effectiveness is an important discipline which “can be learned and must be earned.” In a survey conducted by the Defence Evaluation Research Agency (DERA), 96 % of respondents said they believed their system contributed to meeting the business goals. However, ca. 72 % responded that their organization did not measure this contribution since they did not know methods to fulfill it. 
(1) The balanced scorecard (BSC) is a strategy performance management tool - a semi-standard structured report, supported by design methods and automation tools, that can be used by managers to keep track of the execution of activities by the staff within their control and to monitor the consequences arising from these actions.
(2) A performance indicator or key performance indicator (KPI) is industry jargon for a type of performance measurement. KPIs are commonly used by an organization to evaluate its success or the success of a particular activity in which it is engaged. Sometimes success is defined in terms of making progress toward strategic goals, but often success is simply the repeated achievement of some level of operational goal (for example, zero defects, 10/10 customer satisfaction, etc.).
Currently there are many problems concerning the measurement of QMS effectiveness and each separate process in the QMS of an organization faces the kind of a problem being of both theoretical and practical importance. The organization certified to ISO 9001 heavily needs methods containing extensive data on the process which will be easy and understandable for the process executors.
Inputs to evaluate efficiency of the quality management system should be considered by the customer and other interested parties and should include:
- status and results of quality objectives and improvement activities;
- results of audits and self-assessment of an organization;
- feedback on the satisfaction of interested parties, perhaps even to the point of their participation;
- market-related factors such as technology, research and development, and competitor performance;
- results from benchmarking;
- performance of suppliers;
- new opportunities for improvement;
- control of process and product nonconformities;
- status of strategic partnership activities;
- financial effects of quality related activities;
- other factors which may impact the organization, such as financial, social or environmental conditions, and relevant statutory and regulatory changes. 
The founders BSC of рrofessor Robert Kaplan, from Harvard Business School, and David Norton, a consultant, carried out a research to understand and evaluate new methods for measuring the performance. They formulated a method that would not rely so much on just financial metrics as measure but would show a balance of financial and non-financial perspectives. The outcome of this process is the Balanced Scorecard, a method that translates strategic themes to actionable and measurable objectives that are ready for execution at all levels of the organization . Kaplan and Norton articulated four perspectives that can guide companies as they translate strategy into actionable terms . They do not argue that these perspectives are necessary and sufficient conditions for success, however. In fact, they recommend these perspectives but suggest that organizations are to add any perspectives that are more relevant.
- what are the financial targets?
- what drives these targets?
- what kind of profit and revenue to be achieved?
- who are the customers?
- how do you satisfy them?
- what segments do you aim to address?
- what goals do you aim to achieve with partners?
- what are your goals for the distribution channel?
- what processes must we be the best in to win customers?
- what internal activities do we need to sustain competencies?
Learning and growth perspective:
- how must we train our team?
- what climate and culture nurtures growth?
- what do we have to do to train our people to achieve their objectives?
But the only way to know whether those goals are being delivered is to identify indicators of their successfulness and to use them for keeping an eye on the way the business is being performed. They are called key performance indicators (KPIs). KPIs reflect the organizational goals, in simple an organization has a goal to increase the sales to certain quantity, then KPIs will be the number of sales closed in a month, number of products sold, % of increase in sales from last year attribution.  In this article one of the algorithms of KPI with BCS for the construction industry is presented. It was tested the building and construction firms in Tomsk.
This firm was certified to ISO 9001 in 2006. The company analyzes data on QMS continuously throughout the year. There was a need to expand the list of criteria for an objective assessment and adequate information on company activity.
It was decided to develop criteria by taking into account company strategy during previous years on the basis of balanced scorecard (BSC) principles.
Below, let us consider the "Getting the construction ready for settling" process as an example.
The purpose of "Getting the construction ready for settling" is specified as "Investment cycle reduction".
In the rank the actions, one of purposes to be achieved is the reduction of time from the moment of obtaining permission for the object to be put into operation until the time it gets inhabited.
Interested parties such as consumers, company staff, public services, the companies which provide utilities (supply with electricity, gas, telephone, heating, light and water) participated in the process.
Involvement of the staff of many company departments and their continuous coordination made the process complex as well.
At the beginning the process was described by QMS according to ISO 9001.
After that process is developed (Table 1) on the basis of the strata of the purpose and purposes of the key performance indicators (KPI).(Table 1).
The technique described below has been developed to measure achievement of KPIs.
Initially, the process should be chosen, and then the data on the process such as reports of the department, statements, transactions, quality standard and etc. should be collected. After that, depending on the process peculiar features and the data available the list of indices must be worked out.
Further, the chosen system of process quality indices is divided into four groups of the PDCA cycle. PDCA can be briefly described as follows:
- Plan: to establish the objectives and processes for delivering the results in accordance with customer requirements and the organization policies;
- Do: to implement the processes;
- Check: to monitor and to measure the processes and product against policies, objectives and requirements for the product and to report the results;
- Act: to take actions to continually improve the process performance; 
Next, to define executors of process (referred to as expert team) and to evaluate the expert judgment and calculation weight coefficient by using the formula (1):
where n is the quantity of experts, m is the quantity of "weighed" quality indices,
- is a judgment of j-th quality indices in the points given by i-th expert.
For each quality indices find the relative ratio by this formula (2):
where is the real quality index within specified period,
is the planed quality index within specified period.  If by this formula
After finding relative ratio as the ratio is expressed in different values (percent, quantity of items) the normalized magnitude should be found by this formula (3):
where is relative ratio,
g is weight coefficient for selected quality index.
When all quality indices are considered, the effectiveness of process should be computed by summing up normalized magnitude by PDCA. Finally, the result may be presented as a diagram.
The result should be estimated in the following way:
- If total sum S is closely to one, the process is very effective;
- If total sum S > 0.7 the process is effective, but development of corrective actions and preventive actions are required;
- If total sum S ≥ 0.5, the process is still effective but the active process requires urgent amendment.
- If total sum S is close to zero, the process is not effective.
1. Vnedrenie sbalansirovannoi sistemy pokazatelei / Horvath & Partners; Per. s nem.. – M.: Al'pina Biznes Buks, 2005.
2. GOST R ISO 9000-2008 Sistemy menedzhmenta kachestva. Osnovnye polozheniya i slovar'. – M.: IPK Izd-vo standartov, 2008.
3. GOST R ISO 9001-2008 Sistemy menedzhmenta kachestva. Trebovaniya.– M.: Standartinform, 2008.
4. Razrabotka sbalansirovannoi sistemy pokazatelei. Prakticheskoe rukovodstvo / Pod red. A.M. Gershuna, Yu.S. Nefed'evoi. - M.: ZAO «Olimp-Biznes», 2004.
5. Rampersad H. Universal'naya sistema pokazatelei. Kak dostigat' rezul'tatov, sohranyaya celostnost'. — M.: Al'pina biznes buk, 2006.
6. Robert S. Kaplan, Deivid P. Norton. Organizaciya, orientirovannaya na strategiyu. Kak v novoi biznes srede preuspevayut organizacii, primenyayushie sbalansirovannuyu sistemu pokazatelei. —M.: Olimp-Biznes, 2005.
7. Robert S. Kaplan, Deivid P. Norton. Sbalansirovannaya sistema pokazatelei. Ot strategii k deistviyu. M.: Olimp-Biznes, 2003.
8. Doran, G. T. There's a S.M.A.R.T. way to write management's goals and objectives. // Management Review. 1981. - R. 35-36.