정부에서는 지난 3년 간 중소기업의 인력구조 고도화를 목적으로 교육훈련사업을 실시해 왔다. 이 연구는 퍼지집합이론을 이용하여 중소기업 인력구조 고도화 사업을 평가한 것이다. 연구의 결과, 현장성 있는 기술교육 사업의 사업효과가 높으며 경영관리 교육의 효과는 상대적으로 낮은 것으로 나타났다. 또, 교육훈련 사업의 효과를 제고하기 위해서는 수요자 요구에 부응하는 다양한 교육프로그램의 개발이 필요한 것으로 나타났다. 또한 교육기간이 비교적 장기간일수록, 단위 프로그램의 참여자 수가 작을수록 교육효과가 큰 것으로 나타났다.
이 연구는 시범적으로 정부사업평가에 퍼지집합 이론을 적용한 것이다. 언어적 표현으로 구성된 주관적, 정성적 평가를 가능한 객관화시키는 방법론을 제시한 것으로 정성적 사업평가의 대안으로 활용할 수 있을 것이다.For the past several years, the Korean government has been carrying out training programs (named Programs Designed to Enhance the Personnel Structure: EPS program) for human resource development in SMEs. The objectives of the programs are to strengthen the capability of existing manpower and induce competent workers to join SMEs. Approximately, 100 million dollars of subsidies has been supported every year, and 469 training courses has been offered through 8 business associations. A total of 12,420 SMEs takes part in the programs with 31,296 employees enrolled.
It is difficult to evaluate the performances of the training programs subsidized by government since the goal, which is mainly to enhance the productivity and human resource capability of the trainees, is often hard to measure. That is the reason why qualitative assessment is often conducted to the training programs. However, there are several limitations to this approach. First, the often-used Lickert scale measure, in which the evaluations are described in words to make qualitative assessments, has the ambiguity problem of words undermining the objectivity of numerical scores. The ambiguity associated with the linguistic variable must be reflected for the possibly-accurate evaluation. Second, the results may be distorted if the evaluators lack of sufficient understanding of the program or of the assessment criteria. The difference in the confidence levels measured by each of the evaluators must also be factored into the results.
To address these issues regarding the composition of evaluation indicators and measurements, the fuzzy set theory may be employed to put together the indicators and measurements. The fuzzy set theory is generally used as a powerful tool to reflect or lesson the uncertainty and inaccuracy in the decision making process.
The fuzzy set theory is used to evaluate the EPS programs of SMEs in this study. In order to assess the programs, we make the qualified indices over the process of program, and calculate the program score using the fuzzy methods.
In the program evaluation, we decided three qualified criteria from the Delphi method. The criteria consists of the effectiveness of the training program, the performance of training program and feedback effect. The evaluation indicator of EPS program consists of three qualified indicators : PEI (Program effectiveness indicator), TPI (Training performance indicator) and PFI (Program feedback indicator). These indicators assess the program and calculates the respective composite indices for each type of training programs. The types of training programs can be categorized as the technical fields training course, the technical theory training course, the management training course and the CEO course.
We use three kind of methodology for evaluation: SMS(Simple mean score), FICS(Fuzzy importance and confidence weighting score) and FIS(Fuzzy Importance weighting score). The FICS is the linguistic rating score with the fuzzy calculation method, which takes into consideration of the importance of criteria and confidence of judgement, applied . The FIS is the linguistic rating score with the fuzzy calculation method, which takes into consideration of only the importance of criteria. The SMS is just the mean of the linguistic rating score. In view of all the programs, the program effectiveness score (0.745) and feedback effect score (0.744) are nearly the same, followed by the training performance score (0.726) in the terms of SMS. However, in terms of FIS and FICS, the program feedback index (respectively, 0.625 and 0.538) scored the highest followed by TPI (respectively, 0.621 and 0.536) and PEI (respectively, 0.606 and 0.522). The results differ when taking into consideration the ambiguity of the linguistic expressions and the uncertainty of human judgment. Methodologically, the FICS is likely to be less fit than the FIS since the evaluator has a lower confidence on his/her judgment.
The scores for each type of the training are as follows. The score ranking of composite index is CEO training (0.75) > work-field skill training (0.74) > theoretical knowledge training (0.73) > management training (0.72) in terms of SMS. In terms of FIS, the score rankings of composite index are work-field skill training (0.62) = CEO training (0.62), then theoretical knowledge training, management training (0.59). FICS ranking of composite index are work-field skill (0.54), technical knowledge (0.53) and CEO (0.53), and management (0.52). The score of management training scored the lowest in FICS and FIS. In the three methods, the work-field skill course scored the highest.
We applied the OLS estimates in order to find the characteristics of training course. The OLS estimate is drawn up with the FICS as the dependent variable and the characteristics of the training course such as training periods, number of trainees, government training subsidy, degrees of improvement for human resource technology as the independent variables. The estimation results reveal that the evaluator gives higher scores when the training is done over longer period, the number of participants is smaller, the government subsidies per trainee is smaller (or the individual cost-sharing is larger), and the capacity(or productivity) increase is greater. This indicates that small-scale and long-term training courses are more effective. Furthermore, since the possibility of deadweight loss effect of subsides may be high, the ratio between government subsidies and individual cost-sharing should be well-balanced.
Next question is how the confidence of evaluation is relevant to score? We can suggest that the difference between FICS and FIS is the confidence of evaluator. If the degree of evaluator's confidence is complete, FICS is equal to FIS. If the degree of evaluator's confidence is less complete, FICS is less than FIS.
The difference between FIS and FICS is denoted as dP = FIS/FICS. We may suggest that the difference between FIS and FICS, The dP, is used to accept or not to accept for the evaluation result. That is, the results are the more different, the less acceptable. So, we can define the level of evaluation confidence (LEC). If the LCE is equal to 1, the confidence of evaluator is complete and the decision maker may accept the result of evaluation. However, the LCE is near to 0, we can recognize that the evaluator assesses the training program with weak information or ambiguous judgement. So, we may not accept the evaluation results. The LCE helps the policy maker decide whether to accommodate the results or conduct another round of evaluations.
We find that the training programs, which has the long duration and applied to small groups of trainee, shows more positive performance. And the workplace training program shows the best performance and the management program, the next programs and then, CEO course.
This study reveals that the evaluation indicators may has the impact on the results of a qualitative evaluation for policy assessment. We find two things: one is that the fuzzy importance weighting score infuses partial objectivity by translating the ambiguity of linguistic expression into numerical figures, and the other is that fuzzy importance and confidence weighting score boosts the reliability of the results by adding the confidence of the evaluator. This study suggests that the Fuzzy evaluation methods can suffice as an alternative method of policy evaluation.