Impact of the Self-Assessment Process on the Improvement of the Bachelor’s Level Training Programme

Self-assessment is the 1st phase of the multiphase process of TP analysis started at the School of Sociology in November 2012 with the emphasis on the training results. At this phase, special attention is paid to the implementation of the programme of competence-based approach, which so far has no off-the-shelf solutions in Russia or abroad.

Work on the implementation and development of the new approach in sociological education is supervised by HSE pro-rectors Vadim Radaev and Sergey Roshchin as well as the Academic Supervisor of the School of Sociology Alexander Chepurenko.

Research and methodology development in this field is the result of active cooperation of the employees of the Centre for Development of Sociological Education (headed by Margarita Bershadskaya) and the teams of the School departments with consulting support and continuous creative collaboration with the leading universities and the best employers in the sociological research industry.

This section provides specific examples of the following three important aspects:

• how the process of self-analysis proper impacts TP improvement (subsections 3.3.2 and 3.3.3);
• how the process of self-analysis will be applied for the enhancement of the quality of training in the future (subsection 3.3.8.);
• the process of self-analysis as a powerful tool for involving the F&S in TP assessment  and improvement (subsections 3.3.7, 3.3.8) . 

Development of National Requirements for the Competences of a Bachelor and Master of Sociology

Development of common cultural and professional competences of a sociologist is one of the major goals of the bachelor’s programme TP, which to the most extent corresponds to the institutional goals of the School of Sociology.

According to this goal, self-analysis shall focus on the monitoring of the fulfilment of requirements for the structure of developed competences and methods of their development and assessment.

It should be emphasized here that development of national requirements for the structure of bachelor competences (and master competences as continuation) is the best practice of the previous experience of analytical work of the School of Sociology. In 2009 the School organized and chaired the Consortium of universities that developed the current Fedreral educational standards for bachelor and master training in sociology^[1]. The team of authors of DS 2010 officially included five employees and leading faculty members of the School of Sociology: Vadim Radaev (Head of the Department of Economic Sociology, chairman of the Consortium of FS authors); Alexander Chepurenko (Head of the School, vice chairman of the Consortium); Andrey Kozhanov (Deputy School Head for academic affairs and methodology), Inna Devyatko (Head of the Department of Analysis of Social Institutions, Head of the master’s programme), Margarita Bershadskaya (Head of the Centre for Development of Sociological Education).

The fact that the Ministry of Education and Science of the Russian Federation has approved the FSS developed by the initiative and under supervision of the HSE’s School of Sociology is a sure evidence of the competitive advantage of the Faculty and recognition of its experience in analytical work in the field of TPs for training in sociology.

The same is true about methodological support of the SS implementation, in particular,  recommendations in the form of approximate TPs for bachelor and master training (they include suggested curricula and training programmes for all disciplines)^[2].

These activities of the School are going on: development of DS 2012 by the initiative of the Ministry – bachelor’s, master’s and postgraduate programmes. When performing these highly responsible activities, the School mainly uses the results of analysis of its own TPs.

Implementation of the Requirements for the Competences of a Bachelor in Sociology in the Bachelor Programme TP

The HSE uses its own Original educational standards for training bachelors and masters in sociology, which were developed by the School based on the DS requirements (not lower than the state requirements).

• enhance the qualification of faculty members;
• correct the training programmes of disciplines;
• repeat TP monitoring (second self-assessment phase).

The mechanism of detecting shortcomings is implied in the system of programme self-assessment including:

• development and analysis of the matrix of competences, which distributes the competences a graduate should have by the basic programme disciplines (i.e., partial development of competences in some disciplines);
• creation and analysis of the matrix of knowledge and skills, which distributes them by competences (this forms a basis for creating the resource base for each of the competences of the resource base);
• development of suggestions on the correction of discipline training programmes to eliminate the shortcomings detected.

Taking into account the importance of the specified elements of self-analysis, they are described in more detail below (subsections 3.3.3 and 3.3.4).

This subsection just gives an overview of general results of the first self-assessment phase (Table 3.8) and shows the role of self-analysis and its impact on TP improvement as a consequence of the general results of the second self-analysis phase (Table 3.9).

Table 3.8 lists the TP strengths and weaknesses detected at the first self-assessment phase and gives corresponding suggestions on how to make corrections in the training programmes (see subsection 3.3.2.).

Table 3.8

Table 3.9 gives examples of some shortcomings, detected during the first self-analysis phase, to show how the self-analysis process impacts the programme improvement.

Table 3.9

Results of the Second Self-Assessment Phase: Examples of Eliminating the Shortcomings Detected during the First Phase

Analysis of the Matrix of Competences and its Impact on the Process of TP Improvement

The main tool for assessing the competence-based approach by the entire set of TP elements (training programmes of the basic disciplines, practical training and FSC provisions) is creation and analysis of the matrix of competences.

To illustrate the efficiency of TP self-assessment with aggregation of results and further measures for eliminating the shortcomings detected, below there are the results of monitoring the competence-based approach for mastering the basic professional disciplines.

The TP template, suggested to School members since 2011, includes not only the list of the competences developed when training in the discipline (fully or partially), but also the indicators of competence mastery (descriptors), which are tested both when studying the discipline and within various types of testing by the discipline.

Unfortunately School members do not always understand the new approach and its role in the training process, so when developing training programmes, they often just formally list competences and descriptors trying (intentionally or not) to make the number of competences, developed during the discipline study, minimal. Hence, a real risk of not fulfilling the requirements for the competences to be acquired by a programme graduate. The matrix of competences created during the first self-assessment phase identifies some competences that are insufficiently or not at all covered in the programme. By way of example, let’s give the results of the analysis of the initial matrix of competences developed in the basic part of the programme – when studying the basic disciplines of the professional cycle during theoretical training. The matrix showed that more than five out of 16 competences are not sufficiently covered in the discipline TPs. These include two common cultural competences (CC) and three professional competences (PC):

• CC-12 Understanding of essence and value of information in the development of modern society, recognize risks and dangers, satisfy information security requirement, including protection of official secrets.
• CC-16 Knowledge of safeguards for personnel and public protection from the consequences of accidents and disasters.
• PC-6 Ability to participate in elaboration of recommendations based on professional sociological knowledge aimed at social problems solutions, elaboration of mechanisms for coordinating the interests of social groups and communities.
• PC-10 Ability to apply basic theoretical knowledge and practical skills for participation in scientific and applied research, analytical work and consulting.
• PC-11 Ability to use methods of collection, processing and interpretations of complex social information to solve the administrative tasks including those beyond the immediate sphere of activities.

In addition, two TPs were found to be missing information about the competences to be developed:

• “Economic and social statistics”
• “Social structure and social stratification”

As is seen from the first phase matrix, the shortcomings, specified in items 1, 2, 3 of Table 3.9 in the previous subsection, were identified. This allowed carrying out some work on the programmes during the second phase, which fully eliminated the bottlenecks of the first matrix.

The first and second self-analysis phase matrixes of competences, which give a summary of this subsection, can be found in Annexes A.3.8 and A3.9. 

Analysis of the Matrix of Knowledge and Skills and its Impact on the Process of TP Improvement

Knowledge and skills are the necessary and very important component of any developed competence and its resource base.

At the first self-analysis phase it was found out that the discipline TPs quite fully reflect the process of obtaining knowledge and skills during discipline study. One of the shortcomings is a weak correlation between the knowledge and competences developed when training in the discipline. This results from the distribution of the knowledge and skills by the competences developed, i.e., the matrix of knowledge and skills. The matrix was created based on the appropriate analysis of the competence descriptors that reflect their resource base. The mandatory competences of the professional cycle are described considering the standard requirements for the cycle resource base. Formalabsence of the correlation between the knowledge and skills and mandatory competences is quite significant: the three requirements for the knowledge and skills were not reflected in any competences; 10 out of 16 competences had no descriptors of the resource base.

Although mastery of the knowledge and skills, implied by the training programmes, is undoubtedly the basis for developing the competences, the absence of the formal correlation with the competences did not allow evaluating them as part of the competences (the resource base and the competences are evaluated separately).

As a result of the analysis of the resource base and the content of the discipline training programmes, over 30 descriptors of the resource base were formulated additionally for all the mandatory competences of the professional cycle. Correction of the training programmes in compliance with the suggestions made dramatically changed the matrix (see Table 3.10). The competences that contained the descriptors of the resource base at least in one of the basic disciplines in the original matrix are marked with *. The # character indicates changes in the second programme self-assessment phase.

As is seen, at the second self-analysis phase, the matrix of knowledge and skills has no gaps: all the knowledge and skills are included in the competences (one or several); each of the 16 mandatory competences now contains the descriptors of the knowledge and skills.

Table 3.10

Matrix of Knowledge and Skills at the Second Self-Analysis Phase

Suggestions on Correction of Training Programmes of Other Faculties Based on the Requirements for Sociology Training Programme 

To make sure the requirements of our TP resulting from the analysis of the matrix of competences and matrix of knowledge and skills are met, we were to develop suggestions on how to correct the TPs of the basic professional disciplines of other faculties (“Economic and social statistics”; “Social structure and social stratification”). The aim was to formulate the competences, required by a sociologist, based on the specifics of the content of these courses, which, as our preliminary analysis shows, contains all necessary conditions for their development.

There are two different ways to formulate competences developed by disciplines and their corresponding descriptors. The first approach is to limit the number of such competences to 5-6. Most authors of training programmes of other faculties are committed to this principle. The second approach, which is applied by our School to the basic professional disciplines, implies step-by-step development of system and professional competences in most disciplines of the cycle. We believe (and were supported in the matter by psychologists during one of the meetings on methodological issues), that such approach is more effective, the more so because, as a rule, none of the disciplines can be responsible for the development of all professional competences.

It was this approach that we followed to develop recommendations on how to make changes in the above-named disciplines. Recommendations on how to fill the table of competences in these disciplines’ TPs are given in the Annex. Considering the importance of these courses for sociology students, the recommendations contain the wording of descriptors of all the mandatory competences of the professional cycle (more than 30 descriptors per discipline). The recommended descriptors (see tables in Annexes A.3.10 and A.3.11) are in full compliance with the content of the disciplines and are easy to verify.

3.3.6. Development of the Methodology School at the School of Sociology

The HSE’s School of Sociology, renowned for its academic school, in recent years has gained prominence for its methodology development, too. Promotion of the methodology school at the School is carried out under the guidance and in creative collaboration with the HSE’s Methodology Office.

First recognition of our methodology school is its winning of the competition for the development of state educational standards in sociology (2009) and further development of approximate TPs for Russian universities and colleges that train bachelors and masters in sociology. The Centre for Development of Sociological Education was opened at the School to coordinate the activities on the implementation of new educational standards in Russian universities and colleges.

The next important development phase was working out of our own original standards introduced in 2010.

Then followed a great volume of work on the implementation of the new standards along with the development of methodology materials, organization of professional development courses, in-house and field seminars for faculty members and the members of the Consortium of universities with experience shared during round tables, All-Russian and inter-university conferences, social networks, publication of developed methodologies in journals and conference files.

Professional development of the F&S of the School is mainly aimed to solve the most critical problem with the implementation of standards – that of competence-based approach to training results. The HSE’s Methodology Office has developed very good methodologies, templates and algorithms for competence evaluation^[3]. Every two weeks there are events called “Methodology Wednesdays” where the methodology materials are explained and discussed. The F&S of the School of Sociology take an active part in sharing the experience. It should be noted that there was a clear increase in their activity during TP self-assessment. This is a strong evidence of the fact that the process of TP self-assessment is a powerful tool for integration of the F&S of the School of Sociology in TP evaluation and improvement.Things, that first were not accepted by the faculty members (competences, descriptors, etc.), are gradually not just entering their vocabulary but also more actively used to formulate the course objectives and requirements for mastering it (see sections devoted to TP improvement in the process of self-analysis: 3.3.2.; 3.3.5; 3.3.6). The most active employees and faculty members participate in such projects of the Methodology Office as development and expertise of tests for competence evaluation, development of a competence-based course profile, certificates of system competences and so on.

School events on the issues of sociological education proper are held by the University’s Methodology Office (or jointly). The nature of discussions indicates that the process of TP self-assessment makes the faculty members more and more interested in methodology issues. One should just see the expressions on their faces when they are discussing the tasks in the field of methodology development (slides on page 65) – this is so unlike many meetings and seminars of the kind when the faces of the participants express nothing but indifference, incomprehension and no interest.

These and other similar slides  summarize the results of 2009-2012 and tasks for the future (2013-2016), some of which are already in progress during self-assessment.

[1] http://ecsocman.hse.ru/sociology/bac.html

[2] http://www.hse.ru/data/2011/04/07/1211727134/plan_ba.pdf

[3] http://www.hse.ru/studyspravka/programmauchdisc