ArticlePDF Available

Assessing information technology personnel: Toward a behavioral rating scale

Authors:
Michael A. Chilton at Kansas State University
Michael A. Chilton
Bill C. Hardgrave at Auburn University
Bill C. Hardgrave
Download full-text PDF
Read full-text
Download citation

Abstract

Performance ratings are used as a means to measure and record performance of individuals on any particular job. Results can then be used for promotion, increases in pay and other incentives. They, therefore, can have a large impact on a person's career and are viewed by both employers and employees as extremely important. Performance ratings vary widely and typically include a variety of measures - some of which the employee can control, and others over which the employee has no direct control. Overall, the various types of performance ratings are not perfect; each has its flaws along with its advantages. In this study, one particular type of scale - a behavioral rating scale - is proposed by taking the results of a previous study performed by Green (1989) and empirically testing a set of critical skills for use as a performance assessment instrument for information technology (IT) personnel. The instrument is provided for researchers who wish to incorporate a behavior-oriented measure in their research and for practitioners who may be looking for a behavior-oriented scale to evaluate IT personnel. The psychometric properties of the instrument as observed in this study are also presented.
Content uploaded by Michael A. Chilton
Author content
All content in this area was uploaded by Michael A. Chilton on May 01, 2014
Content may be subject to copyright.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Assessing Information Technology Personnel: Toward A Behavioral Rating Scale
Michael A Chilton; Bill C Hardgrave
Database for Advances in Information Systems; Summer 2004; 35, 3; ABI/INFORM Global
pg. 88
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
... It is not only the quality of the physical resources or the technology that counts, but the people and their skills and most importantly their commitments also make the difference. Allocation of the Information Systems Technology personnel's knowledge and skills is essential for improving performance related to operational activities and project management [22]. [24], [19] and [23] indicated that staff commitment such as cooperation, loyalty to organization, excellent team spirit, and job satisfaction and commitment, loyalty to supervisor and commitment to the organization are the measurements for the job performance. ...
... The regression morale test for the model generally shows that the calculated value of the test (fcal = 43.474) and the level of indication (Sig = 0.000) are less than the level of the specified morale (5%) As shown in table (22), It is inferred that there is a linear moral relationship between the independent variable (information systems Technology) and the intermediate variable (the organizational structure of the information). The relationship model therefore has efficient data reconciliation and can be used to represent the relationship between the study variables. ...
Use of information Systems in Yemeni Universities Future Vision
Article
Full-text available
  • Jan 2019
This research examines the use of computed information systems in Yemeni universities through the use of intermediate variable technology to test the indirect impact of the information systems structure on the relationship between information technology, and the final performance in the computed information systems in Yemeni universities. The applied situation was exemplified by the selection of four universities where the data on the study were collected through special friendly questionnaire, and the number of respondent (224) persons had been used. Many statistical means, percentages, arithmetic mean, standard deviation, and intermediate deviation have been used for the description of the study sample, and for the study variables, as for hypothesis testing, the intermediate variant method was used and the end result was to prove that there was a relationship An incomplete mediation attributed to the structure of information systems that has an indirect impact on the resulting relationship between information technology and the final performance of information systems.
View
... Wynekoop and Walz (1998) examined the differences in personality profiles of key IT personnel and found out that there are clear differences. Similar to this, is the work of Chilton and Hardgrave (2004), which assessed the behavioural skills (Technical and Managerial) and determined the relationship between these skills and personality traits. Houghton (2004) introduced a theoretical framework for understanding the role of personal traits in collaboration in virtual contexts. ...
An Improved Assessment of Personality Traits in Software Engineering
Conference Paper
Full-text available
  • Jan 2007
The success or otherwise of Software Engineering (SE) activities depends on the interactions among software engineers. Consequently, effective interactions depend largely on personality traits, which is a consistent and long-lasting tendency in behaviour. In psychology, five major trait factors (The Big Five Factors) have been generally used to assess personality of people. But, these might not be adequate in SE because of the required technical and cognitive skills. In this work, we first present Cognitive Ability as an additional factor that must be measured in order to adequately assess personality in SE. A research survey was conducted in order to capture personality requirements in SE. Based on the result of the survey conducted, we develop a model for assessing personality traits in SE. We then design an assessment technique that is based on responses to some well-structured and deductive on-line questions. The implementation of the model using Visual Basic resulted in a much-needed tool that can guide intending software engineers in choosing area of specialization in SE based on their personality traits.
View
... about here A measure of consistency of each item and each construct was estimated by reliability. The reliability of each item was estimated by squaring the standardized factor loading of the item and the composite reliability of each construct(Chilton and Hardgrave, 2004) ...
View
Personnel Performance Appraisal Coverage in ITIL, COBIT and CMMi
Chapter
  • Jan 2012
Human capital is of paramount importance in the ICT industry. This paper analyses one of the elements which underlies effective management of human assets: performance assessment. This analysis uses the development of a common standard based on the tasks and processes outlined in People CMM, the standard in human resource management designed by the Software Engineering Institute. This framework is evaluated in different models related to IT Governance, such as COBIT and ITIL, as well as other related IT practices like CMMI for Development, CMMI for Acquisition, and CMMi-SVC. The results indicate that COBIT is the only framework that presents reliable coverage in relation to personnel performance assessment.
View
View
Designing games for dementia care at pre and early stages
Thesis
  • Jul 2019
Dementia is emerging as a major public health concern. Nevertheless, it is still very challenging to the families of people who are diagnosed with dementia - especially the main carers, as the stereotype of dementia in society remains negative. Games for family carers of patients at pre- and early-stages of dementia are designed to engage people, and are valued for their educational benefits, health benefits and community enhancing merits. A conceptual framework is proposed in this thesis to help develop these types of games to support family carers of patients at pre- and early-stages of dementia. The aim of the research is to increase knowledge of health issues in order to help maintain the relationship between patients and carers, since the quality of this relationship will have an effect on both of them. By playing the games, carers can understand more about their patients to change their behaviour and thoughts so they can better support them. The game-based methods for dementia carers include social marketing, knowledge transfer, social media connections and game involvement. The research methods used to test this hypothesis include expert interviews and games analysis. Expert interviews were conducted with a sample of professionals, including organisations and academia, to address the two main areas: carers and digital technologies. The games analysis involved using measures of the effectiveness of the game’s purpose, such as games for health, games for education and social games. Metric instruments were developed to measure the game’s purpose. Exploratory experiments were performed on a selection of games using experienced gaming professionals. The game Make a cup of tea was designed and developed based on the framework and the exploratory experiments results. The use of Make a cup of tea showed improvements to the quality of life of family carers of patients at pre- and early-stages of dementia. This thesis will contribute to the research on dementia family carers through several aspects: the proposed framework will serve as a proven theoretical concept for future research; the metric instruments provide benchmarking measures for health-based games; Make a cup of tea educates and enlightens dementia family carers through in-game experiences. The aim of the project is to help maintain the relationship between dementia patients and their carers which heavily influences the quality of life for both parties. From the design aspect, this project aims to explore ways to design games for dementia research purposes.
View
The effects of relative and criticism-based performance appraisals on task-level escalation in an IT project: a laboratory experiment
Article
Full-text available
  • Feb 2018
While performance appraisals provide an opportunity for managers to review employee performance, prior research points to potential negative consequences associated with performance appraisals. Specifically, prior IS research suggests that relative and criticism-based appraisals can have an undesirable impact on employee behaviour. Motivated by this, in this study we theorised and empirically investigated how relative and criticism-based appraisals can influence an IT project member’s tendency to continue working on a task despite encountering serious setbacks (i.e., task-level escalation). To shed light on the underlying mechanisms through which appraisals influence task-level escalation, we proposed and tested two mediating pathways based on goal orientation theory and prospect theory. In a laboratory experiment, we found that a relative appraisal led to greater task-level escalation as compared to an absolute appraisal. Proving goal orientation and avoiding goal orientation mediated this relationship. In addition, we found that a criticism-based appraisal led to greater task-level escalation as compared to a praise-based appraisal. Risk propensity and self-esteem mediated this relationship. Lastly, we found that the effects of both a relative appraisal and a criticism-based appraisal on task-level escalation were significantly different from a control group that did not involve any form of performance appraisal.
View
Measuring the intellectual capital of individuals: A Key Personal Competency Measurement Framework (KPCMF)
Article
  • Jan 2008
Current literature and empirical evidence highlights the growth of methods and frameworks used to measure Intellectual Capital (IC) at organisational levels but the IC of the individual, and the ability of the individual to create and exploit knowledge is often ignored. This paper contributes to this body of work, through the development of a Key Personal Competency Measurement Framework (KPCMF) that can measure, manage and inform the development of an individuals IC. The paper presents a view of IC emanating from a myriad of activities at lower levels of the organisational structure where careful development of these atomistic components is required for IC to grow and become a significant asset to the individual and organisation. The research was undertaken as an IS professional bringing a comparative perspective informed by experience; a role consistent with that of the social theorist. Content analysis was used to identify and define Key Personal Competencies (KPC's) whilst a survey of IS executives provided external validation. In its first application the KPCMF is applied to the Information Systems Development profession where increasing complexity and criticality of information systems, combined with the growing number of tools and methods, are causing the IS profession to question which competencies are needed to work effectively. This research aligns other researcher's findings in similar ways: The non-financial Balanced Scorecard perspectives and three types of IC (Human, Structural and Relationship) to provide a complementary means of categorising Key Personal Competencies; the value of job market knowledge informing professional development; and the acquisition of new relationship skills. This paper illustrates that a generic model is possible to measure IC, contrary to the views of Shulver et al (2000), but its value, relevance and ownership is only released by aligning it to the context in which it will be used.
View
Curriculum change and alignment with industry: The student perspective
Article
  • Jan 2007
This chapter presents the design and results of a study that focuses on students as stakeholders in the education process. A general framework based on a nomological net is introduced and used to derive the research models underpinning data gathering and the subsequent data analysis. The findings indicate that students realistically evaluate the gaps in their learning but put more emphasis on technical skills, ignoring or undervaluing soft and business skills despite academic efforts to develop these through skill-centered teaching. It was also found that a mismatch existed between student expectations of required skills and skills demanded by employers, to some extent exacerbated by the content of job advertisements.
View
A Competency Model for the Information Technology Workforce: Implications for Training and Selection
Article
  • Jan 2010
With a new wave of demand for information technology (IT) staff, it is time for the IT community to discover what competencies are needed for IT workers in the new millennium. Most organizations have outsourced their IT projects to low-cost centers in developing countries, and this has changed expectations about local workers-organizations need, and expect, to employ highly skilled IT professionals. This article draws on the competency model to explore what competencies are needed to successfully equip the IT workforce. This research uses the example of IT architects to illustrate how to use the competency model to identify important sets of competencies. We interview professional architects, analyze the content of the discussions and derive a list of fourteen competency attributes important to IT architects. Some competencies are easy to develop, others are difficult to change, and this has implications for university education and industry hiring. This article also discusses the general use of competency models for IT workforce training and selection.
View
View
View
Structural Equation Modeling in Practice: A Review of Recommended Two-Step Approach
Article
Full-text available
  • May 1988
In this article, we provide guidance for substantive researchers on the use of structural equation modeling in practice for theory testing and development. We present a comprehensive, two-step modeling approach that employs a series of nested models and sequential chi-square difference tests. We discuss the comparative advantages of this approach over a one-step approach. Considerations in specification, assessment of fit, and respecification of measurement models using confirmatory factor analysis are reviewed. As background to the two-step approach, the distinction between exploratory and confirmatory analysis, the distinction between complementary approaches for theory testing versus predictive application, and some developments in estimation methods also are discussed.
View
Dutch Reactions to Behavioral Observation, Behavioral Expectation, and Trait Scales
Article
Full-text available
  • Sep 1995
Dutch managers and computer programmers appraised the performance of a subordinate or peer, respectively, using Behavioral Observation scales (BOS), Behavioral Expectation scales (BES), and Trait scales, and then evaluated the three appraisal instruments on eight criteria: ability to give feedback, ability to differentiate, objectivity, position differences, ability to provide training, setting corporate-wide standards, ability to set goals, and overall ease of use. Users preferred the BOS to the BES on seven criteria, and to the Trait Scale on all but two criteria. They preferred the Trait Scale to the BES on two criteria, namely, ease of use and position differences, and considered the BES and Trait scale equivalent on the remaining criteria. These findings parallel those found in a similar study conducted in the United States.
View
View
Structural equation modeling in practice: A review and recommended two-step approach.
Article
  • Jan 1988
In this article, we provide guidance for substantive researchers on the use of structural equation modeling in practice for theory testing and development. We present a comprehensive, two-step modeling approach that employs a series of nested models and sequential chi-square difference tests. We discuss the comparative advantages of this approach over a one-step approach. Considerations in specification, assessment of fit, and respecification of measurement models using confirmatory factor analysis are reviewed. As background to the two-step approach, the distinction between exploratory and confirmatory analysis, the distinction between complementary approaches for theory testing versus predictive application, and some developments in estimation methods also are discussed.
View
View
Are Information Systems People Different? An Investigation of How They Are and Should Be Managed
Article
  • Sep 1988
This study, based on seven different samples involving 1005 employees, examines whether IS and non-IS people are or should be managed differently. How IS and non-IS people are managed is measured by three sets of managerial activities: (1) enriching the job, (2) attending to interpersonal relations, involving the employee, and reinforcing work behavior, and (3) attending to production and targeting work behavior. Two research questions are asked: (1) Do work-unit environments differ for IS and non-IS people? (2) Is the relationship of work-unit environment to productivity different for IS and non-IS people? The findings of this study support the conclusion that IS and non-IS employees at the same occupational level are not and should not be managed differently.
View
Structural Equations with Latent Variables.
Article
  • Jan 1991
Model Notation, Covariances, and Path Analysis. Causality and Causal Models. Structural Equation Models with Observed Variables. The Consequences of Measurement Error. Measurement Models: The Relation Between Latent and Observed Variables. Confirmatory Factor Analysis. The General Model, Part I: Latent Variable and Measurement Models Combined. The General Model, Part II: Extensions. Appendices. Distribution Theory. References. Index.
View
Perceived Importance of Systems Analysts' Job Skills, Roles, and Non-Salary Incentives
Article
  • Jun 1989
System analysts are service providers who are required to work closely with users for the purpose of defining, developing and implementing computer-based systems. Analysts and users in private organizations may have different expectations and proficiencies than those in public organizations, in part due to the types of applications required. Beliefs about how others are supposed to perform and what motivates them contribute to a variety of behavioral responses. Conflict between analysts and users may have serious consequences that can be very costly, such as poorly developed systems, behavioral dysfunctions (e.g., mistrust, avoidance, rejection), and negative user satisfaction. An interesting research question is whether perceptual differences exist among systems analysts and users about how systems analysts perform their jobs, as well as whether the perceptions are the same for public and private organizations. In a survey of perceptual differences about job skills, job roles, and non-salary incentives of systems analysts, results from872 questionnaires show that analysts and users differ significantly in their perceptions of skills and roles for systems analysts. Public and private systems analysts and users differ significantly on perception of all three measures. The results provide evidence that analysts, more so than users, recognize the importance of behavioral skills for effective development. This difference may be a major source of conflict, with users expecting analysts to exhibit technical skills in situations where behavioral skills are required. Public and private sector differences suggest that even though the process of systems development may be very similar, users and analysts in public organizations may, in fact, be different than their counterparts in private organizations. An exploration of these sector differences should be addressed by future research. Information systems managers may use the results to guide educational programs for users, develop better assessment measures for analysts, and establish better mechanisms for providing important non-salary incentives for analysts.
View