Total views : 3351

Assessment Engineering Task Model Maps, Task Models and Templates as a New Way to Develop and Implement Test Specifications

Abstract


Assessment engineering is a new way to design and implement scalable, sustainable and ideally lower-cost solutions to the complexities of designing and developing tests. It represents a merger of sorts between cognitive task modeling and engineering design principles—a merger that requires some new thinking about the nature of score scales, item difficulty, and content. This article summarizes some recent developments in developing AE task model maps, task models, and templates as alternative to more traditional test specifications, and discusses some of the necessary quality control mechanisms that can ensure the consistent production of high quality items and test forms over time.

Keywords

Assessment Engineering, Item templates, Evidence centered design

Full Text:

 |  (PDF views: 781)

References


  • Anastasi, A. (1986). Psychological testing (6th ed.). New York, NY: Macmillan.
  • Bejar, I., Braun, H. I., & Tannenbaum, R. J. (2007). A prospective, predictive, and progressive approach to standard setting. In Lissitz, R. (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting. Maple Grove, MN: JAM Press.
  • Bejar, I. I., & Yocom, P. (1991). A generative approach to the modeling of isomorphic hiddenfigure items. Applied Psychological Measurement, 15(2), 129-137.
  • Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals, by a committee of college and university examiners. Handbook 1: Cognitive domain. New York, NY: Longmans.
  • Geerlings, H., Glas, C. A. W., & van der Linden, W. J. (2011). Modeling rule-based item generation. Psychometrika, 76, 337-359.
  • Glas, C. A. W., & van der Linden, W. J. (2003). Computerized adaptive testing with item cloning. Applied Psychological Measurement, 27, 247–261.
  • Guion, R. M. (1977). Content validity: The source of my discontent. Applied Psychological Measurement, 1, 1-10.
  • Hambleton, R. K., & Pitoniak, M. P. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433-470). Westport, CT: American Council on Education and Praeger.
  • Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education and Praeger.
  • Kintsch, W. (1988). The role of knowledge in discourse comprehension construction-integration model. Psychological Review, 95, 163-182.
  • Luecht, R. M. (2006). Engineering the test: From principled item design to automated test assembly. Invited paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, Texas.
  • Luecht, R. M. (2007, October). Assessment engineering: An integrated approach to test design, development, assembly, and scoring. Invited keynote and workshop presented at the Performance Testing Council Summit, Scottsdale, AZ.
  • Luecht, R. M. (2008a, February). Assessment engineering. Session paper at Assessment Engineering: Moving from Theory to Practice, Coordinated panel presentation at the Annual Meeting of the Association of Test Publishers, Dallas, TX.
  • Luecht, R. M. (2008b, February). The application of assessment engineering to and operational licensure testing program. Paper presented at the Annual Meeting of the Association of Test Publishers, Dallas, TX.
  • Luecht, R. M. (2008c, October). Assessment engineering in test design, development, assembly, and scoring. Invited keynote address at the Annual Meeting of the East Coast Organization of Language Testers (ECOLT), Washington, D.C.
  • Luecht, R. M. (2009, June). Adaptive computer-based tasks under and assessment engineering paradigm. Paper presented at the 2009 GMAC Conference on Computerized Adaptive Testing, Minneapolis, MN (published online in Proceedings. http://www.psych.umn.edu/psylabs/catcentral/).
  • Luecht, R. M. (2010, April). Controlling difficulty and security for complex computerized performance exercises using assessment engineering. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Denver, CO. Luecht, R. M. (2011, March). Assessment design and development, version 2.0: From art to engineering. Invited, closing keynote address at the Annual Meeting of the Association of Test Publishers, Phoenix, AZ.
  • Luecht, R. M., Burke, M., & Devore, R. (2009, April). Task modeling of complex computerbased performance exercises. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Diego, CA.
  • Luecht, R. M., Dallas, A., & Steed, T. (2010, April). Developing assessment engineering task models: A new way to develop test specifications. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Denver, CO.
  • Luecht, R. M., & Masters, J. (2010, February). The efficiency of calibrating multiple-item templates and task models using a hierarchical calibration model. Paper presented at the Annual Meeting of the Association of Test Publishers, Orlando, FL.
  • Masters, J. S., & Luecht, R. M. (2010, April). Assessment engineering quality assurance steps: Analyzing sources of variation in task models and templates. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Denver, CO.
  • Messick, S, (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13- 103). New York, NY: American Council on Education and Macmillan.
  • Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
  • Mislevy, R. J. (2006). Cognitive psychology and educational assessment. In R. L. Brennan (ed.), Educational measurement (4th ed., pp. 257-305). Westport, CT: American Council on Education and Praeger.
  • Mislevy, R. J.; & Riconscente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 61-90). Mahwah, NJ: Lawrence Erlbaum.
  • Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67.
  • Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 61-90). Mahwah, NJ: Lawrence Erlbaum.
  • Shu, Z., Burke, M., & Luecht, R. M. (2010, April). Some quality control results of using a hierarchical bayesian calibration system for assessment engineering task models, templates, and items. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Denver, CO.
  • Webb, N. L. (April, 2005). Issues related to judging the alignment of curriculum standards and assessments. Paper presented at the Annual Meeting of the American Educational Research Association Meeting, Montreal, QB, Canada.
  • Wilson, M. (2005). Constructing measures. Mahwah, NJ: Lawrence Erlbaum.

Refbacks

  • There are currently no refbacks.