Psychometrics II 3Statistics

Java Python Psychometrics II Assignment 3

Due Date: March 22, 2023

Part I. Application of BILOG

Use the PISA 2009 data to conduct the following analyses.  There are 23 items in total. The sample size is 2905.

1. Provide the command file necessary to conduct a 3PL analysis in BILOG-MG. (2-points) 

2. Show the item statistics table and use classical item statistics to find the easiest and most difficult items. Provide the ICC plots for these two items by using IRT Graphics. (4-points)

3. Interpret the two correlation coefficients. Please include the formula for each correlation coefficient. What is the relationship between difficulty and correlation in classical item statistics? (4-points)

4. Obtain the item parameter estimates from the output files and denote clearly the column names of discrimination, difficulty, and guessing. (Put the corresponding discrimination, difficulty, and guessing parameters beside the BILOG output names.) (3-points)

5. Which statistics do we use to check the model fit? What conclusion do you draw for each item’s fit to the 3PL model? (2-points)

6. Provide the plots of the standard error function and the information function. What is the relationship between them? (4-points)

7. Find two examinees that correctly answered the same amount of items. Are their ability estimates the same? Why? (4-points)

Part II. Item Development Exercise: 3PL Model Questions

8. Develop six multiple-choice and three-short answer questions based on the IRT topics covered by March 22. Potential topics include the assumptions of the 3PL model, 3PL model calibration, 3PL model parameters, and assessing fit for 3PL models (including model comparison).

a) Please develop six multiple-choice Psychometrics II Assignment 3Statistics questions, including the possible answer choices.  Provide the correct response for each item. Each multiple-choice item is worth 1.5 points for a total of 9 points.

b) Please develop three short-answer questions for a potential graduate student preparing for a final IRT exam and provide a sample response. The sample response should demonstrate what should be included to receive full-credit. The length of the response can vary from two-to-three sentences to two-to-three pages. Please use your best judgment regarding how much information should be provided to demonstrate your depth of understanding and engagement with the topic. Each short-answer item is worth 2 points for a total of 6 points.

You are encouraged to consider engaging with the topic using material other than the lecture notes and textbook. For instance, you could find an article that discusses the theoretical foundations of the topic or uses the topic in an applied fashion. You could also find your own dataset and run an analysis applying the topic. Your short-answer item might include output from the analysis.  

Here is an example of a short-answer response question (corresponding to topics from the first assignment): “G² is the likelihood Ratio Test for nested models. This test is used to compare model fit for hierarchically related models. Explain what the term, ‘hierarchically related models’ mean within the context of both Rasch and IRT.”

Here is another example: “We know that IRT models are concerned with modeling observed responses. However, in working with empirical data one will, at times, encounter situations where some items do not have responses from all individuals in the calibration sample. How should missing responses be handled in 3PL models?”

For all the questions in part 8 a) and b), please list the topics you covered for each item directly below the item