This study examines the reliability and the validity of the Korean National Assessment of Educational Achievement (NAEA) tests since they began to be administered nationwide. In addition, it suggests some improvements to item development by analyzing those characteristics of the items that lower the reliability and the validity of the test. Using the data from 2008～2010 elementary, middle, and high school Korean NAEA tests, the reliability coefficient (Cronbach's alpha), Cronbach's alpha if item deleted, item discrimination, and factor loadings by factor analysis were examined. The results showed that the internal consistency of all the Korean NAEA tests was high：eight items from six of the nine tests lower the Cronbach's alpha. Also, fourteen items from the nine tests had low factor loadings; that is, the correlation coefficients between the items and the factor (Korean ability). These items had both relatively low item discrimination in common and factors which distracted students from answering correctly. These items can be categorized into four types：first, they have limited information；second, they have imprecise statements and ambiguous content；third, they have inappropriate difficulty levels for the students to whom they are administered；and fourth, they have improper factors for the language style；any and all of these may mislead students into answering incorrectly.