Most preliminary IPIP scales were developed with a mixed external - rational/intuitive - internal strategy (Goldberg, 1972; Johnson, 2001, 2014), which is rather common in personality assessment. Here is a general description of this process:
1) Correlate all of the IPIP items available at that time (e.g., there are now over 3,000 items, but obviously there were less available at earlier times) with the external criterion (e.g., the 20-item adjectival marker scale measuring Big-Five Extraversion), and rank order the IPIP items by the size of the absolute value of those correlations.
1a) When developing IPIP scales targeted on constructs from a multi-scale inventory, correlate all IPIP items available at that time with each of the inventory scales, categorize each IPIP item by the scale with which it has its highest correlation, and rank order the IPIP items within each of the resulting categories by the size of those correlations. This will ensure that all of the IPIP items selected for an IPIP scale correlate more highly with its criterion scale than with any of the others.
2) Select the N highest positively correlating IPIP items and the N highest negatively correlating IPIP items for the preliminary scale, with N being ½ the number of items that are desired in the final scale (e.g., 5 + 5 = 10).
2a) If the correlations with the original scale are substantially higher within the set correlating positively than are the correlations within the set correlating negatively, or vice versa, then relax the criterion for equal numbers of positively and negatively keyed items, by trying to balance equal keying direction with high strength of association.
3) Examine the content of the IPIP items selected, noting any item pairs that are essentially identical in content. Omit the lowest correlating item from such redundant pairs. If any item is omitted using this redundancy criterion, add another most highly correlating IPIP item to the set from which it was omitted.
3a) Does the content of all of the selected items tell a coherent story? Are there any items that do not mirror the major story-line? If so, omit any such "subtle" item and add a new one from the set of most highly correlating items.
4) Carry out a Reliability analysis of the items in the scale. Find any item whose addition to the scale actually serves to lower the Coefficient Alpha reliability of the resulting scale. Omit that item, substituting another item from the set of most highly correlating items. Continue this process until Alpha is as high as is reasonable, without sacrificing too much breadth of content. This last stage requires some ingenuity, and thus this is the stage where an exact algorithm would be difficult to formulate.
These IPIP scales are labeled "preliminary" because they should be able to be improved by the use of more sophisticated procedures, such as those based on Item Response Theory (IRT). Members of the worldwide research community are invited to use IRT models and any other techniques to improve the quality of the preliminary IPIP scales.
A few IPIP scales were constructed without correlating items with external criteria. In some cases, existing IPIP items whose content was judged to represent the same construct were assembled into a scale, and the coherence of the scale was verified by internal item intercorrelations. The Physical Attractiveness scale was constructed this way. In other cases, new IPIP items were created by revising items from an existing scale. For example, an IPIP depression scale was created by rewording 20 items from the Center for Epidemiologic Studies Depression Scale (e.g., "I was bothered by things that usually do not bother me" became "Was bothered by things that usually do not bother me") and writing four new items. Two items with relatively low loadings on the first unrotated principal component were dropped, leaving 22 items on the final scale.
All of the scales that were constructed a rational/intuitive and internal consistency strategy are listed on the Single Constructs page under the heading "Constructs for which only IPIP items were administered to the ESCS."
Goldberg, L. R. (1972). Parameters of personality inventory construction and utilization: A comparison of prediction strategies and tactics. Multivariate Behavioral Research Monograph, 7, No. 72-2.
Johnson, J. A. (2001). Methods: personality psychology. In N. J. Smelser & P. B. Baltes (Eds.), International Encyclopedia of the Social and Behavioral Sciences. 16, 11313-113117. Amsterdam and New York: Elsevier. doi:10.1016/B0-08-043076-7/01671-5
Johnson, J. A. (2014). Measuring thirty facets of the five factor model with a 120-item public domain inventory: Development of the IPIP-NEO-120. Journal of Research in Personality, 51, 78-89. DOI: 10.1016/j.jrp.2014.05.003