Neither the Union nor the Confederate armies had regiments composed exclusively of convicts. The first year of the war drew only volunteers on both sides, though many of these men enlisted in response to social coercion from friends, family, or community leaders. In April 1862, the Confederacy passed its Conscription Act and in March 1863 the Union did the same. These acts pulled in only men of military age who were physically fit and outside the exemption categories. Neither act allowed men to be released from jail or prison in order to serve in the armies. That said, it is possible that communities that could not meet their quotas in 1863 may have released men from light sentences early if they obligated themselves to military service. But these men would have been folded into regular conscript regiments, which included a wide array of men from different backgrounds.

Contemporary cinematic treatments of the war (especially the popular Gangs of New York) leave the impression that Northern units filled their ranks with recently arrived immigrants or other people who did not understand the rules of enlistment. While many immigrants served, the immediate enlistment of new arrivals pictured in the film overstates the reality considerably. One real element that could have generated the impression of regiments of convicts was war-time prosecutions. Both Union and Confederate officers punished their men for transgressions of military law. Infractions became particularly common when troops camped near urban areas or occupied cities for long periods of time. The traditional mix of bar, gambling house, and brothel attendance that often accompany armies created a climate ripe for eager officers to arrest, prosecute, and punish transgressors. Certain units developed reputations for lower-class crimes such as theft, drunkenness, and fighting, which could have well colored the impression of other soldiers about the entire regiment.

Other units compiled lengthy lists of men who violated battlefield rules—from cowardice to desertion. One regiment of this sort was the 19th New York Infantry, organized in Cayuga County. After serving their original three-month enlistment term, the unit was remustered for two years. According to the New York State Military Museum and Veterans Research Center, "these orders were received with open dissatisfaction and by refusal of obedience on the part of 206 members, 23 of the greatest offenders being sent to the Dry Tortugas and the others placed under arrest until they were ready to be remustered. The 23 were finally released on condition that they serve the remainder of the two years in the 2nd N.Y. infantry." These men were not convicts going into the war, but became so in response to what they regarded as unfair treatment. Though a small number, their actions may well have been enough to color perceptions of the regiment as a whole.

Another element that might have contributed to a perception that some regiments were filled with criminals was the observations made by officers and upper-class men who looked down their noses at their own units. Class distinctions, especially between officers and enlisted men, contributed to an assumption that criminals and other lowlifes filled the ranks. This was true in both the Union and Confederate armies. William Elzey Harrison, an officer in Lee's Engineer Corps, conveyed this sentiment quite clearly in a letter to his brother in late 1862. "I would not have the captaincy of the company I was assigned to," he wrote, "as the greater part of it was composed of the offscourings of Richmond, Norfolk, &c. I would have nothing to do with such a lot of men.” The "offscourings" in this case were most likely working men from the docks of these port cities or perhaps occasionally employed sailors and watermen. Undoubtedly, these were a lower class of men than William Harrison, but probably not all convicts.