From Battlestar Wiki, the free, open content Battlestar Galactica encyclopedia and episode guide
A Cylon detector is a scientific method of determining whether a subject is human or a humanoid Cylon.
Created by Dr. Gaius Baltar, the test is administered by subjecting a person's blood to plutonium embedded in a carbon nanotube matrix. A filter preferentially ionizes synthetic molecules; this is because Cylons are susceptible to certain kinds of radiation, such as that of the storm around Ragnar Anchorage. The plutonium was taken from one of Galactica's nuclear warheads, at the suggestion of Number Six (Bastille Day). The test results are simple: either "red", indicating Cylon, or "green", indicating human. Yet with a single test cycle lasting approximately eleven hours, Baltar estimates it will take over 60 years (not counting breaks for sleep, food, bathroom, etc.) to test the whole fleet (numbering almost 50,000 people) (Tigh Me Up, Tigh Me Down).
Development work on the Cylon detector had previously been halted when Dr. Baltar was accused by Shelly Godfrey of being a traitor and temporarily thrown in the brig (Six Degrees of Separation). Baltar's "innocence" was proven, however, by Lieutenant Felix Gaeta, who had previously been ordered by Commander Adama to assist Baltar in the development of the device (Water). Although considered a military secret, the Cylon detector is nevertheless the subject of constant rumor and innuendo (Act of Contrition).
Messenger Six is the one who initially suggests the construction of the Cylon detector by using the nuclear materials. Baltar was put in charge of the project because of his known genius and his earlier detection of the fleet's first Cylon infiltrator, Aaron Doral, which was also greatly influenced by the image of Number Six (Miniseries). Ironically, the Cylon detection method that Baltar claimed to have used on Doral was pure technobabble, unlike his later device, so it was very fortunate that he fingered an actual Cylon.
The Cylon detector does work. It confirms that Sharon Valerii is indeed a Cylon; a fact that Baltar hides from Valerii, fearing what her Cylon persona would do were he to tell her (Flesh and Bone). After discovering that it would be easier for him to make all the results "green", Baltar alters the device to mimic this method of thinking (Tigh Me Up, Tigh Me Down). However, after Sharon Valerii shoots Commander Adama, everyone believes the Cylon detector to be a failure, thinking that it didn't pick up her Cylon nature (Resistance, He That Believeth In Me).
Because of the abandonment of the detector, Dr. Sherman Cottle is limited to comparing the DNA of Daniel Novacek and Kara Thrace with their records to check that they are not Cylon duplicates, as such fakes would presumably have some genetic difference from the human originals (Hero, He That Believeth In Me). Tory Foster points out that this method is ineffective if the subject has always been a Cylon (indeed, none of the twelve Cylon models is a duplicate of a human).
The Cylons themselves have accurate and reliable protocols for determining the difference between humans and Cylons. Their testing methods determine that the Thirteenth Tribe were Cylons (Sometimes a Great Notion).