Torbernite is an aqueous phosphate of copper and uranium. It is a very beautiful mineral in appearance, and it is even not the most toxic, but extremely radioactive.
Coloradoite aka mercury telluride is a rare mineral, semi-metal, representing a compound of mercury and tellurium. It can be found in the frozen lava flows and silver mines. Telluride mercury is very toxic and dangerous to humans.
Orpiment is an arsenic sulfide with all the consequences. Despite the gorgeous color, a mineral is extremely toxic and requires a fair amount of skill in mining.
Cinnabar is a mercury sulfide, the most common mercury mineral in the world. Own mercury is made of it as well as the red dye from the time of the Roman Empire. Mineral is obviously toxic, and in ancient times resulted in death of many miners.
Compounds of lead and mercury, arsenic and radioactive elements often turn into great looking multi-colored stones. They do not kill instantly, but the poison can penetrate into the body through the skin or lungs, if you just inhale particles of mineral dust. Prudence will not interfere even with a simple shuffling of ore.