Ruffins attended Cooper Union in New York. In 1954, he co-founded Push Pin Studios with fellow Cooper graduates Glaser, Sorel, and Chwast. In 1963, after leaving Push Pin, Ruffins founded another design studio with Simms Taback, a partnership which lasted for more than thirty years. Commercial clients included IBM, AT&T, Coca-Cola, CBS, Pfizer, the New York Times, Time Life, Fortune, Gourmet Magazine, and the U.S. Post Office.
Ruffins entered the field of children's book illustration in 1969, and throughout the 1970s and early 1980s he frequently collaborated with writer Jane Sarnoff.
In 1991, Ruffins teamed with Whoopi Goldberg and Herbie Hancock to produce "Koi and the Kola Nuts," a highly praised video for children which was part of Rabbit Ears Productions' We All Have Tales series.
Ruffins lives in Sag Harbor, New York, with his wife Joan.
A professor emeritus at CUNY's Queens College, Ruffins has also taught at the School of Visual Arts, Parsons The New School for Design, and was a visiting adjunct professor at Syracuse University.
Ruffins was the recipient of the Augustus St. Gaudens Award (presented by Cooper Union) for outstanding professional achievement in the arts. The Cooper Union Presidential Citation was also presented to Ruffins for his work and prominence in his profession. Ruffins' work led to awards from the New York Art Directors Club and a Silver Medal from the Society of Illustrators. Ruffins won the Coretta Scott King Illustrator Honor Award in 1997 for the book Running the Road to ABC (written by Denize Lauture).