Swarowsky was born in Budapest, Hungary. He studied the art of conducting under Felix Weingartner and Richard Strauss. His teachers in musical theory included Arnold Schoenberg and Anton Webern.

Herbert von Karajan invited him to take on the permanent position as conductor of the Vienna State Opera.

He became a professor of conducting at the Vienna Music Academy. His many conducting students included Claudio Abbado, Mariss Jansons, Alexis Hauser, Alexander Alexeev, Zubin Mehta, Leonid Nikolaev, Paul Angerer, Ádám and Iván Fischer, Avi Ostrowsky Jesús López-Cobos, Gustav Meier, Ewa Michnik, Miltiades Caridis, Aleksandr Alekseyev, Giuseppe Sinopoli, Gianluigi Gelmetti, Brian Jackson, Alfred Prinz, Bryan Fairfax, James Allen Gähres, Albert Rosen and Bruno Weil. Otmar Suitner was Hans Swarowsky's successor at the Vienna Music Academy. Swarowsky's lectures and essays were collected into the publication Wahrung der Gestalt (Keeping Shape), which today serves as an encyclopaedia for performance and conducting. From 1957 to 1959 he was chief conductor of the Scottish National Orchestra (now the Royal Scottish N.O.).

He died in Salzburg, Austria, less than a week before his 76th birthday.