Mertz was born in Pozsony. He was active in Vienna which had been home to various prominent figures of the guitar, including Anton Diabelli, Mauro Giuliani, Wenceslaus Matiegka and Simon Franz Molitor. A virtuoso, he established a solid reputation as a performer. He toured Moravia, Poland, and Russia, and gave performances in Berlin and Dresden.

Pozsony is now the Slovakian capital, Bratislava, where his only notable memorial is an annual competition in his name. He was a child prodigy on the guitar and flute. Little is known of his early life but by 1840 he was ensconced in Vienna, enjoying royal patronage and touring widely in Europe. In 1842 he married Josephine Plantin, a pianist whose unwise administration of strychnine in 1846 aggravated an illness from which he recovered eighteen months later.

In 1846 Mertz nearly died of an overdose of strychnine that had been prescribed to him as a treatment for neuralgia. Over the following year he was nursed back to health in the presence his wife, a concert pianist, Josephine Plantin whom he married in 1842. Some speculation may lead one to the conclusion that listening to his wife performing the Romantic piano pieces of the day during his period of recovery may have had an influence on the sound and unusual right hand technique he adopted for the Bardenklange (Bardic Sounds) Op.13.

Mertz's guitar music, unlike that of most of his contemporaries, followed the pianistic models of Chopin, Mendelssohn,Schubert and Schumann, rather than the classical models of Mozart and Haydn (as did Sor and Aguado), or the bel canto style of Rossini (as did Giuliani).

The Bardenklänge (1847) are probably Mertz's most important contribution to the guitar repertoire — a series of deceptively difficult character pieces in the mould of Schumann.

He died shortly before his magnum opus, the fifteen-volume Bardenklänge was awarded the First Prize in a competition in Brussels, organised by his great admirer Nicolai Petrovich Makaroff.