Hertz was born in Rebrény, Hungary (now Slovakia), and immigrated to New York City in 1884. He was educated at New York City College (BA), Columbia University (PhD) and the Jewish Theological Seminary of America (Rabbi, 1894, the Seminary's first graduate). His first Ministerial post was at Syracuse, New York.

In 1898, he moved to (Transvaal), South Africa, to the Witwatersrand Old Hebrew Congregation in Johannesburg. He stayed there until 1911, despite attempts by President Paul Kruger in 1899 to expel him for his pro-British sympathies and for advocating the removal of religious disabilities of Jews and Catholics in South Africa. He was Professor of Philosophy at Transvaal University College (later known as the University of the Witwatersrand), 1906-8.

In 1911, he returned to New York to the Orach Chayim Congregation.

In 1913, R. Hertz was appointed Chief Rabbi of the United Hebrew Congregations of the British Empire; his rival candidates had included R. Moses Hyamson, R. Lewis Daly, and R. Bernard Drachman. R. Hertz held the post until his death. His period in office was marked by many arguments with a wide variety of people, mainly within the Jewish community; the Dictionary of National Biography describes him as a "combative Conservative". It was said that he was in favour of resolving disagreements by calm discussion - when all other methods had failed.

Chief Rabbi Hertz, Despite his title, he was not universally recognized as the final rabbinical authority, even in Britain. While he was Chief Rabbi of the group of Synagogues known as the United Hebrew Congregations of the British Empire, led by the United Synagogue, some new immigrants who had arrived since the 1880s regarded it as not orthodox enough. Hertz tried both persuasion and such force as he could muster to influence them; he added to his credibility among these immigrants by persuading R. Yechezkel Abramsky to become head of the London Beth Din.

Hertz antagonized others by his strong support for Zionism in the 1920s and 1930s, when many prominent Jews were against it, fearing that it would lead to accusations against the Jewish community of divided loyalty. Hertz was strongly opposed to Reform and Liberal Judaism, though he did not allow this to create personal animosities, and had no objection in principle to attending the funerals of Reform Jews, whom he described as "His Majesty's Loyal Opposition".

He was ex officio President of Jews' College, and Acting Principal, 1939-45. He was President of the Jewish Historical Society of England, 1922-3, and of the Conference of Anglo-Jewish Preachers. He was on the Board of Governors of the Hebrew University of Jerusalem and Chairman of the Governing Body of its Institute of Jewish Studies. He was Vice-President of a wide variety of Jewish and non-Jewish bodies, including the Anglo-Jewish Association, the London Hospital, the League of Nations Union, the National Council of Public Morals and King George's Fund for Sailors. In 1942, with the Archbishop of Canterbury, William Temple, Chief Rabbi Hertz founded the Council of Christians and Jews to combat anti-Jewish bigotry.

His daughter Judith married Rabbi Dr Solomon Schonfeld.

His great granddaughter is the writer Noreena Hertz.

In the 1920s, Hertz successfully organized international opposition to proposed calendar reform. The League of Nations was considering a calendar amendment, such that a given date would fall on the same day of the week every year. This requires that one day every year (two in leap years) is not any day of the week but a "world day". Thus, once or twice a year there would be eight days rather than seven between consecutive Saturdays. Thus the Jewish Sabbath, which must occur every seventh day, would be on a different weekday each year. The same applies to the Christian Sabbath. Hertz realized that this would cause problems for Jews and Christians alike in observing their Sabbaths, and mobilized worldwide religious opposition to defeat the proposal.

Hertz edited notable commentaries on the Torah (1929-36, one volume edition 1937) and the Jewish Prayer Book or Siddur (1946). He also contributed to the Jewish Encyclopedia and the Encyclopedia Britannica.