Harkavy was born in Novogrudok, Belorussia. He had a traditional Jewish education and showed an early interest in languages, acquiring some knowledge not only of Hebrew but also of Russian, Syriac, German, and—particularly-Yiddish in his teens. In 1878 Harkavy went to Vilna, where he was befriended by the Yiddish author Isaac Meir Dick. He wrote his first work in Yiddish. He earned a living as a bookkeeper for Romm, the Hebrew-Yiddish publishing house. After the pogroms of 1881 Harkavy joined the Am Olam movement and emigrated to the United States. He found what work he could, as a stevedore, a farm laborer, a dishwasher, learning English intensively and then tutoring English and Hebrew privately.

Harkavy's love of Yiddish soon crystallized into a vocation. In 1891, his first popular textbook, Englishe Lerer, was published. Almost 100,000 copies were sold. Through this and other books in the "English self-taught" genre, such as his guide to writing letters, Englishe Brifnshteler (1892); through his Yiddish translations of classics; through his classroom lectures and popular expositions of American history and culture; and above all through his Yiddish dictionaries, he became the teacher par excellence of two generations of immigrants. He taught U.S. history and politics for the New York Board of Education and Yiddish literature and grammar at the Jewish Teachers' Seminary in New York, and he lectured as well for the Workmen's Circle. Harkavy wrote a column called "Kol-Boi" ("Everything in It") for the Abend-Post and occasional articles for many Yiddish, Hebrew, and English papers and journals. Harkavy's most lasting achievements were in lexicography. His English-Yiddish and Yiddish-English dictionaries, encompassing about 40,000 Yiddish words, went through at least 22 editions and reprints. His crowning work was the Yiddish-English-Hebrew Dictionary (1925; fourth reprint, 1957).