Olivier began studying classical piano at the age of four. He obtained a doctorate in 1978 for a thesis on urban music in Ghana, directed by cinematographer and ethnologist Jean Rouch.
It was Oscar Peterson’s discovery that led him to devote himself to jazz. He started in 1975 with the group Moravagine, which he formed with Denis Barbier and Mino Cinelu. From 1975 to 1977, he participated in the formation of jazz-rock Chute libre with the brothers Cinelu. Later (1980-1982), he played keyboard in a quartet by Christian Escoudé.
Olivier subsequently participated in numerous musical projects. His collaborators include Tony Scott, Art Farmer, Didier Lockwood, Clifford Jordan, Dee Dee Bridgewater, Turk Mauro, Pepper Adams, Harry Edison, James Moody, Toots Thielemans, Philip Catherine, Eddy Louiss, Éric Le Lann, Joe Lee Wilson, Mundell Lowe, Ted Nash, Jimmy Gourley, Junior Cook, Frank Wess, Marc Bertaux, Claude Barthélemy, Denise King or Tony Rabeson.