The merger of the publishing giant with rival Simon & Schuster was scuppered by competition concerns.