Cengage Learning Holdings II Inc. and McGraw-Hill Education Inc. mutually agreed to abandon their plans to merge after the Department of Justice informed the companies it had serious concerns that the proposed transaction, as structured, would harm competition.