I recently did this quest, and my party eventually figured out that if we clear the way first and then walk backwards while keeping Coyle in our line of sight that he will make his way to the entrance of the library, although he did so in short bursts of movement with long pauses between movements, and it took a minute for him to walk the last 10 feet or so to trigger the cut scene where he finds that the door is locked.
That said, when we went to go deal with those earth elementals, he followed part way and almost got himself killed.
We did the Western Expedition afterwards and had zero problems with Derward. So Coyle was probably just being an ******* like he usually is and influenced the Deneith guards to also be ********.
Btw, when we did Hold for Reinforcements, Coyle was almost instantly KO'd by a reaper. I'm very glad they changed the quest so we don't have to keep his butt alive.