Caoimhin Agyarko was heartbroken when a title eliminator fell through and feared being sidelined before his world title fight against Josh Kelly was arranged.