If there is still an active button on the toolbar, press it again to deactivate the tool.
Tap anywhere on the annotation to open the related menu.
Press "Remove" to delete the annotation.
Tip: If you want to delete multiple annotations at once, you can use the selection tool.