Custom editor function

Since I am working on a plugin for a Wordpress website, I really like the embed link option in the WordPress editor. I like the option so much, that I would like to create a similar option but then slightly else.

The option I talk about is:

What i want is the same, but then with my own popup. How can I do this, in which files can I add this?

This is what I want

Thanks in forward!

via Chebli Mohamed