Yesterday I made some tests after your post (unluckily I could upload my reply, don’t know why).
Yes the website was quite slow but not because of the plugin.
The reason (probably )were the hosting provider and the gzip compression not enabled.
If you want to check how much the plugin needs to load I suggest you to add a new page to your website, add a newsletter form and then make 5/10 tests in different hours of the day with tools like GTmatrix or Pingdom to see what’s the bandwith needed to load the plugin (and the relative time).
Always mind that the time depends on several factors.