OK, replying to myself.
Finally found the way to do this : I marked my main list as Public, the other two (members and staff) as “Private” (the subscriptions there will be handled manually by the admin).
I also had to go into “members – profile page” to add the “lists” fieds into the form (I think this was the part that was missing in my previous trials).
Of course there is only one public list, so only one box to check-uncheck. I also removed the “unsubscribe” link, as I fear this might cause confusion by silently unsubscribing people from ALL lists (even hidden ones).
I adjusted the wording on that page to explain that they can there uncheck the box to unsubscribe (from the public list) or change the email address.
Works for us, thanks.