SQL 2012 :: Removing IP Address From Listener Pool When Demoting One Of Remote Sites

We have a multi-site AG and are demoting one of the remote sites out of the AG. In doing so, we've discovered that no logic exists to remove an IP address from the listener.

It seems that the ALTER AVAILABILITY GROUP MODIFY LISTENER logic lacks functionality other than ADD. [URL] ....

I'm afraid to just remove the IP address from the cluster object as that IP is also stored in the HADR systables.

select * from sys.availability_group_listener_ip_addresses
select * from sys.availability_group_listeners