JSP :: Hide Sensitive System Properties Like Passwords Set By Java Applications?

I am maintaining an existing Java product (which has a HUGE code-base). I discovered that it is setting (and getting) two of its internal passwords as Java system properties, at no less than 4-5 different places (methods). Now, the problem is, the passwords are being stored as plain text in the Java system properties, and so, the same is visible to external entities, as the application is not using any Java Security Manager. For example, if the application (process) is running on port number 1234, we can run the Java command:

jinfo -sysprops 1234

to view both the passwords as values of the corresponding Java system properties. I wish to ask if there is any remedy to this without changing the existing code-base too much? The desired effect would be to "hide" the two Java system properties (denoting the two passwords) from all external entities.

It may be noted that introducing a Java Security Manager into the application may not be a solution, as if we revoke read permissions from the said two Java system properties using the Java Security Manager, the application codes which read those properties would crash. Same is applicable for storing the passwords in encrypted form, as that would crash all codes within the application which are expecting to read the passwords in clear text form.

To give a bit more context, the said two passwords we are storing as Java system properties are actually passwords to access two key-stores, and Tomcat requires that we store the said two passwords in plain-text format. Any workarounds, such that only Tomcat will be able to see the two passwords as-is, while they will be invisible to all other external entities?

Or, is there any way to place the said passwords in other in-memory locations (like static variables) which only Tomcat can (be made to) read instead of placing them as system properties which is exposed to anyone?