I agree with Wildgoose. The law is not clear on this, but having discussed it with a number of people, the general consensus is that as long as you don't approach the bank area itself for any purpose (in other words, just passing by would be okay), you won't run afoul of the law. Still, if someone "made" you and called authorities, you might be in for a hassel at least and possible charges depending on the cop, the prosecutor, and how you respond to the situation. My guess is that as long as you cooperate and show respect for the officer (just doing his job), you'd be free to go soon (unless you have other legal issues like a stack of parking tickets, and outstanding warrant, or a record a mile long). Personally, I never worry about that circumstance, as unless something happens that requires you to pull your weapon, no one will ever know I was carrying anyway. If something does happen that requires me to pull my firearm, I'm not going to care what the law says at that point.
I see metaldoc posted while I was writing. Agreed, the whole concept of preventing a bank robbery (for example) by making guns illegal in them is preposerous.