haha that's awesome.
That actually happened to me a lot back when I worked at Staples, but I wore a red shirt, and was always too lazy to change before going shopping at Super Target (which has a grocery section; I don't think they exist in California?), so it was more understandable.
I'd always just say, "Oh, haha, sorry, I actually work somewhere else and just got off work, but hey, I know the store pretty well, so maybe I can help?" 9 times out of 10, I could tell them what aisle to go to, just because I shopped there enough to know where pretty much everything was.
For most people that was cool, and they were actually super happy that just a random fellow customer would help them out.
One time though a woman just looked at me strange, and said "gluten-free peanut butter".
I didn't realize peanut butter even HAD gluten in it, but I said "Oh, I think they have peanut butter on aisle 2? But again, I don't work here, so don't take my word for it."
She replied, "... Well aren't you going to SHOW me where it is?"
I said, "No? I'm busy shopping right now, just like you. Again, I don't work here."
So she left in a huff.
I wandered like 3 aisles down to look for something else, when she came back up to me. She goes, "It wasn't there! You lied!"
I said, "I'm ... Sorry? I could have sworn it was on aisle 2."
She stomped and yelled, "Not the gluten-free kind! What kind of person ARE you?! Do you want me to die from eating gluten?! I'm telling your manager!!"
I tried to tell her I didn't work there but shrugged and went back to shopping.
Some time later she and a manager shows up to the aisle I was in. They stood at the end of the aisle while she screamed at him about me. I looked up and smiled and shrugged. He shrugged back and walked her off. Hopefully he didn't have too hard of a time figuring out how the fuck you could tell if peanut butter were gluten-free. This was years before "gluten-free" shit was in-style and there weren't special sections for it.
People are morons. It definitely got me to stop wearing my work shirts to Target though. But, funny thing was, I'd still get stopped and asked.
I think that if you look and act like you know where you're going and what you're doing, people just assume you work there.