Google demands HAR files with sensitive info to look into payment issues

6 pointssgrytoyr3y ago9 comments

My company uses Google Workspace for email, and we are happy to pay them for their services. A while back, our old payment card expired and they emailed us saying we have to add a new one. So I logged in and tried adding a new card (a VISA card that works everywhere else, even in other parts of Google), but it repeatedly failed with the error code OR-CCSEH-26 and the message “Your card’s issuer declined this request. Contact your bank or use a different payment method.” So I contacted my bank (the largest bank in Norway) and they said everything is fine and they have heard reports that Google sometimes rejects transactions for no good reason, but there’s nothing they can do about it - I have to resolve it with Google.

I contact Google Workspace support and after the usual introductory pleasantries they seem to understand that I have a legitimate issue that their specialist team needs to look more closely at. However, even though this appears to be strictly a backend issue (after all, they have contacted the card’s issuer), their procedures will not allow their agents to escalate this without receiving a HAR file from me by email, containing complete details about all requests pertaining to my attempt to update the payment card.

Importantly, such a HAR file would contain every single payload and header sent from my browser to Google’s servers, including my authentication token and full credit card details. I balk at this and explain that I am very reluctant to send such a sensitive file to anybody, using any transport, but particularly to some shared email address at Google. Oh, that’s not a problem, the agent says (but only after I resist sending them the raw HAR file), I can just use the HAR analyser in the Google Admin Toolbox to remove any sensitive information. However, it is unclear if this tool requires me to upload the file to Google first, or if it is strictly a client-side tool (on closer inspection, it looks like it may be local, which is good). It is also unclear if it completely removes stuff like credit card details, or just auth headers.

Regardless, there are several things that IMHO are wrong with this:

  - Why is it impossible to escalate a payment issue without a HAR file?
  - Why do customers need to upload a HAR file to debug what is in all likelihood a backend issue?
  - Why is Google, a champion of online security (unironically), asking customers to send files containing their login credentials and full credit card details, by email?
  - I am a developer and could probably figure out how to clean a HAR file manually (even though it is 7MB of JSON), but do they really expect regular users to be able to do this?

My issue has been ongoing for several weeks, with no solution in sight, and when asked point blank, their agent confirmed that there is nothing they can do without a HAR file. The last agent I spoke to even put me on hold while she double-checked with her supervisor that this was the case.

This is a bit of a rant, I know, but I also felt that the big-picture aspects of this might be worth discussing on HN. Is it really Google’s policy that they will not help a customer trying their best to give them their money, without said customer sending them a highly technical file containing extremely sensitive information? And if so, what does that say about their internal security culture?

9 comments

solardev3y ago

The agent is just following some script and if that calls for HAR for troubleshooting, that's what they're going to ask you for, no matter if it's a security concern.

Can you escalate it to a manager, or just use a different credit card? Google's not known for their customer service, and they're probably not very trained on how to handle foreign transaction problems. It's one of the downsides of using Workspace... good services, poor support, and mostly you're on your own to figure things out =/

Wonnk133y ago

This 100% I used to work adjacent to teams in GTech (Google's customer support org) and asking for a HAR file was standard practice back then, whoever the support agent is just doesn't realize that given the context it may contain sensitive information.

theduder993y ago

this situation is not unique to google. support teams asking for har files is a common practice nowadays, yes even for companies processing payments. if I was you I would escalate this thing on the issuer side again. there are a variety of reasons why the issuer would send a generic "customer should contact issuer" response back to google.

existencebox3y ago

Some perspective, since I've been on the other side of exactly this issue, but for another bigco:

While there may well be a backend issue, in many cases, the HAR file contains tons of useful details (in my product's case) for things like "what were responses from other service calls, was anything failing locally, was any state in a weird... state" that can then let me better understand what's going on in the backend, or even know where to be looking in the backend. These large services are _so complex_ nowadays that looking at a slice of backend logs without the frontend to dovetail can often be a very partial view of the world, or be a needle-in-a-haystack scenario.

I'm giving them the benefit of the doubt here that it's similar to my project, clearly, they could maybe just be running a script (since we similarly request a HAR for all reports, since 99% of the time in practice it _is_ very useful), and you won't be harming anyone from trying to get a clear answer as to if the PG/triage group actually needs it to move forward and pushing back if they don't.

I also imagine that, if they're anything like us, you could request explicit deletion of all your support data from that case after the case is done, and they'd have to comply per GDPR/etc (We certainly would) and they likely already have to silo the information in ways that explicitly makes sure that sort of PII doesn't end up in buckets it shouldn't. I don't know if this moves the needle for you, CC #s are still touchy, but just thinking out loud.

Anyway, I hope this doesn't come across as apologetic for lax security practices, just wanted to give this perspective as I just remember the feeling of frustration of the customer refusing to send logs with a similar justification and my going "but it's going to be _much_ harder/impossible to diagnose your issue without that, and I have 0 doubts in my team's ability to properly handle and dispose of secure data as professionals, we are literally legally obligated to."

sgrytoyrOP3y ago

Appreciate the perspective. I do realise that HAR files are very useful, if for nothing else than being able to rule out any client-side issues. However, I don’t agree with their decision to make it impossible to get something looked at without HAR files - especially when there is a legitimate consern that they may contain highly sensitive data (even after their tool’s automated cleaning) and for something that is almost certainly a backend issue.

Also, it’s not so much that I don’t trust Google to handle the files responsibly, I just think it’s principally wrong to ask customers to send highly technical files (that most people won’t understand the implications of) in this day and age, when everywhere else we are all trying our best to educate people how NOT to get tricked into sharing security credentials and credit card info.

How easy wouldn’t it be to call someone you know are having a payment card issue, claim you are from Google Support, and then ask them to follow the procedure to record a HAR file while they are trying to add a new card, and then send it to some Google-like email? Even though many now have learned that they shouldn’t give out their password to anyone or click random links in emails, I suspect that a huge percentage of people would have no idea of what they just emailed to some stranger in this scenario.

Do we really want the major players to teach their customers that it’s perfectly fine to share whatever with someone claiming to be a support rep? Shouldn’t we be moving in the other direction instead?

existencebox3y ago

There's definitely a line to walk there re: consumer education, but I'll give the analogy that if you walk into a bank to obtain a loan, you'll hand over _far_ more sensitive information than is in a HAR file. Typically this is fine though, because we're confident we're talking to a party that actually needs this and is whom they say they are, both to lend legitimacy and potentially follow back if something goes wrong. (The fact that we initiated the interaction as well would seem to lend some legitimacy to otherwise "escalatory" requests) I personally see a similar relationship when I reach out to some service provider/utility with an issue, e.g. I'll tell them my SSN but if someone on the street walks up and says "I'm from the water company tell me your birthdate" I'd... make a very confused face.

Both as such, and to be clear, I am sensitive to the making it impossible part, and stand by my earlier statement that ideally you should be able to push back enough to get a cogent answer from the PG as to why they need it, or get an exception if not. (We should absolutely teach people to have informed reservations. Ideally we'd also have better mechanisms for easily verifying identity and securely sharing and ring-fencing information, but if wishes were nickels etc.)

(To wrap this ramble up, I will grant you a scary addendum though: A slight variation to the phishing attack you described even broaches the "We initiated the communication" trust-exercise, as a more sophisticated phisher may be able to by side channel identify that you're having a certain issue and may have reached out for assistance, and can try to intercede in that by extending help pretending to be the intended respondent. The mitigation to this one is typically "never trust someone who reaches out to you, call the trusted verifiable root-of-identity yourself each time." but it illustrates the balance one has to strike in keeping ahead of the escalating cat and mouse game while still being able to securely exchange information when necessary.)

animitronix3y ago

Oh boy would you hate what one can see with session replay tools like LogRocket

sgrytoyrOP3y ago

Heh, I do know that to some extent, but I feel like it’s a different issue. We are all submitting sensitive data all the time, and trusting that whatever service we are using (and the services they in turn are using) will handle our secrets responsibly.

But that stuff is regulated by laws (GDPR etc.) and, at least to some degree, self-regulated by economic principles (leaking passwords or credit cards should be bad for business). More importantly, though, it isn’t in itself a violation of security best practices. You often have to submit sensitive information to live in the modern world.

What is totally unnecessary, though, is for highly trusted services to teach people to share sensitive files unreservedly, just because they are really nice to have during debugging.

barelysapient3y ago

Use a different card or draw from a bank account.

j / k navigate · click thread line to collapse

Google demands HAR files with sensitive info to look into payment issues

6 pointssgrytoyr3y ago9 comments

Regardless, there are several things that IMHO are wrong with this:

  - Why is it impossible to escalate a payment issue without a HAR file?
  - Why do customers need to upload a HAR file to debug what is in all likelihood a backend issue?
  - Why is Google, a champion of online security (unironically), asking customers to send files containing their login credentials and full credit card details, by email?
  - I am a developer and could probably figure out how to clean a HAR file manually (even though it is 7MB of JSON), but do they really expect regular users to be able to do this?

9 comments

solardev3y ago

The agent is just following some script and if that calls for HAR for troubleshooting, that's what they're going to ask you for, no matter if it's a security concern.

Wonnk133y ago

theduder993y ago

existencebox3y ago

Some perspective, since I've been on the other side of exactly this issue, but for another bigco:

sgrytoyrOP3y ago

existencebox3y ago

animitronix3y ago

Oh boy would you hate what one can see with session replay tools like LogRocket

sgrytoyrOP3y ago

What is totally unnecessary, though, is for highly trusted services to teach people to share sensitive files unreservedly, just because they are really nice to have during debugging.

barelysapient3y ago

Use a different card or draw from a bank account.

j / k navigate · click thread line to collapse