Sorry to hear that, I can definitely and easily see it going wrong, so it was important on my team to emphasize
1. this was completely optional. it was a daily timeslot but we didn't go every day. We were more than free to schedule other meetings over it as it was the lowest priority.
2. Timezones will always be weird, but fwiw ours was at 3pm and slotted for an hour. Usually we could go over if there was a particularly thorny task but we respected everyone's time. Towards the end of the day but not quite around the point where we started to check out for the day.
3. The time should hopefully feel productive in some way. We never did any kind of "team bonding exercise" that'd fit more into some party icebreaker. we were professionals, and outside of some very specific hobbies members shared (e.g. half the team loved music production and would occasionally talk about sound design topics) I'd say 80% of the "active" hours were focused more on getting unblocked from some tasks. I think that's why it was important to frame it as "office hours" and not "social time".
4. It was absolutely okay to have quieter days. There was no pressure to speak out or pretend to be engaged or whatnot. After the first few weeks we'd normally just start with no video on and it'd be a small voice chat. There were days where 1 or more members just cut out early (and since it was a 4-6 person team, if half or more weren't engaged and the rest weren't stuck, we'd just not do it). There were others where it was just maybe 5 minutes of small talk and people just stayed in the ambience. It was common for at least one person at any time to be muted, so again: no pressure to engage. It was time for us to utilize.
Maybe it simply doesn't work for you, but I wouldn't cast off the entire idea just yet.