Gemini Pro is... not great. In one test, I asked what gesture I was making (while showing a thumbs up) -- it said thumbs down and "The image is a commentary on the changing nature of truth".
I just just made a heads-to-heads comparison -- you can watch it here: https://www.youtube.com/watch?v=1RrkRA7wuoE
Code is here: https://github.com/gregsadetsky/sagittarius
I wasn’t going to bother even testing Google’s AI products unless everyone started gushing about how much better they are than GPT4 but with 60 free queries per minute? That’s worth exploring even if only to find out shortly that it’s not worth paying for.
For users that get close to that sustained rate, they’re just as likely to exceed it and actually pay.
Doing this with ChatGPT 4.0 for months and months did not cause this type of behavior.
[1] https://makersuite.google.com/app/prompts/blog-post-creator
Developers have a free quota and access to a full range of features including function calling, embeddings, semantic retrieval, custom knowledge grounding, chat functionality and more. It supports 38 languages across 180+ countries.
Wait only in the US?
Edit: I can access it through the Google Cloud Console.
It looks like a fairly easy swap-in for GPT. "messages" becomes "content". Some of the configuration parameters are slightly different (topP/etc), but I have never put in the effort to understand the practical effect of those so I never tweak their values.
The messages themselves are a list of "parts", which allows mixed media messages. This feels a little cleaner than how GPT has handled messages being extended.
Instead of role: "assistant" they use role: "model". There's no role: "system" – presumably you just shove everything into user messages. You can also leave off the role... and I assume that means default to "user" but it's not clear if it's 100% equivalent...?
There's a bunch of moderation parameters, which seems like a good idea. OpenAI has a moderation endpoint you can use to preflight check your input, but doing it all at once makes more sense. There's four categories and you can adjust your sensitivity to each (and turn off blocking at entirely). The sensitivity is not about how extreme the violation is, but how likely it is a violation. So it's not like a G/PG/PG-13/etc rating. Just a question of how many false positives/negatives you want.
There's functions, though they are in beta (whatever that means): https://ai.google.dev/docs/function_calling – they look very very similar to GPT functions. They don't have the "JSON response" that GPT has, but that seems mostly redundant with functions anyway.
I have no idea how well prompts translate, but it feels like the API is an easy translation. And importantly everything is semantically equivalent, you don't have to make one pretend it is the other, like turning a completion API into a chat API.
Given the generous free tier I feel fairly motivated to swap in Gemini and try to ship experiments that I've sat on until now.
Node.js:
const model = genAI.getGenerativeModel({ model: "gemini-pro-vision"});
const result = model.generateContent({ contents: [{parts: [
{text: "What’s in this photo?"},
{inlineData: {data: imgBase64, mimeType: 'image/png'}}
] }] })
const model = genAI.getGenerativeModel({ model: "gemini-pro-vision"});
const result = await model.generateContent([
"What’s in this photo?",
{inlineData: {data: imgDataInBase64, mimeType: 'image/png'}}
]);updated my comment to paste in what's written in the docs
I will happily let google buy me for that price.
I'm sure it's just an accident.
In the case of this specific Gemini, it's apparently the result of there being two teams involved, and it's a reference to the Gemini space program[0].
Some basic prompts, which are answered correctly most of the time by ChatGPT4:
There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?
> 29 books
Julia has three brothers, each of them has two sisters. How many sisters does Julia have?
> Three
If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?
> Under the plate in the kitchen.
So, not great.
> Consider the following user chat. Reason out loud as needed to provide a reasonable response. User message: "Julia has three brothers, each of them has two sisters. How many sisters does Julia have?"
Gemini Pro:
> Julia's brothers have nothing to do with how many sisters she has. Julia's sisters are also her brothers' sisters, so the number of sisters Julia has is the same as the number of sisters her brothers have. Julia's brothers have two sisters each, so if we add up the number of sisters for each brother, we get: 2 + 2 + 2 = 6. Therefore, Julia has 6 sisters. The answer is 6.
https://poe.com/s/5gtLvuWT52o9LEBAJouo
GPT 4 gets it right.
GPT 3.5 gets it hilariously wrong if I change the question slightly:
User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have?
Assistant:
Julia has 8 brothers, and each of them has 5 sisters. Since Julia is one of those sisters, she has 8 brothers + 1 (herself) = 9 sisters.
https://chat.openai.com/share/3a49800f-7747-44d9-90e7-c1e037...
Also on 3.5 I tried this:
User: Julia has 8 brothers, each of them has 5 sisters. How many sisters does Julia have? Show your work step by step.
Assistant: Julia has 8 brothers, and each of them has 5 sisters, including Julia herself. So, Julia is one of those 5 sisters for each of her 8 brothers.
To calculate how many sisters Julia has, you can simply multiply the number of brothers by the number of sisters each brother has:
8 brothers * 5 sisters per brother = 40 sisters
Therefore, Julia has 40 sisters in total.
> If you read 2 books over the weekend, then there are 31 books in your house. However, if you only read one book, then there would be only 25 books left in your house.
> I apologize for any confusion caused by my previous response. Reading a book doesn't physically remove it from your house. The assumption I made was a misunderstanding. If you read a book, it is still in your house unless you lend it, give it away, or otherwise remove it.
> So, if you started with 31 books and read 2 over the weekend, you would have 31 - 2 = 29 books still in your house.
Now a human can reason because a human can visualize in their mind that a plate is being moved without moving the orange. How is that kind of reasoning possible in LLMs which in my very basic understanding is predicting the next token.
> The orange remains in the living room. Moving the plate to the kitchen does not affect the location of the orange, since it was placed below the plate but not attached to it. Therefore, the orange stays where it was originally placed, which is in the living room.
You don't need to visualize it in your mind to understand the relationship between being _below_ and being _moved with_. Keep in mind that many people cannot visualize anything in their mind!
Q. If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?
A. The orange remains in the living room. Moving the plate to the kitchen does not affect the position of the orange.
----
Q. There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?
A. There are still 31 books in your house. Reading books does not change their quantity
----
Q. Julia has three brothers, each of them has two sisters. How many sisters does Julia have?
A. Julia has one sister. Each of her three brothers has two sisters, and Julia is one of them. The other one is her sister, making it a total of one sister for Julia.
> Julia has three brothers, each of them has two sisters. How many sisters does Julia have?
Julia has two sisters. Each brother having two sisters refers to the same two sisters, Julia and another sister.
Then if I ask to verify its answer, it realizes the issue.
> Apologies for any confusion. If Julia has three brothers and each of them has two sisters, then Julia is one of the sisters. The other sister is common among all three brothers. Therefore, Julia has only one sister.
>>There are 31 books in my house. I read 2 books over the weekend. How many books are still in my house?
You still have 31 books in your house. The number of books doesn't change because you finished reading two of them; they're just no longer unread.
>>Julia has three brothers, each of them has two sisters. How many sisters does Julia have?
Julia has one sister.
Here's the logic behind this:
1. We know that Julia has three brothers. 2. Then we are told that each of her brothers has two sisters. 3. Since all four children (Julia and her brothers) share the same parents, they also share the same siblings. 4. Therefore, if each of Julia's brothers has two sisters, then there must be only one other sister besides Julia herself.
>> If you place an orange below a plate in the living room, and then move the plate to the kitchen, where is the orange now?
The orange is still in the living room, as moving the plate does not affect the location of the orange.
Get API key, takes me to makersuite, where I get a create API key button that errors. Then when I reload the page, I get a straight forbidden page.
HP said it best, you have to isolate the team from the bigger company to allow them to work as an effective startup. How can solo-preneurs provide better UX & onboarding while doing 16 other jobs than Google can with multi-billion dollar budgets?
How can they be so unaware of the fact that people will often prefer english because that's the language with most tutorials/guides/resources and makes interoperability in cross-country remote companies simpler?
Don't they want to sell cloud products to global companies?
How am I supposed to help or receive help from my coworkers?
I have lost days and days trying to set Google Sheets in English and I have been stuck with the Italian version no matter how many changes I did to my Sheets or Google account settings.
There's a 5000+ comments/upvotes discussion on their forums and they simply don't give two damns, I don't think humans even see those threads.
Didn't feel so stressed using a software since programming in Liferay professionally or trying to figure out Autodesk products a decade ago for hobby 3d modelling..
Hey maybe try using, I dunno, the currency of my logged in account / country? Or the currency I used last time? Or the currency of my VPN country?
Just because I’m traveling in Uzbekistan for a few days doesn’t mean I want to buy tickets denominated in Uzbek Som.
At a huge organisation like Google there’s a plethora of people all working according to their incentives. These organisations pay lip service to customer experience and excellence, but when push comes to shove it’s the flashy projects that get rewarded.
So nobody has a great overview of everything going on to the necessary level of detail, and nobody actually cares about onboarding UX.
Microsoft has the same problem, I want to shoot myself every time I need to get something done for a kid’s account on Xbox.
Solo-preneurs don't have to spend time coordinating with the 39 other people assigned to the project plus the 20 execs hovering over their shoulder, all with different ideas and agendas about what the product should be, while navigating the politics to get appropriate resources assigned without upsetting the hundreds of other folks with competing interests.
Simple features so bizarrely complex and hidden that turning an instance off, checking visitors for a page or other trivial matters seriously required hiring a full time "google interface understander", i kid you not.
Kafka on steroids.
Click on link > Get API Key > "We are sorry, but you do not have access to Early Access Apps"
Since I'm the admin, I checked and confirmed that I do have access. It's enabled for both my account and the entire organization.
Whatever.
But I wanted to note API key generation works when you pick an existing Google Cloud project.
- https://cloud.google.com/vertex-ai (marketing page)
- https://cloud.google.com/vertex-ai/docs (docs entry point)
- https://console.cloud.google.com/vertex-ai (cloud console)
- https://console.cloud.google.com/vertex-ai/model-garden (all the models)
- https://console.cloud.google.com/vertex-ai/generative (studio / playground)
VertexAI is the umbrella for all of the Google models available through their cloud platform.
You want the last link if you are looking for a ChatGPT like experience, with the ability to also adjust the parameters, so more like a UI on top of the API
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- 60 queries per minute free - about 1/5th the price of GPT3.5 Turbo - priced per char, not per token - same image pricing as GPT4 150x150
Courtesy of GPT4:
"To determine which option is cheaper, Gemini Pro or GPT-3.5 Turbo, we need to consider the average length difference between tokens and characters and the pricing structure for each.
Gemini Pro Pricing: Input: $0.00025 per 1,000 characters Output: $0.0005 per 1,000 characters
GPT-3.5 Turbo Pricing: Input: $0.0010 per 1,000 tokens Output: $0.0020 per 1,000 tokens
Average Length Difference Between Tokens and Characters: A token in GPT-3.5 can be a single word, part of a word, or a special character. On average, a token in GPT-3 models is roughly equivalent to 4 characters (this is a rough estimate as the length of tokens can vary significantly).
Given this information, let's calculate the effective cost per 1,000 characters for both Gemini Pro and GPT-3.5 Turbo. For GPT-3.5 Turbo, since 1 token ≈ 4 characters, the cost per 1,000 characters would be a quarter of the cost per 1,000 tokens. We'll calculate the cost for both input and output for each and compare.
The total cost per 1,000 characters for both Gemini Pro and GPT-3.5 Turbo, considering both input and output, is the same at $0.00075. Therefore, based on the provided pricing and the average token-to-character ratio, they are equally cost-effective."