Evaluating AI agents: Real-world lessons from building agentic systems at Amazon (opens in new tab)

(aws.amazon.com)

3 pointsbpedro1mo ago1 comments

1 comments

I get the justification but I found it hard to understand how the actual evaluation at each step is carried out. For example, is there any calibration to some human gold standard involved or is the AI evaluating the AI without calibration/oversight?

j / k navigate · click thread line to collapse

Evaluating AI agents: Real-world lessons from building agentic systems at Amazon (opens in new tab)

(aws.amazon.com)

3 pointsbpedro1mo ago1 comments

1 comments

lumpilumpi1mo ago

I get the justification but I found it hard to understand how the actual evaluation at each step is carried out. For example, is there any calibration to some human gold standard involved or is the AI evaluating the AI without calibration/oversight?

j / k navigate · click thread line to collapse