Does your order book need AGI?

In TOP STORIES #189, I covered Google co-founder Sergey Brin’s “RSI treadmill”..... researchers and AI working together to build the next generation of models. Brin expects that pace to continue accelerating ↗ (opens a new tab) .

Now Anthropic has added a warning. Its latest filing says future models may soon meet its threshold for dramatically accelerated AI development..... and that capabilities could advance faster than the corresponding safety and security measures.

BUT..... Anthropic has NOT designated any current model as crossing that threshold. This remains an assessment of what could happen with future models.

RSI means Recursive Self-Improvement where AI helps develop more capable AI, which becomes better at developing the generation after that.

If that loop gathers momentum, progress could move far beyond the pace human researchers could sustain unaided.

My view..... I do not see humans collectively pulling up the handbrake. The commercial and strategic incentives are too strong. This will remain a race (hopefully not to Zero).

BUT..... take a step back and consider what YOUR business actually needs.

Do you really need AGI, (Artificial General Intelligence), to process orders against specified products, agreed prices, stock availability and credit limits?

For a clearly defined process, conventional automation may handle much of the work. A suitable AI model could interpret the customer’s email and extract the order details, while the business system enforces the rules and refers exceptions for human review.

Start with the requirement. Test accuracy, reliability and cost. Choose capability that is fit for purpose.

And security MUST be part of the design..... control what the AI can read and change, protect customer information, record its actions and require approval for sensitive transactions. Enforce those permissions in the connected systems.

A cleverer AI still needs somebody deciding what it is allowed to do!