When you need structured data from a web page — product prices, article metadata, event details — the old approach was brittle: write a custom scraper, maintain CSS selectors, and pray the markup doesn't change. Spatie's laravel-url-ai-transformer flips this model. Instead of selectors, you give an AI agent instructions and a JSON schema. The agent reads the page, extracts what you asked for, and returns validated JSON.
Version 2, released in mid-2025 and recently highlighted by Laravel News, rebuilds the package on top of the official Laravel AI package. Every built-in transformer is now a Laravel AI agent. You get provider-agnostic model selection, structured output validation, and the same testing fakes you use for your own AI features.
Installation and setup
The package requires PHP 8.4+, Laravel 12 or 13, and an API key for your AI provider (OpenAI, Anthropic, etc.).
composer require spatie/laravel-url-ai-transformer
php artisan vendor:publish --tag="laravel-url-ai-transformer-migrations"
php artisan migrate
Add your provider key to .env:
OPENAI_API_KEY=sk-...
The built-in transformers
Two transformers ship with the package.
LdJsonTransformer
Extracts and validates schema.org JSON-LD from the page. Useful for products, articles, events, recipes — anything that exposes structured data.
use Spatie\LaravelUrlAiTransformer\Support\Transform;
use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;
Transform::urls('https://example.com/products/widget')
->usingTransformers(new LdJsonTransformer);
SummaryTransformer
Generates a concise summary. You can customise the prompt:
use Spatie\LaravelUrlAiTransformer\Transformers\SummaryTransformer;
Transform::urls('https://example.com/blog/long-article')
->usingTransformers(new SummaryTransformer(
instructions: 'Summarise this article in three bullet points, focusing on technical decisions.'
));
Running transformations
Transformations run on the queue by default. Dispatch them with the Artisan command:
php artisan transform-urls
For synchronous execution during development or testing:
php artisan transform-urls --now
Filter which queued URLs to process:
php artisan transform-urls --url="*/blog/*"
php artisan transform-urls --url="https://example.com/products/*"
Force re-processing URLs that already have results:
php artisan transform-urls --force
Retrieving results
Results are stored in the transformation_results table. Retrieve them via the TransformationResult model:
use Spatie\LaravelUrlAiTransformer\Models\TransformationResult;
$result = TransformationResult::forUrl(
'https://example.com/products/widget',
'ldJson' // the transformer key
);
if ($result) {
$data = json_decode($result->output, true);
// $data['name'], $data['price'], etc.
}
The output column contains the raw AI response (text for summaries, JSON for structured transformers). The meta column holds token usage, model name, and timing.
Building a custom transformer
When the built-ins don't fit, extend Spatie\LaravelUrlAiTransformer\Transformers\Transformer and implement instructions():
namespace App\Transformers;
use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
class EventTransformer extends Transformer
{
public function instructions(): string
{
return <<<INSTRUCTIONS
Extract the event name, start date (ISO 8601), venue name, and city
from this webpage. If the event has multiple dates, return the first one.
INSTRUCTIONS;
}
}
Register it:
Transform::urls('https://example.com/events/conference-2026')
->usingTransformers(new EventTransformer);
The package fetches the URL, strips scripts, styles, and HTML tags, collapses whitespace, and truncates to ~6,000 characters before sending to the AI. This keeps token costs predictable.
Structured output with JSON Schema
For reliable machine-readable data, implement HasStructuredOutput from the Laravel AI package and define a schema() method using Laravel's JsonSchema builder.
namespace App\Transformers;
use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
class ProductTransformer extends Transformer implements HasStructuredOutput
{
public function instructions(): string
{
return 'Extract the product name, price in EUR, currency, availability, and an array of image URLs.';
}
public function schema(JsonSchema $schema): array
{
return [
'name' => $schema->string()->required(),
'price' => $schema->number()->required(),
'currency' => $schema->string()->enum(['EUR', 'USD', 'GBP'])->required(),
'in_stock' => $schema->boolean(),
'images' => $schema->array()
->items($schema->string()->format('uri'))
->minItems(1),
];
}
}
The AI response is validated against this schema. If validation fails, the transformation is marked as failed and a TransformerFailed event is fired — you can listen for it and alert or retry.
Retrieve validated JSON directly:
$result = TransformationResult::forUrl($url, 'product');
$product = json_decode($result->output, true); // already matches your schema
Nested objects and arrays
JsonSchema supports nested structures:
public function schema(JsonSchema $schema): array
{
return [
'name' => $schema->string()->required(),
'variants' => $schema->array()->items($schema->object(fn ($s) => [
'sku' => $s->string()->required(),
'price' => $s->number()->required(),
'attributes' => $s->array()->items($s->string()),
])),
];
}
Conditional execution
Sometimes you only want to transform when certain conditions hold — for example, only if the page was updated since the last run. Override shouldRun() on your transformer:
public function shouldRun(): bool
{
$lastResult = TransformationResult::forUrl($this->url, $this->getKey());
if (!$lastResult) {
return true;
}
// Only re-run if the page's last-modified header changed
$headers = Http::head($this->url)->headers();
return $headers->get('last-modified') !== $lastResult->meta['last_modified'] ?? null;
}
The --force flag on the command bypasses this check.
Model and provider selection
You can configure the default provider/model in config/laravel-url-ai-transformer.php, or control it per-transformer with attributes:
use Laravel\Ai\Facades\Ai;
use Laravel\Ai\Attributes\UseSmartestModel;
#[UseSmartestModel] // picks the most capable configured model
class ProductTransformer extends Transformer implements HasStructuredOutput
{
// ...
}
Or at runtime:
$transformer = (new ProductTransformer)
->usingProvider('anthropic')
->usingModel('claude-3-5-sonnet-latest');
Registering URLs in bulk
Instead of calling Transform::urls() inline, register URLs in a service provider — either statically or lazily via a closure:
// AppServiceProvider.php
use Spatie\LaravelUrlAiTransformer\Support\Transform;
use App\Transformers\ProductTransformer;
public function boot(): void
{
// Static list
Transform::urls([
'https://shop.example.com/product/a',
'https://shop.example.com/product/b',
])->usingTransformers(new ProductTransformer);
// Lazy: resolves at runtime
Transform::urls(fn () => Product::published()
->where('needs_sync', true)
->pluck('url')
->toArray()
)->usingTransformers(new ProductTransformer);
}
The closure is evaluated when the transform-urls command runs, so it always picks up the current set of URLs.
Crawling entire sites
Combine with spatie/crawler to transform every page on a domain:
composer require spatie/crawler
use Spatie\Crawler\Crawler;
use Spatie\LaravelUrlAiTransformer\Support\Transform;
use App\Transformers\ArticleTransformer;
Crawler::create()
->setCrawlProfile('subdomains')
->startUrl('https://blog.example.com')
->addCrawlObserver(new class implements \Spatie\Crawler\CrawlObservers\CrawlObserver {
public function crawled(\Spatie\Crawler\CrawlData $data): void
{
if ($data->responseStatusCode === 200) {
Transform::urls($data->url)
->usingTransformers(new ArticleTransformer);
}
}
})
->startCrawl();
Each discovered URL gets queued for transformation.
Handling failures
Failed transformations store the exception in the exception column and fire Spatie\LaravelUrlAiTransformer\Events\TransformerFailed. Listen for it to log, notify, or schedule a retry:
// EventServiceProvider.php
protected $listen = [
\Spatie\LaravelUrlAiTransformer\Events\TransformerFailed::class => [
\App\Listeners\NotifyOnTransformFailure::class,
],
];
// NotifyOnTransformFailure.php
public function handle(TransformerFailed $event): void
{
Log::error('Transform failed', [
'url' => $event->url,
'transformer' => $event->transformerKey,
'error' => $event->exception->getMessage(),
]);
}
Regenerate a result later (queued):
$event->transformationResult->regenerate();
Or immediately (synchronous):
$event->transformationResult->regenerateNow();
Testing
The package provides fakes for both HTTP and AI responses so your tests run fast and offline.
use Spatie\LaravelUrlAiTransformer\Transformers\SummaryTransformer;
use Illuminate\Support\Facades\Http;
test('summary transformer extracts key points', function () {
Http::fake([
'https://example.com/article' => Http::response(
'<html><body><article>Real content here...</article></body></html>',
200,
['Content-Type' => 'text/html']
),
]);
SummaryTransformer::fake([
"• First key point\n• Second key point\n• Third key point",
]);
Transform::urls('https://example.com/article')
->usingTransformers(new SummaryTransformer);
$this->artisan('transform-urls --now');
$result = TransformationResult::forUrl('https://example.com/article', 'summary');
expect($result->output)->toContain('First key point');
});
For structured transformers, fake the JSON that matches your schema:
ProductTransformer::fake([
json_encode([
'name' => 'Test Product',
'price' => 29.99,
'currency' => 'EUR',
'in_stock' => true,
'images' => ['https://example.com/img.jpg'],
]),
]);
When to use this (and when not to)
Good fits:
- Extracting product data from supplier/partner sites where you don't control the markup
- Aggregating article metadata (author, date, tags) from multiple sources
- Pulling event details from conference pages
- Any scenario where the structure is semantic but the HTML varies
Avoid when:
- You need real-time data — AI transformation adds latency (queue it)
- The source has a stable API — use the API instead
- You're scraping at massive scale — token costs add up; consider a dedicated scraper
- The content is behind authentication — the package fetches public URLs only
Cost considerations
Each transformation sends the cleaned page content (~6,000 chars max) plus your instructions to the model. A typical product extraction with GPT-4o-mini costs fractions of a cent. At scale, cache aggressively: the shouldRun() method and TransformationResult model make it straightforward to only re-transform when the source actually changes.
Takeaway
laravel-url-ai-transformer v2 turns "scrape this page" into "ask an agent to read this page and give me JSON." The shift from selectors to schemas makes the code more resilient to redesigns and easier to maintain. Because it's built on Laravel AI, you get multi-provider support, structured output validation, and first-class testing tools without extra work.
If you've been maintaining fragile scrapers, this is worth a look for your next extraction task.