Skip to content

Commit

Permalink
AI: Review follow-up
Browse files Browse the repository at this point in the history
  • Loading branch information
mnocon committed Feb 6, 2025
1 parent 26a5af1 commit f51d84f
Show file tree
Hide file tree
Showing 5 changed files with 25 additions and 18 deletions.
2 changes: 1 addition & 1 deletion code_samples/ai_actions/config/services.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ services:

App\Command\AddMissingAltTextCommand:
arguments:
$projectDir: '%kernel.project_dir%'
$binaryDataHandler: '@Ibexa\Core\IO\IOBinarydataHandler\SiteAccessDependentBinaryDataHandler'

App\AI\Handler\LLaVATextToTextActionHandler:
tags:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,9 @@ protected function execute(InputInterface $input, OutputInterface $output): int
$this->actionConfigurationService->createActionConfiguration($actionConfigurationCreateStruct);

$action = new RefineTextAction(new Text([
<<<TEXT
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
and which usually results in protein folding into a specific 3D structure that determines its activity.
<<<TEXT
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
and which usually results in protein folding into a specific 3D structure that determines its activity.
TEXT
]));
$actionConfiguration = $this->actionConfigurationService->getActionConfiguration('rewrite_casual');
Expand Down
17 changes: 10 additions & 7 deletions code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\Operator;
use Ibexa\Contracts\Core\Repository\Values\Filter\Filter;
use Ibexa\Core\FieldType\Image\Value;
use Ibexa\Core\IO\IOBinarydataHandler;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
Expand All @@ -42,23 +43,23 @@ final class AddMissingAltTextCommand extends Command

private ActionServiceInterface $actionService;

private string $projectDir;
private IOBinarydataHandler $binaryDataHandler;

public function __construct(
ContentService $contentService,
PermissionResolver $permissionResolver,
UserService $userService,
FieldTypeService $fieldTypeService,
ActionServiceInterface $actionService,
string $projectDir
IOBinarydataHandler $binaryDataHandler
) {
parent::__construct();
$this->contentService = $contentService;
$this->permissionResolver = $permissionResolver;
$this->userService = $userService;
$this->fieldTypeService = $fieldTypeService;
$this->actionService = $actionService;
$this->projectDir = $projectDir;
$this->binaryDataHandler = $binaryDataHandler;
}

protected function configure(): void
Expand All @@ -75,7 +76,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int

/** @var \Ibexa\Contracts\Core\Repository\Values\Content\Content $content */
foreach ($modifiedImages as $content) {
/** @var ?Value $value */
/** @var \Ibexa\Core\FieldType\Image\Value $value */
$value = $content->getFieldValue(self::IMAGE_FIELD_IDENTIFIER);

if ($value === null || !$this->shouldGenerateAltText($value)) {
Expand Down Expand Up @@ -126,11 +127,13 @@ private function getSuggestedAltText(string $imageEncodedInBase64, string $langu

private function convertImageToBase64(?string $uri): string
{
$file = file_get_contents($this->projectDir . \DIRECTORY_SEPARATOR . 'public' . \DIRECTORY_SEPARATOR . $uri);
if ($file === false) {
throw new \RuntimeException('Cannot read file');
if ($uri === null) {
throw new \DomainException('Image field type is missing the uri property');
}

$id = $this->binaryDataHandler->getIdFromUri($uri);
$file = $this->binaryDataHandler->getContents($id);

return 'data:image/jpeg;base64,' . base64_encode($file);
}

Expand Down
18 changes: 11 additions & 7 deletions docs/ai_actions/extend_ai_actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ month_change: true

# Extend AI Actions

By extending AI Actions, you can make regular content management and editing tasks more appealing and less demanding.
By extending [AI Actions](ai_actions_guide.md), you can make regular content management and editing tasks more appealing and less demanding.
You can start by integrating additional AI services to the existing action types or develop custom ones that impact completely new areas of application.
For example, you can create a handler that connects to a translation model and use it to translate your website on-the-fly, or generate illustrations based on a body of an article.

Expand All @@ -14,7 +14,7 @@ For example, you can create a handler that connects to a translation model and u
You can execute AI Actions by using the [ActionServiceInterface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-ActionServiceInterface.html) service, as in the following example:

``` php
[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php', 101, 120) =]]
[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php', 102, 121) =]]
```

The `GenerateAltTextAction` is a built-in action that implements the [ActionInterface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-ActionInterface.html), takes an [Image](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-Action-DataType-Image.html) as an input, and generates the alternative text in the response.
Expand Down Expand Up @@ -43,7 +43,7 @@ You can influence the execution of an Action with two events:
Below you can find the full example of a Symfony Command, together with a matching service definition.
The command finds the images modified in the last 24 hours, and adds the alternative text to them if it's missing.

``` php hl_lines="87 100-125"
``` php hl_lines="88 101-126"
[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php') =]]
```

Expand Down Expand Up @@ -114,7 +114,7 @@ Create a class implementing the [ActionHandlerInterface](../api/php_api/php_api_

See the code sample below, together with a matching service definition:

``` php hl_lines="21 29-31 34-69 71-74"
``` php hl_lines="21 29-32 34-69 71-74"
[[= include_file('code_samples/ai_actions/src/AI/Handler/LLaVaTextToTextActionHandler.php') =]]
```

Expand Down Expand Up @@ -146,10 +146,14 @@ The created Form Type adds the `system_prompt` field to the Form.
Use the `Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper` class together with the `ibexa.connector_ai.action_configuration.form_mapper.options` service tag to make it part of the Action Handler options form.
Pass the Action Handler identifier (`LLaVATextToText`) as the type when tagging the service.

The Action Handler and Action Type options are rendered in the back office using the built-in Twig option formatter.
The Action Handler and Action Type options are rendered in the back office using the built-in Twig options formatter.

![Custom Action Handler options rendered using the default Twig options formatter](img/action_handler_options.png "Custom Action Handler options rendered using the default Twig options formatter")


You can create your own formatting by creating a class implementing the [OptionsFormatterInterface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-ActionConfiguration-OptionsFormatterInterface.html) interface and aliasing it to `Ibexa\Contracts\ConnectorAi\ActionConfiguration\OptionsFormatterInterface`.

The following service definition switches the options rendering to the other built-in option formatter, displaying the options as JSON.
The following service definition switches the options rendering to the other built-in options formatter, displaying the options as JSON.

``` yaml
[[= include_file('code_samples/ai_actions/config/services.yaml', 64, 66) =]]
Expand Down Expand Up @@ -192,7 +196,7 @@ See the built-in Action Types like Generate Alt Text or Refine Text for an examp

### Create custom Data classes

The `TranscribeAudio` Action Type requires adding two data classes that exists in its definition:
The `TranscribeAudio` Action Type requires adding two data classes that exist in its definition:

- an `Audio` class, implementing the [DataType interface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-DataType.html), to store the input data for the Action

Expand Down
Binary file added docs/ai_actions/img/action_handler_options.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f51d84f

Please sign in to comment.