Install the Data Exchange Jobs feature

This document describes how to install the Data Exchange Jobs feature: the engine that stores import jobs and runs, processes uploaded files, and provides the Back Office Import & Export pages. The engine imports and exports no data on its own; a data type is added by registering a schema plugin. To install the product data type, see Install the Back Office Product CSV Import and Export feature. To add your own data type, see Add a custom data type to Data Exchange Jobs.

Prerequisites

Install the required features:

Install feature core

1) Install the required modules

composer require spryker/data-exchange:"202610.0" spryker/data-exchange-extension:"202610.0" --update-with-dependencies

2) Set up configuration

Filesystems

The engine reads uploaded files from and writes exported files to two named filesystem entries: data-exchange-imports and data-exchange-exports. The names are returned by getImportFileSystemName() and getExportFileSystemName() of DataExchangeConfig; override the methods on the project level if you need different names.

Production (S3)

config/Shared/config_default.php

To enable an IAM role configuration, use IamAws3v3FilesystemBuilderPlugin:

use Spryker\Service\FlysystemAws3v3FileSystem\Plugin\Flysystem\IamAws3v3FilesystemBuilderPlugin;
use Spryker\Shared\FileSystem\FileSystemConstants;

$config[FileSystemConstants::FILESYSTEM_SERVICE] = [
    // ... existing entries ...
    'data-exchange-imports' => [
        'sprykerAdapterClass' => IamAws3v3FilesystemBuilderPlugin::class,
        'bucket' => getenv('SPRYKER_S3_DATA_EXCHANGE_IMPORT_BUCKET') ?: '',
        'region' => getenv('AWS_REGION') ?: '',
        'path' => 'data-exchange-imports/',
    ],
    'data-exchange-exports' => [
        'sprykerAdapterClass' => IamAws3v3FilesystemBuilderPlugin::class,
        'bucket' => getenv('SPRYKER_S3_DATA_EXCHANGE_EXPORT_BUCKET') ?: '',
        'region' => getenv('AWS_REGION') ?: '',
        'path' => 'data-exchange-exports/',
    ],
];

Development (local filesystem)

config/Shared/config_default-docker.dev.php

use Spryker\Service\FlysystemLocalFileSystem\Plugin\Flysystem\LocalFilesystemBuilderPlugin;
use Spryker\Shared\FileSystem\FileSystemConstants;

$config[FileSystemConstants::FILESYSTEM_SERVICE]['data-exchange-imports'] = [
    'sprykerAdapterClass' => LocalFilesystemBuilderPlugin::class,
    'root' => '/data',
    'path' => '/data/data-exchange-imports',
];

$config[FileSystemConstants::FILESYSTEM_SERVICE]['data-exchange-exports'] = [
    'sprykerAdapterClass' => LocalFilesystemBuilderPlugin::class,
    'root' => '/data',
    'path' => '/data/data-exchange-exports',
];

Engine limits and error display

The following settings are defined in Spryker\Zed\DataExchange\DataExchangeConfig:

To change the defaults, extend the module configuration on the project level:

src/Pyz/Zed/DataExchange/DataExchangeConfig.php

<?php

namespace Pyz\Zed\DataExchange;

use Spryker\Zed\DataExchange\DataExchangeConfig as SprykerDataExchangeConfig;

class DataExchangeConfig extends SprykerDataExchangeConfig
{
    protected const int IMPORT_ERROR_DISPLAY_THRESHOLD = 20;
}

3) Set up the database schema and transfer objects

Apply database changes and generate entity and transfer changes:

console propel:install
console transfer:generate

4) Register the schema plugins

The engine learns about job types from the schema plugins registered in the DataExchange dependency provider. Register one plugin per data type. The following example registers the product data type of the Product Experience Management capability; see Add a custom data type to Data Exchange Jobs for your own types.

src/Pyz/Zed/DataExchange/DataExchangeDependencyProvider.php

<?php

namespace Pyz\Zed\DataExchange;

use Spryker\Zed\DataExchange\DataExchangeDependencyProvider as SprykerDataExchangeDependencyProvider;
use SprykerFeature\Zed\ProductExperienceManagement\Communication\Plugin\ProductCsvImportSchemaPlugin;

class DataExchangeDependencyProvider extends SprykerDataExchangeDependencyProvider
{
    /**
     * @return array<\Spryker\Zed\DataExchangeExtension\Dependency\Plugin\ImportSchemaPluginInterface>
     */
    protected function getImportSchemaPlugins(): array
    {
        return [
            new ProductCsvImportSchemaPlugin(),
        ];
    }

    /**
     * @return array<\Spryker\Zed\DataExchangeExtension\Dependency\Plugin\ExportSchemaPluginInterface>
     */
    protected function getExportSchemaPlugins(): array
    {
        return [
            new ProductCsvImportSchemaPlugin(),
        ];
    }
}

getImportSchemaPlugins() serves the import, the template download, and the job creation; getExportSchemaPlugins() serves the export with data. A type that is not exportable is registered in the import stack only. The same dependency provider offers getImportPreProcessorPlugins() and getImportPostProcessorPlugins() for plugins that run once before and once after the rows of a run are processed.

5) Configure navigation

Add the following entry to the Back Office navigation:

config/Zed/navigation.xml

<import-export>
    <label>Import &amp; Export</label>
    <title>Import &amp; Export</title>
    <icon>swap_vert</icon>
    <pages>
        <import>
            <label>Import</label>
            <title>Import</title>
            <bundle>data-exchange</bundle>
            <controller>job</controller>
            <action>index</action>
            <visible>1</visible>
        </import>
        <export>
            <label>Export</label>
            <title>Export</title>
            <bundle>data-exchange</bundle>
            <controller>export</controller>
            <action>index</action>
            <visible>1</visible>
        </export>
    </pages>
</import-export>

Build the navigation cache:

console navigation:cache:remove

6) Set up console

Register the console command

Register the import:job:run console command that processes pending import job runs:

src/Pyz/Zed/Console/ConsoleDependencyProvider.php

use Spryker\Zed\DataExchange\Communication\Console\ImportJobRunConsole;

// In getConsoleCommands():
new ImportJobRunConsole(),

console import:job:run

Configure the scheduler

Register the command to run on a schedule so that pending runs are picked up automatically:

src/Pyz/SymfonyScheduler/src/Pyz/Client/SymfonyScheduler/SymfonySchedulerConfig.php

'import-job-run' => [
    'command' => '$PHP_BIN vendor/bin/console import:job:run',
    'schedule' => '* * * * *',
],

Add the entry to the array that getJobs() returns. If the other entries in your file prefix the command with the $logger variable to redirect the output to the scheduler log, prefix this one in the same way.

console scheduler:setup

7) Build the Back Office frontend

The Export page ships a JavaScript entry point. Build the Back Office frontend to publish it:

console frontend:zed:build