Install the Data Exchange Jobs feature

Edit on GitHub

This document describes how to install the Data Exchange Jobs feature: the engine that stores import jobs and runs, processes uploaded files, and provides the Back Office Import & Export pages. The engine imports and exports no data on its own; a data type is added by registering a schema plugin. To install the product data type, see Install the Back Office Product CSV Import and Export feature. To add your own data type, see Add a custom data type to Data Exchange Jobs.

Prerequisites

Install the required features:

NAME VERSION REQUIRED
Spryker Core 202610.0 Required
File System 202610.0 Required

Install feature core

1) Install the required modules

composer require spryker/data-exchange:"202610.0" spryker/data-exchange-extension:"202610.0" --update-with-dependencies
Verification

Make sure the following modules have been installed:

MODULE EXPECTED DIRECTORY
DataExchange vendor/spryker/data-exchange
DataExchangeExtension vendor/spryker/data-exchange-extension

2) Set up configuration

Filesystems

The engine reads uploaded files from and writes exported files to two named filesystem entries: data-exchange-imports and data-exchange-exports. The names are returned by getImportFileSystemName() and getExportFileSystemName() of DataExchangeConfig; override the methods on the project level if you need different names.

Production (S3)

config/Shared/config_default.php

To enable an IAM role configuration, use IamAws3v3FilesystemBuilderPlugin:

use Spryker\Service\FlysystemAws3v3FileSystem\Plugin\Flysystem\IamAws3v3FilesystemBuilderPlugin;
use Spryker\Shared\FileSystem\FileSystemConstants;

$config[FileSystemConstants::FILESYSTEM_SERVICE] = [
    // ... existing entries ...
    'data-exchange-imports' => [
        'sprykerAdapterClass' => IamAws3v3FilesystemBuilderPlugin::class,
        'bucket' => getenv('SPRYKER_S3_DATA_EXCHANGE_IMPORT_BUCKET') ?: '',
        'region' => getenv('AWS_REGION') ?: '',
        'path' => 'data-exchange-imports/',
    ],
    'data-exchange-exports' => [
        'sprykerAdapterClass' => IamAws3v3FilesystemBuilderPlugin::class,
        'bucket' => getenv('SPRYKER_S3_DATA_EXCHANGE_EXPORT_BUCKET') ?: '',
        'region' => getenv('AWS_REGION') ?: '',
        'path' => 'data-exchange-exports/',
    ],
];
IAM authentication required

These filesystems authenticate through the IAM role attached to the workload, which resolves AWS credentials automatically. Access keys (key and secret) are not supported. For more details, see AWS S3 filesystem builder plugins.

Development (local filesystem)

config/Shared/config_default-docker.dev.php

use Spryker\Service\FlysystemLocalFileSystem\Plugin\Flysystem\LocalFilesystemBuilderPlugin;
use Spryker\Shared\FileSystem\FileSystemConstants;

$config[FileSystemConstants::FILESYSTEM_SERVICE]['data-exchange-imports'] = [
    'sprykerAdapterClass' => LocalFilesystemBuilderPlugin::class,
    'root' => '/data',
    'path' => '/data/data-exchange-imports',
];

$config[FileSystemConstants::FILESYSTEM_SERVICE]['data-exchange-exports'] = [
    'sprykerAdapterClass' => LocalFilesystemBuilderPlugin::class,
    'root' => '/data',
    'path' => '/data/data-exchange-exports',
];
Verification

Make sure the filesystem directories exist and are writable by the application. For local development, verify that the /data/data-exchange-imports and /data/data-exchange-exports directories are created inside the Docker container.

Engine limits, run processing, and error display

The following settings are defined in Spryker\Zed\DataExchange\DataExchangeConfig:

SETTING DEFAULT DESCRIPTION
IMPORT_CSV_BATCH_SIZE 1000 Number of CSV rows handed to the import steps at once.
EXPORT_CSV_BATCH_SIZE 1000 Number of rows the export data provider returns at once.
IMPORT_MAX_FILE_SIZE_MB 50 Maximum size of an uploaded CSV file in megabytes.
IMPORT_JOB_RUNS_PER_INVOCATION 1 Number of pending import job runs one import:job:run invocation processes, oldest first, before the process exits. The invocation stops earlier when no pending run is left. Must be 1 or greater; the command throws an exception otherwise.
IMPORT_JOB_RUN_ERROR_ROW_LIMIT 1000 Maximum number of error rows stored per import job run. Rejected rows past the limit are counted in the failed line counter and in number_of_omitted_error_rows of the run but not stored; errors without a row number are always stored. Must be 1 or greater; a run fails with an exception otherwise.
IMPORT_ERROR_DISPLAY_THRESHOLD 10 Maximum number of import errors displayed inline on the import job run detail page. When a run produces more errors than this threshold, the errors are offered as a downloadable summary instead.
getAllowedMimeTypesWithExtensions() text/csv, text/plain MIME types accepted by the upload form.
IMPORT_JOB_RUN_LEASE_DURATION_IN_MINUTES 5 How long a worker holds an import job run after claiming it. The lease is renewed after every batch; once it expires, another worker can reclaim the run. Must exceed the longest time between two renewals: the pre-processors plus the first batch, or any later batch. If one batch can take longer, raise the lease or lower IMPORT_CSV_BATCH_SIZE.
IMPORT_JOB_RUN_MAX_ATTEMPTS 3 How many times an import job run can be claimed. A run that reached this number of attempts is not claimed again.
IMPORT_JOB_RUN_LEASE_TIME_ZONE UTC Time zone in which lease timestamps are written and compared. Every worker uses it regardless of its PHP default time zone.

To change the defaults, extend the module configuration on the project level:

src/Pyz/Zed/DataExchange/DataExchangeConfig.php

<?php

namespace Pyz\Zed\DataExchange;

use Spryker\Zed\DataExchange\DataExchangeConfig as SprykerDataExchangeConfig;

class DataExchangeConfig extends SprykerDataExchangeConfig
{
    protected const int IMPORT_ERROR_DISPLAY_THRESHOLD = 20;

    protected const int IMPORT_JOB_RUNS_PER_INVOCATION = 5;

    protected const int IMPORT_JOB_RUN_ERROR_ROW_LIMIT = 5000;
}

3) Set up the database schema and transfer objects

Apply database changes and generate entity and transfer changes:

console propel:install
console transfer:generate
Verification

Make sure that the following changes have been applied by checking your database:

DATABASE ENTITY TYPE EVENT
spy_import_job table created
spy_import_job_run table created
spy_import_job_run_error table created
spy_import_job_run.claimed_by column created
spy_import_job_run.lease_expires_at column created
spy_import_job_run.attempt_count column created
spy_import_job_run.number_of_processed_bytes column created
spy_import_job_run.number_of_omitted_error_rows column created
Verification

Make sure the following changes in transfer objects have been applied:

TRANSFER TYPE EVENT PATH
ImportJob class created src/Generated/Shared/Transfer/ImportJobTransfer
ImportJobRun class created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.numberOfProcessedBytes property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.progressPercentage property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.claimedBy property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.leaseExpiresAt property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.attemptCount property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRun.numberOfOmittedErrorRows property created src/Generated/Shared/Transfer/ImportJobRunTransfer
ImportJobRunFileInfo class created src/Generated/Shared/Transfer/ImportJobRunFileInfoTransfer
ImportJobRunError class created src/Generated/Shared/Transfer/ImportJobRunErrorTransfer
ImportJobCollection class created src/Generated/Shared/Transfer/ImportJobCollectionTransfer
ImportJobRunCollection class created src/Generated/Shared/Transfer/ImportJobRunCollectionTransfer
ImportJobRunErrorCollection class created src/Generated/Shared/Transfer/ImportJobRunErrorCollectionTransfer
ImportJobRunCollectionRequest class created src/Generated/Shared/Transfer/ImportJobRunCollectionRequestTransfer
ImportJobRunCollectionResponse class created src/Generated/Shared/Transfer/ImportJobRunCollectionResponseTransfer
ImportJobCollectionRequest class created src/Generated/Shared/Transfer/ImportJobCollectionRequestTransfer
ImportJobCollectionResponse class created src/Generated/Shared/Transfer/ImportJobCollectionResponseTransfer
ImportJobCriteria class created src/Generated/Shared/Transfer/ImportJobCriteriaTransfer
ImportJobConditions class created src/Generated/Shared/Transfer/ImportJobConditionsTransfer
ImportJobRunCriteria class created src/Generated/Shared/Transfer/ImportJobRunCriteriaTransfer
ImportJobRunConditions class created src/Generated/Shared/Transfer/ImportJobRunConditionsTransfer
ImportJobRunErrorCriteria class created src/Generated/Shared/Transfer/ImportJobRunErrorCriteriaTransfer
ImportJobRunErrorConditions class created src/Generated/Shared/Transfer/ImportJobRunErrorConditionsTransfer
ImportJobExportResult class created src/Generated/Shared/Transfer/ImportJobExportResultTransfer
ImportStepResponse class created src/Generated/Shared/Transfer/ImportStepResponseTransfer
ImportStepError class created src/Generated/Shared/Transfer/ImportStepErrorTransfer
ImportPublishEvent class created src/Generated/Shared/Transfer/ImportPublishEventTransfer

4) Register the schema plugins

The engine learns about job types from the schema plugins registered in the DataExchange dependency provider. Register one plugin per data type. The following example registers the product data type of the Product Experience Management capability; see Add a custom data type to Data Exchange Jobs for your own types.

PLUGIN SPECIFICATION PREREQUISITES NAMESPACE
ProductCsvImportSchemaPlugin Provides the products-csv-import job type: its column schema, import steps, export steps, and placeholder values. Product Experience Management feature SprykerFeature\Zed\ProductExperienceManagement\Communication\Plugin

src/Pyz/Zed/DataExchange/DataExchangeDependencyProvider.php

<?php

namespace Pyz\Zed\DataExchange;

use Spryker\Zed\DataExchange\DataExchangeDependencyProvider as SprykerDataExchangeDependencyProvider;
use SprykerFeature\Zed\ProductExperienceManagement\Communication\Plugin\ProductCsvImportSchemaPlugin;

class DataExchangeDependencyProvider extends SprykerDataExchangeDependencyProvider
{
    /**
     * @return array<\Spryker\Zed\DataExchangeExtension\Dependency\Plugin\ImportSchemaPluginInterface>
     */
    protected function getImportSchemaPlugins(): array
    {
        return [
            new ProductCsvImportSchemaPlugin(),
        ];
    }

    /**
     * @return array<\Spryker\Zed\DataExchangeExtension\Dependency\Plugin\ExportSchemaPluginInterface>
     */
    protected function getExportSchemaPlugins(): array
    {
        return [
            new ProductCsvImportSchemaPlugin(),
        ];
    }
}

getImportSchemaPlugins() serves the import, the template download, and the job creation; getExportSchemaPlugins() serves the export with data. A type that is not exportable is registered in the import stack only. The same dependency provider offers getImportPreProcessorPlugins() and getImportPostProcessorPlugins() for plugins that run once before and once after the rows of a run are processed.

Verification

Log in to the Back Office, go to Import & Export > Import, and click Create Import Job. Make sure that every registered job type is offered in the Type field.

5) Configure navigation

Add the following entry to the Back Office navigation:

config/Zed/navigation.xml

<import-export>
    <label>Import &amp; Export</label>
    <title>Import &amp; Export</title>
    <icon>swap_vert</icon>
    <pages>
        <import>
            <label>Import</label>
            <title>Import</title>
            <bundle>data-exchange</bundle>
            <controller>job</controller>
            <action>index</action>
            <visible>1</visible>
        </import>
        <export>
            <label>Export</label>
            <title>Export</title>
            <bundle>data-exchange</bundle>
            <controller>export</controller>
            <action>index</action>
            <visible>1</visible>
        </export>
    </pages>
</import-export>

Build the navigation cache:

console navigation:cache:remove
Verification

Log in to the Back Office and verify that the Import & Export section appears in the main navigation with the Import and Export sub-items.

6) Set up console

Register the console command

Register the import:job:run console command that processes pending import job runs:

src/Pyz/Zed/Console/ConsoleDependencyProvider.php

use Spryker\Zed\DataExchange\Communication\Console\ImportJobRunConsole;

// In getConsoleCommands():
new ImportJobRunConsole(),
Verification

Execute the following and verify it completes without errors:

console import:job:run

The command prints Processing next pending import job run... for every run it picks up, No pending import job run left. when the queue is empty, and Done. Processed N import job run(s)., and exits with code 0. It claims and processes up to IMPORT_JOB_RUNS_PER_INVOCATION claimable runs (default 1), oldest first; when no run is claimable, nothing is processed. After a run, the claimed_by, lease_expires_at, and attempt_count columns of the run are filled in spy_import_job_run.

Configure the scheduler

Register the command to run on a schedule so that pending runs are picked up automatically:

src/Pyz/SymfonyScheduler/src/Pyz/Client/SymfonyScheduler/SymfonySchedulerConfig.php

'import-job-run' => [
    'command' => '$PHP_BIN vendor/bin/console import:job:run',
    'schedule' => '* * * * *',
],

Add the entry to the array that getCronJobs() returns. If the other entries in your file prefix the command with the $logger variable to redirect the output to the scheduler log, prefix this one in the same way.

Each run is claimed by one worker at a time, so you can run the command from several workers in parallel. For how runs are claimed, leased, and reclaimed, see Import run processing.

Every invocation processes up to getImportJobRunsPerInvocation() claimable runs (default 1) and exits; the next scheduled invocation continues the queue. With the default, runs that are queued faster than one per minute wait for a later invocation. To drain the queue faster, raise IMPORT_JOB_RUNS_PER_INVOCATION in the project configuration as shown in Engine limits and error display.

The bound exists because PHP releases the memory a run leaves behind only when the process exits: the publish events collected during a run, and anything a third-party step keeps, stay in the process until then. Keep the value small enough that the runs of one invocation fit into the memory_limit of the CLI process. A larger value also makes an invocation last longer than the one-minute schedule. Overlapping invocations do not process the same run, because each run is claimed by one worker at a time. A value below 1 is a configuration error: the command throws an exception and processes nothing.

Verification

Verify that the scheduler configuration is loaded:

console scheduler:setup

Check that import-job-run appears in the list of scheduled jobs.

7) Build the Back Office frontend

The Export page ships a JavaScript entry point. Build the Back Office frontend to publish it:

console frontend:zed:build
Verification

Log in to the Back Office, go to Import & Export > Export, select a job, and make sure that the download link changes to the selected job.