Writing Custom Metadata For Objects In Object Storage - IBM Aspera HST Admin Manual

A one-time (non-continuous) push that is run from a local disk to an S3 bucket using SSH keys (for more information
on using SSH keys, see
# async -N sync-to-s3 -d /data/data-2017-01 -r ec2_user@192.0.4.24:/data
-i /bobcat/.ssh/private_key -K push -B /mnt/ephemeral/data/db --transfer-
threads=8 --remote-fs-threads=16
One-time bidi from S3 to local disk:
A one-time bidirectional sync that is run from the S3 client to a local disk:
# async -L /mnt/ephemeral/data/log --apply-local-docroot -N bidi_london -d /
data -r bear@192.0.12.442:/data -K bidi -b /mnt/ephemeral/data/db -B /async/
log --transfer-threads=8 --local-fs-threads=16
One-time pull from S3 to S3:
A one-time pull by ec2_user from s3host to /data/2017 in the client S3 storage:
# async -L /mnt/ephemeral/data/log --apply-local-docroot -N s3sync -d /
data/2017 -r ec2_user@s3host:/data/2017-01 -K pull -b /tmp --transfer-
threads=8 --local-fs-threads=16 --remote-fs-threads=16

Writing Custom Metadata for Objects in Object Storage

Files that are uploaded to metadata-compatible storage (S3, Google Cloud, and Azure) can have custom metadata
written with them by using the --tags or --tags64 option. The argument is a JSON payload that specifies the
metadata and that is base64 encoded if it is used as an argument for --tags64.
Metadata Behavior
• All objects that are uploaded in a session have the same metadata.
• If an upload resumes, the metadata of the original transfer is used.
• Multi-session transfers must specify the same metadata.
• Metadata are not retrieved when downloading objects; use the REST API associated with the storage.
• Transfers to object storages that do not support metadata (such as HDFS and Azure Files) fail if metadata is
specified.
Specifying Metadata in JSON
The JSON payload has the general syntax of key-value pairs in a "cloud-metadata" section:
{
"aspera": {
"cloud-metadata": [
{"key1":"value1"},
{"key2":"value2"},
...
] } }
Restrictions on key-value pairs:
• key cannot be ctime, mtime, or atime. These keys are reserved and the transfer fails if they are used.
• key might be case-sensitive, depending on the destination storage type.
• The key-value pair must be less than 1024 characters.
Sample Async Session with Metadata
Creating SSH Keys on page 269), where ec2_user is the transfer user:
| Aspera Sync | 255