IBM Aspera HST Admin Manual

High-speed transfer server

Table of Contents

Quick Links

High-Speed Transfer

Server Admin Guide 3.9.1

PowerLinux

Revision:1978 Generated:04/05/2019 10:17

Table of Contents

Troubleshooting

Summary of Contents for IBM Aspera HST

Page 1 High-Speed Transfer Server Admin Guide 3.9.1 PowerLinux Revision:1978 Generated:04/05/2019 10:17...
Page 2: Table Of Contents
| Contents | ii Contents Introduction....................... 7 What's New?......................9 Get Started with an Aspera Transfer Server............9 Get Started as a Transfer Client................10 Comparison of Aspera File Delivery and Synchronization Tools...... 11 Installation and Upgrades..................13 Requirements...............................13 Installing HST Server............................14 Configuring the Firewall............................ 15 Securing Your SSH Server..........................
Page 3 Ascp General Examples............................114 Ascp File Manipulation Examples........................116 Ascp Transfers with Object Storage and HDFS....................118 Transfers with IBM Aspera On Demand and Cloud-Based HST Servers........... 118 Writing Custom Metadata for Objects in Object Storage..............120 Multi-Session Transfers............................ 121 Using Standard I/O as the Source or Destination....................122 Using Filters to Include and Exclude Files......................126...
Page 4 | Contents | iv Watch Folders and the Aspera Watch Service..........159 Introduction to Watch Folders and the Aspera Watch Service................ 159 Choosing User Accounts to Run Watch Folder Services................160 Creating, Managing, and Configuring Services....................161 Watch Folders..............................164 Getting Started with Watch Folders.....................
Page 5 Overview: Aspera Node API..........................272 Node API Setup..............................272 Node Admin Tool............................. 275 Configuring the IBM Aspera NodeD Service....................277 Securing the Node Service Behind a Proxy.....................281 Backing up and Restoring the Node User Database Records................282 Backing up and Restoring Access Keys (Tenant Data)................... 282 Backing up and Restoring a Node Database....................
Page 6 Securing Content in your Workflow....................341 Testing and Optimizing Transfer Performance....................343 Create an SSL Certificate (Apache).........................345 Enable SSL (Apache)............................347 Log Files................................348 Preserving IBM Spectrum Scale ACLs of Transferred Files................352 Product Limitations............................352 Technical Support....................353 Legal Notice......................353...
Page 7: Introduction
HST Server installed as part of IBM Aspera Faspex, or • an HST Server deployed in object storage as an IBM Aspera On Demand instance, an IBM Aspera on Cloud transfer service node, or an IBM Aspera Transfer Cluster Manager node.
Page 8 HST Server can be incorporated into a scalable Aspera data transfer ecosystem that meets your needs. Your Aspera server can be monitored and managed by IBM Aspera Console, and added as a node to IBM Aspera Faspex, IBM Aspera Shares, IBM Aspera on Cloud, and IBM Aspera Application for Microsoft SharePoint.
Page 9: What's New
| What's New? | 9 When everyone needs to see the same files or you need to be sure that every file is replicated, Aspera Sync provides a high-speed tool to do it. Unique among Aspera's transfer tools, Aspera Sync supports bidirectional synchronization for optimum collaboration and consistency between computers.
Page 10: Get Started As A Transfer Client
| Get Started as a Transfer Client | 10 b) If you prefer to have your users authenticate to the server using SSH keys rather than with passwords, gather their public keys and install them on the server. For instructions, see Setting Up a User's Public Key on the Server on page 38.
Page 11: Comparison Of Aspera File Delivery And Synchronization Tools
| Comparison of Aspera File Delivery and Synchronization Tools | 11 • If you want to synchronize files with your server, with the ability to synchronize bidirectionally, see Aspera Sync on page 218. The async tool requires an additional license on each to run. For a comparison of automatic transfer tools, see Comparison of Aspera File Delivery and Synchronization Tools page 11.
Page 12 | Comparison of Aspera File Delivery and Synchronization Tools | 12 Hot Folders Watch Folders Aspera Sync File delivery or File delivery: File delivery: Synchronization: synchronization Files and folders added to Files and folders added to All file system changes or modified within a Hot or modified within a watch (additions, deletions,...
Page 13: Installation And Upgrades
Product-specific Aspera license file. • Ubuntu 16.04.2 LTS; Kernel: Linux 4.4.0-116-generic; architecture: ppc64-le Note: Your OS version must support Little Endian (LE) and it must run on IBM Power hardware that supports • SSH Server. Version 7.0 or higher is recommended.
Page 14: Installing Hst Server
| Installation and Upgrades | 14 Installing HST Server To install HST Server, log into your computer with root permissions. 1. Download the HST Server installer. Use the credentials provided to your organization by Aspera to access: https://downloads.asperasoft.com/en/downloads/4 If you need help determining your firm's access credentials, contact your Aspera account manager. 2.
Page 15: Configuring The Firewall
| Installation and Upgrades | 15 Copy and paste your license key string into it, then save and close the file. To verify the license information, run the following command: $ ascp -A To update your product license after the installation, see Updating the Product License on page 21.
Page 16: Securing Your Ssh Server
| Installation and Upgrades | 16 Important: Aspera strongly recommends running the SSH server on a non-default port (allowing inbound SSH connections on TCP/33001, and disallowing inbound connections on TCP/22) to ensure that your server remains secure from SSH port scan attacks. For instructions on how to change your SSH port, see Securing Your SSH Server on page 16.
Page 17: Changing And Securing The Tcp Port
| Installation and Upgrades | 17 Changing and Securing the TCP Port SSH servers, including the OpenSSH suite included with your product, listen for incoming connections on TCP Port 22 by default. As such, Port 22 is subject to numerous unauthorized login attempts by hackers who attempt to access unsecured servers.
Page 18 | Installation and Upgrades | 18 Depending on your sshd_config file, you might have additional instances of AllowTCPForwarding that are set to the default Yes. Review your sshd_config file for other instances and disable if necessary. Disabling TCP forwarding does not improve security unless users are also denied shell access, because they can still install their own forwarders.
Page 19: Configuring Transfer Server Authentication With The Host-Key Fingerprint
| Installation and Upgrades | 19 10.0.111.200:53865 173.194.202.188:5228 ESTABLISHED 10.0.111.200:53876 10.0.9.16:445 TIME_WAIT 10.0.111.200:55164 208.85.40.20:443 ESTABLISHED 10.0.111.200:55335 207.200.35.240:443 ESTABLISHED 10.0.111.200:55444 67.199.110.81:443 ESTABLISHED 10.0.111.200:56278 104.24.11.90:443 ESTABLISHED If your server is under attack, you might see output similar to the following, in which the same IP address attempts to connect to contiguous ports (hundreds or thousands of times) and the connection is timing out (reporting a status of TIME_WAIT): 10.0.111.200:53402...
Page 20: Testing A Locally Initiated Transfer
| Installation and Upgrades | 20 b) Set the SSH host key fingerprint in aspera.conf. (Go to the next step to set the host key path instead). # asconfigurator -x "set_server_data;ssh_host_key_fingerprint,fingerprint" This command creates a line similar to the following example of the <server> section of aspera.conf: <ssh_host_key_fingerprint>7qdOwebGGeDeN7Wv+2dP3HmWfP3 </ssh_host_key_fingerprint>...
Page 21: Updating The Product License
| Installation and Upgrades | 21 This message provides the following information: Item Description 100 MB The name of the file that is being transferred. The percentage completed. 28 MB The amount transferred. 2.2 Gbps The current transfer rate. 01:02 ETA The estimated time the transfer will complete.
Page 22: Set Up The Hst Server Web Ui
Aspera Technical Support on page 353. Note: If HST Server and IBM Aspera Faspex versions 4.1.0 or newer are installed on the same computer, they cannot use the same Apache. 1. Locate and open your Apache configuration file.
Page 23 | Set up the HST Server Web UI | 23 Add the following section at the end of the configuration file if it is not already there: #BEGIN_ASPERA <Directory /opt/aspera/var/webtools> AllowOverride All Require all granted </Directory> <Directory /opt/aspera/var/webtools/scripts> AddHandler cgi-script .pl SetHandler cgi-script Options +ExecCGI AllowOverride All...
Page 24 Important: Do not paste the example output shown below into your sudoers file. Paste the output generated when you ran the enablesecure script as described above. # BEGIN IBM Aspera High-Speed Transfer Server # The user account that runs the web server will impersonate # the logged-in user to present that user's files and folders.
Page 25: Configuring Your Web Ui Settings
| Set up the HST Server Web UI | 25 Configuring your Web UI Settings Configure transfer settings for the HST Server web UI by manually editing aspera.conf. 1. Open aspera.conf in a text editor with admin or root privileges. /opt/aspera/etc/aspera.conf 2.
Page 26 HST Server allows the user to transfer to and from the file system on the indicated transfer server machine. MinimumConnectVersion Set the minimum version of IBM Aspera Connect Version 2.8.0.0 that must be installed in order for users to be able number to use HST Server.
Page 27: Customize The Appearance Of The Web Ui
| Set up the HST Server Web UI | 27 • Require strong SSL. Run the following command to require that the SSL security protocol be TLS version 1.2 or higher: # /opt/aspera/bin/asconfigurator -x "set_server_data;ssl_protocol,tlsv1.2" This setting applies to HTTP(S) fallback transfers as well. 6.
Page 28 13, and cookies must be enabled in the client's browser. a) Go to the following address in the browser: HTTP http://server_ip_or_name/aspera/user HTTPS https://server_ip_or_name/aspera/user b) The IBM Aspera Connect banner appears with a link to download and install the latest version of Connect:...
Page 29: Configuring Http And Https Fallback
| Set up the HST Server Web UI | 29 Click Download latest version. c) Once the installation is complete, refresh the page and the web UI for appears: d) In the web UI, click Upload and select one or more files to upload to HST Server. Note: When you add files to the web UI, do not use the following characters in the filenames: / \ "...
Page 30 | Set up the HST Server Web UI | 30 Limitations: • Folders that are symbolic links cannot be downloaded directly by using HTTP fallback. Folders that are symbolic links are processed correctly when their parent folder is the source. •...
Page 31 | Set up the HST Server Web UI | 31 <http_port>8080</http_port>  <https_port>8443</https_port>  </http_server> </CONF> To manually inspect and edit aspera.conf, open it from the following directory: /opt/aspera/etc/aspera.conf 3. Review additional HTTP fallback settings. Additional HTTP fallback settings can be set in aspera.conf: Field Description...
Page 32 | Set up the HST Server Web UI | 32 <https_port>8443</https_port> </http_server> </CONF> 4. Set a token encryption key. If HTTP/HTTPS fallback is enabled, a token encryption key is required. If HTTP/HTTPS is configured without the encryption key, initiating a transfer with the download button generates the following error: Error: internal error - unable to start token generation The token encryption key is the secret text string used for authorizing transfers configured to require a token.
Page 33: Server Set Up Methods
334. • If users connect to the server by providing IBM Aspera Shares credentials or by providing Node API credentials that are associated with the transfer user, changes to a user's configuration, such as their docroot, are not applied to the user until asperanoded is restarted.
Page 34 | Set up Users and Groups | 34 Note: If you have Apache 2.4.4, you may get authentication errors when trying to provide a password to view the site. As a workaround, run htpasswd with the -b option and enter the password on the command line as follows: # htpasswd -b /opt/aspera/etc/webpasswd asp1 password 2.
Page 35: Setting Up Transfer Groups
| Set up Users and Groups | 35 Confirm that the user's shell updated by running the following command and looking for /bin/aspshell at the end of the output: # grep username /etc/passwd username:x:501:501:...:/home/username:/bin/aspshell Note: If you use OpenSSH, sssd, and Active Directory for authentication: To make aspshell the default shell for all domain users, first set up a local account for server administration because this change affects all domain users.
Page 36 | Set up Users and Groups | 36 Ensure that you have an existing user group on your operating system, or create a new user group. Please refer to your operating system's documentation for information on creating user groups. HST Server reads group information from the following file: /etc/group 2.
Page 37 | Set up Users and Groups | 37 Settings Description Configuration Precedence on page 38 When a user is a member of multiple groups, the precedence setting can be used to determine priority. aspera.conf - Authorization Configuration Connection permissions, token key, and encryption page 40 requirements.
Page 38: Configuration Precedence
| Set up Users and Groups | 38 or for Linux systems that use init.d: # service asperacentral restart Configuration Precedence HST Server applies configuration settings in this order: 1) user settings, 2) group settings (and if a user belongs to more than one group, precedence can be set for each group), 3) global settings, 4) default settings.
Page 39: Testing A User-Initiated Remote Transfer
HST Server and upload a file. Prerequisites: • Client: Install an Aspera client application, such as the freely available IBM Aspera Desktop Client or IBM Aspera Command-Line Interface, on the client computer. • Server: HST Server must have at least one Aspera transfer user (a system user account that is configured to authenticate Aspera transfers) configured on it.
Page 40: Configure The Server From The Command Line
| Configure the Server from the Command Line | 40 This command specifies the following values for the transfer: Item Argument Example Value TCP Port -P port -P 33001 Set the TCP port to start the transfer session. Transfer Direction --mode=direction --mode=send Specify if the server is the...
Page 41 Token-based transfers are typically used by web applications such as IBM Aspera Faspex and IBM Aspera Shares and require a Token Encryption Key. Incoming External Provider URL Set the URL of the external authorization...
Page 42: Aspera.conf - Transfer Configuration
| Configure the Server from the Command Line | 42 Field Description Values Default Outgoing Transfers To enable users to transfer from this computer, allow, allow leave the default setting of allow. Set to deny deny, or to prevent transfers from this computer. Set to token token to allow only transfers initiated with valid tokens from this computer.
Page 43 | Configure the Server from the Command Line | 43 User, Group and Default Configurations on page 315 and run the following command to retrieve a complete default aspera.conf that includes the asconfigurator syntax for each setting: # /opt/aspera/bin/asuserdata -+ 1.
Page 44 | Configure the Server from the Command Line | 44 <target_queue>unset</target_queue>  </network_rc> </flow> </bandwidth> </out> <encryption> <allowed_cipher>any</allowed_cipher>  <fips_mode>false</fips_mode>  <content_protection_required>false </content_protection_required>  <content_protection_secret></content_protection_secret>...
Page 45 | Configure the Server from the Command Line | 45 Field Description Values Default Incoming Target Rate The default initial rate for incoming positive integer 10000 Default (Kbps) transfers, in kilobits per second. If allowed ("Incoming Target Rate Lock" is set to false), clients can modify this rate in real time.
Page 46 | Configure the Server from the Command Line | 46 Field Description Values Default to determine bandwidth allocation among transfers. • any - The server does not deny any transfer based on policy setting. Note: Setting to any allows clients to request a fixed bandwidth policy.
Page 47 | Configure the Server from the Command Line | 47 Field Description Values Default rate is reduced to the minimum rate until other traffic decreases. • fixed - Attempt to transfer at the specified target rate, regardless of network or storage capacity. This can decrease transfer performance and cause problems on the target storage.
Page 48 | Configure the Server from the Command Line | 48 Field Description Values Default many FASP sessions run at the same time), and shallow buffers (limited packet queuing capability of a router). When LAQ is set, then it uses the FD31 RTT predictor unless a different RTT predictor is explicitly set.
Page 49 | Configure the Server from the Command Line | 49 Field Description Values Default setting of unlimited applies no target rate cap. Outgoing Target Rate The default initial rate for outgoing positive integer 10000 Default (Kbps) transfers, in kilobits per second. If allowed ("Outgoing Target Rate Lock"...
Page 50 | Configure the Server from the Command Line | 50 Field Description Values Default Outgoing Bandwidth The bandwidth policies that outgoing high, fair, Policy Allowed transfers can use. Aspera transfers can use low, or any high, fair, low, or fixed bandwidth policies to determine bandwidth allocation among transfers.
Page 51 | Configure the Server from the Command Line | 51 Field Description Values Default but less aggressive when sharing bandwidth with other network traffic. When congestion occurs, the transfer rate is reduced to the minimum rate until other traffic decreases. •...
Page 52 | Configure the Server from the Command Line | 52 Field Description Values Default Note: The LAQ module is an experimental rate control module that is designed to solve issues with target rate overdrive, high concurrency (when many FASP sessions run at the same time), and shallow buffers (limited packet queuing capability of a router).
Page 53 | Configure the Server from the Command Line | 53 Field Description Values Default a passphrase–the file remains encrypted. If HTTP fallback occurs during upload, then– despite entering a passphrase–the files are not encrypted. Strong Password Set to true to require that the password true or false false Required for Content...
Page 54 | Configure the Server from the Command Line | 54 Field Description Values Default the setting. These settings use the CFB or GCM mode depending on the client version and cipher requested. Supports all client versions. • aes-128-cfb, aes-192-cfb, or aes-256-cfb - require that transfers use the CFB encryption mode and a cipher key that is as long or longer than...
Page 55 | Configure the Server from the Command Line | 55 Field Description Values Default Disable Packet Batching Set to true to send data packets back-to- true or false false back (no sending a batch of packets). This results in smoother data traffic at a cost of higher CPU usage.
Page 56 | Configure the Server from the Command Line | 56 Field Description Values Default Run File Validation at Validate files by using the specified uri, none File Stop method when file transfer is complete and lua_script, or file is closed. For more information, see none Inline File Validation on page 83.
Page 57 | Configure the Server from the Command Line | 57 Field Description Values Default validation_threshold for the same file. Validation URI Use the specified external URL none for validation calls. When this parameter is defined, at least two validations, validation_file_start and validation_file_stop will happen for every file.
Page 58: Controlling Bandwidth Usage With Virtual Links (Command Line)
| Configure the Server from the Command Line | 58 Run the following command to confirm that the XML is correctly formatted and the parameter settings are valid: # /opt/aspera/bin/asuserdata -v Controlling Bandwidth Usage with Virtual Links (Command Line) FASP transfers attempt to transfer at the maximum transfer rate available. However, too many simultaneous transfers can overwhelm your storage or leave little bandwidth available for other network activity.
Page 59 | Configure the Server from the Command Line | 59 Description Values Default the same virtual bandwidth cap needs to have the same port number. To avoid port conflicts, it is recommended to use the default UDP port 55001. Do NOT set the port number to the same one used by FASP data transfer (33001).
Page 60 | Configure the Server from the Command Line | 60 # asconfigurator -x "set_group_data;group_name,groupname;transfer_out_bandwidth_aggregate_trunk_id,id" # asconfigurator -x "set_user_data;user_name,username;transfer_out_bandwidth_aggregate_trunk_id,id" For example, to set Vlink 108 as the default for transfers out and set Vlink 109 to the user aspera_user_1 for transfers out, run the following commands: # asconfigurator -x "set_node_data;transfer_out_bandwidth_aggregate_trunk_id,108"...
Page 61: Global Bandwidth Settings (Command Line)
| Configure the Server from the Command Line | 61 b) Lock the minimum default transfer rate for select users or globally. The following commands lock minimum incoming and outgoing transfer rates for all users: # asconfigurator -x "set_node_data;transfer_in_bandwidth_flow_min_rate_lock,true" # asconfigurator -x "set_node_data;transfer_out_bandwidth_flow_min_rate_lock,true"...
Page 62: Increasing Transfer Performance By Using An Rtt Predictor
| Configure the Server from the Command Line | 62 The capacity of the Vlink is set within a <schedule> tag because the capacity can be scheduled as one value during a specified time period, and a default value at all other times. For more information on this configuration, see the knowledge base article Specifying a time varying schedule for a Vlink at https://support.asperasoft.com/hc/en-us/ articles/216127698-Specifying-a-time-varying-schedule-for-a-Vlink.
Page 63: Aspera.conf - File System Configuration
| Configure the Server from the Command Line | 63 To enable dynamic target queuing for incoming (transfer_in) or outgoing (transfer_out) transfers, run the following command: # asconfigurator -x "set_node_data;transfer_{in| out}_bandwidth_flow_network_rc_target_queue,dynamic" Command line options override server settings. If no predictor is specified on the client command line, in the client's aspera.conf, or in the server's aspera.conf, then no predictor is used for the transfer.
Page 64 | Configure the Server from the Command Line | 64 <use_file_cache>true</use_file_cache>  <max_file_cache_buffer>0</max_file_cache_buffer>  <resume_suffix>.aspx</resume_suffix>  <symbolic_links>follow,create</symbolic_links>  <preserve_attributes> </preserve_attributes>  <overwrite>allow</overwrite> ...
Page 65 | Configure the Server from the Command Line | 65 Field Description Values Default do so, edit the absolute path setting by adding the IP address using the following syntax: <absolute peer_ip="ip_address">path</ absolute> File Restriction undefined Note: A configuration (global, group, or user) can (total have a docroot or a file restriction;...
Page 66 | Configure the Server from the Command Line | 66 Field Description Values Default Write Block Size (bytes) Set the maximum bytes within a block that an ascp positive receiver can write to disk. The default of zero causes integer, the Aspera receiver to use its default internal buffer where size, which may vary by operating system.
Page 67 | Configure the Server from the Command Line | 67 Field Description Values Default Compression Method for Set the compression method to apply to transfers. It lz4, qlz, File Transfer applies to both the client and server. zlib, or none Use File Cache Set to true (default) to enable per-file memory true or...
Page 68 | Configure the Server from the Command Line | 68 Field Description Values Default File Manifest Set to text to generate a text file "receipt" of all text, none files within each transfer session. Set to disable to disable, not create a File Manifest. The file manifest is a file or none containing a list of everything that was transferred in a given transfer session.
Page 69: Aspera.conf - Transfer Server Configuration
| Configure the Server from the Command Line | 69 Field Description Values Default Partial File Name Suffix Set the filename extension on the destination computer text string blank while the file is being transferred. Once the file has been completely transferred, this filename extension is removed.
Page 70 | Configure the Server from the Command Line | 70 <validation_timeout>300</validation_timeout>  </central_server> 3. Edit settings as needed. Central Server Settings Reference Setting Description Values Default Address The network interface address on which the transfer Valid IPv4 127.0.0.1 server listens.
Page 71: Aspera.conf - Filters To Include And Exclude Files
| Configure the Server from the Command Line | 71 Run the following command to confirm that the XML is correctly formatted and the parameter settings are valid: # /opt/aspera/bin/asuserdata -v aspera.conf - Filters to Include and Exclude Files Filters refine the list of source files (or directories) to transfer by indicating which to skip or include based on name matching.
Page 72 | Configure the Server from the Command Line | 72 Note: When a directory is excluded, directories and files in it are also excluded and are not compared to any following rules. 3. If the file does not match, it is compared to the next rule and repeats the process for each rule until a match is found or until all rules have been tried.
Page 73 | Configure the Server from the Command Line | 73 • Set global include and exclude filters: # asconfigurator -x "set_node_data;file_filters,|+ file.txt|- *.txt" Results in aspera.conf: <default> <file_system> <filters> <filter>+ file.txt</filter> <filter>- *.txt</filter> </filters> </file_system> </default> • Sets filters for user asp1: # asconfigurator -x "set_user_data;user_name,asp1;file_filters,|+ abc/wxy/ tuv/**|- abc/**/def"...
Page 74: Server-Side Encryption-At-Rest (Ear)
| Configure the Server from the Command Line | 74 Server-Side Encryption-at-Rest (EAR) When files are uploaded from an Aspera client to HST Server, server-side encryption-at-rest (EAR) saves files on disk in an encrypted state. When downloaded from HST Server, server-side EAR first decrypts files automatically, and then the transferred files are written to the client's disk in an unencrypted state.
Page 75: Reporting Checksums
The checksum reported by the source can be retrieved in the destination logs, a manifest file, in IBM Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.
Page 76 | Configure the Server from the Command Line | 76 Description asconfigurator Option ascp Option sha384 - Calculate and report a SHA-384 checksum. sha512 - Calculate and report a SHA-512 checksum. Note: The default value for the ascp option is none, in which case the reported checksum is the one configured on the server, if any.
Page 77 | Configure the Server from the Command Line | 77 For example: # ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=/ tmp file aspera_user_1@189.0.202.39:/destination_path Setting up a Pre/Post-processing Script An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a pre/post- processing script to report the values.
Page 78: Server Logging Configuration For Ascp And Ascp 4
| Configure the Server from the Command Line | 78 Mac OS X: $ md5 filepath Linux and Linux on z Systems: # md5sum filepath AIX: # csum -h MD5 filepath Solaris: # digest -a md5 -v filepath 5. Compare the checksum of the file on the source with the one reported by Aspera. •...
Page 79 | Configure the Server from the Command Line | 79 Object Description information). Default: 10 MB. Applies to ascp and ascp4 transfers. level The logging level. Valid values are log (default), dbg1, or dbg2. Applies only to ascp transfers. These commands modify the <logging> sub-section of the <default> section of aspera.conf (or you can manually edit the file): <default>...
Page 80: Out-Of-Transfer File Validation
More efficient use of system resources because the ascp sessions can close before validation is completed. • Out-of-transfer file validation is applied to transfers that use HTTP(S) fallback transport. • Files are explicitly reported as "validating" to IBM Aspera Faspex through asperacentral. Files that are validated inline are reported as "transferring".
Page 81 | Configure the Server from the Command Line | 81 1. Associate the transfer user with a Node API username and password, if not already configured. # /opt/aspera/bin/asnodeadmin -a -u node_username -p node_password - x transfer_user To view existing Node API users and the transfer users associated with them, run the following command: # /opt/aspera/bin/asnodeadmin -l 2.
Page 82 | Configure the Server from the Command Line | 82 Note: If a validator does not update the file status within the validation timeout, the file status is reset to "to_be_validated" and the file is released from the validator so that the file can be validated by a different validator.
Page 83: Inline File Validation
| Configure the Server from the Command Line | 83 In the output, locate the value for "persistent_store". If it is not set to "enable", run the following command: # asconfigurator -x "set_central_server_data;persistent_store,enable" 4. Ensure that empty files and files that exist at the destination (and are skipped by the transfer session) are not ignored.
Page 84 | Configure the Server from the Command Line | 84 a) Open web.xml and edit the <servlet> and <servlet_mapping> sections to provide the necessary information for validation. The <servlet-name> (URL handler) value is also configured in aspera.conf (in the next step) and any custom code (such as file filtering, see Inline File Validation with URI on page 85).
Page 85: Inline File Validation With Uri
| Configure the Server from the Command Line | 85 4. If you schedule validation at a file size threshold, set the threshold. # asconfigurator -x "set_user_data;user_name,username;validation_threshold_kb,size" 5. Configure multi-threaded validation. By default, inline validation is set to use 5 threads. If the number of validation threads is not set to 1, then multiple threads may perform different types of validations for different (or the same) files at the same time.
Page 86 | Configure the Server from the Command Line | 86 "error" : { "code" : "1022", "message" : "The file fails validation" Custom Code for Including and Excluding Files Administrators can include or exclude files by enabling whitelisting, blacklisting, or another method of their own design.
Page 87: Inline File Validation With Lua Script
| Configure the Server from the Command Line | 87 response.getOutputStream().println(jsonObject.toString()); response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); else { JsonObject jsonObject = new JsonObject(); jsonObject.addProperty("success", true); jsonObject.addProperty("data", "File is ok to transfer"); jsonObject.addProperty("code", 1); response.getOutputStream().println(jsonObject.toString()); response.setStatus(HttpServletResponse.SC_OK); return; Inline File Validation with Lua Script To use a Lua script for inline file validation, the administrator creates a base-64 encoded Lua action script and sets the path to that script in the <transfer>...
Page 88 | Configure the Server from the Command Line | 88 Field Description Values Value used to identify a validation String "session_id" session Server hostname or IP address Hostname or IP address "host" "client_ip" Client IP address IP address "user" SSH account login name String "user_id"...
Page 89 | Configure the Server from the Command Line | 89 Status Lua return value transfer start, returning it at any other state results in an error. Validation failure LRET_INVALID, optionally followed by a failure description string Script error LRET_ERROR followed by an error number or error description string Lua File Interfaces Three Lua file interfaces allow Lua scripts to reference files: lua_stat, lua_file_delete, and lua_rename.
Page 90: File Pre- And Post-Processing (Prepost)
| File Pre- and Post-Processing (Prepost) | 90 • as_dbg2 • as_dbg3 • as_dbg4 To use the ascp log functions in your Lua script, replace as with lua. Miscellaneous Lua Interfaces • lua_override_ear_secret("secret") Override the server-side encryption-at-rest (EAR) secret that is set in aspera.conf with the specified secret. File Pre- and Post-Processing (Prepost) HST Server can run file processing scripts that run before or after a transfer session or file transfer.
Page 91: Pre/Post Variables
| File Pre- and Post-Processing (Prepost) | 91 This file runs the Perl script aspera-notif.pl, which is an email notification script that sends emails (according to user-defined filters) to one or more recipients. Filters and lists are defined in the Aspera configuration file aspera.conf, which is located in /opt/aspera/etc.
Page 92 | File Pre- and Post-Processing (Prepost) | 92 Variable Description Values Example PEER The peer name or IP address. string or valid "$PEER" == 10.0.0.1 IPv4 address SECURE Transfer encryption. "$SECURE" == no • • SESSIONID The session id. string "$SESSIONID"...
Page 93: Pre/Post Script Examples
| File Pre- and Post-Processing (Prepost) | 93 Variable Description Values Example TOTALSIZE The total size of files being transferred positive "$TOTALSIZE" >= 500000000 in bytes. integer For Files Variable Description Values Example DELAY The measured network delay, in ms. positive "$DELAY"...
Page 94 | File Pre- and Post-Processing (Prepost) | 94 In the shell script, change file and directory permissions after receiving, and log into the file /tmp/p.log: #!/bin/bash if [ $TYPE == File ]; then if [ $STARTSTOP == Stop ]; then echo "The file is: $FILE"...
Page 95: Email Notifications
| Email Notifications | 95 Email Notifications Email notifications are a special type of prepost processing that can be configured on HST Server . Setting Up Email Notifications The email notification feature is a built-in pre- and post-processing application that generates customized emails based on transfer events.
Page 96 | Email Notifications | 96 MAILCONF Field Description Values Example The email address to a valid email FROM FROM="admin@example.com" send notifications from. address (Required) MAILSERVER The outgoing mail server A valid URL MAILSERVER="mail.example.com" (SMTP). (Required) SUBJECT General subject of the email. text string SUBJECT="Transfer:%{STATE}"...
Page 97: Email Notification Examples
| Email Notifications | 97 FILTER Field Description Values Example The email subject, preceded by the text string SUBJECTPREFIX SUBJECTPREFIX="Sub" SUBJECT in <MAILCONF />. The email body, preceded by the text string BODYPREFIX BODYPREFIX="Txt" BODYTEXT in <MAILCONF />. TOTALBYTESOVER Send email when total bytes positive TOTALBYTESOVER="9000"...
Page 98 /> </EMAILNOTIF> 3. Send different notifications for regular transfers and for IBM Aspera Sync transfers. In the example below, when Aspera Sync triggers a transfer (assuming only Aspera Sync uses the folder /sync- folder), an email message is sent to "mediaGroup". When a regular transfer occurs (files are sent to /upload), a different notification is sent to "mediaLead"...
Page 99: Ascp: Transferring From The Command Line With Ascp
| ascp: Transferring from the Command Line with Ascp | 99 " /> </EMAILNOTIF> ascp: Transferring from the Command Line with Ascp Ascp is a scriptable FASP transfer binary that enables you to transfer to and from Aspera transfer servers to which you have authentication credentials.
Page 100 | ascp: Transferring from the Command Line with Ascp | 100 Specifying Files, Directories, and Paths • Specify paths on the remote computer relative to the transfer user's docroot. If the user has a restriction instead of a docroot, specify the full path, which must be allowed by the restriction. •...
Page 101 | ascp: Transferring from the Command Line with Ascp | 101 Note: If the local token is a basic or bearer token, the access key settings for cipher and preserve_time are not respected and the server settings are used. To set the cipher and timestamp preservation options as a client, set them in the command line.
Page 102 | ascp: Transferring from the Command Line with Ascp | 102 Galois/counter mode (GCM). The GCM mode encrypts data faster and increases transfer speeds compared to the CFB mode, but the server must support and permit it. Cipher rules The encryption cipher that you are allowed to use depends on the server configuration and the version of the client and server: •...
Page 103 | ascp: Transferring from the Command Line with Ascp | 103 Server, v3.9.0+ Server, v3.9.0+ Server, v3.9.0+ Server, v3.8.1 or older AES-XXX-GCM AES-XXX-CFB AES-XXX AES-XXX Client, v3.9.0+ AES-XXX Client, v3.8.1 or server refuses older transfer AES-XXX --check-sshfp=fingerprint Compare fingerprint to the server SSH host key fingerprint that is set with <ssh_host_key_fingerprint>...
Page 104 | ascp: Transferring from the Command Line with Ascp | 104 scripts, Perl scripts, Windows batch files, and executable binaries that can invoke a variety of environment variables. For instructions, see File Pre- and Post-Processing (Prepost) on page 90. --exclude-newer-than=mtime, --exclude-older-than=mtime Exclude files (but not directories) from the transfer, based on when the file was last modified.
Page 105 | ascp: Transferring from the Command Line with Ascp | 105 • Only one --file-list or --file-pair-list option is allowed per ascp session. If multiple lists are specified, only the last one is used. • Only files and directories specified in the file list are transferred; any sources specified on the command line are ignored.
Page 106 | ascp: Transferring from the Command Line with Ascp | 106 • Only files from the file pair list are transferred; any additional source files specified on the command line are ignored. • If the source paths are URIs, the file list cannot exceed 24 KB. For additional examples, see Ascp General Examples on page 114.
Page 107 | ascp: Transferring from the Command Line with Ascp | 107 Note: If the destination is a URI path, then the only valid options are -k 0 and -k 1 and no .aspx file is created. -L local_log_dir[:size] Log to the specified directory on the client computer rather than the default directory. Optionally, set the size of the log file (Default: 10 MB).
Page 108 | ascp: Transferring from the Command Line with Ascp | 108 If --multi-session-threshold is not used, the threshold value is taken from the setting for <multi_session_threshold_default> in the aspera.conf file on the client. If not found in aspera.conf on the client, the setting is taken from aspera.conf on the server. The command-line setting overrides any aspera.conf settings, including when the command-line setting is 0 (zero).
Page 109 | ascp: Transferring from the Command Line with Ascp | 109 Preserve file timestamps for access and modification time. Equivalent to setting --preserve- modification-time and --preserve-access-time (and --preserve-creation- time on Windows). Timestamp support in object storage varies by provider; consult your object storage documentation to determine which settings are supported.
Page 110 | ascp: Transferring from the Command Line with Ascp | 110 Preserve Access Control Lists (ACL) data for macOS, Windows, and AIX files. To preserve ACL data for other operating systems, use --preserve-xattrs. See also --remote-preserve- acls. Default: none. • native - Preserve attributes using the native capabilities of the file system.
Page 111 | ascp: Transferring from the Command Line with Ascp | 111 • Extended attributes are not preserved for directories. • If Ascp is run by a regular user, only user-level attributes are preserved. If run as superuser, all attributes are preserved. •...
Page 112 | ascp: Transferring from the Command Line with Ascp | 112 at the start of each second and incremented for each additional file saved during that second. The saved copies retain the attributes of the original. Not supported for URI path destinations. --skip-special-files Skip special files, such as devices and pipes, without reporting errors for them.
Page 113 | ascp: Transferring from the Command Line with Ascp | 113 Note: If you are authenticating on a Windows computer as a domain user, the transfer server strips the domain from the username. For example, Administrator is authenticated rather than DOMAIN\Administrator.
Page 114: Ascp General Examples
| ascp: Transferring from the Command Line with Ascp | 114 -y {0|1} If set to "1", use the HTTP fallback transfer server when a UDP connection fails. (Default: 0) Ascp General Examples Use the following Ascp examples to craft your own transfers. To describe filepaths, use single-quote (' ') and forward-slashes (/) on all platforms.
Page 115 | ascp: Transferring from the Command Line with Ascp | 115 This command and file pair list create the following directories within the transfer user's docroot on the destination: /tmp1/folder1 /tmp2/folder2 Network shared location transfer • Send files to a network shares location \\1.2.3.4\nw-share-dir, through the computer 10.0.0.2: # ascp local-dir/files root@10.0.0.2:"//1.2.3.4/nw-share-dir/"...
Page 116: Ascp File Manipulation Examples
| ascp: Transferring from the Command Line with Ascp | 116 • Transfer random data and do not save result to disk (no source or destination storage required) Transfer 10 MB of random data from 10.0.0.2 as user root and do not save result to disk: #ascp --mode=send --user=root --host=10.0.0.2 faux:///dummy?10m faux:// Ascp File Manipulation Examples Ascp can manipulate files and directories as part of the transfer, such as upload only the files in the specified source...
Page 117 | ascp: Transferring from the Command Line with Ascp | 117 Upload a file, /monday/file1, and a directory, /tuesday/*, to the /storage/ directory on the server, while stripping the srcbase path and preserving the rest of the file structure. The content is saved as /storage/ monday/file1and /storage/tuesday/* on the server.
Page 118: Ascp Transfers With Object Storage And Hdfs
If your transfer user account has a docroot set that includes credentials or credentials are configured in the .properties file, ascp transfers to and from Alibaba Cloud, Amazon S3, IBM COS - S3, Google Cloud Storage, Akamai, SoftLayer, Azure, and are the same as regular ascp transfers.
Page 119 | ascp: Transferring from the Command Line with Ascp | 119 Storage Platform ascp Syntax and Examples Upload example: # ascp --mode=send --user=bear -- host=s3.asperasoft.com bigfile.txt s3://1K3C18FBWF9902:GEyU...AqXuxtTVHWtc@s3.amazonaws.com/ demos2014 Download syntax: # ascp options --mode=recv --user=username -- host=s3_server_addr s3://access_id:secret_key@s3.amazonaws.com/my_bucket my_source_path destination_path Download example: # ascp --mode=recv --user=bear --host=s3.asperasoft.com s3://1K3C18FBWF9902:GEyU...AqXuxtTVHWtc@s3.amazonaws.com/...
Page 120: Writing Custom Metadata For Objects In Object Storage
Download example: # ascp --mode=recv --user=bear --host=10.0.0.5 gs:///2017_transfers/data/bigfile.txt /data HDFS Aspera recommends running ascp transfers with HDFS with a docroot configured. IBM COS - S3 Upload syntax: # ascp options --mode=send --user=username -- host=server_address source_files s3://access_id:secret_key@accessor_endp Upload example: # ascp --mode=send --user=bear -- host=s3.asperasoft.com bigfile.txt...
Page 121: Multi-Session Transfers
| ascp: Transferring from the Command Line with Ascp | 121 The JSON payload has the general syntax of key-value pairs in a "cloud-metadata" section: "aspera": { "cloud-metadata": [ {"key1":"value1"}, {"key2":"value2"}, ] } } Restrictions on key-value pairs: • key cannot be ctime, mtime, or atime. These keys are reserved and the transfer fails if they are used. •...
Page 122: Using Standard I/O As The Source Or Destination
IAM role. • If you are self-managing an Aspera server or cluster of Aspera servers in the cloud (you installed IBM Aspera High-Speed Transfer Server on a VM), you must configure the server for multi-session transfers. File-spliting with multi-session threshold The value of the multi-session threshold depends on the target rates that a single ascp transfer can achieve on your system for files of a given size, as well as the typical distribution of file sizes in the transfer list.
Page 123 | ascp: Transferring from the Command Line with Ascp | 123 A named pipe can be specified as a stdio destination, with the syntax stdio:///path for single files, or stdio- tar:///path for multiple files, where path is the path of the named pipe. If a docroot is configured on the destination, then the transfer goes to the named pipe docroot/path.
Page 124 | ascp: Transferring from the Command Line with Ascp | 124 Size: file_size file 2 data To download one or more files to stdout, set the destination as stdio-tar://. Normal status output to stdout is suppressed during downloads because the transfer output is streamed to stdout. The data sent to stdout has the same encoding as described for uploads.
Page 125 | ascp: Transferring from the Command Line with Ascp | 125 • Upload two files, myfile1 and myfile2, to the named pipe /tmp/mypipe (streaming output) on the server at 10.0.0.2. Transfer at 100 Mbps. ascp -l 100m --mode=send --user=username --host=10.0.0.2 myfile1 myfile2 stdio-tar:////tmp/mypipe This sends an encoded stream of myfile1 and myfile2 (with the format of sourcefile in the upload example) to the pipe /tmp/mypipe.
Page 126: Using Filters To Include And Exclude Files
| ascp: Transferring from the Command Line with Ascp | 126 Where sourcefile contains the following: File: myfile1 Size: 1025 << 1025 bytes of data>> File: myfile2 Size: 20 <<20 bytes of data>> Using Filters to Include and Exclude Files Filters refine the list of source files (or directories) to transfer by indicating which to skip or include based on name matching.
Page 127 | ascp: Transferring from the Command Line with Ascp | 127 Example Transfer Result To transfer only the files and directories with names that do not match rule1 but do match rule2 use: ascp -E 'rule1' -N 'rule2' -N '/**/' -E '/**' Filtering Rule Application Filters can be specified on the ascp command line and in aspera.conf.
Page 128 | ascp: Transferring from the Command Line with Ascp | 128 Standard Globbing: Wildcards and Special Characters The only recognized path separator. Quotes any character literally, including itself. \ is exclusively a quoting operator, not a path separator. Matches zero or more characters, except "/" or the . in "/.". Matches any single character, except "/"...
Page 129 | ascp: Transferring from the Command Line with Ascp | 129 Wildcard Example Matches Does Not Match /** at end of rule abc/** abc/.file abc/d/e/ abc/ / at end of rule abc/*/ abc/dir abc/file no / at end of rule abc (file) abc/ / at start of rule...
Page 130 | ascp: Transferring from the Command Line with Ascp | 130 1. Transfer everything except files and directories starting with ".": -N '*' -E 'AAA/**' Results: AAA/abc/def AAA/abc/wxy/def AAA/abc/wxy/tuv/def AAA/abc/xyz/def/wxy AAA/wxyfile AAA/wxy/xyx/ AAA/wxy/xyxfile < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/wxy/.def 2. Exclude directories and files whose names start with wxy: -E 'wxy*' Results: AAA/abc/def...
Page 131 | ascp: Transferring from the Command Line with Ascp | 131 AAA/wxyfile AAA/wxy/xyx/ < AAA/abc/def < AAA/abc/.def < AAA/abc/.wxy/def < AAA/abc/wxy/def < AAA/abc/wxy/.def < AAA/abc/wxy/tuv/def < AAA/wxy/xyxfile * Even though wxy is included, def is excluded because it's a file. 5.
Page 132: Symbolic Link Handling
| ascp: Transferring from the Command Line with Ascp | 132 < AAA/wxy/xyxfile Symbolic Link Handling When transferring files using FASP (ascp, ascp4, or async), you can configure how the server and client handle symbolic links. Note: Symbolic links are not supported on Windows. Server settings are ignored on Windows servers. If the transfer destination is a Windows computer, the only supported option that the client can use is skip.
Page 133: Creating Ssh Keys
| ascp: Transferring from the Command Line with Ascp | 133 Receive to Client from Server (Download) Server setting Server setting = Server setting = Server setting = Server setting = = create, follow create follow follow_wide none (default) Client setting = Follow Skip Follow...
Page 134: Reporting Checksums
The checksum reported by the source can be retrieved in the destination logs, a manifest file, in IBM Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.
Page 135 | ascp: Transferring from the Command Line with Ascp | 135 Overview of Checksum Configuration Options Description asconfigurator Option ascp Option Enable checksum reporting and specify the type of checksum to file_checksum calculate for transferred files. --file-checksum=type any - Allow the checksum format to be whichever format the client requests.
Page 136 | ascp: Transferring from the Command Line with Ascp | 136 Enabling checksum reporting in an ascp session To enable checksum reporting on a per-transfer-session basis, run ascp with the --file-checksum=hash option, where hash is sha1, md5, sha-512, sha-384, sha-256, or none (the default). Enable the manifest with --file-manifest=output where output is either text or none.
Page 137: Client-Side Encryption-At-Rest (Ear)
| ascp: Transferring from the Command Line with Ascp | 137 Windows: > CertUtil -hashfile filepath MD5 Mac OS X: $ md5 filepath Linux and Linux on z Systems: # md5sum filepath AIX: # csum -h MD5 filepath Solaris: # digest -a md5 -v filepath 5.
Page 138: Comparison Of Ascp And Ascp 4 Options
| ascp: Transferring from the Command Line with Ascp | 138 # ascp --mode=recv --file-crypt=decrypt user@host:/source_path/file.aspera- env local_destination For more command line examples, see Ascp General Examples on page 114. Note: When a transfer to HST Server falls back to HTTP or HTTPS, client-side EAR is no longer supported. If HTTP fallback occurs while uploading, then the files are NOT encrypted.
Page 139 | ascp: Transferring from the Command Line with Ascp | 139 Ascp Ascp 4 --delete-before --delete-before-transfer** --delete-before-transfer** --dest64 -E pattern -E pattern -e prepost_filepath --exclude-newer-than=mtime --exclude-older-than=mtime -f config_file --faspmgr-io --file-checksum=hash --file-crypt={encrypt|decrypt} --file-list=filepath** --file-list=filepath** --file-manifest={none|text} --file-manifest-path=directory --file-manifest-inprogress-suffix=suffix --file-pair-list=filepath -G write_size -g read_size -h, --help -h, --help -i private_key_file_path**...
Page 140 | ascp: Transferring from the Command Line with Ascp | 140 Ascp Ascp 4 --no-write -O fasp_port -O fasp_port --overwrite=method** --overwrite=method** -P ssh-port -P ssh-port --partial-file-suffix=suffix --policy={fixed|high|fair|low} --policy={fixed|high|fair|low} --precalculate-job-size --preserve-access-time --preserve-acls=mode --preserve-creation-time --preserve-file-owner-gid --preserve-file-owner-gid --preserve-file-owner-uid --preserve-file-owner-uid --preserve-modification-time --preserve-source-access-time --preserve-xattrs=mode --proxy=proxy_url -R remote_log_dir -R remote_log_dir --read-threads=num...
Page 141: Ascp Faqs
| ascp: Transferring from the Command Line with Ascp | 141 Ascp Ascp 4 --src-base=prefix --src-base=prefix --symbolic-links=method** --symbolic-links=method** -u user_string -u user_string --user=username --user=username -W token_string | @token_filepath -w{r|f} -X rexmsg_size -X rexmsg_size -Z dgram_size -Z dgram_size Differences in Option Behavior --delete-before-transfer With ascp4, --delete-before-transfer can be used with URI storage.
Page 142 | ascp: Transferring from the Command Line with Ascp | 142 You can specify a transfer policy that determines how a FASP transfer utilizes the network resource, and you can specify target and minimum transfer rates where applicable. In an ascp command, use the following flags to specify transfer policies that are fixed, fair, high, or low: Policy Command template...
Page 143 | ascp: Transferring from the Command Line with Ascp | 143 3. How do I ensure that if the transfer is interrupted or fails to finish, it will resume without re-transferring the files? Use the -k flag to enable resume, and specify a resume rule: -k 0 –...
Page 144: Ascp4: Transferring From The Command Line With Ascp 4
Both Ascp 4 and Ascp are automatically installed with IBM Aspera High-Speed Transfer Server, IBM Aspera High- Speed Transfer Endpoint, and IBM Aspera Desktop Client.
Page 145 | ascp4: Transferring from the Command Line with Ascp 4 | 145 ascp4 Syntax ascp4 options [[user@]srcHost:]source_file1[,source_file2,...] [[user@]destHost:]dest_path User The username of the Aspera transfer user can be specified as part of the as part of the source or destination, whichever is the remote server or with the --user option.
Page 146 | ascp4: Transferring from the Command Line with Ascp 4 | 146 • Files that are open for write by another process on a Windows source or destination cannot be transferred and return a "sharing violation" error. On Unix-like operating systems, files that are open for write by another process are transferred without reporting an error, but may produce unexpected results depending on what data in the file is changed and when relative to the transfer.
Page 147 | ascp4: Transferring from the Command Line with Ascp 4 | 147 compression (0 = no compression). A higher value results in greater, slower compression. Valid values are -1 to 9, where -1 is "balanced". Default: -1. -D | -DD | -DDD Log at the specified debug level.
Page 148 | ascp4: Transferring from the Command Line with Ascp 4 | 148 • If more than one read thread is specified (default is 2) for a transfer that uses --file-list, the files in the file list must be unique. Duplicates can produce unexpected results on the destination.
Page 149 | ascp4: Transferring from the Command Line with Ascp 4 | 149 --mode={send|recv} Transfer in the specified direction: send or recv (receive). Requires --host. -N pattern Protect ("include") files or directories from exclusion by any -E (exclude) options that follow it.
Page 150 | ascp4: Transferring from the Command Line with Ascp 4 | 150 time. Timestamp support in object storage varies by provider; consult your object storage documentation to determine which settings are supported. On Windows, modification time may be affected when the system automatically adjusts for Daylight Savings Time (DST).
Page 151 | ascp4: Transferring from the Command Line with Ascp 4 | 151 Note: If more than one read thread is specified for a transfer that uses --file-list, the files in the file list must be unique. Duplicates can produce unexpected results on the destination. --remote-memory=bytes Allow the remote ascp4 process to use no more than the specified memory.
Page 152: Ascp 4 Transfers With Object Storage
| ascp4: Transferring from the Command Line with Ascp 4 | 152 Use the specified number of storage "write" threads (receiver only). Default: 2. To set "read" threads on the sender, use --read-threads. For transfers to object or HDFS storage, write threads cannot exceed the maximum number of jobs that are configured for Trapd.
Page 153: Ascp 4 Examples
| ascp4: Transferring from the Command Line with Ascp 4 | 153 Ascp 4 Examples The command options for ascp4 are generally similar to those for ascp. The following examples demonstrate options that are unique to Ascp 4. These options enable reading management commands, transfer TCP and UDP data streams, and enable read/write concurrency.
Page 154: Data Streaming Command Syntax
URI paths. The license prohibits video streaming. A separate product, IBM Aspera Streaming for Video, is available for video streaming. For more information, see the IBM Aspera Streaming for Video User Guide.
Page 155 | ascp4: Transferring from the Command Line with Ascp 4 | 155 The restriction can be set to allow all access (*) or limited by protocol, hostname or path: Restriction Format Example By protocol udp://* tcp://* By protocol and hostname udp://hostname* By protocol, hostname, and port tcp://hostname:5000*...
Page 156: Ascp 4 Data Streaming Examples
| ascp4: Transferring from the Command Line with Ascp 4 | 156 Option Description Default Maximum stream length No default maxsize=maximum_size maxtime=maximum_time Maximum stream duration, in No default seconds maxidle=maximum_time Maximum idle duration, in seconds No default rcvbufsz=buffer_size Receive buffer size 10MB sndbufsz=buffer_size Send buffer size...
Page 157: Configuring Macos Server For Multicast Streams
| ascp4: Transferring from the Command Line with Ascp 4 | 157 # ascp4 -L/opt/test-local-03 -R/opt/test-remote-03 -DD -m 12m -l 15m --mode send --host 10.132.117.2 --user root --read-threads 1 --write-threads 1 --compression none "udp://233.33.3.1:3003? sndbufsz=100MB&ifaddr=10.131.117.1" "udp://233.44.4.1:4003? rcvbufsz=100M&loopback=0" # ascp4 -L/opt/test-local-04 -R/opt/test-remote-04 -DD -m 12m -l 15m --mode send --host 10.132.117.2 --user root --read-threads 1 --write-threads 1 --compression none "udp://233.33.3.1:3004? sndbufsz=100MB&ifaddr=10.131.117.1"...
Page 158: Troubleshooting Stream Transfers
| ascp4: Transferring from the Command Line with Ascp 4 | 158 en0) to receive and send the multicast streams, we must set the interface receiving and sending the multicast streams as the server default interface. Run the following command on a4_1: # route delete default 10.215.0.1 # route add default 10.215.0.1 Run the following command on a4_2:...
Page 159: Watch Folders And The Aspera Watch Service
They can be configured to push from the local server or pull from a remote server. Remote servers can be HST Server, HST Endpoint, and IBM Aspera Shares servers, as well as servers in object storage.
Page 160: Choosing User Accounts To Run Watch Folder Services
Aspera recommends using the Node API to configure services and manage Watch Folders in a multi-user context. You can interact with the Node API by using IBM Aspera Console or using curl commands from the command line.
Page 161: Creating, Managing, And Configuring Services
| Watch Folders and the Aspera Watch Service | 161 Creating, Managing, and Configuring Services Both asperawatchd and asperawatchfolderd are managed by asperarund, which stores asperawatchd and asperawatchfolderd configurations in its database. It automatically starts services when they are added and restarts services if they fail.
Page 162 | Watch Folders and the Aspera Watch Service | 162 Starting asperarund If asperarund is not running, then you cannot create Watch Folders or start a watch. The service is started automatically during installation, but you might have to start it if it was disabled or stopped. # ps ax | grep aspera Locate asperarund in the output.
Page 163 | Watch Folders and the Aspera Watch Service | 163 "state_changed_at":"2016-10-20T19:14:34Z" "id":"d109d1bd-7db7-409f-bb16-ca6ff9abb5f4", "configuration": { "enabled":true, "run_as":{ "pass": "*****", "user":"root" "type":"WATCHFOLDERD" "state":"RUNNING", "state_changed_at":"2016-10-20T00:11:19Z" The Watch Service configuration includes the string "type":"WATCHD" and, before this entry in the output, a value for "id". The Watch Folder service includes the string: "type":"WATCHFOLDERD". Disable a Service Disabling a service stops the service but saves its configuration in the database.
Page 164: Watch Folders
They can be configured to push from the local server or pull from a remote server. Remote servers can be HST Server, HST Endpoint, and IBM Aspera Shares servers, as well as servers in object storage.
Page 165: Creating A Push Watch Folder With Aswatchfolderadmin
Push Watch Folders can still be created from JSON configuration files that follow the 3.7 version syntax by using the Watch Folder API. To create and manage Watch Folders by using the Watch Folder API or IBM Aspera Console, see Creating a Push...
Page 166 Source file archiving is not supported if the Watch Folder source is in object storage. • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version. To create a push Watch Folder: 1.
Page 167 | Watch Folders and the Aspera Watch Service | 167 Use the aswatchadmin and aswatchfolderadmin utilities to retrieve a list of running daemons. Daemons have the same name as the user for which they are running. For example, if you used the root user to run your services, you should see the root daemon listed when you run the following commands: # /opt/aspera/bin/aswatchadmin query-daemons [aswatchadmin query-daemons] Found a single daemon:...
Page 168 SSH, (the value for tcp_port in the "transport" section) is used. If the then default is authentication type is NODE_BASIC, 9092 is used. For Shares, IBM the value for Aspera Transfer Cluster Manager, or IBM Aspera on Cloud endpoints, tcp_port in enter 443.
Page 169 | Watch Folders and the Aspera Watch Service | 169 Field Description Default file system scans triggered by the scan period are used to detect file changes. In this case, set the scan period to frequently scan for changes. On operating systems that support file notifications (Linux, Windows, macOS), asperawatchd uses the file notifications as the primary means for detecting changes, and the scan period serves as a backup.
Page 170: Creating A Pull Watch Folder With Aswatchfolderadmin
Source file archiving is not supported if the Watch Folder source is in object storage. • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version. Restrictions on Pull Watch Folders •...
Page 171 | Watch Folders and the Aspera Watch Service | 171 b) Confirm that the service was created. # /opt/aspera/bin/aswatchadmin query-daemons If the service exists, the following output is returned (in this example, the user is "root"): # /opt/aspera/bin/aswatchadmin query-daemons [aswatchadmin query-daemons] Found a single daemon: root If other services are running on the server, other daemons are also returned.
Page 172 (the value for tcp_port in the "transport" section) then default is is used. If the authentication type is NODE_BASIC, 9092 is used. the value for For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera tcp_port in the on Cloud endpoints, enter 443. "transport" section (default: 22).
Page 173 | Watch Folders and the Aspera Watch Service | 173 Field Description Default username and password, Shares credentials, or an access key ID and secret. The username for authentication. Required. Depending on the user type of authentication, it is the transfer user's username, Node API username, Shares username, or access key ID.
Page 174: Watch Folder Service Configuration
| Watch Folders and the Aspera Watch Service | 174 Daemons have the same name as the user for which they are running. For example, if you used the root user to run your services, you should see the root daemon listed. For example, using the root daemon and a valid JSON file, watchfolderconf.json, the output of the aswatchfolderadmin command should look like the following: # /opt/aspera/bin/aswatchfolderadmin create-folder root -f...
Page 175: Watch Folder Json Configuration File Reference
| Watch Folders and the Aspera Watch Service | 175 To view the current settings, run the following command and look for settings that start with watch and watchfolderd: # /opt/aspera/bin/asuserdata -a Configuring asperawatchfolderd Settings Configure asperawatchfolderd by using asconfigurator commands with this general syntax: # /opt/aspera/bin/asconfigurator -x "set_server_data;option,value"...
Page 176 | Watch Folders and the Aspera Watch Service | 176 "port": 9092, "authentication": { "type": "NODE_BASIC", "user": "nodeuser1", "pass": "watchfoldersaregreat", "target": { "path": "/projectA" "id":"b394d0ee-1cda-4f0d-b785-efdc6496c585", "cool_off":"30s", "snapshot_creation_period":"10s", "meta":{ "version":0, "name":"aspera_watchfolder" "drop":{ "detection_strategy":"COOL_OFF_ONLY", "cool_off":"5s" "post_processing":{ "source":{ "type":"TRANSFER_NONE", "archive_dir":"/watchfolder_sessions/{$UUID}_{$DATETIME}" "filters":[ "type":"GLOB", "pattern":"*.txt", "rule":"INCLUDE"...
Page 177 | Watch Folders and the Aspera Watch Service | 177 "proxy":"dnat://aspx1:passwordsarecool@localhost:9001", "keypath":"", "fingerprint":"", "cookie":"", "tags":{ "error_handling":{ "file":{ "max_retries":3, "retry_timeout":"3s" "drop":{ "retry_period":"1m" "regular":{ "max_parallel":10, "connect_timeout":"10s", "policy":"FAIR", "min_rate":"0B", "target_rate":"10M", "tcp_port":22, "udp_port":33001, "read_blk_size":"", "write_blk_size":"", "datagram_size":"", "rexmsg_size":"", "cipher":"AES128", "overwrite":"DIFF", "resume":"NONE", "preserve_uid":false, "preserve_gid":false, "preserve_time":false, "preserve_creation_time":false, "preserve_modification_time":false, "preserve_access_time":false, "queue_threshold":"5s",...
Page 178 API user authentication, the path is relative to the user's docroot, configured, or the absolute or UNC path if the user is configured with a restriction. For IBM Aspera Shares authentication, the path is the share name and, optionally, a path within the share.
Page 179 (the value for tcp_port in the "transport" section) default is the value is used. If the authentication type is NODE_BASIC, 9092 is used. for tcp_port in the For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera "transport" section on Cloud endpoints, enter 443. (default: 22). If...
Page 180 | Watch Folders and the Aspera Watch Service | 180 Field Description Default "Remote host is not who we expected". For more information, see Securing Your SSH Server on page 16 ("Configuring Transfer Server Authentication"). Value used to identify a Watch Folder. If this field is not configured at creation, a UUID is automatically generated for and assigned to the Watch Folder.
Page 181 | Watch Folders and the Aspera Watch Service | 181 "type": "NODE_BASIC", "user": "access_key_id", "pass": "access_key_secret", Meta Fields "meta":{ "version":0, "name":"aspera_watchfolder" Field Description Default version Specifies the current version of the configuration. When updating the configuration, this value must match the version stored by the server. Otherwise, the update is rejected.
Page 182 | Watch Folders and the Aspera Watch Service | 182 Post Processing Fields Optionally, specify post-processing to do after a drop or file is successfully transferred. "post_processing":{ "source":{ "type":"TRANSFER_NONE", "archive_dir":"/watchfolder_sessions/{$UUID}_{$DATETIME}" Field Description Default type The type of post-transfer processing. Files can be archived, deleted, TRANSFER_NONE or retained after transfer of a drop.
Page 183 | Watch Folders and the Aspera Watch Service | 183 Filter Fields Each filter object must include values for "type", "pattern", and "rule". Filters are applied in order. Watch Folders supports glob and Regex filters. The glob filter system is the same as Ascp; see Using Filters to Include and Exclude Files on page 126.
Page 184 | Watch Folders and the Aspera Watch Service | 184 Field Description Default timeout How long to wait for file dependencies to be satisfied (files that must be transferred before the last file are transferred) before considering the dependency as unsatisfied. final_transfer Define the file to transfer last.
Page 185 If required, the token string. Not valid for use with growing files. proxy If using, the address of an IBM Aspera Proxy server. The proxy syntax is: dnat(s)://user:password@server:port keypath If authenticating by SSH user and key, the path to the SSH user's private key file.
Page 186 | Watch Folders and the Aspera Watch Service | 186 Option Description Default max_retries How many times to try transferring a file before the file is marked as failed. retry_timeout How frequently to retry file transfers. retry_period If no bytes are transferred during the specified period and no file is completed, the drop and all remaining incomplete files in the drop are marked as failed.
Page 187 | Watch Folders and the Aspera Watch Service | 187 Field Description Default • HIGH • min_rate Attempt to transfer no slower than the specified minimum transfer rate. target_rate The target transfer rate. Transfer at rates up to the specified target rate.
Page 188 | Watch Folders and the Aspera Watch Service | 188 Field Description Default Cipher Values Value Description Support Use the GCM or CFB All client and server AES128 encryption mode, versions. AES192 depending on the server AES256 configuration and version (see cipher negotiation matrix).
Page 189 | Watch Folders and the Aspera Watch Service | 189 Field Description Default overwrite Specify whether a file is overwritten if it already exists at the DIFF destination. Valid options are: • NEVER • ALWAYS • DIFF • OLDER • DIFF+OLDER resume Specify if and how partial transfers are resumed.
Page 190 | Watch Folders and the Aspera Watch Service | 190 Field Description Default options, they must be enabled in the client's aspera.conf by running the following command: # asconfigurator -x "set_central_server_data;raw_options,enable" symbolic_links Set the symbolic link handling policy, as allowed by the server. FOLLOW Value can be FOLLOW, COPY, or SKIP.
Page 191 | Watch Folders and the Aspera Watch Service | 191 Option Description Default policy Defines how FASPStream manages the bandwidth. The policy can FAIR be set to the following values: • FIXED • FAIR • HIGH • min_rate Attempt to transfer no slower than the specified minimum transfer rate.
Page 192 | Watch Folders and the Aspera Watch Service | 192 Option Description Default scan_period The time between file system scans of the watches (from end of one to start of the next). These scans are independent of the snapshot minimum interval and snapshot minimum changes to ensure that changes are identified.
Page 193: Managing Watch Folders With Aswatchfolderadmin
| Watch Folders and the Aspera Watch Service | 193 Managing Watch Folders with aswatchfolderadmin The aswatchfolderadmin tool can be used to retrieve a list of Watch Folders, update the configuration of Watch Folder, and delete a Watch Folder. Retrieve a List of Running Daemons Use the aswatchadmin and aswatchfolderadmin utilities to retrieve a list of running daemons.
Page 194: Configuring Linux For Many Watch Folders
| Watch Folders and the Aspera Watch Service | 194 Delete a Watch Folder # /opt/aspera/bin/aswatchfolderadmin delete-folder daemon watchfolder_id For example: # /opt/aspera/bin/aswatchfolderadmin update-folder root 3354f360- dfa6-4789-930e-074cd9d4551b [aswatchfolderadmin update-folder] Successfully deleted instance b394d0ee-1cda-4f0d-b785-efdc6496c585 Configuring Linux for Many Watch Folders To run many (>100) push Watch Folders on Linux computers, adjust three system settings and then reload the sysctl.conf file to activate them.
Page 195: Creating A Push Watch Folder With The Api
Source file archiving is not supported if the Watch Folder source is in object storage. • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version. To create a push Watch Folder with the API: 1.
Page 196 | Watch Folders and the Aspera Watch Service | 196 "type": "WATCHD", "run_as": { "user": "username", "pass": "password" "enabled": true For the Watch Folder service: "type": "WATCHFOLDERD", "run_as": { "user": "username", "pass": "password" "enabled": true The username and password are for a transfer user with permissions to the source path. Save the files, with the .json extension.
Page 197 SSH, (the value for tcp_port in the "transport" section) is used. If the then default is authentication type is NODE_BASIC, 9092 is used. For Shares, IBM the value for Aspera Transfer Cluster Manager, or IBM Aspera on Cloud endpoints, tcp_port in enter 443.
Page 198 | Watch Folders and the Aspera Watch Service | 198 Field Description Default Required for SSH authentication if "keypath" is not specified keypath For SSH authentication with an SSH key, the path to the transfer user's SSH private key file. Required for SSH authentication if "pass"...
Page 199: Creating A Pull Watch Folder With The Api
Source file archiving is not supported if the Watch Folder source is in object storage. • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version. Restrictions on Pull Watch Folders •...
Page 200 | Watch Folders and the Aspera Watch Service | 200 For example: # /opt/aspera/bin/asnodeadmin -a -u watchfolder_user -p X245lskd3 -x root --acl-set "admin,impersonation" Adding, modifying, or deleting a node-user triggers automatic reloading of the user database and the node's configuration and license files. For more information on the Node API, see your transfer server's administrator guide.
Page 201 | Watch Folders and the Aspera Watch Service | 201 "enabled": true The username and password are for a transfer user with permissions to the source path. Save the files, with the .json extension. b) Create the service. # curl -ki -u node_username:node_password -X POST -d @config_file "https://localhost:9092/rund/services"...
Page 202 (the value for tcp_port in the "transport" section) then default is is used. If the authentication type is NODE_BASIC, 9092 is used. the value for For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera tcp_port in the on Cloud endpoints, enter 443. "transport" section (default: 22).
Page 203: Managing Watch Folders With The Api
| Watch Folders and the Aspera Watch Service | 203 Field Description Default Lower scan periods detect changes faster but can result in greater resource consumption, particularly for object storage. Note: The value for scan period cannot be empty, otherwise the configuration is rejected.
Page 204 | Watch Folders and the Aspera Watch Service | 204 For example: # curl -k --user watchfolder_admin:XF324cd28 -H "X-aspera-WF- version:2017_10_23" -X GET https://198.51.100.22:9092/v3/watchfolders "ids" : [ "b394d0ee-1cda-4f0d-b785-efdc6496c585" If there are no running Watch Folders, the server returns the following output. "ids"...
Page 205 | Watch Folders and the Aspera Watch Service | 205 Query and save a configuration for a specific Watch Folder # curl -ks -u node_api_user:node_api_password -H "X-aspera- WF-version:2017_10_23" -X GET https://host:node_api_port/v3/ watchfolders/watchfolder_id > config_file.json For example: # curl -ks --user watchfolder_admin:XF324cd28 -H "X-aspera- WF-version:2017_10_23"...
Page 206 | Watch Folders and the Aspera Watch Service | 206 1. Get the ID of the Watch Folder that you want to update by getting a list of Watch Folders: # curl -k --user node_api_user:node_api_password -H "X-aspera-WF- version:2017_10_23" -X GET https://host:node_api_port/v3/watchfolders 2.
Page 207: Configuring Custom Watch Folder Permissions Policies
| Watch Folders and the Aspera Watch Service | 207 For example, to change the user to admin2, run the following: # curl -k --user admin2:XF324cd28 -H "X-aspera-WF-version:2017_10_23" -X POST -d @~/watchfolder_conf.json https://198.51.100.22:9092/v3/watchfolders "id": "b394d0ee-1cda-4f0d-b785-efdc6496c585" To verify that the configuration was updated, retrieve the configuration file again and look for your changes. Deleting a Watch Folder To remove a Watch Folder, run the following command: # curl -sk --user node_api_user:node_api_password -X DELETE...
Page 208 | Watch Folders and the Aspera Watch Service | 208 "resource_id" The placeholders take the following values: • policy_name: A descriptive name for the policy, such as "only-wfd-aspera". If no value is specified, a UUID is generated and returned in the output when the policy is created. •...
Page 209 | Watch Folders and the Aspera Watch Service | 209 WF_UPDATE_WATCHFOLDER WF_RETRY_DROP To allow all Watch Folder actions on all Watch Folders, enter "WF_*" as the action and "arn:watchfolder:wfd:*" as the resource. Sample Policies Allow the user to view policies and user permissions: "id": "read-permissions", "statements": [ "effect": "ALLOW",...
Page 210: Updating The Docroot Or Restriction Of A Running Watch Folder Service
| Watch Folders and the Aspera Watch Service | 210 To view the permissions policies that are assigned to a user, run the following command: # curl -k --user node_api_user:node_api_password -X GET https:// localhost:9092/access_control/users/username/policies To view the users that are assigned to a permissions policy, run the following command: # curl -k --user node_api_user:node_api_password -X GET https:// localhost:9092/access_control/policies/policy_id/users Editing Policies...
Page 211: The Aspera Watch Service
• drive root: file:////* For Windows OS: • specific folder: file:///c%3A/folder/* • drive root: file:///c* Amazon S3 and IBM Cloud Object Storage - S3 s3://* Azure azu://* Azure Files azure-files://* Azure Data Lake Storage adl://* Alibaba Cloud...
Page 212 • drive root: file:////* For Windows OS: • specific folder: file:///c%3A/folder/* • drive root: file:///c* Amazon S3 and IBM Cloud Object Storage - S3 s3://* Azure azu://* Azure Files azure-files://* Azure Data Lake Storage adl://* Alibaba Cloud...
Page 213: Watch Service Configuration
| Watch Folders and the Aspera Watch Service | 213 or for Linux systems that use init.d: # service asperanoded restart 2. Ensure the user has permissions to write to the default log directory if no directory is specified. For more information about configuring log directories, seeWatch Service Configuration on page 213.
Page 214 | Watch Folders and the Aspera Watch Service | 214 <snapshot_min_interval>3s</snapshot_min_interval> <snapshot_min_changes>100</snapshot_min_changes> <scan_threads>16</scan_threads> </watchd> <watchfolderd>...</watchfolderd> </watch> </server> To view current settings without opening aspera.conf, run the following command and look for settings that start with watch and watchd: # /opt/aspera/bin/asuserdata -a Note: Logging and database settings apply to both the Watch Service and Watch Folders services.
Page 215: Setting Custom Watch Scan Periods
| Watch Folders and the Aspera Watch Service | 215 Description Default asconfigurator option aspera.conf setting The number of snapshots that are 10000 watchd_max_snapshots stored in the database before the <max_snapshots> oldest are overwritten. The maximum amount of time watchd_snapshot_min_interval between snapshots.
Page 216: Transferring And Deleting Files With The Aspera Watch Service
| Watch Folders and the Aspera Watch Service | 216 Create a new subscription to a watch # /opt/aspera/bin/aswatchadmin subscribe daemon filepath [options] Options include: --db-spec=type:host:port Use the specified Redis database, which is defined with the syntax redis:ip_address:port[:domain]. -L, --logdir=log_dir Specify the location for watch logging, particularly if the user does not have access to the default log location (/var/log/messages) or the location specified in aspera.conf.
Page 217 | Watch Folders and the Aspera Watch Service | 217 Creating a Subscription, Snapshots, and Snapshot Differential 1. Create a subscription and decide how to manage its expiration. # /opt/aspera/bin/aswatchadmin subscribe daemon filepath [options] By default, subscriptions expire in 24 hours. If your snapshot comparisons will be spaced more than 24 hours apart, either set the expiration time to a duration longer than the time between snapshots (add -- expire_in=seconds to the command) or send a resubscribe command periodically to maintain the subscription.
Page 218: Aspera Sync
Learn about the key features and capabilities of IBM Aspera Sync. Overview IBM Aspera Sync is a software application that provides high-speed, highly-scalable, multi-directional, file- based replication and synchronization. Aspera Sync is designed to fill the performance gap of uni-directional file synchronization tools like rsync, which are often slow for synchronizing large files and large sets of files over the WAN.
Page 219 | Aspera Sync | 219 • Utilizes high-speed Aspera FASP transport for moving data at maximum speed over the WAN, whereas traditional synchronization tools are built on TCP. Aspera Sync transfers new data between remote hosts at full bandwidth capacity, regardless of round-trip delay and packet loss, and does not degrade in performance for large file sizes. •...
Page 220: Synchronization And Direction Modes
| Aspera Sync | 220 Sample Sync Deployment Diagram Synchronization and Direction Modes Aspera Sync offers two modes of operation: one-time ("on demand") synchronization and continuous synchronization, as well as three direction modes: uni-directional, bi-directional, and multi-directional. One-time vs. Continuous Synchronization One-time synchronization In this mode, async performs synchronization of the endpoints, and exits.
Page 221: Aspera Sync Faq
| Aspera Sync | 221 Continuous Aspera Sync Direction Supported Aspera Sync Client OS Supported Aspera Sync Server OS PUSH Linux, Windows, macOS PULL Linux, Windows, macOS BIDI Linux, Windows, macOS Linux, Windows, macOS Aspera Sync Direction Modes Uni-directional Similar to rsync, the uni-directional mode supports replication of files and directories, and any updates to these (including deletions, renames, moves, and copies) from a source to a target.
Page 222: Aspera Sync Set Up
| Aspera Sync | 222 For a comparison of async options versus rsync options, see rsync vs. async Uni-directional Example. How is one-time mode different from continuous mode? Aspera Sync offers two modes of operation: one-time ("on-demand") synchronization and continuous synchronization.
Page 223 228). The Aspera Sync database should not be located on CIFS, NFS, or other shared file systems mounted on Linux, unless you are synchronizing through IBM Aspera Proxy. If server data are stored on a mount, specify a local location for the Aspera Sync database. Aspera recommends setting the database location...
Page 224 | Aspera Sync | 224 See an example aspera.conf following the settings reference table. For an example of the asperawatchd configuration, see Watch Service Configuration on page 213. After manually editing aspera.conf, validate that its XML syntax is correct by running the following command: # /opt/aspera/bin/asuserdata -v This command does not check if the settings are valid.
Page 225 | Aspera Sync | 225 Description and Value Options asconfigurator option aspera.conf setting Enable (set to true, default) or disable (set to false) Sync. When set async_enabled to false, the client async session fails with the error "Operation <async_enabled> 'sync' not enabled or not permitted by license". Specify an alternative location for the async server's log files.
Page 226: Viewing Aspera Sync Transfers In The Aspera Gui
| Aspera Sync | 226 Example Sync Configuration in aspera.conf <file_system> <directory_create_mode> </directory_create_mode> <directory_create_grant_mask>755</directory_create_grant_mask> <preserve_acls>none</preserve_acls> <preserve_xattrs>none</preserve_xattrs> </file_system> <default> <async_db_dir> </async_db_dir> <async_db_spec> </async_db_spec> <async_enabled>true</async_enabled> <async_connection_timeout>20</async_connection_timeout> <async_session_timeout>20</async_session_timeout> <async_log_dir>AS_NULL</async_log_dir> <async_log_level>log</async_log_level> </default> Viewing Aspera Sync Transfers in the Aspera GUI The HST Server GUI shows async-initiated transfers if Aspera Sync is run on the machine (as client) by default, whereas server async transfers are not shown.
Page 227: Symbolic Link Handling
| Aspera Sync | 227 Symbolic Link Handling When transferring files using FASP (ascp, ascp4, or async), you can configure how the server and client handle symbolic links. Note: Symbolic links are not supported on Windows. Server settings are ignored on Windows servers. If the transfer destination is a Windows computer, the only supported option that the client can use is skip.
Page 228: The Aspera Sync Database
| Aspera Sync | 228 Server setting Server setting = Server setting = Server setting = Server setting = = create, follow create follow follow_wide none (default) Client setting = Follow Skip Follow Follow even Skip follow if the target is outside the user's (default for ascp docroot.
Page 229 | Aspera Sync | 229 Each async session must have a unique name. If multiple sessions synchronize the same directory or specify the same database directory (-b/-B), then the session names must be unique. For example, you run an async session named job1 that synchronizes the local directory /data and the remote directory /data1, and that stores the database in /sync/db on both endpoints.
Page 230: Running Async
-N pair is required in async commands. The value for pair is a name that uniquely identifies the Aspera Sync session and is visible in IBM Aspera Console. -N pair must follow any instance options and must precede all session arguments. Names can only use standard alphanumeric characters, plus "_" and "-" characters.
Page 231 Basic token: Basic tokens are used for synchronizing to Aspera products that require access key authentication, such as IBM Aspera on Cloud transfer service (AoCts) or IBM Aspera Transfer Cluster Manager (ATC Manager). For instructions on creating the basic token, see...
Page 232 | Aspera Sync | 232 For example, use -d to set the local directory to Morgan's data folder: async -L "C:\Users\Morgan\Aspera jobs\log" -N job1 -i c:/users/ morgan/.ssh/id_rsa -b "C:\Users\Morgan\Aspera jobs\db" -l 500m -d c:/ users/morgan/data 9. Specify the transfer username, remote host, and remote directory for synchronization. Unlike previous options for which one short option flag was equivalent to one long option flag, when specifying the username, remote host, and remote directory, the short flag option is the equivalent of one to three long option flags.
Page 233 | Aspera Sync | 233 • pull: The contents of rdir are synchronized to ldir, with the rdir content overwriting the ldir content, by default. • bidi (bi-directional): The contents of ldir and rdir are synchronized, with newer versions of files and directories overwriting older versions in either ldir or rdir, by default.
Page 234: Async Command Reference
-N pair is required in async commands. The value for pair is a name that uniquely identifies the Aspera Sync session and is visible in IBM Aspera Console. -N pair must follow any instance options and must precede all session arguments. Names can only use standard alphanumeric characters, plus "_" and "-" characters.
Page 235 | Aspera Sync | 235 • When scanning or monitoring a file system for changes, async skips over files with names that end in one of the special suffixes specified in aspera.conf with <resume_suffix> and <partial_file_suffix> . To disable this behavior, you can set these values to the empty string. <resume_suffix> defaults to .aspx. The <partial_file_suffix>...
Page 236 | Aspera Sync | 236 Instance Options -A, --version Display the async version information and license information. --apply-local-docroot Prepend the local docroot to the local directory. -D[D..] Log at the specified debug level. Default is 0. Additional Ds return more messages. -h, --help Display help for command-line options.
Page 237 | Aspera Sync | 237 -C, --continuous Run continuous synchronization. Default: disabled. Usage notes: • Continuous mode is supported only when the file source is Windows or Linux. Continuous pulls can be run from any operating system if the source is Windows or Linux. Continuous push can be run only by Windows or Linux.
Page 238 | Aspera Sync | 238 Value Description Support Do not encrypt data in transit. Aspera All client and server versions. none strongly recommends against using this setting. • NONE - Do not encrypt data in transit. Aspera strongly recommends against using this setting. •...
Page 239 | Aspera Sync | 239 Delay the start of the transfer. For example, if --cooloff=5, async waits 5 seconds before copying a file. If --cooloff=0 transfers start immediately. The client and server use the same cooloff period. Valid range for sec: integers 0-60. Default: 3. --cooloff-max=sec Wait up to the specified time (in seconds) for a file to stop changing before skipping synchronization of the file.
Page 240 | Aspera Sync | 240 mtime may be specified in any one of the following ways: • As a positive number of seconds since 1970-01-01 00:00:00, for Unix and POSIX-compliant operating systems. Note: Some file servers, such as Windows NT, use a different epoch for the recursive modified time.
Page 241 | Aspera Sync | 241 Authenticate with the specified SSH private key file. For information on creating a key pair, see Creating SSH Keys on page 269. --ignore-delete Do not copy removals to the peer. This option is used mostly with uni-directional syncs. In bi- directional sync, a deletion on one side is ignored but the next time async is run, the file is recopied from the other end.
Page 242 Assign a name for the synchronization session. The value can contain only ASCII alphanumeric, hyphen, and underscore characters. This value is stored in the session cookie and can be used in IBM Aspera Console to identify the transfer session. Note: -N must precede all session options.
Page 243 | Aspera Sync | 243 --pending-max=N Allow the maximum number of files that are pending transfer to be no more than the specified number. This option acts as a buffer. Default: 2000. --preserve-access-time Preserve file access time from the source to the destination. Default: disabled. --preserve-acls={native|metafile|none} Preserve Access Control Lists (ACL) data for macOS, Windows, and AIX files.
Page 244 If Aspera Sync is run by a regular user, only user-level attributes are preserved. If run as superuser, all attributes are preserved. --proxy proxy_url Synchronize using the specified IBM Aspera Proxy address. The Proxy URL is specified with the following syntax: dnat[s]://proxy_username:proxy_password@proxy_ip_address[:port] The default port for DNAT is 9091, and for DNATS is 9092.
Page 245 | Aspera Sync | 245 --remote-scan-threads=N Use the specified number of directory scanning threads on the remote computer. More threads decrease the time it takes for async to scan the directory after the initial synchronization, and increase the number of pending files. Default: 1. To specify the number of scanning threads on the local computer, see --scan-threads.
Page 246 | Aspera Sync | 246 --tags64=string User-defined metadata tags in JSON format and base64-encoded that can be used in transfer session reporting and searches. --transfer-threads=N[:size] Use the specified number of dedicated transfer threads and optionally specify the file size at which files are assigned groups of threads.
Page 247: Examples Of Async Commands And Output
| Aspera Sync | 247 Examples of Async Commands and Output Examples of common Aspera Sync use cases and a description of async output. Async Command Examples 1. Continuous synchronization of a daily archive of large files on a Windows computer to Linux computer, preserving Windows ACLs, run as an async pull on the Linux computer: $ async -L /sync/logs -N backup -d /sync/backup -r alligator@everglades.company.com:"C:\data\"...
Page 248: Include And Exclude Filtering Rules
The async session uses two threads, one for files larger than 100 KB and one for files less than or equal to 100 KB, specified with the --transfer-threads option. 4. Non-continuous push synchronization through reverse IBM Aspera Proxy: $ async -N pushproxy -d /images -r lion@10.0.0.1:/data/images -- proxy=dnats://gazelle:password@10.0.0.4 -K push...
Page 249 | Aspera Sync | 249 Basic usage The filtering options can be intermixed and have the following behavior: • Filtering rules are applied in the order they appear on the command line. If filtering rules are configured in aspera.conf, they are applied before the rules on the command line. •...
Page 250 | Aspera Sync | 250 Specifying Rules in a File Rules can be specified in a text file, with each rule on a separate line. • Rules in the file are applied to the file set as if from a series of --include A --exclude B options. •...
Page 251 | Aspera Sync | 251 Quotes any character literally, including itself. \ is exclusively a quoting operator, not a path separator. Matches zero or more characters, except "/" or the . in "/.". Matches any single character, except "/" or the . in "/.". [ …...
Page 252: Filtering Examples
| Aspera Sync | 252 Globbing Extensions Example Matches Does Not Match / at end of rule abc/*/ abc/dir abc/file no / at end of rule file file / at start of rule /abc/def /abc/def .../abc/def Filtering Examples Filtering examples that demonstrate the effects of adding more filter rules to the command and show how to format a filter rule file.
Page 253: Bidirectional Example
| Aspera Sync | 253 Bidirectional Example Bidirectional synchronization syntax is similar to push or pull async sessions, as show in the following example. Note: You can synchronize Windows, Linux, macOS, and other Unix-based endpoints and servers, but must take care with path separators.
Page 254 • An IBM Aspera On Demand instance in AWS S3, or HST Server for Linux or Windows version 3.7.3 or later installed on a virtual machine instance in AWS with Trapd enabled. For instructions on setting up a HST Server...
Page 255: Writing Custom Metadata For Objects In Object Storage
| Aspera Sync | 255 A one-time (non-continuous) push that is run from a local disk to an S3 bucket using SSH keys (for more information on using SSH keys, see Creating SSH Keys on page 269), where ec2_user is the transfer user: # async -N sync-to-s3 -d /data/data-2017-01 -r ec2_user@192.0.4.24:/data -i /bobcat/.ssh/private_key -K push -B /mnt/ephemeral/data/db --transfer- threads=8 --remote-fs-threads=16...
Page 256: Aspera Sync With Basic Token Authorization
Aspera Sync with Basic Token Authorization Aspera nodes that require access key authentication, such as IBM Aspera Transfer Cluster Manager or IBM Aspera on Cloud transfer service (AoCts), can be used as synchronization endpoints by configuring the async database on the node and authenticating the async session with a basic token.
Page 257: Using The Aspera Watch Service With Aspera Sync
For Unix-like OS: • specific folder: file:////folder/* • drive root: file:////* For Windows OS: • specific folder: file:///c%3A/folder/* • drive root: file:///c* Amazon S3 and IBM Cloud Object Storage - S3 s3://* Azure azu://* Azure Files azure-files://* Azure Data Lake Storage adl://*...
Page 258 | Aspera Sync | 258 Storage Type Format Example Alibaba Cloud oss://* Google Cloud gs://* HDFS hdfs://* With a docroot or restriction set up, the user is now an Aspera transfer user. Restart asperanoded to activate your change: Run the following commands to restart asperanoded: # systemctl restart asperanoded or for Linux systems that use init.d: # service asperanoded restart...
Page 259: Starting The Aspera Watch Service
| Aspera Sync | 259 Starting the Aspera Watch Service Aspera Sync can be configured to use asperawatchd for fast synchronization of very large numbers of files without scanning the directory. Aspera Sync can push files from a local directory, pull files from a remote directory, or create a bi-directional session between two directories (as long as asperawatchd is properly configured to monitor both directories).
Page 260 | Aspera Sync | 260 Configuring Watch Service Settings Configure the Watch Service by using asconfigurator commands with this general syntax: # /opt/aspera/bin/asconfigurator -x "set_server_data;option,value" Options and values are described in the following table. Configuration Options and Values Description Default asconfigurator option aspera.conf setting Log to the specified directory.
Page 261: Aspera Sync With Aspera Watch Service Session Examples
| Aspera Sync | 261 Description Default asconfigurator option aspera.conf setting The number of threads to use to scan watchd_scan_threads the watched folder. More threads <scan_threads> increase the speed of the scan, particularly for folders with large numbers of files, but require more of your computer's resources.
Page 262: Aspera Sync Monitoring And Logging
| Aspera Sync | 262 # If you want you can bind a single interface, if the bind option is not # specified all the interfaces will listen for incoming connections. bind 10.54.44.194 Save your configuration file and then run the asperaredisd service with the location of your configuration file. # /opt/aspera/sbin/asperaredisd /filepath/redis_configuration.conf.
Page 263 | Aspera Sync | 263 General asyncadmin usage: # asyncadmin -d path [-N name][options] The -N name option is required if multiple Aspera Sync sessions are running; you must specify the name of the session to which the asyncadmin command should apply. Note: When records are deleted using the -M or -E options, Aspera Sync recalculates file counters for the entire database.
Page 264: Logging
| Aspera Sync | 264 Display only the requested information. Use with -f / --file-info to disable abbreviating file names in the output. -s, --summary Report the number of files in each state. When -s is used alone, a brief summary from the async database's counters table is reported back (same as the cached counters as in the -t option).
Page 265 | Aspera Sync | 265 Causes: Possible causes include the following: • async binary doesn't exist (or is not in the path and sshd cannot find/execute it). • async binary cannot be run. • async binary cannot initialize properly (such as when the system is out of memory or socket resources). •...
Page 266: Troubleshooting Continuous Aspera Sync Errors
| Aspera Sync | 266 Solution: Specify -n skip or --symbolic-links=skip when performing the synchronization. Error returned when you synchronize two locations on the same computer You can synchronize files between two locations on the same computer. If you only enter the "remote" directory, such as -r /tmp/, then async fails with the following error: Failed - Error, must specify remote-host name Solution: Specify the remote host and path as -r username@127.0.0.1:filename.
Page 267: Resolving Bidirectional Aspera Sync File Conflicts
| Aspera Sync | 267 How to recover: You must modify the kernel parameters on the Linux computer to increase the maximum number of user watches. The following procedure might differ between Linux versions; consult your operating system Administrator's guide for more information. 1.
Page 268: Appendix
| Aspera Sync | 268 If the checksums match, then you can run the async session again and the files are synchronized without conflict. async -N my_bidi_sync -d /my_documents -r colleague@B:/home/my_documents -w pass -K bidi SYNCHRONIZED /Document1 SYNCHRONIZED /Document2 SYNCHRONIZED /Document3 SYNCHRONIZED 2.
Page 269: Creating Ssh Keys
| Aspera Sync | 269 Handling Moves in Scan Mode • If a new file has only one link, Aspera Sync checks whether it is a move. • If a new file has two or more links, Aspera Sync does not check whether it is a move (regardless of whether the other links are inside or outside the Aspera Sync directory).
Page 270 | Aspera Sync | 270 Example 1 Options: • Recursively synchronize the contents of a directory, /media/ to the remote system directory /backups/ media • Preserve access and modification time stamps on files • Preserve the owner and group ID •...
Page 271: Configuring For Other Aspera Products
HST Server can be configured as the transfer server for IBM Aspera Faspex, IBM Aspera Shares, and IBM Aspera Application for Microsoft SharePoint. It can also be configured and added as a node to Shares, IBM Aspera on Cloud (AoC), and IBM Aspera Console.
Page 272: Set Up Hst Server For Node Api
HTTP or HTTPS. The Node API allows you to connect nodes to Aspera web applications, such as IBM Aspera Faspex, IBM Aspera Shares, and IBM Aspera on Cloud, as well as integrate Aspera applications with your web application. It is supported by all Aspera server products and across multi-cloud and hybrid storage systems.
Page 273 • drive root: file:////* For Windows OS: • specific folder: file:///c%3A/folder/* • drive root: file:///c* Amazon S3 and IBM Cloud Object Storage - S3 s3://* Azure azu://* Azure Files azure-files://* Azure Data Lake Storage adl://* Alibaba Cloud...
Page 274 Then open /etc/sssd/sssd.conf and change default_shell from /bin/bash to / bin/aspshell. 5. Set the IBM Aspera Connect public SSH key as an authorized key for the transfer user and ensure that they own the file. a) Create the .ssh directory in the user's home folder.
Page 275: Node Admin Tool
• For a description of other settings, see Configuring the IBM Aspera NodeD Service on page 277. 9. Restart asperanoded to activate your changes. Run the following commands to restart asperanoded: # systemctl restart asperanoded or for Linux systems that use init.d:...
Page 276 | Set up HST Server for Node API | 276 All Options -h,--help Display usage. -A,--version Display version. -a,--add Add a user (also reloads configuration). --access-key access_key Specify the access_key. Use with --transfer options, --bearer-create, --bearer-verify, and --access-key-backup. --access-key-backup filename Backup tenant data to an AOF file.
Page 277: Configuring The Ibm Aspera Noded Service
Specify system transfer user. Configuring the IBM Aspera NodeD Service The IBM Aspera NodeD Service handles HTTP/HTTPS requests to HST Server. You can configure server settings including the hostname, HTTP/HTTPS ports, the address and port of the Redis database, and SSL certificates.
Page 278 | Set up HST Server for Node API | 278 Description and Values To Activate asconfigurator option Changes... aspera.conf setting Hostname or IP address. Restart server_name asperanoded <server_name> Default: hostname HTTP service port. Value is an integer 1 - Restart http_port 65535, default 9091.
Page 279 | Set up HST Server for Node API | 279 Description and Values To Activate asconfigurator option Changes... aspera.conf setting Full pathname of the SSL certificate, which Restart cert_file must be in .pem format. asperanoded <cert_file> Default: /opt/aspera/etc/ aspera_server_cert.pem Maximum number of entries to return in a Reload node max_response_entries response.
Page 280 IBM asperanoded <files_recursive_counts_enabled> Aspera on Cloud. The server configuration can be overridden by the access key configuration. Default is false. If true, enable reporting to the IBM Aspera on Restart aej_logging Cloud Activity app. The server configuration asperanoded <aej_logging>...
Page 281: Securing The Node Service Behind A Proxy
Securing the Node Service Behind a Proxy If your HST Server must expose asperanoded to the internet, such as when setting it up as a IBM Aspera on Cloud (AoC) node, Aspera strongly recommends protecting it with a reverse proxy and keeping the SSL ciphers up-to-date (see https://cipherli.st/...
Page 282: Backing Up And Restoring The Node User Database Records
| Set up HST Server for Node API | 282 to AoC must have asperanoded run on port 443, the standard HTTPS port for secure browser access. Configuring a reverse proxy in front of asperanoded provides additional protection (such as against DOS attacks) and resource handling for requests to the node's 443 port.
Page 283: Backing Up And Restoring A Node Database
These instructions describe how to back up and restore the entire Redis database of a node, which includes Node API users, their access keys, and transfer history. If your transfer server is an IBM Aspera on Cloud (AoC) node, migrate AoC data from one node to another by backing up the Redis database on the original node and restoring the database on a new node.
Page 284: Setting Up Ssl For Your Nodes
The Aspera Node API provides an HTTPS interface for encrypted communication between nodes (on port 9092, by default). For example, if you are running the IBM Aspera Faspex web UI or the IBM Aspera Shares web UI on one computer, you can encrypt the connection (using SSL) with your transfer server or file-storage node on another computer.
Page 285 | Set up HST Server for Node API | 285 Organization Name (eg, company) [Internet Widgits Pty Ltd]:Your_Company Organizational Unit Name (eg, section) []:Your_Department Common Name (i.e., your server's hostname) []:secure.yourwebsite.com Email Address []:johndoe@yourwebsite.com You are also prompted to input "extra" attributes, including an optional challenge password. Note: Manually entering a challenge password when starting the server can be problematic in some situations, for example, when starting the server from the system boot scripts.
Page 286: Installing Ssl Certificates
6. Enable SSL options in aspera.conf. For information about enabling specific SSL protocols with <ssl_protocol> and enabling specific encryption ciphers with <ssl_ciphers>, see Configuring the IBM Aspera NodeD Service on page 277. 7. Restart asperanoded by running the following command:...
Page 287 | Set up HST Server for Node API | 287 • You place the certificate bundle (chained or intermediary certificates) from the CA in this file. Changing Filenames and Locations: If desired, the default filenames and locations of the certificate files and chain files can be changed by configuring settings in the transfer server's aspera.conf file, using asconfigurator commands: # asconfigurator -x "set_http_server_data;cert_file,path/certfile.pem"...
Page 288 Failure: The following sample output shows that verification failed because verify return is 1. depth=0 C = US, ST = California, L = Emeryville, O = IBM, OU = Aspera Inc IT Department, CN = *.asperafiles.com verify error:num=20:unable to get local issuer certificate...
Page 289: Authentication And Authorization
HTTPS: HTTPS (Node API) authentication was introduced to support browsing and transfers that are initiated through Aspera web applications (IBM Aspera Faspex, IBM Aspera Shares, and IBM Aspera on Cloud), and uses a token-based authorization security layer in addition to SSH.
Page 290: Require Token Authorization: Set From The Command Line
The user should not have a password. If the system does not allow this, create a very large password. 2. Set the IBM Aspera Connect public SSH key as an authorized key for the transfer user and ensure that they own the file.
Page 291: Transfer Token Creation (Node Api)
Note: When generating tokens with an IBM Aspera Shares server, the endpoints are /node_api/files/ upload_setup and /node_api/files/download_setup. Upload token...
Page 292 | Authentication and Authorization | 292 -i Include the HTTP header in the output. -X POST Specify a POST request to the HTTP server, rather than the default GET request. (This option is not required when -d is used, but is included here for completeness). -u node_username:node_user_password Authenticate using the Node API username and password that are associated with the transfer user who has been configured for token authorization.
Page 293: Transfer Token Generation (Astokengen)
| Authentication and Authorization | 293 "destination" : "/archive/monday/texts/next_thing" "source" : "/monday/next_thing.txt", "destination" : "/archive/monday/texts/last_thing", "source" : "/monday/last_thing.txt" To use this file in the curl command, specify the path to the file in the -d option, as follows: -d @upload_setup.json Download token The method for generating a download token is the same as for an upload token, except that you use the /files/ download_setup (or /node_api/files/download_setup in the case of Shares) endpoint.
Page 294 | Authentication and Authorization | 294 Option (short Option (long form) Description form) --source-prefix=prefix Prepend the given path to each source path. Store the entire path set in the token. --full-paths Note: This option is required when creating tokens for A4 transfers. --file-list=filename Specifies a file name that contains a list of sources for a download token.
Page 295 | Authentication and Authorization | 295 • Authorize downloads of one or more files or directories from a specific destination: # astokengen --mode=recv [options] -u user -p path [-p path …] [-v token] • Authorize downloads of files specified in a file list: # astokengen --mode=recv [options] -u user --file-list=filename [-v token] •...
Page 296: Access Key Authentication
Access key authentication can by used by Aspera client products such as IBM Aspera Desktop Client, HST Server, HST Endpoint, and IBM Aspera Drive. It can also be used by IBM Aspera Faspex, IBM Aspera Shares, and AIBM Aspera on Cloud transfer service. For details about using access key authentication with these products, see their documentation.
Page 297 | Authentication and Authorization | 297 2. Assign a Node API username and password to the system user. This command requires admin permissions. # /opt/aspera/bin/asnodeadmin -au node_username -p node_password - x system_user For example, to assign the Node API username nodeuser to the system user xfer: # /opt/aspera/bin/asnodeadmin -au nodeuser -p asperaissofast -x xfer This command automatically reloads the node configuration.
Page 298 | Authentication and Authorization | 298 "files_filelock_enabled" : true|false, "files_filelock_restriction" : "restriction" Element Required Type Description Optional String ID of the access key. Returns 209 (conflict) if it already exists. If it is not provided, the Node API creates an ID and returns the value in the response. secret Optional String...
Page 299 | Authentication and Authorization | 299 Element Required Type Description • none - require unencrypted transfers (not recommended). • aes-128, aes-192, or aes-256 - allow transfers that use an encryption cipher key that is as long or longer than the setting. These settings use the CFB or GCM mode depending on the client version and cipher requested.
Page 300 Set to true to enable recursive counts. The access key configuration overrides the server configuration. This option must be enabled for event reporting to IBM Aspera on Cloud. Default is unset, such that the access key inherits the server configuration. Available as of 3.8.0.
Page 301 | Authentication and Authorization | 301 Because local storage objects are simple, you can create your access key by specifying the storage in the command line: # curl -ki -u nodeadmin:superP@55wOrD -X POST https://localhost:9092/access_keys -d'{"storage": {"type":"local","path":"/projects/project1"}} Amazon S3 {"storage" : { "type"...
Page 302 | Authentication and Authorization | 302 "type" : "azure-datalake", "path" : "container/path", "storage_endpoint" : "data_lake_store_name.azuredatalakestore.net", "credentials" : { "type" : "ClientCredential", "client_id" : "client_application_id", "refresh_url" : "https://login.windows.net/directory_id/ oauth2/token", "client_secret" : "secret" Azure SAS {"storage" : { "type" : "azure_sas", "container" : "container", "path"...
Page 303 "token_expiration" : "token_lifetime_seconds" "auth_uri" : "https://accounts.google.com/o/oauth2/auth", "token_uri" : "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url" : "https:// www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url" : "https://www.googleapis.com/robot/ v1/metadata/x509/client_id%40developer.gserviceaccount.com" IBM Cloud Object Storage (COS) - S3 {"storage" : { "type" : "ibm-s3", "path" : "bucket/path", "endpoint" : "s3-api.us- geo.objectstorage.service.networklayer.com", "credentials" : { "type"...
Page 304: Basic Tokens
Basic tokens are less restrictive than transfer tokens. They can be used to transfer with any Aspera server that supports access keys (all but IBM Aspera on Cloud).
Page 305: Bearer Tokens
A bearer token is created from an access key ID, access key secret, and an SSL private-public key pair. Bearer token authentication is required for transfers to and from IBM Aspera on Cloud, but can be used for transfers with all other Aspera servers, too.
Page 306: Highly-Available Redis Backend For An Hst Server Cluster
Aspera with continuous availability and automatic failover if one node goes down. A cluster consists of nodes. A node is a machine (physical, virtual, or container) on which IBM Aspera High-Speed Transfer Server and Redis are installed. A Redis node can be a primary (also called a "master") or replica ("slave").
Page 307: Requirements
To sucessfully deploy a highly-available HST Server cluster, your system must meet the following requirements: • At least three nodes, preferably an odd number. A node is a machine (physical, virtual, or container) on which IBM Aspera High-Speed Transfer Server is installed. • Glibc 2.5 or higher.
Page 308 | Highly-Available Redis Backend for an HST Server Cluster | 308 2. On the Redis primary node, create (or edit an existing) /opt/aspera/etc/redis.conf with the following values: bind private_ip_address port 31415 daemonize yes pidfile /opt/aspera/var/run/redis.31415.pid /opt/aspera/var/run/redis_31415.log tcp-keepalive 300 dbfilename /opt/aspera/var/redis.31415.rdb dir /opt/aspera/var save 900 1 protected-mode no...
Page 309 | Highly-Available Redis Backend for an HST Server Cluster | 309 • private_ip_address is the private IP address of the local node. • primary_name is a name of the primary node. The name can include A-z, 0-9, and ".", "-", or "_"; the name cannot include special characters or spaces.
Page 310: Start And Test The Cluster
| Highly-Available Redis Backend for an HST Server Cluster | 310 tcp-check send PING\r\n tcp-check expect string +PONG tcp-check comment role\ check tcp-check send info\ replication\r\n tcp-check expect string role:master tcp-check comment QUIT\ phase tcp-check send QUIT\r\n tcp-check expect string +OK server redis-1 primary_ip_address:31415 maxconn 1024 check inter 1s server redis-2 replica1_ip_address:31415 maxconn 1024 check inter 1s server redis-3 replica2_ip_address:31415 maxconn 1024 check inter 1s...
Page 311: Redis Sentinel Operation
| Asconfigurator Reference | 311 Redis Sentinel Operation Use the Redis Sentinel API to get information about the Redis nodes, test failover, and reconfigure Sentinel at runtime, among other tasks. For more information, see https://redis.io/topics/sentinel. Logging Logs for Redis, HAProxy, and HST Server are found in the following locations: Redis database: /opt/aspera/etc/redis_31415.log or syslog, depending on the Redis configuration.
Page 312 | Asconfigurator Reference | 312 Commands for Setting Parameter Values Command Description set_user_data Sets data in the user section. For parameters and values, User, Group and Default Configurations on page 315. set_group_data Sets data in the group section. For parameters and values, see User, Group and Default Configurations page 315.
Page 313: Examples
| Asconfigurator Reference | 313 The command below takes a path to a file to modify. If the file does not exist, it is created. # asconfigurator -x "command[;parameter,value;parameter,value]" /path/to/ file The command below takes paths to two files. The first file is used as a base, and the modifications are written to the second file.
Page 314: Reading Output
| Asconfigurator Reference | 314 • Setting the docroot of your transfer user # asconfigurator -x "set_user_data;user_name,transferuser;absolute,/path/ to/docroot" • Enabling HTTP Fallback using HTTPS on port 8444. # asconfigurator -x "set_http_server_data;enable_https,true" # asconfigurator -x "set_http_server_data;https_port,8444" Note: You can also chain two or more parameters to set within the same command. The two commands above can be combined as follows (separated by semi-colons): # asconfigurator -x "set_http_server_data;enable_https,true;https_port,8444"...
Page 315: User, Group And Default Configurations
| Asconfigurator Reference | 315 Syntax Error: Syntax error. Valid values are "assert_current","server" or"option_mask", got "enable_htt" Reading aspera.conf configuration settings with asuserdata You can view the current configuration settings by section and all the possible parameters with their default values and corresponding asconfigurator syntax by running asuserdata.
Page 316 | Asconfigurator Reference | 316 Setting or getting user/group data requires you to specify the username or group name as the first parameter of the asconfigurator command. Note: Not all available parameters are listed below, only the most commonly used. To view a complete list, run the following command: # /opt/aspera/bin/asuserdata -+ Transfer Authorizations...
Page 317 | Asconfigurator Reference | 317 Values: (Number 0-255) transfer_out_bandwidth_aggregate_trunk_id The ID of the Vlink to apply to outgoing transfers. A value of 0 disables the Vlink. Values: (Number 0-255) transfer_in_bandwidth_flow_target_rate_cap The maximum value to which the target rate for incoming transfers can be set. Values: (Number) transfer_out_bandwidth_flow_target_rate_cap The maximum value to which the target rate for outgoing transfers can be set (in Kbps).
Page 318 | Asconfigurator Reference | 318 Values: false (default), true transfer_out_bandwidth_flow_min_rate_lock A value of false allows users to adjust the minimum rate for outgoing transfers. A value of true prevents users from adjusting the minimum rate for outgoing transfers. Values: false (default), true transfer_in_bandwidth_flow_policy_default The default bandwidth policy for incoming transfers.
Page 319 | Asconfigurator Reference | 319 Whether a strong passphrase is required for content protection (6 characters long, at least one letter, number and special symbol). Values: false (default), true Transfer File System Options resume_suffix The extension of files used to store metadata and enable resumption of partially completed transfers. Include a '.' in the suffix, such as: .aspera Values: (String), default .aspx preserve_attributes...
Page 320 | Asconfigurator Reference | 320 Extension to be added to the names of files that are currently only partially transferred. Include a '.' in the suffix, such as: .aspera Values: (String) file_checksum Type of checksum to compute while reading a file. Checksums are used to verify that file contents on the destination match what was read on the destination.
Page 321: Trunk (Vlink) Configurations
| Asconfigurator Reference | 321 Trunk (Vlink) Configurations General Syntax This collection of commands configures settings related to Vlinks, which are aggregate bandwidth caps applied to transfer sessions. The syntax for setting trunk configurations is the following : # asconfigurator -x "set_trunk_data;id,trunk_id;parameter,value" Setting or getting trunk data requires you to specify the ID number of the Vlink as the first parameter of the asconfigurator command.
Page 322: Http Server Configurations
| Asconfigurator Reference | 322 The network interface address on which the Aspera Central listens. The default 127.0.0.1 enables the transfer server to accept transfer requests from the local computer. Setting the value to 0.0.0.0 allows the transfer server to accept transfer requests on all network interfaces. Values: (Network interface address, default 127.0.0.1) port The port on which the Aspera Central service listens.
Page 323 | Asconfigurator Reference | 323 The syntax for setting HTTP server parameters is the following : # asconfigurator -x "set_http_server_data;parameter,value" Note: Not all available parameters are listed below, only the most commonly used. To view a complete list, run the following command: # /opt/aspera/bin/asuserdata -+ HTTP Server Configurations...
Page 324: Database Configurations
| Asconfigurator Reference | 324 Database Configurations General Syntax This collection of commands configures settings related to the MySQL database that stores transfer data (for use with Aspera Console before version 3.0). The syntax for setting database parameters is the following: # asconfigurator -x "set_database_data;parameter,value"...
Page 325: Server Configurations
| Asconfigurator Reference | 325 Values: true (default), false file_progress Whether file status, such as bytes transferred, should be logged (true) or not (false). Values: true (default), false file_progress_interval The frequency with which an Aspera node logs file transfer data, in seconds. Values: (Number 1-65535, default 1) files_per_session The number of file names to be recorded for any transfer session.
Page 326 | Asconfigurator Reference | 326 Values: (String) transfers_multi_session_default The default value for the number of sessions in a multi-session transfer. Values: (Number, default 1) transfers_retry_duration The time duration during which transfer retries are attempted. Values: (Time value, default 20m) transfers_retry_all_failures Whether a transfer should be retried after all failures (true) or not (false).
Page 327 | Asconfigurator Reference | 327 max_response time_sec The time limit in seconds before an unresponsive Node API response times out. Values: (Number, default 10) db_dir The path to the directory where the redis database file for the Node API is saved. Values: (Absolute path) db_port The port on which the redis database for the Node API listens.
Page 328 | Asconfigurator Reference | 328 The upper bound of the port range for the forward proxy. Values: (Number, default 10000) proxy_cleanup_interval The interval in seconds at which the forward proxy server scans and cleans up expired sessions. Values: (Number, default 0) proxy_keepalive_internal The interval in seconds at which the ascp client sends keep-alive requests.
Page 329: Client Configurations
| Troubleshooting | 329 Values: 0 (default), 1, 2 rproxy_log_directory The reverse proxy server log file location. If no value is set, the proxy logs to syslog. Values: (Absolute path) Client Configurations General Syntax Guidelines This collection of commands configures settings related to client transfers, which are transfers you initiate with ascp on the command line or the GUI of your product.
Page 330 | Troubleshooting | 330 a) On the client computer, run the following command: # telnet server_ip_address port For example, to test connection to 10.0.1.1 through TCP/33001, you run the following command: # telnet 10.0.1.1 33001 b) If the client cannot establish connections to the ports, verify the port number and the firewall configuration of HST Server.
Page 331: Error: Session Timeout During Ascp Transfers
| Troubleshooting | 331 7. Verify that the user credentials are correct, and the user has sufficient access permissions to their docroot a) Attempt to establish an SSH connection: # ssh username@server_ip_address -p port For example: $ ssh aspera_user_1@10.0.1.1 -p 33001 b) Enter the user's password.
Page 332: Node Api Transfers Of Many Small Files Fails
| Troubleshooting | 332 Node API Transfers of Many Small Files Fails Ascp transfers that are started through the Node API or Watch Folders to or from servers that have Unix-like OS can fail when transferring many (millions) of small files because the Redis database exceeds available number of file descriptors.
Page 333: Appendix
When you change product settings, you might need to restart certain Aspera services in order for the new values to take effect. IBM Aspera Central If asperacentral is stopped, or if you have modified the <central_server> or <database> sections in aspera.conf, then you need to restart the service.
Page 334: Docroot Vs. File Restriction
• Complex file-system access rules (docroot in URI format) • Creating access keys with the • Connecting the node to IBM Node API Aspera Faspex, IBM Aspera • Connecting the node to IBM Shares, IBM Aspera Console, Aspera on Cloud...
Page 335: Aspera Ecosystem Security Best Practices
Aspera Ecosystem Security Best Practices Your Aspera applications can be configured to maximize system and content security. The following sections describe the recommended settings and practices that best protect your content when using IBM Aspera High-Speed Transfer Server and IBM Aspera High-Speed Transfer Endpoint.
Page 336 | Appendix | 336 Rarely, security vulnerabilities are detected in the operating systems and third-party components that are used by Aspera. Aspera publishes security bulletins immediately that describe the affected products and recommended remediation steps. Security Configuration Recommended security settings vary depending on the products you are using and how they interact. See the following subsections for your Aspera products.
Page 337 # service sshd restart g) Review your logs periodically for attacks. For information on identifying attacks, see IBM Aspera IBM Aspera High-Speed Transfer Server Admin Guide: Securing Your SSH Server. 2. Configure your server's firewall to permit inbound access to only Aspera-required ports.
Page 338: Securing The Aspera Applications
4. If asperanoded is exposed to internet traffic, run it behind a reverse proxy. If your Aspera server must expose asperanoded to the internet, such as when setting it up as a IBM Aspera on Cloud (AoC) node, Aspera strongly recommends protecting it with a reverse proxy. Normally, asperanoded runs on port 9092, but nodes that are added to AoC must have asperanoded run on port 443, the standard HTTPS port for secure browser access.
Page 339 Replace username with the username and storage_path with the path to which the user has access. Restriction syntax is specific to the storage: Storage Type Format Example local storage file:////* S3 and IBM Cloud Object Storage s3://* Swift storage swift//* Azure storage azu://*...
Page 340 | Appendix | 340 Allow transfers for specific users by running the following commands for each user: # asconfigurator -x "set_user_data;user_name,username;authorization_transfer_in_value,allow" # asconfigurator -x "set_user_data;user_name,username;authorization_transfer_out_value,allow" Note: For a user that is used by Shares or Faspex (usually xfer), allow transfers only with a token by setting authorization_transfer_{in|out}_value to token.
Page 341: Securing Content In Your Workflow
| Appendix | 341 a) Retrieve the server's SHA-1 fingerprint. # cat /etc/ssh/ssh_host_rsa_key.pub | awk '{print $2}' | base64 - | sha1sum b) Set the SSH host key fingerprint in aspera.conf. # asconfigurator -x "set_server_data;ssh_host_key_fingerprint,fingerprint" This command creates a line similar to the following example of the <server> section of aspera.conf: <ssh_host_key_fingerprint>7qdOwebGGeDeN7Wv+2dP3HmWfP3 </ssh_host_key_fingerprint>...
Page 342 | Appendix | 342 containing both the encrypted contents of the file and the encryption metadata, and it also changes the name of the file by adding the file extension .aspera-env.) • It works with both regular transfers (FASP) and HTTP fallback transfers. Limitations and Other Considerations •...
Page 343: Testing And Optimizing Transfer Performance
| Appendix | 343 GUI: Go to Connections > connection_name > Security. Select Encrypt uploaded files with a password and set the password. Select Decrypt password-protected files downloaded and enter the password. Ascp command line: Set the encryption and decryption password as the environment variable ASPERA_SCP_FILEPASS.
Page 344 | Appendix | 344 To send random data in place of a source file (do not read from the source), you can specify the file as faux:///fname?fsize.fname is the name assigned to the file on the destination and fsize is the number of bytes to send.
Page 345: Create An Ssl Certificate (Apache)
| Appendix | 345 To send random data to a faux target, run the following command: # ascp --mode=send --user=username --host=host_ip_address faux:///fname?fsize faux:// Testing Transfer Performance 1. Start a transfer with fair transfer policy and compare the transfer rate to the target rate. On the client computer, open the user interface and start a transfer (either from the GUI or command line).
Page 346 | Appendix | 346 You can also visit http://www.openssl.org/source for a repository of all OpenSSL distribution tarballs. Aspera recommends that you review your specific operating system's documentation for information on installing or upgrading OpenSSL packages. 2. Generate your Private Key and Certificate Signing Request using OpenSSL by running the following command: $ openssl req -new -nodes -newkey rsa:2048 -keyout my_key_name.key - out my_csr_name.csr Where my_key_name.key is the name of the unique key you are creating and my_csr_name.csr is the name of your...
Page 347: Enable Ssl (Apache)
| Appendix | 347 Important: Some Certificate Authorities provide a Certificate Signing Request generation tool on their Website. Please check with your CA for additional information. 5. Generate a Self-Signed Certificate (Optional). At this point, you might need to generate a self-signed certificate for either of the following reasons: •...
Page 348: Log Files
| Appendix | 348 Listen 443 </IfDefine> d) Update the SSLCertificateFile and SSLCertificateKeyFile values to the correct certificate paths and file names. For example: SSLCertificateFile /path/to/my_cert_name.crt SSLCertificateKeyFile /path/to/my_key_name.key e) Save your changes and close the file. 3. Restart the Apache Web Server. Linux OS Restart Command Debian 8 or newer...
Page 349 | Appendix | 349 Note: To view logs from the command line in Linux, you must have a functional web-browser or other default application for opening HTML files. To set the logging level for transfers, open the My Preferences dialog by clicking Tools > Preferences or by clicking Preferences in the upper-right corner of the application window.
Page 350 | Appendix | 350 To redirect Aspera logging, modify /etc/syslog.conf (/etc/rsyslog.conf in the case of Red Hat or CentOS 6.XA) and add local2.none to the /var/log/messages line. For example, if you have the following line: *.info;mail.none;authpriv.none;cron.none /var/log/messages Change it to: *.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages Next, forward local2.info log messages to your new file.
Page 351 | Appendix | 351 To test this, run the following commands: # logger -p local2.info aspera test # cat /var/log/aspera.log The cat command should display something similar to the following: Jun 13 10:30:33 linux-kua5 root: aspera test Rotating Your Aspera Log File There are several ways to rotate Aspera logs in Linux: 1.
Page 352: Preserving Ibm Spectrum Scale Acls Of Transferred Files
Preserving IBM Spectrum Scale ACLs of Transferred Files Ascp and Aspera Sync can preserve NFSv4 and POSIX ACLs and immutability attributes when transferring files from an IBM Spectrum Scale (formerly GPFS) cluster to another cluster. Preserving Spectrum Scale ACLs To preserve file attributes and permissions when transferring from one Spectrum Scale cluster to another, run ascp or async with the --preserve-xattrs=native option.
Page 353: Technical Support
Aspera, the Aspera logo, and FASP transfer technology are trademarks of Aspera, Inc., registered in the United States. Aspera Drive, IBM Aspera High-Speed Transfer Server (a merger of IBM products formerly named Aspera Connect Server and Aspera HST Server, 2008 and 2007), IBM Aspera High-Speed Endpoint (formerly Aspera Point-to-Point,...

IBM Aspera HST Admin Manual

Introduction

What's New

Get Started with an Aspera Transfer Server

Get Started as a Transfer Client

Comparison of Aspera File Delivery and Synchronization Tools

Installation and Upgrades

Set up the HST Server Web UI

Server Set up Methods

Set up Users and Groups

Configure the Server from the Command Line

File Pre- and Post-Processing (Prepost)

Email Notifications

Ascp: Transferring from the Command Line with Ascp

Ascp4: Transferring from the Command Line with Ascp 4

Watch Folders and the Aspera Watch Service

Aspera Sync

Appendix

Configuring for Other Aspera Products

Set up HST Server for Node API

Authentication and Authorization

Highly-Available Redis Backend for an HST Server Cluster

Asconfigurator Reference

Troubleshooting

Appendix

Technical Support

Legal Notice

Quick Links

Troubleshooting

Related Manuals for IBM Aspera HST

Summary of Contents for IBM Aspera HST