Technical article 127376

I need to transfer a very large file, but my transfers timeout or fail due to corruption.

OpenServer 6.0.0OpenServer 5.0.0OpenServer 5.0.2OpenServer 5.0.4OpenServer 5.0.5OpenServer 5.0.6OpenServer 5.0.7UnixWare 7.0.0UnixWare 7.0.1UnixWare 7.1.0UnixWare 7.1.1UnixWare 7.1.3UnixWare 7.1.4Open UNIX 8

Written · Last revised

Problem

My file is too large to transfer over ftp, how can I get around this problem?

Cause

Transferring large files over ftp can fail for any of these reasons:

* Some ftp servers won't store files larger than 1GB. * Network connection poor or unstable. * Long inactive terminal session times out. * Transfer completes, but results in corrupted files.

Solution

Use compression utilities like compress(C), gzip(C), or bzip2(C) to reduce the size of the file being transferred.

You can use the split(C) utility to break the file into smaller pieces and then once transferred to the target machine, use cat(C) to combine the pieces back into a single file.

To keep the transferring terminal session active use ftp's 'hash' option to provide hash marks as transfer status.

To avoid corruption use ftp's 'bin' option so the transfer uses binary mode.

Here is a step-by-step outline showing how to transfer a large file.

1) First reduce the file sizes by using compress and split:

$ compress big_file
$ split -b 500m big_file.Z big_file.Z.part

This will create separate files, each 500MB with the names: big_file.Z.partaa big_file.Z.partab big_file.Z.partac big_file.Z.partad

2) transfer these smaller files through ftp in binary mode with hash marks:

$ ftp -p ftp.destination.com
ftp> bin
ftp> hash
ftp> cd /target
ftp> mput big_file.Z.part*

3) On the target machine, use cat to combine the files into one:

$ cat big_file.Z.partaa big_file.Z.partab big_file.Z.partac
  big_file.Z.partad > big_file.Z
$ uncompress big_file.Z

If you are having problems retrieving a file through ftp due to network timeouts, try using ftp's 'reget' command along with 'bin' and 'hash':

$ ftp -p ftp.server.com
ftp> bin
ftp> hash
ftp> reget big_file.Z.partaa

The reget command will continue from where the last transfer failed and append to the end of the existing file on the local system so the transfer doesn't need to start over at the beginning.

NOTE:     split(C) is limited to files 2GB or less at this time, a
          request has been filed with SCO Engineering to have split
          made largefile aware.  On Openserver6 and UnixWare7
          compress is largefile aware, so if you can compress the
          file to less than 2GB, you can still split it and use this
          method to transfer the file.
NOTE:     No additional utilities such as BZIP2, GZIP, INFOZIP or ZIP
          support file sizes greater than 2Gb.

See also

TA110169 "How to enable large file support?" TA126131 "I see the error "value too large for defined data type" TA105852 "What are the filesystem types that can be used in SCO OpenServer 5?"

From the Xinuos technical article database, record 127376. Reproduced with permission. The original is at www.sco.com/ta/127376, which is reachable but blocked from search engines.

Next step

Need this fixed rather than explained?

Tell us the release, what the system runs and what it is doing. We will reply by email with what we would do next.

  • You deal with the people who do the work
  • No obligation, and no upgrade required before we will help
  • We tell you plainly if something is not worth doing

Send an enquiry

We reply by email. If a server is down, say so in the first line. You can also write to help@scoexpert.com.