Large Ruby File Downloads Done Right!

The post explains that large file transfers need to account for limited RAM. In the example, a Ruby utility moves files from Rackspace Cloud to Amazon S3 on a small EC2 instance, so the download has to be written to disk as it arrives instead of being kept in memory.

Reading notes

#

• The utility downloads large files from Rackspace and uploads them to Amazon S3.
• Some of the files are 1 GB or larger.
• The server used is an Amazon Micro EC2 instance with 613 MB of RAM.
• Loading the full HTTP response into memory would risk exhausting RAM and stopping the download.
• The code uses Net::HTTP to request the file.
• The response body is read in segments.
• Each segment is written directly to a file on disk.
• The file handle is closed in an ensure block.