Skip to content
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.

Commit

Permalink
Add fog_options to configuration to be passed to fog #create
Browse files Browse the repository at this point in the history
We found that uploading large files to S3 would result in a socket error
("connection reset by peer") occasionally and lately much more
consistently. In researching this I saw that many people got this error
when uploading too large of a file without multipart chunking. I would
have assumed fog did this automatically but the default chunk size may
be too high. In order to address this I wanted to drop the chunk size to
100MB.

Rather than hard-code this I opted to expose a `fog_option`
configuration option that lets me pass any additional options I want to
the fog's `#create` call. This is similar to the `fog_attributes` option
implemented in CarrierWave which [addresses the same
problem](http://stackoverflow.com/a/11867978/201911).

We've been running this now for a week in production and it seems to
resolve the issue.

#2135
  • Loading branch information
jeremywadsack authored and tute committed Mar 23, 2016
1 parent e60f000 commit 65ef0d3
Show file tree
Hide file tree
Showing 3 changed files with 29 additions and 2 deletions.
2 changes: 2 additions & 0 deletions NEWS
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
master:

* Improvement: Add `fog_options` configuration to send options to fog when
storing files.
* Drops support to end-of-life'd ruby 2.0.
* Improvement: Paperclip now supports aws-sdk v2
@betesh, @davetchen,
Expand Down
10 changes: 8 additions & 2 deletions lib/paperclip/storage/fog.rb
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ module Storage
# that is the alias to the S3 domain of your bucket, e.g.
# 'http://images.example.com'. This can also be used in
# conjunction with Cloudfront (http://aws.amazon.com/cloudfront)
# * +fog_options+: (optional) A hash of options that are passed
# to fog when the file is created. For example, you could set
# the multipart-chunk size to 100MB with a hash:
# { :multipart_chunk_size => 104857600 }

module Fog
def self.extended base
Expand Down Expand Up @@ -98,12 +102,14 @@ def flush_writes
log("saving #{path(style)}")
retried = false
begin
directory.files.create(fog_file.merge(
attributes = fog_file.merge(
:body => file,
:key => path(style),
:public => fog_public(style),
:content_type => file.content_type
))
)
attributes.merge!(@options[:fog_options]) if @options[:fog_options]
directory.files.create(attributes)
rescue Excon::Errors::NotFound
raise if retried
retried = true
Expand Down
19 changes: 19 additions & 0 deletions spec/paperclip/storage/fog_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,25 @@ def custom_method
assert_equal @dummy.avatar.fog_credentials, @dynamic_fog_credentials
end
end

context "with custom fog_options" do
before do
rebuild_model(
@options.merge(fog_options: { multipart_chunk_size: 104857600 }),
)
@dummy = Dummy.new
@dummy.avatar = @file
end

it "applies the options to the fog #create call" do
files = stub
@dummy.avatar.stubs(:directory).returns stub(files: files)
files.expects(:create).with(
has_entries(multipart_chunk_size: 104857600),
)
@dummy.save
end
end
end

end
Expand Down

0 comments on commit 65ef0d3

Please sign in to comment.