How to Integrate CarrierWave, Fog, and Google Cloud Storage for all of your Photo Upload needs
In this post I will outline the steps necessary to implement a photo upload feature in a Rails application using CarrierWave, Fog, and Google Cloud Storage. CarrierWave is "a simple and extremely flexible way to upload files from Ruby applications." Fog "is the Ruby cloud services library", and Google Cloud Storage is something I seem to be obsessed with while everyone else prefers AWS.
Gems
Let's start by adding some gems to our Gemfile:
gem 'carrierwave' gem 'fog' gem 'mini_magick'
Mini_magick will be used to handle any image processing we may want to implement. There are other options, I went with Mini_magick. Next from terminal run:
Bundle
TESTING
Next we'll write a feature test to drive our development.
# spec/features/photo__upload_spec.rb require 'rails_helper' RSpec.feature "admin uploads photo" do context "completes the photos#new form" do it "redirects to the new photo's show page" do Fog.mock! Fog::Mock.delay = 0 service = Fog::Storage.new({ provider: 'Google', google_storage_access_key_id: ENV['google_storage_access_key_id'], google_storage_secret_access_key: ENV['google_storage_secret_access_key'] }) service.directories.create(:key => 'photo-of-the-day') admin = create :admin sign_in admin visit new_photo_path fill_in "Title", with: "My photo title" fill_in "Caption", with: "My photo caption" fill_in "Date", with: "2015-12-29" attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png" click_button "Upload" expect(current_path).to eq(photo_path(Photo.last)) end end end
Ignoring the Fog.mock! and admin stuff for a second and just starting with:
# spec/features/photo__upload_spec.rb require 'rails_helper' RSpec.feature "admin uploads photo" do context "completes the photos#new form" do it "redirects to the new photo's show page" do ... visit new_photo_path fill_in "Title", with: "My photo title" fill_in "Caption", with: "My photo caption" fill_in "Date", with: "2015-12-29" attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png" click_button "Upload" expect(current_path).to eq(photo_path(Photo.last)) end end end
This should look like a fairly straightforward RSpec feature test with capybara. We are having capybara fill in a title, caption, date, and then attach a file, which we have stored in our spec/fixtures directory. Lastly we have capybara click the "upload" button and then we assert that the current path should be the photo show page for the photo we just created.
Next let's take a closer look at the admin piece.
# spec/features/photo__upload_spec.rb require 'rails_helper' RSpec.feature "admin uploads photo" do context "completes the photos#new form" do it "redirects to the new photo's show page" do ... admin = create :admin sign_in admin ... end end end
These two lines of code are just meant to create an admin type user using FactoryGirl and then sign the admin into the application using the Devise sign_in method. In my application, I want only admin to be able to use this feature, hence the authorization portion of this feature test. If all users should be able to upload photos, these lines would not be necessary.
Now the fun part. Fog has a mock feature which allows you to run your test without actually uploading a file to your cloud storage destination of choice. It is important to mock this feature in order to speed up the test suite and keep your online cloud storage bucket free of detritus.
The configuration for Fog.mock! is actually very simple. You can extract this into a spec_helper file if that makes more sense to you, but since I only have one photo upload feature, I only have this one test to mock.
# spec/features/photo__upload_spec.rb require 'rails_helper' RSpec.feature "admin uploads photo" do context "completes the photos#new form" do it "redirects to the new photo's show page" do Fog.mock! Fog::Mock.delay = 0 service = Fog::Storage.new({ provider: 'Google', google_storage_access_key_id: ENV['google_storage_access_key_id'], google_storage_secret_access_key: ENV['google_storage_secret_access_key'] }) service.directories.create(:key => 'photo-of-the-day') ... end end end
The above configuration says that first, we are going to run Fog in mock mode. Next we are going to apply a delay of 0, which I believe means that Fog will not mimic the actual delay required to upload a photo to a cloud storage provider. Next we provide our cloud storage credentials. This is the most important part. The credentials must match the credentials that we use for the actual upload. I have chosen to store my credentials with the Figaro gem as environment variables. There are other ways to do this, but this is my preferred method.
CarrierWave Configuration
Next let's set up our CarrierWave initializer. Start by creating a config/carrierwave.rb file.
#config/carrierwave.rb CarrierWave.configure do |config| config.fog_credentials = { provider: 'Google', google_storage_access_key_id: ENV['google_storage_access_key_id'], google_storage_secret_access_key: ENV['google_storage_secret_access_key'] } config.fog_directory = 'photo-of-the-day' end
Add a credentials hash to your config (config.fog_credentials = {}). Again I use my environment variables just as I did in the test. The config.fog_directory is the name of the cloud storage bucket into which I would like to upload my files.
Google Cloud
Setting up Cloud Storage on Google's Cloud platform is nicely documented here. Once you have a Cloud Storage instance up and running, you just need to get an API key and secret, which can be a little hard to find.
From Cloud Storage, go to settings, and then interoperability. Click "Create a new key". Boom. Again, I put this information in my application.yml file using Figaro so I can reference these codes securely throughout my application.
Generate an Uploader
I've already set up my photo model with a handy:
rails g model Photo title caption date:date image
The above command after running your migration will yield the following schema:
#db/schema.rb create_table "photos", force: :cascade do |t| t.string "title" t.string "caption" t.date "date" t.string "image" t.datetime "created_at", null: false t.datetime "updated_at", null: false end
For my application I want every photo object to have a title, caption, and date (which could be different from the created_at / updated_at timestamps). Then I will actually store the photo under the attribute "image" which is of datatype string.
So now we'll follow the CarrierWave documentation and run:
rails generate uploader Image
"Image" corresponds to the image attribute on our photo model. Running this command will create an "app/uploaders" directory with a "image_uploader.rb" file. Here are the contents of my image_uploader.rb file:
# app/uploaders/image_uploader.rb class ImageUploader < CarrierWave::Uploader::Base include CarrierWave::MiniMagick storage :fog def store_dir end version :thumb do process resize_to_fit: [300, 300] end def extension_whitelist %w(jpg jpeg gif png) end def filename "#{model.date}-#{model.title}-#{Time.now.getutc.to_s}" end end
When the file generates, there will be lots of commented out documentation to help you customize the CarrierWave configuration. Let's go line by line to understand what is going on.
First I'm including MiniMagick to handle image processing because in addition to the original version of the image, I also want to have a thumbnail of the image created upon upload.
Next I am telling CarrierWave that I want to use ":fog" as my storage. CarrierWave plays nicely with Fog and a great number of cloud storage providers.
Next I describe the thumbnail version of the photo that I want to create which is sized at 300 x 300.
Next I list the file types I would like CarrierWave to handle. So with this configuration CarrierWave will reject non-image file types.
Last I describe the file naming convention I would like to apply. In this case it is a concatenation of the image's date, title, and then the current time.
Add Uploader to the Model
Next we have to add the uploader we just created to our Photo model. To do so we'll just add the following code.
# app/models/photo.rb class Photo < ApplicationRecord mount_uploader :image, ImageUploader validates_presence_of :title validates_presence_of :caption validates_presence_of :date end
The validations are clearly application specific, but there is one notable point to make here. Notice how I validate for presence of all of my photo attributes except :image. Calling photo.image will actually return nil, so the validation test with shoulda_matchers will fail. To determine if an image is attached to the photo, the CarrierWave documentation suggests calling photo.image.file.nil?
Routes
Let's add the following line to our config/routes.rb file.
resources :photos, only: [:new, :create, :show]
Photo Controller
Moving on to our app/controller/photos_controller.rb we see there is not anything special we need to do, CarrierWave and Fog make the magic happen.
# app/controller/photos_controller.rb class PhotosController < ApplicationController def new @photo = Photo.new end def create @photo = Photo.new(photo_params) if @photo.save redirect_to photo_path(@photo) end end def show end private def photo_params params.require(:photo).permit(:title, :caption, :date, :image) end end
Add Views
First lets add an app/view/photos/new.html.erb view to hold our upload form.
# app/views/photos/new.html.erb <div class="container"> <h3>New Photo of the Day</h3> <div class="row"> <%= form_for @photo, multiple: true do |f| %> <div class="input-field"> <%= f.label :title %> <%= f.text_field :title %> </div> <div class="input-field"> <%= f.label :caption %> <%= f.text_area :caption, class: "materialize-textarea" %> </div> <%= f.label :date %> <%= f.date_field :date, class: "datepicker"%> <div class="file-field input-field"> <div class="btn btn-flat waves-effect waves-light cyan accent-2"> <span>File</span> <%= f.file_field :image %> </div> <div class="file-path-wrapper"> <input class="file-path validate" type="text"> </div> </div> <div class="input-field center-align"> <%= f.submit "Upload", class: "btn waves-effect waves-light cyan darken-3" %> </div> <% end %> </div> </div>
And we'll just add app/view/photos/show.html.erb view, which for now we'll leave blank because our test does not specifically look for any content on this page.
Run the test
Next I comment out the Fog.mock!, run the test, and check my Cloud Storage for the file. I do this to make sure that the test runs and actually uploads an image into Cloud Storage.
require 'rails_helper' RSpec.feature "admin uploads photo" do context "completes the photos#new form" do it "redirects to the new photo's show page" do # Fog.mock! # Fog::Mock.delay = 0 # service = Fog::Storage.new({ # provider: 'Google', # google_storage_access_key_id: ENV['google_storage_access_key_id'], # google_storage_secret_access_key: ENV['google_storage_secret_access_key'] # }) # service.directories.create(:key => 'photo-of-the-day') admin = create :admin sign_in admin visit new_photo_path fill_in "Title", with: "My photo title" fill_in "Caption", with: "My photo caption" fill_in "Date", with: "2015-12-29" attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png" click_button "Upload" expect(current_path).to eq(photo_path(Photo.last)) end end end
Notice that the run time below is 1.61 seconds without mocking the actual file upload.
And here are the two files, the uploaded original copy and the processed thumbnail in my cloud storage.
Then uncommenting out the Fog.mock! lines and running the test gives us:
Notice that the run this time took only .66 seconds, a full second faster when we mock the file upload. Also note, if you look at your cloud storage after running the test with Fog.mock! you will not see any new files.
If you made it this far, thanks for sticking with me. CarrierWave, Fog and Google Cloud Storage is a great solution for photo uploading needs and I hope that this post elucidates the process slightly for newcomers to the challenge.