CarrierWave, Fog, and Google Cloud Storage

How to Integrate CarrierWave, Fog, and Google Cloud Storage for all of your Photo Upload needs

In this post I will outline the steps necessary to implement a photo upload feature in a Rails application using CarrierWave, Fog, and Google Cloud Storage. CarrierWave is "a simple and extremely flexible way to upload files from Ruby applications." Fog "is the Ruby cloud services library", and Google Cloud Storage is something I seem to be obsessed with while everyone else prefers AWS.

Gems

Let's start by adding some gems to our Gemfile:

gem 'carrierwave'
gem 'fog'
gem 'mini_magick'

Mini_magick will be used to handle any image processing we may want to implement. There are other options, I went with Mini_magick. Next from terminal run:

Bundle

TESTING

Next we'll write a feature test to drive our development.

# spec/features/photo__upload_spec.rb
require 'rails_helper'

RSpec.feature "admin uploads photo" do
  context "completes the photos#new form" do
    it "redirects to the new photo's show page" do
      Fog.mock!
      Fog::Mock.delay = 0
      service = Fog::Storage.new({
        provider: 'Google',
        google_storage_access_key_id: ENV['google_storage_access_key_id'],
        google_storage_secret_access_key: ENV['google_storage_secret_access_key']
      })
      service.directories.create(:key => 'photo-of-the-day')
      admin = create :admin
      sign_in admin
      visit new_photo_path

      fill_in "Title", with: "My photo title"
      fill_in "Caption", with: "My photo caption"
      fill_in "Date", with: "2015-12-29"
      attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png"
      click_button "Upload"

      expect(current_path).to eq(photo_path(Photo.last))
    end
  end
end

Ignoring the Fog.mock! and admin stuff for a second and just starting with:

# spec/features/photo__upload_spec.rb
require 'rails_helper'

RSpec.feature "admin uploads photo" do
  context "completes the photos#new form" do
    it "redirects to the new photo's show page" do
      ...
      visit new_photo_path

      fill_in "Title", with: "My photo title"
      fill_in "Caption", with: "My photo caption"
      fill_in "Date", with: "2015-12-29"
      attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png"
      click_button "Upload"

      expect(current_path).to eq(photo_path(Photo.last))
    end
  end
end

This should look like a fairly straightforward RSpec feature test with capybara. We are having capybara fill in a title, caption, date, and then attach a file, which we have stored in our spec/fixtures directory. Lastly we have capybara click the "upload" button and then we assert that the current path should be the photo show page for the photo we just created.

Next let's take a closer look at the admin piece.

# spec/features/photo__upload_spec.rb

require 'rails_helper'

RSpec.feature "admin uploads photo" do
  context "completes the photos#new form" do
    it "redirects to the new photo's show page" do
      ...
      admin = create :admin
      sign_in admin
      ...
    end
  end
end

These two lines of code are just meant to create an admin type user using FactoryGirl and then sign the admin into the application using the Devise sign_in method. In my application, I want only admin to be able to use this feature, hence the authorization portion of this feature test. If all users should be able to upload photos, these lines would not be necessary.

Now the fun part. Fog has a mock feature which allows you to run your test without actually uploading a file to your cloud storage destination of choice. It is important to mock this feature in order to speed up the test suite and keep your online cloud storage bucket free of detritus. 

The configuration for Fog.mock! is actually very simple. You can extract this into a spec_helper file if that makes more sense to you, but since I only have one photo upload feature, I only have this one test to mock.

# spec/features/photo__upload_spec.rb

require 'rails_helper'

RSpec.feature "admin uploads photo" do
  context "completes the photos#new form" do
    it "redirects to the new photo's show page" do
      Fog.mock!
      Fog::Mock.delay = 0
      service = Fog::Storage.new({
        provider: 'Google',
        google_storage_access_key_id: ENV['google_storage_access_key_id'],
        google_storage_secret_access_key: ENV['google_storage_secret_access_key']
      })
      service.directories.create(:key => 'photo-of-the-day')
      ...
    end
  end
end

The above configuration says that first, we are going to run Fog in mock mode. Next we are going to apply a delay of 0, which I believe means that Fog will not mimic the actual delay required to upload a photo to a cloud storage provider. Next we provide our cloud storage credentials. This is the most important part. The credentials must match the credentials that we use for the actual upload. I have chosen to store my credentials with the Figaro gem as environment variables. There are other ways to do this, but this is my preferred method.

CarrierWave Configuration

Next let's set up our CarrierWave initializer. Start by creating a config/carrierwave.rb file. 

#config/carrierwave.rb
CarrierWave.configure do |config|
  config.fog_credentials = {
    provider: 'Google',
    google_storage_access_key_id: ENV['google_storage_access_key_id'],
    google_storage_secret_access_key: ENV['google_storage_secret_access_key']
  }
  config.fog_directory = 'photo-of-the-day'
end

Add a credentials hash to your config (config.fog_credentials = {}). Again I use my environment variables just as I did in the test. The config.fog_directory is the name of the cloud storage bucket into which I would like to upload my files.

Google Cloud

Setting up Cloud Storage on Google's Cloud platform is nicely documented here. Once you have a Cloud Storage instance up and running, you just need to get an API key and secret, which can be a little hard to find.

From Cloud Storage, go to settings, and then interoperability. Click "Create a new key". Boom. Again, I put this information in my application.yml file using Figaro so I can reference these codes securely throughout my application.

Generate an Uploader

I've already set up my photo model with a handy:

rails g model Photo title caption date:date image

The above command after running your migration will yield the following schema:

#db/schema.rb
  create_table "photos", force: :cascade do |t|
    t.string   "title"
    t.string   "caption"
    t.date     "date"
    t.string   "image"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end

For my application  I want every photo object to have a title, caption, and date (which could be different from the created_at / updated_at timestamps). Then I will actually store the photo under the attribute "image" which is of datatype string.

So now we'll follow the CarrierWave documentation and run:

rails generate uploader Image

"Image" corresponds to the image attribute on our photo model. Running this command will create an "app/uploaders" directory with a "image_uploader.rb" file. Here are the contents of my image_uploader.rb file:

# app/uploaders/image_uploader.rb
class ImageUploader < CarrierWave::Uploader::Base

  include CarrierWave::MiniMagick
  storage :fog
  def store_dir
  end
  version :thumb do
    process resize_to_fit: [300, 300]
  end
  def extension_whitelist
    %w(jpg jpeg gif png)
  end
  def filename
    "#{model.date}-#{model.title}-#{Time.now.getutc.to_s}"
  end
end

When the file generates, there will be lots of commented out documentation to help you customize the CarrierWave configuration. Let's go line by line to understand what is going on.

First I'm including MiniMagick to handle image processing because in addition to the original version of the image, I also want to have a thumbnail of the image created upon upload.

Next I am telling CarrierWave that I want to use ":fog" as my storage. CarrierWave plays nicely with Fog and a great number of cloud storage providers.

Next I describe the thumbnail version of the photo that I want to create which is sized at 300 x 300.

Next I list the file types I would like CarrierWave to handle. So with this configuration CarrierWave will reject non-image file types.

Last I describe the file naming convention I would like to apply. In this case it is a concatenation of the image's date, title, and then the current time.

Add Uploader to the Model

Next we have to add the uploader we just created to our Photo model. To do so we'll just add the following code.

# app/models/photo.rb
class Photo < ApplicationRecord
  mount_uploader :image, ImageUploader

  validates_presence_of :title
  validates_presence_of :caption
  validates_presence_of :date
end

The validations are clearly application specific, but there is one notable point to make here. Notice how I validate for presence of all of my photo attributes except :image. Calling photo.image will actually return nil, so the validation test with shoulda_matchers will fail. To determine if an image is attached to the photo, the CarrierWave documentation suggests calling photo.image.file.nil?

Routes

Let's add the following line to our config/routes.rb file.

resources :photos, only: [:new, :create, :show]

Photo Controller

Moving on to our app/controller/photos_controller.rb we see there is not anything special we need to do, CarrierWave and Fog make the magic happen.

# app/controller/photos_controller.rb
class PhotosController < ApplicationController
  def new
    @photo = Photo.new
  end

  def create
    @photo = Photo.new(photo_params)
    if @photo.save
      redirect_to photo_path(@photo)
    end
  end

  def show
  end

  private

  def photo_params
    params.require(:photo).permit(:title, :caption, :date, :image)
  end
end

Add Views

First lets add an app/view/photos/new.html.erb view to hold our upload form.

# app/views/photos/new.html.erb
<div class="container">
  <h3>New Photo of the Day</h3>
  <div class="row">
    <%= form_for @photo, multiple: true do |f| %>

      <div class="input-field">
        <%= f.label :title %>
        <%= f.text_field :title %>
      </div>

      <div class="input-field">
        <%= f.label :caption %>
        <%= f.text_area :caption, class: "materialize-textarea" %>
      </div>

      <%= f.label :date %>
      <%= f.date_field :date, class: "datepicker"%>

      <div class="file-field input-field">
        <div class="btn btn-flat waves-effect waves-light cyan accent-2">
          <span>File</span>
          <%= f.file_field :image %>
        </div>
        <div class="file-path-wrapper">
          <input class="file-path validate" type="text">
        </div>
      </div>

      <div class="input-field center-align">
        <%= f.submit "Upload", class: "btn waves-effect waves-light cyan darken-3"  %>
      </div>

    <% end %>
  </div>
</div>

And we'll just add app/view/photos/show.html.erb view, which for now we'll leave blank because our test does not specifically look for any content on this page.

Run the test

Next I comment out the Fog.mock!, run the test, and check my Cloud Storage for the file. I do this to make sure that the test runs and actually uploads an image into Cloud Storage.

require 'rails_helper'

RSpec.feature "admin uploads photo" do
  context "completes the photos#new form" do
    it "redirects to the new photo's show page" do
      # Fog.mock!
      # Fog::Mock.delay = 0
      # service = Fog::Storage.new({
      #   provider: 'Google',
      #   google_storage_access_key_id: ENV['google_storage_access_key_id'],
      #   google_storage_secret_access_key: ENV['google_storage_secret_access_key']
      # })
      # service.directories.create(:key => 'photo-of-the-day')
      admin = create :admin
      sign_in admin
      visit new_photo_path

      fill_in "Title", with: "My photo title"
      fill_in "Caption", with: "My photo caption"
      fill_in "Date", with: "2015-12-29"
      attach_file "photo[image]", Rails.root + "spec/fixtures/dummy.png"
      click_button "Upload"

      expect(current_path).to eq(photo_path(Photo.last))
    end
  end
end

Notice that the run time below is 1.61 seconds without mocking the actual file upload.

And here are the two files, the uploaded original copy and the processed thumbnail in my cloud storage.

Then uncommenting out the Fog.mock! lines and running the test gives us:

Notice that the run this time took only .66 seconds, a full second faster when we mock the file upload. Also note, if you look at your cloud storage after running the test with Fog.mock! you will not see any new files.

If you made it this far, thanks for sticking with me. CarrierWave, Fog and Google Cloud Storage is a great solution for photo uploading needs and I hope that this post elucidates the process slightly for newcomers to the challenge.