Let's make a Hot Dog App: A Google AutoML Tutorial for N00bs

A Google AutoML Tutorial for N00bs

By the end of this tutorial you will be able to submit pictures to a trained machine learning algorithm and get the image's classification as a response. In other words, we are making Jian Yang's Not Hot Dog app.

With Google’s new AutoML beta, making a hot dog classifier app is actually possible even if you do not know anything about machine learning.

This tutorial is divided into two parts. First we will navigate Google’s Cloud Console and AutoML UI to create a model, upload training data, and train it. This requires no coding at all.

The second part of this tutorial is about using Ruby to create a POST request that sends a picture to our newly created AutoML model’s predict endpoint and returns our model’s prediction. If all goes well, we’ll be able to send a picture of a hot dog to our model and get a confident hot_dog classification. When we send other pictures, say of trucks or pizza, we should get a confident not_hot_dog classification.

During part two, I try to demonstrate how I use test driven development to build out api requests. I know a segment of this blogs readership may appreciate having some tests to go along with this code.

You can find all the code written for part 2 in this Github repository.

Part 1: Creating an AutoML Model

Step 1: Visit your Google Cloud Console (gmail / Google Apps account required). If this is your first Google Cloud Console rodeo, you will likely have to agree to Google’s terms of service.

Step 2: Click the “Select Project” dropdown at the top left corner of the screen and select “New Project.”

Screen Shot 2018-12-23 at 11.54.10 AM.png

Step 3: Name your project whatever you want! I am naming mine “Hot Dog App.” Next click create and wait as Google creates your project.

Screen Shot 2018-12-23 at 11.57.36 AM.png

Step 4: Select your app from the same “Select Project” drop down from step 2. In the “Getting Started” panel (bottom left) click “Explore and enable APIs.” Click the “ENABLE APIS AND SERVICES” at the top of the Dashboard. Type “automl” into the search bar and click on the “Cloud AutoML API” card that shows up. Click the blue “ENABLE” button. You will likely have to enable billing at this point.

Step 5: Next we are going to need a JSON file that contains the credentials we need to make requests to the AutoML endpoint. Click “CREATE CREDENTIALS”. Choose “Cloud AutoML API” from the “Which API are you using?” dropdown. Choose “No, I’m not using them” from the “Are you planning to use this API with App Engine or Compute Engine?” radio buttons. Click “What Credentials Do I need?”

Step 6: Enter a service account name, I’m using “T1000,” but I think you can put whatever you want in there. For Role select “AutoML Predictor.” Make sure JSON is the selected “Key type” because lord only knows what P12 is. Click the “Create” button and you will be prompted to download a JSON file. Keep this secret. Keep it safe.

{
  "type": "service_account",
  "project_id": "hot-dog-app-226418",
  "private_key_id": "private_key",
  "private_key": "-----BEGIN PRIVATE KEY-----PRIVATE KEY----END PRIVATE KEY-----\n",
  "client_email": "t1000@example.com",
  "client_id": "12345",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/............"
}

Step 7: Visit the Auto ML UI to create the dataset. You will have to grant Auto ML access to your Google Account, so click “Allow.” Select your project from the drop down. Click “Set up now.” This may take a few moments, so go ahead and Google at least 100 pictures of hot dogs and 100 pictures of not hot dogs. The quick start guide actually says all you need is 10 pictures, but I was not able to train my model with 10 pictures per label. Plus, we want our hot dog app to be super duper accurate.

Step 8: Click “NEW DATASET” and give type “hot_dog” for the label. Click “Select Files” and upload at least 100 pictures of hot dogs. Click “CREATE DATASET.”

Step 9: Click “Add label” on the left. Create a label “hot_dog.” Click “Select all images” and choose the newly created “hot_dog” label from the label drop down.

Step 10: Click “ADD IMAGES”. Select your non hot dog images from your computer. Add a new label (see step 9) called “not_hot_dog” and add your new images to this label.

You should now have a data set with at least 200 images, half labeled as examples of hot dogs and the other half labeled as examples of not hot dogs.

Screen Shot 2018-12-23 at 4.25.24 PM.png

Step 11: Click “Train” at the top left of your screen. If you have enough labeled pictures, you should then be able to click “Start Training.” Give your model a name (the default is fine) and then click “Start Training.” This process can take “15 minutes to several hours” depending on how much computer power you are paying for. I went with the free tier and training took around 10 minutes with roughly 200 pictures.

Step 12: Once the model has been trained, we can test it manually. Find a new picture of either a hot dog or non hot dog. Click on the model you just trained. Click on the predict tab. Upload an image and viola. My model had no problem identifying a hot dog as a hot dog, a truck as not a hot dog, but it was only 53.6% sure the picture of french fries was not a hot dog.

Screen Shot 2018-12-23 at 8.06.23 PM.png
Screen Shot 2018-12-23 at 8.07.10 PM.png
Screen Shot 2018-12-23 at 8.06.11 PM.png


Part 2: Using Ruby to maker REST requests to your AutoML model

Step 1: Create a project folder and files. In terminal type:

 ~/codes  mkdir hot_dog_app
 ~/codes  mkdir hot_dog_app/lib
 ~/codes  mkdir hot_dog_app/spec
 ~/codes  touch hot_dog_app/lib/client.rb
 ~/codes  touch hot_dog_app/spec/client_spec.rb
 ~/codes  cd hot_dog_app

Step 2: Our request is going to need a header and a body. Let’s start by writing a test for the header. Let’s describe what the header should look like and then assert when we call client.headers we get them. Here google_authentication_token is just a place holder because we’ll stub this value in our test. This token will change on some unknown interval, so if this is not an app with millions of users, you likely do not have to worry and you can just request a new token before making each AutoML request.

RSpec.describe Client do
  subject { described_class.new }

  describe "headers" do
    let(:headers) do
      {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer google_authentication_token'
      }
    end


    it 'returns the request headers' do
      expect(subject.headers).to eq headers
    end
  end
end

You can run this test from the project root with rspec spec/client_spec.rb

Step 3: This test fails, so let’s get it passing! First we need to require our client file in our test:

require './lib/client.rb'

RSpec.describe Client do
  subject { described_class.new }

Next let’s actually create a client class that can pass this test:

class Client
  def headers
    {
      'Content-Type': 'application/json',
      'Authorization': "Bearer #{google_authentication_token}"
    }
  end
end

Let’s write a method to get our authentication token. We will need to use the Google::Auth::ServiceAccountCredentials which can be included in our client file from this gem.

require 'googleauth'

class Client
  SCOPE = 'https://www.googleapis.com/auth/cloud-platform'

  def headers
    {
      'Content-Type': 'application/json',
      'Authorization': "Bearer #{google_authentication_token}"
    }
  end

  def google_authentication_token
    authorizer = Google::Auth::ServiceAccountCredentials.make_creds(
      json_key_io: File.open('./secret_authorization.json'),
      scope: SCOPE
    )

    authorizer.fetch_access_token!['access_token']
  end
end

We need to move our secret json file that we got in step 5 of part 1 into our project. I am just putting it in the root of the project and calling it secret_authorization.json. You will likely want to encrypt this file if you are going to be putting this on Github. I recommend Figaro for Rails.

The make_creds method reads in this JSON and gives us an authorizer which we can then call fetch_access_token! on. This will return a hash with a key access_token with the value we will need for our header. It should look something like:

ya29.c.ABCDEFG123456…

Step 4: Let’s update our test to stub this authentication token method, since under the hood it is making an api call.

require './lib/client.rb'

RSpec.describe Client do
  subject { described_class.new }
  # mock an authorizer that we can call fetch_access_token! on
  let(:authorizer) { double }

  describe '#headers' do
    let(:headers) do
      {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer google_authentication_token'
      }
    end

    before do
      # stub #make_creds method and have it return our stub authorizer
      allow(Google::Auth::ServiceAccountCredentials).to receive(:make_creds) { authorizer }
      # stub #fetch_access_token! and have it return a hash with the correct key and
      # a fake value
      allow(authorizer).to receive(:fetch_access_token!) {
        { 'access_token' => 'google_authentication_token' }
      }
    end

    it 'returns the request headers' do
      expect(subject.headers).to eq headers
    end
  end
end

And we can update our implementation with:

require 'googleauth'

class Client
  SCOPE = 'https://www.googleapis.com/auth/cloud-platform'.freeze
  AUTH_FILE_PATH = './secret_authorization.json'.freeze

  def headers
    {
      'Content-Type': 'application/json',
      'Authorization': "Bearer #{google_authentication_token}"
    }
  end

  def google_authentication_token
    authorizer = Google::Auth::ServiceAccountCredentials.make_creds(
      json_key_io: File.open(AUTH_FILE_PATH),
      scope: SCOPE
    )

    authorizer.fetch_access_token!['access_token']
  end
end

Step 5: Now let’s set up a test describing what we expect the body of our request to look like. Google requires that we take the image we are submitting to the predict API, read it as a string, and then Base64 encode it.

In this test, notice that I’ve updated how we initialize method to now take in a file path. I wan our client object to get instantiated with an image file to send to AutoML.

require './lib/client.rb'

RSpec.describe Client do
  subject { described_class.new('./spec/fixtures/hot_dog.jpg') }
  # mock an authorizer that we can call fetch_access_token! on
  let(:authorizer) { double }

  describe '#headers' do
    let(:headers) do
      {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer google_authentication_token'
      }
    end

    before do
      # stub #make_creds method and have it return our stub authorizer
      allow(Google::Auth::ServiceAccountCredentials).to receive(:make_creds) { authorizer }
      # stub #fetch_access_token! and have it return a hash with the correct key and
      # a fake value
      allow(authorizer).to receive(:fetch_access_token!) {
        { 'access_token' => 'google_authentication_token' }
      }
    end

    it 'returns the request headers' do
      expect(subject.headers).to eq headers
    end
  end

  describe '#body' do
    let(:body) do
      {
        payload: {
          image: {
            imageBytes: 'base64_encoded_image'
          }
        }
      }.to_json
    end

    before do
      allow(Base64).to receive(:strict_encode64) { 'base64_encoded_image' }
    end

    it 'returns the request body' do
      expect(subject.body).to eq body
    end
  end
end

Step 6: Now let’s implement.

require 'googleauth'

class Client
  SCOPE = 'https://www.googleapis.com/auth/cloud-platform'.freeze
  AUTH_FILE_PATH = './secret_authorization.json'.freeze

  attr_reader :image_path

  def initialize(image_path)
    @image_path = image_path
  end

  def headers
    {
      'Content-Type': 'application/json',
      'Authorization': "Bearer #{google_authentication_token}"
    }
  end

  def google_authentication_token
    authorizer = Google::Auth::ServiceAccountCredentials.make_creds(
      json_key_io: File.open(AUTH_FILE_PATH),
      scope: SCOPE
    )

    authorizer.fetch_access_token!['access_token']
  end

  def body
    {
      payload: {
        image: {
          imageBytes: Base64.strict_encode64(open(image_path).read)
        }
      }
    }.to_json
  end
end

Step 7: Now let’s write our last test describing the making of our post request to the AutoML predict API. I am familiar with HTTPClient so that is the gem we will use here.

require './lib/client.rb'

RSpec.describe Client do
  subject { described_class.new('./spec/fixtures/hot_dog.jpg') }
  # mock an authorizer that we can call fetch_access_token! on
  let(:authorizer) { double }

  describe '#headers' do
    let(:headers) do
      {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer google_authentication_token'
      }
    end

    before do
      # stub #make_creds method and have it return our stub authorizer
      allow(Google::Auth::ServiceAccountCredentials).to receive(:make_creds) { authorizer }
      # stub #fetch_access_token! and have it return a hash with the correct key and
      # a fake value
      allow(authorizer).to receive(:fetch_access_token!) {
        { 'access_token' => 'google_authentication_token' }
      }
    end

    it 'returns the request headers' do
      expect(subject.headers).to eq headers
    end
  end

  describe '#body' do
    let(:body) do
      {
        payload: {
          image: {
            imageBytes: 'base64_encoded_image'
          }
        }
      }.to_json
    end

    before do
      allow(Base64).to receive(:strict_encode64) { 'base64_encoded_image' }
    end

    it 'returns the request body' do
      expect(subject.body).to eq body
    end
  end

  describe '#post' do
    let(:response) { double }
    let(:json) { "Hello World".to_json }

    before { allow(response).to receive(:body) { json } }

    it 'sends a post request' do
      expect(HTTPClient).to receive(:post).with(
        described_class::URL,
        body: subject.body,
        header: subject.headers
      ) { response }

      subject.post
    end
  end
end

Step 8: And now let’s implement! I’ve dotted out a part of the url for my test app because I am paranoid. You can find this URL on the predict tab of your AutoML project below where we were making test requests. There should be a sample Curl request with the full URL you need to send your post request to. Please note, the URL ends with “:predict.”

require 'googleauth'
require 'httpclient'

class Client
  SCOPE = 'https://www.googleapis.com/auth/cloud-platform'.freeze
  AUTH_FILE_PATH = './secret_authorization.json'.freeze
  URL = 'https://automl.googleapis.com/v1beta1/projects/hot-dog-app-226418/locations/us-central1/models/ICN1498268009069992673:predict'.freeze

  attr_reader :image_path

  def initialize(image_path)
    @image_path = image_path
  end

  def headers
    {
      'Content-Type': 'application/json',
      'Authorization': "Bearer #{google_authentication_token}"
    }
  end

  def google_authentication_token
    authorizer = Google::Auth::ServiceAccountCredentials.make_creds(
      json_key_io: File.open(AUTH_FILE_PATH),
      scope: SCOPE
    )

    authorizer.fetch_access_token!['access_token']
  end

  def body
    {
      payload: {
        image: {
          imageBytes: Base64.strict_encode64(open(image_path).read)
        }
      }
    }.to_json
  end

  def post
    response = HTTPClient.post(URL, body: body, header: headers)
    MultiJson.load(response.body)
  end
end

Step 9: Open up pry, require the client.rb file and test your hot dog app in console.

 ~/codes/hot_dog_app  pry
[1] pry(main)> require './lib/client.rb'
=> true
[2] pry(main)> client = Client.new('../../Downloads/pizza.jpg')
=> #<Client:0x007fd893e9e3c8 @image_path="../../Downloads/pizza.jpg">
[3] pry(main)> client.post
=> {"payload"=>[{"classification"=>{"score"=>0.7432721}, "displayName"=>"not_hot_dog"}]}
[4] pry(main)> client = Client.new('../../Downloads/hotdogs.jpg')
=> #<Client:0x007fd893e659b0 @image_path="../../Downloads/hotdogs.jpg">
[5] pry(main)> client.post
=> {"payload"=>[{"classification"=>{"score"=>0.9999999}, "displayName"=>"hot_dog"}]}
[6] pry(main)>

From here you can incorporate your client.rb file into a new or existing Rails app or other Ruby project. You might consider storing these classifications and or displaying them to your user.

I hope that you have fun with this exciting new API from our friends at Google. I’m looking forward to see what you create!