Isnor Creative

Searching Amazon S3 files using the aws cli

2020-03-30T03:00:00Z

I recently had a need to search Amazon S3 files and was very excited to discover how powerful the aws cli tool was when combined with the jmes query language

The first step was to install the aws cli tool. There is a graphic installer available for macOS, but I installed via the command line:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

I checked to make sure it had worked, and then removed the installer:

aws --version
rm AWSCLIV2.pkg

The second step was configuration – just a matter of issuing a command and then providing response to configuration questions (key, secret, region, etc).

aws configure

Once that was done, it was very easy to query files. Here are a few basic examples:

Search S3 files by owner

aws s3api list-objects --bucket your-bucket --query "Contents[?Owner.DisplayName == 'someone']"

Search S3 files not by owner

aws s3api list-objects --bucket your-bucket --query "Contents[?Owner.DisplayName != 'someoneelse']"

Search S3 files by key

aws s3api list-objects --bucket your-bucket --query "Contents[?contains(Key, 'path/to/your/file')]"

Search S3 files by date

aws s3api list-objects --bucket your-bucket --query "Contents[?LastModified > '2020-03-26']"

Keep in mind that, as far as I know, the searching is happening on your end, so the requests can be a bit slow, but still a heck of lot faster and more effective than manually combing through files in Transmit which is what I wondered if I might have to do before checking to see what other options were available!

The cli has options that allow for various ouput formats, coloured output, and so on.

Happy searching!

Integrating Zoho API v2 in a Ruby on Rails application

2019-06-21T03:00:00Z

I had previously integrated a client’s web application with the Zoho API and learned that they were planning to sunset the version 1 of the API at the end of 2019. So I set about figuring out a way to integrate V2 with our application. v2 of the Zoho API now uses oAuth, and is substantially different from v1. I took a look to see if there was a gem that would serve us as well as the rubyzoho gem had for v1 but didn’t find anything that looked easy to use or comprehensive. You could take a look at the zohohub gem, it looked promising.

Step 1: Register a client

The first step was to register a client as explainined in the Zoho docs. I then took note of the key and secret and stored those in environment variables. I use the dotenv gem to manage ENV variables in the development environment.

Step 2: Self-client

After that I used the self-client of generating a grant token request as explained in the Zoho API docs…. The scopes field could look something like this:

ZohoCRM.modules.custom.all,ZohoCRM.settings.all,ZohoCRM.modules.contacts.all,ZohoCRM.modules.all,ZohoCRM.users.all

Step 3: Getting access and refresh tokens

I then built myself a little utility class to get access and refresh tokens, and I plugged in the grant token that I had got from the self-client option above:

# Can be used to get access and refresh tokens when provided with a grant token from the self-client option
# Usage: Zoho::UseGrantTokenUtil.call(grant_token_goes_here)
class Zoho::UseGrantTokenUtil
  include HTTParty
  # debug_output - uncomment to get verbose output from HTTParty

  # @param code [String] get a code from self -client util of Zoho CRM and then use it here 
  def self.call(code)
    options = {
      body: {
        grant_type: "authorization_code",
        client_id: ENV['ZOHO_CLIENT_ID'],
        client_secret: ENV['ZOHO_SECRET'],
        code: code,
        redirect_uri: "http://localhost:3000"
      }
    }
    self.post('https://accounts.zoho.com/oauth/v2/token', options)
  end

end

I used this utility class to generate a refresh token which I then also stored in an ENV variable

Step 4: Getting access tokens using the refresh token

I decided to store access tokens in Redis along with an indication of when they expired. If I don’t have a currently active access token stored in Redis, we get a fresh one from Zoho using the refresh token that is stored in our ENV variable, and then store fresh one in Redis.

# This is used to get access tokens out of Redis, 
# or renew them using the refresh token and store them to Redis
#
# Usage: 
# token = Zoho::Tokens.get
#
class Zoho::Tokens
  include HTTParty
  # debug_output
  require "redis"

  def self.get
    redis = Redis.new
    token = redis.get("zoho:access_token")
    if token && JSON.parse(token)["expires"].to_i >= Time.zone.now.to_i
      JSON.parse(token)["token"]
    else
      Zoho::Tokens.refresh_token
    end
  end

  def self.refresh_token
    redis = Redis.new
    options = {
      body: {
        grant_type: "refresh_token",
        client_id: ENV['ZOHO_CLIENT_ID'],
        client_secret: ENV['ZOHO_SECRET'],
        refresh_token: ENV['ZOHO_REFRESH_TOKEN']
      }
    }
    result = self.post('https://accounts.zoho.com/oauth/v2/token', options)
    if result.code == 200 
      token = result.parsed_response["access_token"]
      expires_in_sec = result.parsed_response["expires_in_sec"]
      token_json = {
        token: token, 
        expires: (Time.zone.now + expires_in_sec).to_i 
      }.to_json
      redis.set("zoho:access_token", token_json)
      return token
    else
      return false
    end
  end
end

Step 5: Base class for interacting with Zoho modules

Once I had a way of getting an access token easily set up, I built a base class that I could inherit from for most of what I needed to do with Zoho, and I can then subclass it for various modules. Some require more customization than others – users for example is inexplicably different than other modules and Zoho users may have custom modules and fields… This base class works much like ActiveRecord, I can instantiate a new object, passing in the required attrs, I can then save it and update it. I can create directly, can find based on an object’s id, can list all items for a module, or can do dynamic findby and findall_by method to look up a certain attribute.

# Any class that inherits from Zoho base
# should have a module name that matches the module in Zoho
# eg Zoho::Contacts
# and should define attributes that it wants to work with eg attributes :email, :name, etc...
# 
# Classes that inherit from this get: 
# find_by_{field} class method eg Zoho::Contacts.find_by_email "foo@bar.com"
# find_all_by_{field} class method eg Zoho::Contacts.find_by_email "foo@bar.com"
# find(id) class method eg Zoho::Contacts.find abc123
# all class method eg Zoho::Contacts.all
# save instance method
# update instance method
# create instance method
# 
# Usage: 
# z = Zoho::{module_name}.new(attrs hash goes here eg email: "foo@bar.com)
# z.save
#
class Zoho::Base
  include HTTParty
  include ActiveModel::AttributeAssignment
  # debug_output

  attr_accessor :id

  def initialize(h = {})
    h.each {|k,v| public_send("#{k}=",v) }
  end

  # @return [String] The base Zoho API url
  def self.zoho_api_url
    "https://www.zohoapis.com/crm/v2"
  end

  # @return [String] The Zoho API url for this module eg Contacts
  def self.module_url
    @module_url ||= File.join(self.zoho_api_url, self.module_name)
  end

  # Subclasses that require special processing for some attrs eg relationships
  # @return [Array] Subclasses will use this to specify an array of attributes that require special handling
  def self.special_attributes
    []
  end

  def self.attributes(*attributes)
    @attributes ||= []
    return @attributes - special_attributes unless attributes
    a = attributes - special_attributes
    attr_accessor(*a)
    @attributes += a
  end

  def attributes
    self.class.attributes - self.class.special_attributes
  end

  # Httparty options
  # @return [Hash] a hash of options for Httparty request
  def self.options
    {headers: {"Authorization": "Zoho-oauthtoken #{Zoho::Tokens.get}"}}
  end

  # Making it easier to add Zoho search criteria to Httparty options
  # @param q [String] A Zoho search criteria string
  # @return [Hash] Httparty options with query params added
  def self.query(q)
    options.merge(query: {criteria: q})
  end

  # This is getting the Zoho module name from our class eg Zoho::Contacts become Contacts
  # @return [String] The Zoho module name
  def self.module_name
    name.split('::')[1]
  end

  # @return [Array] Array of all Zoho items for this class
  def self.all(**args)
    url = module_url
    # allowing extra query params to be added
    if args.present?
      url << "?#{args.first[0]}=#{args.first[1]}"
      args.shift
      args.each do |arg|
        url << "&#{args.first[0]}=#{args.first[1]}"
      end
    end
    result = get(url, options)
    if result.code == 200 
      result_to_objects_array(result)
    else
      raise "Error getting data from Zoho: #{result.message}"
    end
  end

  # Finds a single item from Zoho
  # @param criteria [String] Valid Zoho search criteria
  # @return [Zoho::{module}] Instance of the object
  def self.find(id)
    url = File.join(module_url, id)
    result = get(url, options)
    if result.code == 200
      hash_to_object result.parsed_response["data"].first
    else
      raise "Error getting data from Zoho: #{result.message}"
    end
  end

  # @param criteria [String] Valid Zoho search criteria
  # @return [Hash] Non-symbolized hash of data
  def self.search(criteria)
    result = get(
      File.join(module_url, "search"), 
      query(criteria)
    )

    if result.code == 200
      result_to_objects_array(result)
    else
      return false
    end
  end

  # This is taking HTTParty result data array and for each item in the array it creates a new object eg a Zoho::Contact object
  # @return [Array] An array of objects
  def self.result_to_objects_array(result)
    data = result.parsed_response["data"]
    data.inject([]) { |arr, item| arr << hash_to_object(item) }
  end

  # Using method missing here to define dynamic find by methods such as find_by_email
  def self.method_missing(m, *args, &block)
    if m.to_s.match("find_all_by")
      field = m.to_s.split("find_all_by_")[1].split('_').map(&:titlecase).join('_')
      a = args[0]
      result = search("(#{field}:equals:#{a})")
      return result
    elsif m.to_s.match("find_by")
      field = m.to_s.split("find_by_")[1].split('_').map(&:titlecase).join('_')
      a = args[0]
      result = search("(#{field}:equals:#{a})")
      return result.first if result
    else 
      raise "NO MATCH ON FINDER METHOD #{m}"
    end
  end

  # this takes httparty hash and returns an instance object of this class with attrs assigned
  # @param hash_item [Hash] Object in hash form
  # @return [Zoho::Base] instance of the object
  def self.hash_to_object(hash_item)
    h = attributes.inject({}) { |attrs, a| 
      c = a.to_sym == :id ? "id" : converted_attribute(a) 
      val = hash_item[c] 
      attrs[a] = val
      attrs
    }
    new(h) 
  end

  # This is taking a symbol attr such as :account_name
  # and converting it to a string that Zoho understands eg Account_Name
  # @param a [Symbol] A symbolized attribute name
  # @return [String] attribute name in Zoho form
  def self.converted_attribute(a)
    a.to_s.split('_').map(&:titlecase).join('_')
  end

  # Subclasses that require special processing will overwrite this
  # @return [Hash] Subclasses will use this to modify the params for Zoho
  def params_plus_special_processing(data)
    return data
  end

  # this build JSON data for Zoho to ingest
  # @return [String] A string of JSON data for Zoho
  def params_for_zoho
    params = attributes.inject({}) { |hash, attribute|
      c = Zoho::Base.converted_attribute(attribute)
      val = self.public_send(attribute)
      hash[c] = val if val.present?
      hash
    }
    params = params_plus_special_processing(params)
    data = {data: [params]}
    data.to_json
  end

  # Saves a record to Zoho
  # @return [Zoho::{module}] returns the object being saved
  def save
    result = self.class.post(self.class.module_url, 
      self.class.options.merge(body: params_for_zoho)
    )
    if [200,201].include? result.code
      created_item_id = result.parsed_response['data'].first['details']['id']
      self.id =  created_item_id
      return self
    else
      raise "#{result.code} Error doing save to Zoho: #{result.message}"
    end
  end

  # Creates a record in Zoho when passes in attrs for a new object
  # @param attrs [Hash] Hash of params to create the object
  # @return [Zoho::{module}] returns the object being saved
  def self.create(attrs)
    new(attrs).save
  end

  # updates the current record in Zoho
  # @return [Zoho::{module}] returns the object being saved
  def update
    raise "ID REQUIRED" and return unless id.present?
    result = self.class.put(
      File.join(self.class.module_url, id), 
      self.class.options.merge(body: params_for_zoho)
    )
    if [200, 201].include? result.code
      return self
    else
      raise "#{result.code} Error getting contacts from Zoho: #{result.message}"
    end
  end

end

Step 6: Inheriting from the base class for individual modules

I can then subclass it like this, for example:

# Usage
#
# Get all accounts: 
# accounts = Zoho::Acounts.all
#
# Find one account 
# account = Zoho::Accounts.find_by_account_name({account_name})
#
class Zoho::Accounts < Zoho::Base

  # any Zoho fields you want assigned to found objects and sent to Zoho should be specified here
  attributes :id, :account_name, :account_type

end

I have a good deal of experience integrating various APIs – including Zoho – into Ruby on Rails and Elixir/Phoenix applications. I would be happy to talk if you need something like this for your project. Get in touch!

Bailar! salsa and latin dancing website

2019-03-05T04:00:00Z

I’ve released a new salsa and latin dancing website! The main feature to start is a weekly-ish newsletter, highlighting a new move that has been added to the site. We’ll aggregate the best videos available demonstrating the move.

The site was built with Elixir, Phoenix and Vue.js. It was a joy working with Elixir and Phoenix again, and seeing how Webpack integrates with Vue and SASS. I am finding that the more I work with Vue the more I like it. Love how easy it is to integrate into server-rendered web applications. It would have been nice if Ember hadn’t abandoned this approach so completely back around the introduction of ember-cli, but what can you do? Onwards and upwards!

Using SASS with Phoenix 1.4 and Webpack

2019-02-14T04:00:00Z

I recently created a new Elixir 1.8/Phoenix 1.4 application. One of the first things I do when creating a new application is ensuring I have SASS set up and ready to use. I was surprised at how easy it was to set up SASS with Phoenix 1.4, which now uses Webpack instead of Brunch. Here’s a quick walkthrough that worked for me.

1. First, push node_modules a few steps closer to infinite:

yarn add sass-loader node-sass style-loader —-dev

2. Rename app.css to app.scss

‘Nuff said…

3. assets/webpack.config.js - append to modules => rules:

{
  test: /\.scss$/,
  use: [
    process.env.NODE_ENV !== 'production' ? 'style-loader' : MiniCssExtractPlugin.loader,
    "css-loader",
    "sass-loader"
  ]
}

4. assets/js/app.js:

- import css from "../css/app.css"
+ import css from "../css/app.scss"

Bingo bango! You should be very sassy now.

Developing an Ember application with local storage

2019-02-14T04:00:00Z

I recently released a tea timer progressive web application built with the Ember Octane blueprint circa Feb 2019 (prerelease). The application stores a list of teas in the user’s local storage, and allows people to add, remove, edit or restore from a preset list of teas.

I started out building out the local storage using vanilla JS. It was remarkably easy to do, and worked surprisingly well. At a certain point though, I decided I wanted to add a few validations as it made no sense to have teas without a name or a brewing time. At this point I wanted the convenience of a familiar validation library such as ember-changeset-validations and I decided to switch my own implementation of the local storage management to Ember Data and the excellent ember-local-storage addon which had served me well in the past.

Here’s a walkthrough of both ways of doing this…

Version 1: No ember data, local storage implemented with vanilla JS

Setup

I started with a preset list of teas that was an array of POJOs.

let Teas = [
  {id: 1, name: "Green", ...etc   },
  ...etc
];
export default Teas;

Application route

The application route here checks to see if data exists in local storage, if it doesn’t it stringifies the JSON seed data and saves it to local storage, if it does, it parses the the locally stored string to JSON. I use the application route’s model throughout the application.

import Route from '@ember/routing/route';
import Teas from 'tea-time/src/teas';

export default class ApplicationRoute extends Route {
  model() {
    let stored =  window.localStorage.getItem('teas');
      if (stored !== null) {
      return JSON.parse(stored);
    } else {
      window.localStorage.setItem('teas', JSON.stringify(Teas));
      return Teas;
    }
  }
}

Record detail route

This route loads a single item. Here I am finding the item from my already loaded application route model. I am adding parseInt on the ID param.

import Route from '@ember/routing/route';

export default class TeaRoute extends Route {
  model(params) {
    let teas = this.modelFor('application');
    return teas.findBy('id', parseInt(params.id));
  }
}

Update an item route

I’m not using a controller here. The form’s save action is hitting the route, which hooks into the model for the application route. It removes the item we are modifying, adds it back in modified form, and then stringifies the model and saves it back to local storage.

The delete action is doing a similar thing to save – removing the current item, not then not adding anything, just persisting the shortened list to local storage.

import Route from '@ember/routing/route';
import { action } from '@ember-decorators/object';

export default class EditTeaRoute extends Route {

  model(params) {
    let teas = this.modelFor('application');
    return teas.findBy('id', parseInt(params.id));
  }

  @action
  save(model) {
    let teas = this.modelFor('application')
    teas.rejectBy('id', model.id);
    teas.pushObject(model);
    window.localStorage.setItem('teas', JSON.stringify(teas));
    this.transitionTo('index');
  }

  @action 
  delete(tea) {
    let teas = this.modelFor('application')
    teas.rejectBy('id', model.id);
    window.localStorage.setItem('teas', JSON.stringify(teas));
    this.transitionTo('settings')
  }

}

Add an item route

Our model here is just returning an empty object. The save action is pushing that object onto the application route’s model, and then saving the stringified model back to local storage.

import Route from '@ember/routing/route';
import { action } from '@ember-decorators/object';

export default class AddTeaRoute extends Route {

  model() {
    return {}
  }

  @action
  save(model) {
    let teas = this.modelFor('application')
    teas.pushObject(model);
    window.localStorage.setItem('teas', JSON.stringify(teas));
    this.transitionTo('index');
  }

}

Restore to defaults route

This route allows the user to wipe any changes they have made, and restore to “Factory Defaults”. It is similar to the original application route in storing the original presets to local storage, but then it also refreshes the application route so that the changes are instantly visible.

import Route from '@ember/routing/route';
import { action } from '@ember-decorators/object';
import Teas from 'tea-time/src/teas';
import { getOwner } from '@ember/application';

export default class SettingsRoute extends Route {

  @action 
  restoreDefaults() {
    window.localStorage.setItem('teas', JSON.stringify(Teas));
    getOwner(this).lookup('route:application').refresh();
  }

}

Version 2: Bringing in Ember data, the ember-local-storage addon, and ember-changeset-validations

There were a few setup details required here that were not necessary in the version without Ember Data and ember-local storage. I had to add a model, adapter, serializer, and the storage ember-local-storage storage object.

I removed the id property from my preset list of teas as it was not necessary and caused problems with Ember Data…

src/teas.js

let Teas = [
  {name: "Green", ...etc   },
  ...etc
];
export default Teas;

storages/teas.js

This is a necessary setup step for using ember-local-storage

import StorageObject from 'ember-local-storage/local/object';
const Storage = StorageObject.extend();
export default Storage;

data/models/adapter.js

Necessary for ember-local-storage to work with Ember Data

export { default } from 'ember-local-storage/adapters/local';

data/models/serializer.js

Necessary for ember-local-storage to work with Ember Data

export { default } from 'ember-local-storage/serializers/serializer';

data/models/tea.js

import DS from 'ember-data';
const { Model, attr } = DS;
export default Model.extend({
  name: attr(),
  ...etc
});

Application route

Similar to the version 1 route above, the application route here checks to see if data exists in local storage. If it doesn’t it seeds it from JSON.

import Route from '@ember/routing/route';
import { storageFor } from 'ember-local-storage';
import Teas from 'tea-time/src/teas';

export default class ApplicationRoute extends Route {
  teas = storageFor('teas');

  model() {
    return this.store.findAll('tea').then(teas => {
      if (teas.length === 0) {
        Teas.map(tea => {
          let newTea = this.store.createRecord('tea', tea);
          newTea.save();
        });
        return this.store.findAll('tea');
      } else {
        return teas;
      }
    })
  }

}

Adding, editing or deleting items is typical Ember Data:

One thing I had to do when I started using Ember Data is ensuring that unsaved records were destroyed to prevent them from hanging around…. So I added this willTransition to my create item route…

import Route from '@ember/routing/route';
import { action } from '@ember-decorators/object';

export default class AddTeaRoute extends Route {

  model() {
    return {}
  }
  @action
  save(model) {
    model.save().then(()=> this.transitionToRoute('index'));    
  }
  @action
  willTransition() {
    if (this.get('controller.model.isNew')) {
      this.get('controller.model').deleteRecord();
    } else {
      return true;
    }
  }
}

Restoring to factory defaults

I had an option to restore local storage from the “factory default”, with Ember Data it looks like this:

import Route from '@ember/routing/route';
import { action } from '@ember-decorators/object';
import Teas from 'tea-time/src/teas';

export default class SettingsRoute extends Route {
  @action 
  restoreDefaults() {
    this.modelFor('application').map(tea=> tea.destroyRecord());
    Teas.map(tea => {
      let newTea = this.store.createRecord('tea', tea);
      newTea.save();
    });
  }
}

It had occurred to me when I started to feel a need to add validations that I might look into using a non-Ember javascript object validation library – validate.js looked promising – but I was already very familiar with ember-changeset-validations and decided to go that route instead.

The application can be found at Perfect Cup Tea Timer.

Any questions or comments welcome.

I am available for Ember.js consulting, as well as Ruby on Rails and Elixir/Phoenix + Elm, React, and Vue.js.

Perfect Cup Tea timer

2019-02-11T04:00:00Z

My local tea shop always serves teas with a timer, and I appreciate that it keeps my teas – especially green or black teas from getting nasty as I zone out with web development. I decided to make a tea timer app to get the same benefits at home.

Perfect Cup Tea Timer is the result. It’s a progressive web application built with Ember.js, and I used the prerelease Octane blueprint, which was a pleasure to work with – installing packages was a cinch, sparkles components were great, and the native javascript classes and decorates were nice too. Module unification had some issues with generators putting things in strange places, but I was able to sort it out with a bit of reading scouring through Discord and moving things around. I also found it very odd that components within a route seemingly could not be tested in isolation.

At any rate, it’s released and it will allow you to add, delete or edit teas – they’re stored in local storage, and then start a timer when you’re teas is ready to brew. Pretty basic!

Save it to your home screen to get the full “app” experience. Works great on Android, iOS still hasn’t really figured out PWAs as nicely (no splash screen on iOS for example) but it basically works. Other weird thing with iOS is that the bell that goes off when your tea is ready will only play once initially - and in my experience it seemed to randomly start up again after quite some time. This has partially to do with iOS blocking video and audio that are not triggered by a user action – in this case it is triggered by a user action, but because the bell is ringing over and over, it’s getting blocked as if it was not user-initiated. Frustrating, but that’s Apple for you, these days.

If you like tea, give it a try!

Perfect Cup Tea Timer

I am available for Ember.js consulting work – get in touch to learn more.