API.AI - Conversational User Interface


  • Sign in with google (integration with Google Home/Assistant/Allo?)
  • Create new Agent.
  • Settings - Restore from zip. (use reference/sample zip)
  • Add Fulfillment URL (if needed), and Check/enable "Use webhook", if/where needed.
  • Try it now (type & test)
  • Enable Integration: Assign (pronounceable) Invocation name, add Additional triggering intents, Authorize.
  • Try it on Actions on Google Web-Simulator.

Default / Fallback Intents

Default Welcome Intent: Initiate conversation to start a "Game"
  Input Context=None; Output Context=in_dialog flag.

Fallback Intents must have unique Input Context::

Unrecognized Deep Link Fallback: attempt to continue the conversation
  Trigger=Action called with unrecognized invocation argument;
  Input Context=inside Action/None; Output Context=Redirected to "Game" intent/in_dialog flag.
Default Fallback Intent: attempt to continue the conversation
  Trigger=Unrecognized argument during "Game";
  Input Context="Game"; Output Context="Game".
In Dialog Fallback: attempt to continue the conversation
  Trigger=in_dialog no other intent can be triggered;
  Input Context=in_dialog; Output Context=Redirected to "Game" intent/in_dialog flag.

TODO: Unrecognized Deep Link Fallback - Should the input context be something inside (and, not Welcome)?

Intent, Lifespan, Context

  • Intents can be triggered by Event names instead of User queries. (Welcome events are automatically supported for some one-click integrations).
  • Lifespan for a context is set on the Output Context in Intent. This is useful for retrying/resuming the conversation if user's response is invalid/not understood, by giving user chance(s) to make more attempt(s).
    • However, it could also create ambiguity and allow unexpected reentry into a state which has been handled/exited previously. For example: In Google-Facts sample, following sequence of response "history / quit / yes" shows "next fact" even after "quit" as the response "yes" gets handled in still alive "google-facts" context.
  • Output Context can also be set by webhook service app.js in ask() call:

  // Add google-facts context to outgoing context list
  assistant.setContext(GOOGLE_CONTEXT, DEFAULT_LIFESPAN, parameters);
  // Replace outgoing cat-facts context with lifespan = 0 to end it
  assistant.setContext(CAT_CONTEXT, END_LIFESPAN, {});
  assistant.ask('Looks like you\'ve heard all there is to know ' +
  'about cats. Would you like to hear about Google?');

Florist - Example walkthru

Florist Example: Using Contexts:

Intent transitions with output-context:

  • greetings -> compose (compose) -> yes-compose (yes-compose) -> yes-compose-choose (compose-add) -> yes-compose-choose-add (compose-add) [self loop]
  • greetings -> compose (compose) -> yes-compose (yes-compose) -> yes-compose-choose (compose-add) -> no-compose-choose-add (none)
  • greetings -> compose (compose) -> no-compose (no-compose) -> no-compose-anniversary (none)
  • greetings -> compose (compose) -> no-compose (no-compose) -> no-compose-birthday (none)

Florist - Agent - intent entries:

Intent: greetings
Context: none/none
Input: Hi; Hello
Response: Hello! Would you like to buy a bouquet?

Intent: compose
Context: none / compose, bouquet
Input: Yes; I want to buy a bouquet
Response: Would you like to compose the bouquet yourself?

Intent: yes-compose
Context: compose, bouquet / yes-compose, bouquet
Input: Yes; I want to make a bouquet myself.
Response: What kind of flowers would you like to add first?

Intent: yes-compose-choose
Context: yes-compose, bouquet / compose-add, bouquet
Input: Add @sys.number:amount-1 @sys.color:color-1 @flower:flower-1
Response: I put $amount-1 $color-1 $flower-1.original in your lovely bouquet. Should I add anything else?

Intent: yes-compose-choose-add
Context: compose-add, bouquet / compose-add, bouquet
Input: Add @sys.number:amount-2 @sys.color:color-2 @flower:flower-2
Response: I add $amount-2 $color-2 $flower-2.original to your bouquet. Anything else?

Intent: no-compose-choose-add
Context: compose-add, bouquet / none
Input: No; No, that's all
Response: You can purchase and get your wonderful bouquet at our office. Thank you for the order!

Intent: no-compose
Context: compose, bouquet / no-compose, bouquet
Input: No; No, give me usual bouquet
Response: I can offer you a bouquet for different occasions. For which occasion do you need a bouquet?

Intent: no-compose-anniversary
Context: no-compose, bouquet / none
Input: I need a bouquet for anniversary
Response: You can purchase and get the bouquet for anniversary at our office. Thank you for the order!

Intent: no-compose-birthday
Context: no-compose, bouquet / none
Input: I need a bouquet for birthday
Response: You can purchase and get the bouquet for birthday at our office. Thank you for the order!

More reading/links:

API.AI - Natural Language Interface
https://docs.api.ai/docs/get-started -- Create Intent

API.AI "How to design" examples:
Number Genie - Simple conversation number guess game:
Facts about Google - Simple conversation action with fulfillment:

Google Home "one click" "Actions on Google Integration":

Actions on Google - Voice User Interface - Design Principles and Methodology

Simple examples:
https://github.com/actions-on-google/apiai-silly-name-maker (sample without Fulfillment)
https://github.com/actions-on-google/apiai-silly-name-maker-webhook-nodejs (sample with Webhook Fulfillment)

Fulfillment, and continue Ask; Pass data between Api.ai and Webhook service:

Voice to Natural language processing: Node.js

RVM - Multiple versions of Ruby, Rails gemsets


- RVM allows to install multiple versions of ruby/rails/gems and switch between them.
- RVM installs ruby/gem/rails/command tools under $HOME/.rvm/.
- RVM intercepts shell commands and "redirects" commands/uses gems from chosen Ruby/Gemset Rails.
- Rails is installed as a "gemset" under Ruby: e.g. ruby_version@rails_version => ruby-1.9.2-p290@rails-3.2.11.
- Don't use "sudo" for commands (say "gem") under RVM.

RVM - Install desired Ruby and Rails:

# Install RVM

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -sSL https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm requirements

# Install Ruby under RVM

rvm install 1.9.2-p290  # ruby_version
rvm list
//     rvm rubies
//     =* ruby-1.9.2-p290 [ x86_64 ]
rvm use 1.9.2-p290 # ruby_version
//     Using $HOME/.rvm/gems/ruby-1.9.2-p290

# Create gemset under it for installing Rails

rvm gemset create rails-3.2.11 # gemset_name
//     ruby-1.9.2-p290 - #gemset created $HOME/.rvm/gems/ruby-1.9.2-p290@rails-3.2.11
//     ruby-1.9.2-p290 - #generating rails-3.2.11 wrappers........
rvm gemset list
//     gemsets for ruby-1.9.2-p290 (found in $HOME/.rvm/gems/ruby-1.9.2-p290)
//     => (default)
//        global
//        rails-3.2.11
rvm gemset use rails-3.2.11
//     Using ruby-1.9.2-p290 with gemset rails-3.2.11
rvm gemset list
//     gemsets for ruby-1.9.2-p290 (found in $HOME/.rvm/gems/ruby-1.9.2-p290)
//        (default)
//        global
//     => rails-3.2.11
rvm ruby-1.9.2-p290@rails-3.2.11 #ruby_version@gemset_name
//     Using $HOME/.rvm/gems/ruby-1.9.2-p290 with gemset rails-3.2.11

# Repeat following until all errors are gone!

gem install rails -v 3.2.11 # rails_version

# Errors/Solutions:

//     Fetching: rack-cache-1.7.0.gem (100%)
//     ERROR: Error installing rails: rack-cache requires Ruby version >= 1.9.3.
# Find the needed/working gem version number from pre-installed/heroku "gem list".
gem install rack-cache -v 1.2 # as used in our saasbook
gem install i18n -v 0.6.1
bundle install
//     pg 0.18.0 - Error pg_config, libpq-fe.h missing
sudo apt-get install libpq-dev

### Use RVM:

# Load RVM and set/check desired gemsets:
source $HOME/.rvm/scripts/rvm;

# Choose Ruby/Rails gem combo to use:

rvm ruby-1.9.2-p290@rails-3.2.11;
rvm gemset list;

### Install heroku-CLI & pull (usual way!)

sudo add-apt-repository "deb https://cli-assets.heroku.com/branches/stable/apt ./"
curl -L https://cli-assets.heroku.com/apt/release.key | sudo apt-key add -
sudo apt-get install heroku
heroku git:clone -app railsapp

### Build/Run app (usual way!):

cd railsapp
bundle install
rake db:migrate
rake db:seed
rails server

/** Not tried; Not needed?
If you want to setup your rails app to utilize the RVM gemset 2.2.1@rails_2_1_1, similar to the RVM command below...
rvm use ruby-1.9.2-p290@rails-3.2.11
In your Gemfile, specify right below the source line the following two commented lines:
    source 'https://rubygems.org'

### Other alternative:
# In app directory:
gemset ruby-1.9.2-p290@rails-3.2.11
## $ echo "ruby-1.9.2-p290" > .ruby-version
## $ echo "rails-3.2.11" > .ruby-gemset

Brillo - Exploring BDK code

     kconfig, fconfig, sensor, sepolicy, wifi_driver_hal
     examples (dbus, lights, service)
     .mk & .sh setup files

bdk/device/generic/goldfish/: qemu peripherals (audio, camera, fingerprint, lights, sensors, vibrator)

bdk/device/generic/mini_emulator_ALL and generic/x86_64 (installed HW): .mk files

bdk/device/intel/edison/flash_tools/ brillo-flashall-edison.*


     daemons, dbus, glib, http, minijail, and individual utils/support files?

bdk/external/libbrillo/policy/: device_policy, libpolicy

class LedFlasherHandler {
  void Register(weave::Device* device) {
    device->AddCommandDefinitionsFromJson(R"({      "_ledflasher": {         "_set":{           "parameters": {             "_led": {"minimum": 1, "maximum": 3},             "_on": "boolean"           }         },         "_toggle":{           "parameters": {             "_led": {"minimum": 1, "maximum": 3}           }        }      }    })");
"_ledflasher._toggle", ...);
"_ledflasher._set", ...);
void OnFlasherSetCommand(const std::weak_ptr<weave::Command>& command) {
    UpdateLedState() ==> device_->SetStateProperty("_ledflasher._leds", list, nullptr);

int main(int argc, char** argv) {
  Daemon daemon{opts};
  LedFlasherHandler handler;
  return 0;
Above appears to be implementation of

bdk/external/libweave/examples/daemon/README: "Acquire Registration Ticket"
Google Developers - OAuth 2.0 Playground
Obtain URL to initialize the playground with the current configuration (includes OAuth credentials and OAuth tokens in the link): https://developers.google.com/oauthplayground/

     commands, notification, privet, states
     device_manager.cc: StartPrivet, GetGcdState, AddCommandDefinitions, AddCommand, SetStateProperty, ..
     config.cc: Load(), Config::Transaction::LoadState()

bdk/external/libweave/third_party/: chromium, modp_b64

bdk/external/: dbus, iptables, jpeg, lib*, rootdev, selinux, toybox, wpa_supplicant_8

bdk/frameworks/av/: camera, drm, media, radio, services, sound_trigger

bdk/frameworks/native/: cmds
bdk/frameworks/native/data/etc/: android.hardware.*.xml
             <feature name="android.hardware.* " />
bdk/frameworks/native/include/: android, diskusage, gui, input, media, powermanager, private, ui
bdk/frameworks/native/libs/: binder, diskusage, gui, input, ui
bdk/frameworks/native/services/: battery, powermanager, sensor

bdk/frameworks/rs/: Render Script (?)
bdk/frameworks/wilhelm/: OpenMAXAL, OpenSLES

     bsp/intel/peripheral/: audio, bluetooth, libmraa, libupm, light, sensors, wifi
     bsp/intel/soc/edison/: bootctrl, sepolicy
     libhardware/modules/: audio, camera, input, nfc, power, radio, sensors, soundtrigger, vibrator

bdk/libnativehelper/: JNI

     system/bin - Linux?
     system/etc/os-release.d/bdk_version & product_version
     boot.img, kernel, ramdisk.img, system.img, userdata.img


     common/brillo_gpios/src/gpio_playground/gpio_playground.cpp: Export "/sys/class/gpio" & file stream
     common/brillo_leds/src/brillo_led_tool/brillo_led_tool.cpp: Toggle "/sys/class/leds/led*/brightness"
     common/sensors_example: Accelerometer and other sensors hw_get_module(SENSORS_HARDWARE_MODULE_ID?

     example-ledflasher/config/brillo/product_id: developer-boards:ledflasher
         Used by the backends to route metrics/crash reports to the right customer.

  "_ledflasher": {
    "_set": {
      "parameters": {
        "_led": {"minimum": 1, "maximum": 4},
        "_on": "boolean"      }
    "_toggle": {
      "parameters": {
        "_led": {"minimum": 1, "maximum": 4}
    "_animate": {
      "parameters": {
        "_duration": {"minimum": 0.1, "maximum": 100.0},
        "_type": ["none", "marquee_left", "marquee_right", "blink"]
  "_ledflasher": {
    "_status": {
      "enum": ["idle", "animating"],
      "default": "idle"    },
    "_leds": {"items": "boolean"}
     These two look identical (except additional "animation" in ledflasher.cpp):
     But I couldn't decide whether they are interacting with same device & using identical mode of interaction.

    <node name="/com/android/LEDService/Service"
      <interface name="com.android.LEDService.Service">
        <method name="SetLED">

  "service_name": "com.android.LEDService",
  "object_manager": {
    "name": "com.android.LEDService.ObjectManager",
    "object_path": "/com/android/LEDService"  }

Brillo - Building from Source Code

Building Brillo from Source Code:

Need 40GB VM diskspace (ubuntu, tools, code, build), 2GB swap, 2GB RAM.
Took 4+hrs. More RAM should make it much much faster.

Follow Android build/environment steps:

# Update Ubuntu 10.04
sudo apt-get install git-core gnupg flex bison gperf build-essential \
zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 \
lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache \
libgl1-mesa-dev libxml2-utils xsltproc unzip
# for lunch
sudo apt-get install python-lunch
# for java and javac
sudo apt-get install openjdk-7-jre-headless
sudo apt-get install openjdk-7-jdk

ccache is a compiler cache for C and C++ that can help make builds faster. In the root of the source tree, do the following:
export USE_CCACHE=1
export CCACHE_DIR=$HOME/.ccache # /<path_of_your_choice>/.ccache
prebuilts/misc/linux-x86/ccache/ccache -M 50G
# The suggested cache size is 50-100G.
# watch ccache being used by doing the following:
watch -n1 -d prebuilts/misc/linux-x86/ccache/ccache -s

error: ro.build.fingerprint cannot exceed 91 bytes:

Brillo/brilloemulator_x86_64/brilloemulator_x86_64:6.0.60/MASTER/shreekant02040032:eng/test-keys (96)
To avoid this error, use suitable shorter user login name such that fingerprint doesn't exceed 91 bytes.

For instance, above becomes 89 char fingerprint for user "sk":

Building From Official (Stable) Release:

# Ref https://developers.google.com/brillo/docs/reference/downloads
wget https://dl.google.com/dl/brillo/bdk/latest/bdk-latest.tar.gz
tar -xzf bdk-latest.tar.gz
{BDK_PATH}/tools/bdk/brunch/brunch bsp list
{BDK_PATH}/tools/bdk/brunch/brunch bsp download <bsp-name>

bdk$ source build/envsetup.sh && lunch
selected 9. brilloemulator_x86_64-eng

bdk$ make -j
TARGET_DEVICE brilloemulator_x86_64 (build-brilloemulator_x86_64.ninja??)

? /home/sk/brillo/bdk/hardware/bsp/kernel/common/android-3.18
? /home/sk/brillo/bdk/prebuilts/gcc/linux-x86/x86/x86_64-linux-android-4.9/bin

[100% 10445/10445] Install system fs image: out...target/product/brilloemulator_x86_64/system.img
out/target/product/brilloemulator_x86_64/system.img+out/target/product/brilloemulator_x86_64/recovery.img maxsize=802897920 blocksize=2112 total=793860096 reserve=8110080

# Brillo emulator - Runtime setup

export PATH=\

export ANDROID_BUILD_TOP=/home/sk/brillo/bdk

# Start brillo emulator

Building from Latest Development Branch: (Just for info; Not actually tried/tested)

# Ref http://www.cnx-software.com/2015/11/23/how-to-build-and-run-brillo-operating-system-from-source-code/
mkdir brillo-master
cd brillo-master
# for repo
sudo apt-get install phablet-tools
repo init -u https://android.googlesource.com/brillo/manifest -b master
many more commands & useful fixes in Ref.

Brillo - Basics

Brillo is an IoT Platform Eco-system involves:

  • Embedded OS running on a device,
  • Client/interface App running on Android Phone, and
  • Cloud service to coordinate between devices, owners & users.

Brillo: Embedded OS based on Android

  • BSP for specific board
  • Supports ARM, Intel, MIPS architecture)
  • Smallest configuration 32MB of RAM, and 128MB of storage
  • Supports Wi-Fi and Bluetooth Low Energy
  • Runs on battery-powered devices

Brillo Core platform services:

  • Weave (helps devices to securely connect to the network)
  • Metrics (collect usage data from devices based on the user permission)
  • Console to view & analyze data,
  • Collect crash report & debug data from field
  • Over-the-air (OTA) Push updates,
  • Developer kit: build, debug, test device
    • Android.mk build architecture
    • Runs on Ubuntu 14.04
    • Local Unit test, individual Integration tests, or Own Continuous Build Integration Testing infrastructure.
    • Andoid Debug Bridge (adb) over Ethernet or USB (without any special equipment)

Weave library maintained in Brillo OS:

  • Provides:
    • Setup, Interaction, and communication with wide range of connected devices.
    • Ability to communicate with device directly or through Cloud servicess.
    • Common Language for app or services to use across all of user devices. [based on JSON]
  • Protocol:
    • Device Discovery
    • Provisioning
    • Authentication & Interaction implemented across device-side libraries, mobile SDKs and web service that enables remote interactions.
  • Schema: Commands and States that are common for device types
    • customized & extended for a particular product
  • Cloud Service: Remote communication service integrated in Weave.
    • provides aggregated metrics in developer console to analyze all devices.
  • Client app/services on mobile/desktop/web to setup/interact with devices.
    • Secure setup experience common to all devices that works from both Android & iOS devices.
    • Communicate with your and devices made by other developers using same standardized schemas.

Brillo Boards:


  • Intel Edison Board Made for Brillo:

    Intel EDI2ARDUIN.AL.K Dual Core IA-32 @ 500 MHz, 32-bit Intel Atom Processor Z34xx Series @ 100 MHz Board with 70-pin connector Motherboard/CPU Combo
        http://www.newegg.com/Intel-Brillo ==> "Edison kit for Arduino" http://promotions.newegg.com/intel/16-0304/index.html
    Flashing Brillo on Edison (plus some links on right)


[Need to obtain access permissions! Apply at https://developers.google.com/brillo/?hl=en]
    Intel Edison and Peripherals
        Edison and Grove (Plus/Gen2 sensors, actuators and shields)
            http://johnny-five.io/examples/grove-relay-edison/ Grove Software Library for Edison

  • MIPS-based Imagination Creator board for Brillo:
  • ARM-based PICO-i.MX6UL and DWARF board for Brillo:

Switching deprecated attr_accessible to Rails 4+ Strong Parameters with gem 'cancancan'

class User < ActiveRecord::Base
  ##attr_accessible :name, :full_name, :family, :roles_mask, :password
  validates :name, :presence => true,

            :uniqueness => { :case_sensitive => false,
              :message => "is already used! Error: Duplicate Login Name." }
  validates :password, :on => :create, :presence => true
  def roles=(roles) # roles array to bitmask
    self.roles_mask = (roles & ROLES).map { |r| 2**ROLES.index(r) }.sum

class UsersController < ApplicationController
  # Use cancan authorization in all methods with support for Rails 4+ Strong Parameters
  load_and_authorize_resource param_method: :user_params
  def update

  def create_multiple
          new_params[:family]         = fields[0].strip
          new_params[:name]           = fields[1].strip
          new_params[:roles]          = fields[3].strip.split(/,/)  # actually sets :roles_mask
          new_params[:password] = new_params[:password_confirmation] = new_params[:name]
          @user = User.new(user_params(ActionController::Parameters.new({user: new_params}) ))

    def user_params(custom_params = nil)
      custom_params ||= params
      # roles_mask, password_confirmation and password_digest

      # are not included in the permit list here:
      custom_params.require(:user).permit(:name, :family, :full_name, :valid_upto,
                            :password, roles:[]) # "roles:[]" must be the last param here.
             # See http://api.rubyonrails.org/classes/ActionController/Parameters.html