Huginn

Logged in using

ssh -i deployment_key.txt [email protected] Source

1. GUIDE

Will be following this guide:

Other guides; https://www.digitalocean.com/community/questions/how-to-deploy-huginn-on-my-own-server-without-docker https://github.com/cantino/huginn/wiki/Novice-setup-guide

2. DEPENDENCIES

Already logged in as root, apt-get pre-installed, but did these,

apt-get update -y
apt-get upgrade -y

Import node.js repo,

curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -

Installing required packages:

sudo apt-get install -y runit build-essential git zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake nodejs graphviz

3. RUBY

Removing old Ruby versions,

sudo apt-get remove -y ruby1.8 ruby1.9

ruby1.8 was installed and removed.

However, getting the runit error again. So Doin’ it:

Downloading Ruby and compiling it:

mkdir /tmp/ruby && cd /tmp/ruby
curl -L --progress http://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.bz2 | tar xj
cd ruby-2.3.1
./configure --disable-install-rdoc
make -j`nproc`
sudo make install

sudo gem install rake bundler foreman --no-ri --no-rdoc

4. SYSTEM USERS

sudo adduser –disabled-login –gecos ‘Huginn’ huginn

5. DATABASE

sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev

# Pick a MySQL root password (can be anything), type it and press enter,
# retype the MySQL root password and press enter
# password is test

Password is *****.

mysql --version
mysql -u root -p

mysql> CREATE USER 'huginn'@'localhost' IDENTIFIED BY '$password';
CREATE USER 'huggin'@'localhost' IDENTIFIED BY 'test';

Setting InnoDB engine to support long indexes done.

SET default_storage_engine=INNODB;

Granting Huginn user permissions:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `huginn_production`.* TO 'huginn'@'localhost';

Went smoothly. Quitting database with ‘\q’. Connecting to database with new user:

sudo -u huginn -H mysql -u huginn -p -D huginn_production

#type in password test

Crap! Doesn’t work. Multiple tries made, next, tried all lower-case and kyleJackson with a space at the end. Will have to reset password. Look it up.

So log back into to root, mysql -u root -p, AND FIND THE CULPRIT, a typo in username, CREATE USER ‘huggin’@’localhost’ IDENTIFIED BY ‘kyleJackson’;

instead of

CREATE USER ‘hugginn’@’localhost’ IDENTIFIED BY ‘kyleJackson’;

Granting permissions again:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON huginn_production.* TO ‘huginn’@’localhost’;

Try logging in again

sudo -u huginn -H mysql -u huginn -p -D huginn_production

#Getting error

ERROR 1049 (42000): Unknown database 'huginn_production'

#log back into mysql and re-try "GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `huginn_production`.* TO 'huginn'@'localhost';"

#\q and try logging in again


#Referring to [http://stackoverflow.com/questions/12118627/error-1049-42000-unknown-database](http://stackoverflow.com/questions/12118627/error-1049-42000-unknown-database)

#run in mysql>

create database huginn_production

#grant permissions (again):

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `huginn_production`.* TO 'huginn'@'localhost';

#didn;t work so try

mysql> GRANT ALL PRIVILEGES ON huginn_production.* TO 'huginn'@'localhost' IDENTIFIED BY 'test';


mysql> SHOW GRANTS FOR 'huginn'@'localhost'
    -> ;
+---------------------------------------------------------------------------------------------------------------+
| Grants for huginn@localhost                                                                                   |
+---------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'huginn'@'localhost' IDENTIFIED BY PASSWORD '*94BDCEBE19083CE2A1F959FD02F964C7AF4CFC29' |
| GRANT ALL PRIVILEGES ON `huginn_production`.* TO 'huginn'@'localhost'                                         |
+---------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

#DID NOT WORK

#so removing mysql, and trying again:

sudo apt-get remove -y mysql-server mysql-client libmysqlclient-dev

and installed again

mysql> DROP USER 'huginn'@'localhost';
Query OK, 0 rows affected (0.00 sec)

CREATED DATABASE MANUALLY

mysql> create database huginn_production
    -> ;
Query OK, 1 row affected (0.00 sec)

And it worked.

We do ‘pwd’ : /tmp/ruby/ruby-2.3.1 and take a break.

6. HUGINN

First, we switch directories. Clone the Huginn repo, update the .env file, create ‘log’, ‘pids’, ‘sockets’ directories. Give access rights (write and executable) to directories. Set permissions. Copy unicorn.rb config file.

Clone the source

# We'll install Huginn into the home directory of the user "huginn"
cd /home/huginn

# Clone Huginn repository
sudo -u huginn -H git clone https://github.com/cantino/huginn.git -b master huginn

# Go to Huginn installation folder
cd /home/huginn/huginn

# Copy the example Huginn config
sudo -u huginn -H cp .env.example .env

# Create the log/, tmp/pids/ and tmp/sockets/ directories
sudo -u huginn mkdir -p log tmp/pids tmp/sockets

# Make sure Huginn can write to the log/ and tmp/ directories
sudo chown -R huginn log/ tmp/
sudo chmod -R u+rwX,go-w log/ tmp/

# Make sure permissions are set correctly
sudo chmod -R u+rwX,go-w log/
sudo chmod -R u+rwX tmp/
sudo -u huginn -H chmod o-rwx .env

# Copy the example Unicorn config
sudo -u huginn -H cp config/unicorn.rb.example config/unicorn.rb

Configure it

Now we configure it,

# Update Huginn config file and follow the instructions
sudo -u huginn -H editor .env

# ==== Required configuration settings for Huginn ====

# Replace the following with the output from "rake secret"
APP_SECRET_TOKEN=REPLACE_ME_NOW!

# This is the domain where your Huginn instance will be running. The default should $
# for development, but it needs to be changed to your Huginn domain when you deploy $
# production environment (e.g., yourdomain.com, possibly including a port).
DOMAIN=localhost:3000
PORT=3000

# Optionally set an asset host
# ASSET_HOST=http://assets.example.com

############################
#      Database Setup      #
############################

DATABASE_ADAPTER=mysql2
DATABASE_ENCODING=utf8
DATABASE_RECONNECT=true
DATABASE_NAME=huginn_production //we changed this
DATABASE_POOL=20
DATABASE_USERNAME=huginn //we changed this
DATABASE_PASSWORD="test" //we changed this
#DATABASE_HOST=your-domain-here.com
#DATABASE_PORT=3306
#DATABASE_SOCKET=/tmp/mysql.sock

Regarding step,

“Change the Unicorn config if needed, the requirements.md has a section explaining the suggested amount of unicorn workers:”

# Increase the amount of workers if you expect to have a high load instance.
# 2 are enough for most use cases, if the server has less then 2GB of RAM
# decrease the worker amount to 1
sudo -u huginn -H editor config/unicorn.rb

#looks like this:

wd = File.expand_path(File.join(File.dirname(__FILE__), '..'))

app_path = wd

worker_processes 2
preload_app true
timeout 180
listen "#{wd}/tmp/sockets/unicorn.socket"

working_directory app_path

rails_env = ENV['RAILS_ENV'] || 'production'

# Log everything to one file
stderr_path "log/unicorn.log"
stdout_path "log/unicorn.log"

# Set master PID location
pid "#{wd}/tmp/pids/unicorn.pid"

before_fork do |server, worker|
  ActiveRecord::Base.connection.disconnect!
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exist?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

We move to installing Gems.

Install Gems

There was an error (below, error parsing Gemfile: Line “ RAILS_ENV=production” doesn’t match format.). Fixed because we’re in development mode.

[!] There was an error parsing `Gemfile`: Line " RAILS_ENV=production" doesn't match format. Bundler cannot continue.

 #  from /home/huginn/huginn/Gemfile:8
 #  -------------------------------------------
 #  require File.join(File.dirname(__FILE__), 'lib/gemfile_helper.rb')
 >  GemfileHelper.load_dotenv do |dotenv_dir|
 #    path dotenv_dir do
 #  -------------------------------------------

source 'https://rubygems.org'

# Ruby 2.2.2 is the minimum requirement
ruby ['2.2.2', RUBY_VERSION].max

# Load vendored dotenv gem and .env file
require File.join(File.dirname(__FILE__), 'lib/gemfile_helper.rb')
GemfileHelper.load_dotenv do |dotenv_dir|
  path dotenv_dir do
    gem 'dotenv'
    gem 'dotenv-rails'
  end
end

Gem installation now went smoothly.

root@bezos:/home/huginn/huginn# sudo -u huginn -H bundle install --deployment --without development test
The git source `git://github.com/dsander/weibo_2.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/cantino/twitter-stream.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/tumblr/tumblr_client.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/dsander/delayed_job_active_record.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/lostisland/faraday_middleware.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/amatsuda/kaminari.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
Fetching gem metadata from https://rubygems.org/......
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Fetching git://github.com/amatsuda/kaminari.git
Fetching git://github.com/dsander/delayed_job_active_record.git
Fetching git://github.com/lostisland/faraday_middleware.git
Fetching git://github.com/tumblr/tumblr_client.git
Fetching git://github.com/cantino/twitter-stream.git
Fetching git://github.com/dsander/weibo_2.git
Installing rake 11.2.2
Installing ace-rails-ap 2.0.1
Installing concurrent-ruby 1.0.2
Installing i18n 0.7.0
Installing minitest 5.9.0
Installing thread_safe 0.3.5
Installing builder 3.2.2
Installing erubis 2.7.0
Installing mini_portile2 2.1.0
Installing rack 2.0.1
Installing nio4r 1.2.1 with native extensions
Installing websocket-extensions 0.1.2
Installing mime-types 2.99.3
Installing arel 7.1.1
Installing addressable 2.3.8
Installing extlib 0.9.16
Installing multi_json 1.12.1
Installing jmespath 1.1.3
Installing bcrypt 3.1.11 with native extensions
Using bundler 1.13.6
Installing method_source 0.8.2
Installing thor 0.19.1
Installing buftok 0.2.0
Installing coffee-script-source 1.10.0
Installing execjs 2.6.0
Installing cookiejar 0.3.2
Installing daemons 1.1.9
Installing orm_adapter 0.5.0
Installing unf_ext 0.0.7.1 with native extensions
Using dotenv 2.0.1 from source at `vendor/gems/dotenv-2.0.1`
Installing hashie 3.4.6
Installing oauth 0.4.7
Installing eventmachine 1.0.7 with native extensions
	Installing http_parser.rb 0.6.0 with native extensions
Installing equalizer 0.0.11
Installing polyglot 0.3.5
Installing ffi 1.9.10 with native extensions
Installing evernote-thrift 1.25.1
Installing multipart-post 2.0.0
Installing sax-machine 1.3.2
Installing sass 3.4.14
Installing jwt 1.4.1
Installing retriable 2.0.2
Installing uuidtools 2.1.5
Installing haversine 0.3.0
Using json 1.8.3
Installing multi_xml 0.5.5
Installing mimemagic 0.3.1
Installing kgio 2.10.0 with native extensions
Installing kramdown 1.3.3
Installing libv8 3.16.14.15
Installing liquid 3.0.6
Installing rb-fsevent 0.9.7
Installing systemu 2.6.4
Installing mini_magick 4.2.3
Installing mqtt 0.3.1
Installing mysql2 0.3.20 with native extensions
Installing naught 1.0.0
Installing net-ftp-list 3.2.8
Installing netrc 0.10.3
Installing raindrops 0.17.0 with native extensions
Installing ref 2.0.0
Installing tilt 2.0.5
Installing simple_oauth 0.3.1
Installing slack-notifier 1.0.0
Installing xmpp4r 0.5.6
Installing tzinfo 1.2.2
Installing memoizable 0.4.2
Installing nokogiri 1.6.8.1 with native extensions
Installing rack-test 0.6.3
Installing sprockets 3.7.0
Installing warden 1.2.6
Installing websocket-driver 0.6.4 with native extensions
Installing mail 2.6.4
Installing launchy 2.4.2
Installing autoparse 0.3.3
Installing geokit 1.8.5
Installing jsonpathv2 0.0.8
Installing aws-sdk-core 2.2.15
Installing huginn_agent 0.4.0
Installing select2-rails 3.5.9.3
Installing coffee-script 2.4.1
Installing unf 0.1.4
Using dotenv-rails 2.0.1 from source at `vendor/gems/dotenv-2.0.1`
Installing foreman 0.63.0
Installing omniauth 1.3.1
Installing dropbox-api 0.4.2
Installing em-socksify 0.3.0
Installing http 0.6.4
Installing treetop 1.5.3
Installing ethon 0.7.1
Installing rb-inotify 0.9.5
Installing rb-kqueue 0.2.4
Installing evernote_oauth 0.2.3
Installing faraday 0.9.2
Installing font-awesome-sass 4.3.2.1
Installing twilio-ruby 3.11.6
Installing uglifier 2.7.2
Installing httparty 0.13.7
Installing macaddr 1.7.1
Installing unicorn 5.1.0 with native extensions
Installing therubyracer 0.12.2 with native extensions
Using twitter-stream 0.1.15 from git://github.com/cantino/twitter-stream.git (at huginn@f7e7edb)
Installing activesupport 5.0.0.1
Installing rufus-scheduler 3.0.9
Installing loofah 2.0.3
Installing domain_name 0.5.24
Installing omniauth-oauth 1.1.0
Installing em-http-request 1.1.2
Installing erector 0.10.0
Installing typhoeus 0.6.9
Installing listen 3.0.5
Using faraday_middleware 0.10.0 from git://github.com/lostisland/faraday_middleware.git (at master@59088da)
Installing forecast_io 2.0.1
Installing signet 0.5.1
Installing oauth2 1.2.0
Installing twitter 5.14.0
Installing hipchat 1.2.0
Installing httmultiparty 0.3.16
Installing wunderground 1.2.0
Installing uuid 2.3.7
Installing rails-dom-testing 2.0.1
Installing globalid 0.3.7
Installing activemodel 5.0.0.1
Installing delayed_job 4.1.2
Installing rails-html-sanitizer 1.0.3
Installing http-cookie 1.0.2
Installing omniauth-dropbox 0.2.0
Installing omniauth-evernote 1.2.1
Installing omniauth-tumblr 1.2
Installing omniauth-twitter 1.2.1
Installing feedjira 2.1.0
Using tumblr_client 0.8.5 from git://github.com/tumblr/tumblr_client.git (at master@0c59b04)
Installing google-api-client 0.7.1
Installing omniauth-oauth2 1.3.1
Installing hypdf 1.0.10
Installing ruby-growl 4.1
Installing activejob 5.0.0.1
Installing activerecord 5.0.0.1
Installing actionview 5.0.0.1
Installing rest-client 1.8.0
Installing omniauth-37signals 1.0.5
Installing omniauth-wunderlist 0.0.2
Using delayed_job_active_record 4.1.1 from git://github.com/dsander/delayed_job_active_record.git (at rails5@b314972)
Installing actionpack 5.0.0.1
Installing rturk 2.12.1
Using weibo_2 0.1.7 from git://github.com/dsander/weibo_2.git (at master@e5b77f2)
Installing actioncable 5.0.0.1
Installing actionmailer 5.0.0.1
Using kaminari 0.17.0 from git://github.com/amatsuda/kaminari.git (at 0-17-stable@abbf93d)
Installing railties 5.0.0.1
Installing sprockets-rails 3.2.0
Installing coffee-rails 4.2.1
Installing responders 2.3.0
Installing jquery-rails 4.2.1
Installing spectrum-rails 1.3.4
Installing rails 5.0.0.1
Installing sass-rails 5.0.6
Installing devise 4.2.0
Installing bootstrap-kaminari-views 0.0.5
Installing geokit-rails 2.2.0
Bundle complete! 104 Gemfile dependencies, 161 gems now installed.
Gems in the groups development and test were not installed.
Bundled gems are installed into ./vendor/bundle.
Post-install message from httparty:
When you HTTParty, you must party hard!
Post-install message from rufus-scheduler:

***

Thanks for installing rufus-scheduler 3.0.9

It might not be 100% compatible with rufus-scheduler 2.x.

If you encounter issues with this new rufus-scheduler, especially
if your app worked fine with previous versions of it, you can

A) Forget it and peg your Gemfile to rufus-scheduler 2.0.24

and / or

B) Take some time to carefully report the issue at
   https://github.com/jmettraux/rufus-scheduler/issues

For general help about rufus-scheduler, ask via:
http://stackoverflow.com/questions/ask?tags=rufus-scheduler+ruby

Cheers.

***

Initialize database

# Create the database
sudo -u huginn -H bundle exec rake db:create RAILS_ENV=production

Database 'huginn_production' already exists
(ofcourse)

# Migrate to the latest version
sudo -u huginn -H bundle exec rake db:migrate RAILS_ENV=production

# Create admin user and example agents using the default admin/password login
sudo -u huginn -H bundle exec rake db:seed RAILS_ENV=production SEED_USERNAME=admin SEED_PASSWORD=password

#Receive message:
/*
NOTE: The example 'SF Weather Agent' will not work until you edit it and put in a free API key from http://www.wunderground.com/weather/api/
See the Huginn Wiki for more Agent examples!  https://github.com/cantino/huginn/wiki
*/

Compile Assets

sudo -u huginn -H bundle exec rake assets:precompile RAILS_ENV=production

Install Init Script

sudo -u huginn -H editor Procfile

#Comment out (disable) these two lines

web: bundle exec rails server -p ${PORT-3000} -b ${IP-0.0.0.0}
jobs: bundle exec rails runner bin/threaded.rb

#Enable (remove the comment) from these lines or those

# web: bundle exec unicorn -c config/unicorn.rb
# jobs: bundle exec rails runner bin/threaded.rb

#Export the init scripts:

sudo bundle exec rake production:export

Note: We have to re-export the init script (sudo bundle exec rake production:export) every time you change the configuration in .env or your Procfile!

Setup Logrotate

sudo cp deployment/logrotate/huginn /etc/logrotate.d/huginn

Ensure Your Huginn Instance Is Running

sudo bundle exec rake production:status

6. NGINX

Installation; sudo apt-get install -y nginx

Site Configuration. Copy the example site config:

sudo cp deployment/nginx/huginn /etc/nginx/sites-available/huginn sudo ln -s /etc/nginx/sites-available/huginn /etc/nginx/sites-enabled/huginn

Change YOUR_SERVER_FQDN to the fully-qualified

domain name of your host serving Huginn.

sudo editor /etc/nginx/sites-available/huginn 162.243.124.112

Remove the default nginx site, if huginn is the only enabled nginx site (it is):

sudo rm /etc/nginx/sites-enabled/default

Test Configuration. Validate your huginn or huginn-ssl Nginx config file with the following command:

sudo nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

Restart

sudo service nginx restart

ERROR - tail /var/log/nginx/huginn_error.log

*2 connect() to unix:/home/huginn/huginn/tmp/sockets/unicorn.socket failed (2: No such file or directory) while connecting to upstream

re-checking procfile and unicorn configuration

Edited procfile and re-exported bundle scripts

ERROR still - tail /var/log/nginx/huginn_error.log

2 connect() to unix:/home/huginn/huginn/tmp/sockets/unicorn.socket failed (2: No such file or directory) while connecting to upstream

RAKE PRODUCTION STATUS sudo bundle exec rake production:status (unicorn is running) run: huginn-web-1: (pid 1945) 4s; run: log: (pid 31732) 268s run: huginn-jobs-1: (pid 1962) 4s; run: log: (pid 31730) 268s

editing .env file DOMAIN=162.243.124.112:3000 PORT=3000 bundle exec rake secret b81f311b8984a2b99221f8c935d258ff533623217277a948bb764c1c24ac456edc2edfac44ded487e95497f23387453dfd38c40346f57d48b4824b5a4b9e8621

sudo bundle exec rake production:export sudo bundle exec rake production:export sudo bundle exec rake production:export sudo bundle exec rake production:export

in .env uncomment RAILS_ENV=production be mindful of formatting no spaces before rails_env, there was a formatting error Important: Uncomment the RAILS_ENV setting to run Huginn in the production rails environment