Watson analytics with nodejs integration

Get a key,from here

https://developer.ibm.com/api/view/id-160:title-Watson_Analytics

Go to the manage api section and get these two parameters.

X-IBM-Client-Id c1a4d636-51c1-4aad-
X-IBM-Client-Secret gC3pL3sG2eC0cK4kB3mK

Open postman and make a put request to

https://api.ibm.com/watsonanalytics/run/oauth2/v1/config

In body of put,enter like  this.

{
     "ownerName": "asdf",
     "clientName": "sdfasd",
     "redirectURIs": "https://localhost:6001",
     "ownerPhone": "9999999999",
     "ownerCompany": "asdf",
     "ownerEmail": asdfasdfs@sdfsd.com
}

replace junk with your info, this needs to be done once and put response will get the info you need.

Next time you can do a get on above url with headers, it will spit out the client info to you.

You are all set to push docs to analytics.

But we  have client id only and we need auth  code also, if you try without that you get below error.

"Failed to verify OAuth information. Transaction ID=42

so lets get the auth code

Open a browser and paste this.

https://api.ibm.com/watsonanalytics/run/clientauth/v1/auth?scope=userContext&response_type=code&client_id=-4cf7-89e2-94af7fd6180c&redirectui=http://realhost.org

Make sure you redirect url is same and is reachable, else you will get some background database not reachable error.

give permissions to the app.

You will get a call back with authcode likethis.

http://sdfsdfsdf:6001/?code=7qI2rTRi%2FrqOZkAvX61p835QFP7zkclb1NOyp3RI6NwN7wF%2Bm%2BaRUYFpN5ALvYK4RrN0netLwyFpBQmFjv%2Fc0t%2BC%2B4jOxEsnHW%2F5hPAJxBvI2YU3h3Nuk16iM0%2BW9AfP%2FF9AfK%2BrZgsHHEvMB9r1YAGq3m%2BqGR%2FtnfWrDptChBzhZyMv5LEnpm3p6O8cTI7yExGH%2FecXouP2KpwPB6y3ATDoIacoZUUcKZZlcSebqchDrx%2F5n2RWIxk5fSnWO%2BkQJwo%2BrLggmjLUXpMPeaA0qCOfX5P5gd3iU7spOgh5eA%3D%3D#/

Use this code in our node app, valid for 1 hour.

Don’ know why ibm confused so much with auths and tokens but friends.. one more step

we need access token before we get the data assets i know it sucks and sounds redundant.

use a decoder service to decode encoded code

Now we use this auth code to get token

https://api.ibm.com/watsonanalytics/run/oauth2/v1/token

Ref these docs, i could pass from here, tried different grant_type, all of them which were assigned to me in config , but none worked.

{
     "error": "invalid_client"
}

===================Not working….. from here…..so i moved to another analytics solution.. mailed ibm india team but no reply..

Ref.

github repo. followed instructions but didnt work .

https://github.com/WatsonAnalytics/oauth2-nodejs

https://developer.ibm.com/watson-analytics/#getstarted

https://community.watsonanalytics.com/discussions/questions/23305/automatic-upload-of-datasets-through-rest-api.html

Other errors encountered.

These were due to different tenants, i was using for  calling the auth requests or may be i was using localhost instead of actual reachalbe url of my app.(This one got fixed after using reachable url)

error_description": "Invalid grant type. TransactionI

couldn’t proceed after this and tried lot of things, but didn’t work.I tried with both Partner accounts and general public free accounts.

Embed dynamic JavaScript in angular page using controller injection

To insert a widget, lets take trading view widget if you want to insert javascript widget which needs to  be loaded on a page, you need to inject it inside the html page.

It won’t load if you are inserting the script directly in the html page.

You need to insert it via controller.

look at the controller code

app.controller('ChartsController',function($scope){

$scope.init = function(){

console.log("called on scope init");

$(' <div class="tradingview-widget-copyright"><a href="https://www.tradingview.com/symbols/NSE-ICICIBANK/" rel="noopener" target="_blank"><span class="blue-text">ICICIBANK</span> <span class="blue-text">chart</span> by TradingView</a></div> <script type="text/javascript" src="https://s3.tradingview.com/tv.js"></script> <script type="text/javascript"> new TradingView.widget( { "autosize": true, "symbol": "NSE:ICICIBANK", "interval": "D", "timezone": "Asia/Kolkata", "theme": "Light", "style": "8", "locale": "en", "toolbar_bg": "#f1f3f6", "enable_publishing": false, "withdateranges": true, "hide_side_toolbar": false, "allow_symbol_change": true, "watchlist": [ "NSE:RNAM", "NSE:LUPIN" ], "details": true, "hotlist": true, "calendar": true, "news": [ "stocktwits", "headlines" ], "studies": [ "IchimokuCloud@tv-basicstudies", "BB@tv-basicstudies", "RSI@tv-basicstudies", "Volume@tv-basicstudies" ], "container_id": "tradingview_7e6c5"} ); </script>').appendTo('#tradingview_7e6c5');

}

});

Add a route in the router section of the app.js

when('/charts', {

templateUrl: 'views/charts.html',

controller: 'ChartsController'

}).

Add this  to the chart.html

<div ng-init="init()" class="tradingview-widget-container">

<div id="tradingview_7e6c5"></div>

<div class="tradingview-widget-copyright"><a href="https://www.tradingview.com/symbols/NSE-ICICIBANK/" rel="noopener" target="_blank"><span class="blue-text">ICICIBANK</span> <span class="blue-text">chart</span> by TradingView</a></div>

</div>

You can embed javascript widgets from trading view like this.

nodejs gzip response on chunks are junk

If you are trying to call a request and trying to parse the response and getting

junk characters on the buffer concat of chunks, do this.

Change the header accept encoding.

"accept-encoding": "br" // remove gzip if present

You can check the encoding of response sent via server using this.

console.log('server encoded the data as: ' + (res.headers['content-encoding'] || 'identity'))

once you know what kind of encoding, remove it from the header and fire request again.

instead of pushing chunks to the array you can directly concat the string to the body.

body+=chunk.toString();

// Sample code

sample post code on server.js (node)

app.post('/app/call, function(req,res){

var http = require("https");

var response;

var options = {

"method": "POST",

"hostname": "xx",

"port": null,

"path": "/v3/topics",

"gzip":"true",

"headers": {

"origin": "https://xxi.com",

"accept-encoding": "br",

"accept-language": "en-US,en;q=0.8",

"authorization": "Bearer 7f850f07-xxxxxx",

"content-type": "application/json",

"accept": "*/*",

"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",

"referer": https://sxxxx,

"authority": "xxxxx",

"cache-control": "no-cache",

}

};

var req = http.request(options, function (data) {

var chunks = [];

var body;

data.on("data", function (chunk) {

//console.log('server encoded the data as: ' + (res.headers['content-encoding'] || 'identity'))

chunks.push(chunk);

body+=chunk.toString();

});

data.on("end", function () {

// body = Buffer.concat(chunks);

console.log(body.toString());

response = body.toString();

res.write(response);

res.end();

});

});

req.end();

});

Technorati Tags: junked characters on gzip response via nodejs request module

Run nodejs applications on cloud with all dependencies available on cloud

https://npm.runkit.com

How to post data from angular to nodejs backend and parse response

Make the request in angular

app.js code , client side in our case.

$scope.getTor = function(){ // getTor is bound with ng-click in my html with my controller which has this method.

var torMcItems=$http({ // i want response in this variable.

method: 'GET',

url:'/api/tor' // my backend in server.js

})

.success(function(response){

response = JSON.parse(response);

torItemms=response.items;

}).error(function(err){

}

);

};

///////////////////////////////////////////////////////////

Now server side of the nodejs app , the below method is in server.js.

app.get('/api/tor', function(req, res){ //handle which was called in app.js

var request = require('request');

var options = {

url: 'https://theoldreader.com/reader/api/0/stream/contents?output=json&s=user/-/label/mcrecos',

headers: {

'Authorization':'GoogleLogin auth=emxxxxxxxC'

}

};

let body = [];

request(options).on('data', function(data)

{

//console.log(data.toString());

body.push(data);

}).on('end', function(){

body = Buffer.concat(body).toString();

res.setHeader('Content-Type', 'application/json');

res.write(JSON.stringify(body));

// want to convert the body to string , we will make it json object in the app.js using  JSON.parse

res.end();

});

} );

Unexpected token o in JSON at position

Try out on jslint and see if the json is valid, try to fire a JSON.parse(your string) in the chrome developer tool and see if you get the same error.

If you are using nodejs and this is a request from the server.js to fetch data from external system, you might be getting the data in chunks and you will have to use a buffer to accumulate the body.

Once the body is created you can set response headers and send the body back to the angular.

Use Json.stringify to send it back to client and use JSON.parse(incomingstring) to make the json object out of the string.

server.js =======Json.stringified(body) == going as string to client ====JSON.parse(incoming response)==> use the json object in app.

theoldreader , how to get the api working?

Get the token.

Use windows curl and fire.

curl –k -d "client=YourAppName&accountType=HOSTED_OR_GOOGLE&service=reader&Email=test@krasnoukhov.com&Passwd=..." https://theoldreader.com/accounts/ClientLogin

You will get the token save it for further calls.

To get list of items,fire the below

E:\tools\curl>curl -k  "https://theoldreader.com/reader/api/0/stream/items/ids?output=json&s=user/-/state/com.google/reading-list" -H "Authorization: GoogleLogin auth=emJtzTPu8xyYqxxxxx"

Now loop through the items contains with a for loop and operate on the content of the feeds.

Get all items list for a label(ET) in oldreader

curl -k  "https://theoldreader.com/reader/api/0/stream/items/ids?output=json&s=user/-/label/ET" -H "Authorization: GoogleLogin auth=emJtzTPu8xxx”

Get a specific item with id

curl -k  "https://theoldreader.com/reader/api/0/stream/items/contents?output=json&i=5ad626345f45b7b2c2013aa5" -H "Authorization: GoogleLogin auth=emJtxxxxxxxxxx”

Hope it helps.

How to install openshift origin on laptop locally in 5 minutes

Install vagrant

Run t

$ vagrant init openshift/origin-all-in-one

$ vagrant up --provider=virtualbox

///////////////////

default: You can now access the OpenShift console on: https://10.2.2.2:8443/console
   default:
   default: Configured users are (<username>/<password>):
   default: admin/admin
   default: user/user
   default: But, you can also use any username and password combination you would like to create
   default: a new user.
   default:
   default: You can find links to the client libraries here: https://www.openshift.org/vm
   default: If you have the oc client library on your host, you can also login from your host.
   default:
   default: To use OpenShift CLI, run:
   default: $ oc login https://10.2.2.2:8443

////////////////////////////////

Ssh in to machine

$ vagrant ssh

If you get any cygwin warning from windows, do this.

> set CYGWIN=nodosfilewarning

> vagrant ssh

You will get ssh to the machine.

[root@localhost vagrant]#yum install wget

[root@localhost vagrant]#wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

[root@localhost vagrant]# yum install apache-maven

First time maven on a pom based project will take a long time to update.

kali apt-update slow internet connection

IF you are also struggling with following symptoms, please keep reading.

• Slow apt-get update on kali
• Slow apt-get upgrade on kali rolling repo
• The repository 'http://http.kali.org/kali kali-rolling Release' does not have a Release file.
• Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
• Slow internet connection on kali

you need to install the apt-get https package to enable faster speeds for apt-get

once installed edit /etc/apt/sources.list file to

deb https://http.kali.org/kali kali-rolling main contrib non-free

Try the apt-get update after this speed should go in mega bytes.

Also I found synaptic package manager to fix all my repository issues.

#apt-get install synaptic // will install the package manager, also has provision for fixing broken packages, it fixed a lot of issues my repos.

If you are facing issues with release file not available or repository is not secured and its dangerous to install from unsecured registries. you can simply add the key using the below command.

wget -q -O - archive.kali.org/archive-key.asc | apt-key add

not able to run apt --fix-broken install in kali

if you have multiple versions of a  package and there is a conflict of dependencies on your system.

It might happen that you wont be able to install anything or do an apt-get autoremove or apt-get –f install  or apt-get install any thing.

You can copy the package status file

$cp /var/lib/pkg/status /backup loc

Now $ leafpad /var/lib/dpkg/status

Edit the controversial package, make sure you maintain the syntax else you will get parse error

while running apt-get again from cmdline.

Remove the dependencies for a while, we can do an apt-get update/install later on.

Once you have removed the problematic entry, for example the package or its dependency which was creating the problem.

Run the apt --fix-broken install –f again and apt-get update after this.

Install a package without apt-get on kali

Get the zip/tarball from wget

wget tar/bz2
bzip2 file
tar xzf // will give tar
tar xvf // will give folder
./configure
make
make check
make install

Build whatsapp plugin on wireshark

The old code on GitHub for building whatsapp.so isnt working anymore with the new libwireshark-dev.

You need to clone the git on your kali system using

$git clone https://github.com/davidgfnet/wireshark-whatsapp

mkdir build

cd build

cmake ..

but this will error out because you need to update the CMakefile.txt with the wireshark include dirs and cmake install directories(if you dont have cmake, do an apt-get install cmake and make sure you include the dir in the cmake file)

Sample entries

set(WIRESHARK_INCLUDE_DIRS /usr/include/wireshark)
set(GCRYPT_INCLUDE_DIR /usr/local/include)

Just include them in the CMakeLists.txt top section, make sure Gcrypt is installed)

Good idea to read the readme again on the above git, you need some kind of dependencies before this build can succeed.

There were some modifications required to make this git build work with the newest version of libwireshark-dev, I created a pull request on the git and patched the code.

If the owner accepts my pull you can use the git mentioned above, you can also use my forked one here

$ make install

  P.S Make sure you do apt-get update before doing any thing and include dependencies before running cmake or make.

// Wireshark version 2.4.5 libwireshark-dev 2.4.5

Minishift ( the openshift origin local installation) not starting on windows

To start minishift on windows, you need to have two prereq.

ssh client should be there in path, either cygwin or ssh runtime which comes with windows 10, you can enable it from the features.

also you will need to set the cygwin waring flat either in the command prompt from where you are starting minishift or from the profile. cmd > set CYGWIN=nodosfilewarning // now start mini

cmd> minishift start –vm-driver=virtualbox

This should start the cluster and , minishift status should give you that openshift is running.

You can also try minishift update , it might fix errors you getting.

Also you can set the size of the instances if you are getting space issues while starting minishift, this can be set as a command line parameter while starting minishift. //--disk-size 10g

Troubleshoot:

If you want to recreate the machine, you can delete it from the ~/.minishift/machines/minishift folder and its json and restart minishift, it will pull the openshift 3 again with layers and will start afresh, this is just in case your current config has gone bonkers.

Host private docker registry and connect to it ( Windows version)

If you don’t want to use dockerhub you can host your own private registry.

Get a registry image from docker hub and host it in a container as shown below.

$docker pull registry // also works from toolbelt

I used kitematic, you can dirctly pull from command line also, make sure you map port 5000,

where the docker registry api listens for calls.

so my registry now listening on

Lets connect to it and see the content.

Use curl to fire api call to registry endpoint

$ curl -v -X GET http://192.168.99.102:32768/v2/_catalog
Note: Unnecessary use of -X or --request, GET is already inferred.
* timeout on name lookup is not supported
*   Trying 192.168.99.102...
* Connected to 192.168.99.102 (192.168.99.102) port 32768 (#0)
> GET /v2/_catalog HTTP/1.1
> Host: 192.168.99.102:32768
> User-Agent: curl/7.46.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 30 Mar 2018 14:04:22 GMT
< Content-Length: 20
<
{"repositories":[]}
* Connection #0 to host 192.168.99.102 left intact

I used docker tool belt, i was on windows, it has curl on it.

So i could connect but there are no repositories currently.

Tag the image to puch to local repository

So the way to push a docker image locally to another private registry is by

naming it in such a way that the name has the url port and the image information.

so lets say we have hello-world image in > docker images

To pull it form dockerhub we do

docker run hello-world , it brings the image and run it and saves an image locally.

To push this hello-world image to our private repository , we will tag it like this.

> docker tag hello-world 192.168.99.102:32768/hello-world

Now it will automatically push it to the url  port and the image name specified, this is kind of a

trick to push images from local space to a private registry.

=======================

Lets try to push the image

> docker push 192.168.99.102:32768/hello-world

It ended up with error

 http: server gave HTTP response to HTTPS client

Multiple ways to solve this

If you natively on docker for windows, you can directly specify in preferences.

==============================

If you are using docker on unix, you can try to ssh into the minishift vm and add the daemon.json file in

the /etc/docker/daemon.json and sudo service docker restart(Should work)

// daemon.json file content

{
  "insecure-registries" : ["registry:5000"]
}

^{Tip *To ssh in to minishift vm or any docker vm running via tool belt on windows,you can do this}

^{locate the id_rsa for that machine in the user profile folder and fire a ssh.}

^//

ssh -i ~/.minishift/machines/minishift/id_rsa docker@url

=============================

If you are on windows with toolbelt, you can ssh into your boot2docker vm and update the /var/lib/boot2docker/profile  and add docker opts for insecure registry as shown below

DOCKER_OPTS="--insecure-registry url:5000"

restart docker service and test again by pushing the image.

You should be able to push your image now.

Ref url

you can test your repo availability by  

$curl url:Port/v2/_ping

$curl url:Port/v2/_catalog

Please not , you can also pull repository image in same docker environment and refer to the

registry container on localhost:5000 , that way you should be able to push it easily, pushing to external ip could have this allow unsecure access issues, locally it should work just fine. I fired up a vagrant box for openshift and i could pull a registry image , run a registry locally and push on localhost:5000 just like that, hey vagrant is awsome.. I just started using it.

Play an mp3 file on raspberry pi remotely, use it as an alarm or to call some one

Get the apache2 and php from here

https://www.raspberrypi.org/documentation/remote-access/web-server/apache.md

Create an index.php and place it in root, create the mp3 file and sh file which

will be played by the php, you can use mp3 recorder online to create a custom message

or alarm sound.

Use this link to create php file and I used omxplayer isntead of mpg123 mentioned, i got pi3

https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=72417

Well i could open this local ip of my pi and it played my sound remotely through a webpage,

I saved a bookmark on my android phone on home screen to use it handy.

Tip* You could also test your script and mp3 locally by running in shell before inserting it in php page.

I cam going to use it frequently to call my wife, she doesn't check her phone often and I am

too lazy to move from my couch.

In case you get some weird vchiq error  

* failed to open vchiq instance fix this here 

Oh and yes, i did a mac/ip binding on my dlink router so that my pi always picks up same

address, if you want to use it from outside home(remotely) you can do a port forward also.

Send http post webhook to Iftt maker channel service using motion library on Raspberry pi

In case you are using motion on raspberry pi and want to make a notification on

your android phone whenever some motion is detected on camera, please read this.

Go to ifttt and login and create a new applet , in this part select webhook 

enter the event name , ilke the front_door_movement and select a notification in that

part, for simple android notification on the phone select the ifttt notification (simple one)

Now go to settings and know your api key for this webhook.

The trigger url for webhook will look something like this.

ifttt.py  // run apt-get update & apt-get install python-requests if you dont have requests module on pi

////////

import requests

requests.post("https://maker.ifttt.com/trigger/front_door_movement/with/key/{yourkey}", data="")

////////////

Go to /etc/motion/motion.conf and locate on_event_start event 

on_event_start  python /home/pi/ifttt.py

uncomment and add this trigger, make sure python file is executable by running chmod +x

try running file first to see if ifttt hook is setup properly.

Now whenever the motion is detected on the camera, this will send an android notification from the ifttt app.

Tested and it works perfectly.

Openshift Basics

Openshift is a PAAS solution by Redhat, platform as a service.

Openshift uses kubernetes and docker technology to provision containers.

Openshift is a wrapper around docker and kubernetes 

Openshift can be installed locally using minishift.

To run an  application you will need the following :

Project: A project need to be created for any app to run, it has its own namespace where resources will reside.

Route: To expose the service to outside world.

Pod: The node where containers run , one pod can have multiple containers.

ImageStream: is a link to the image from local registry or dockerhub 

YAML: The meta information for the resources , configuration file for resources

Services: The app hosted on pod can be termed as service.

Once the image is deployed either from docker repo or external registry, it is available in local repo as well 

Internal docker repo will have an assigned repo ip from the cluster.

Db pods usually share the gluster/ceph storage as shared SAN, they also have replication provision to achieve HA

Services running on one node can reference other services on another node of the cluster with the spec names.

Run a docker image on Openshift training environment from dockerhub

Get the client VM and the shared environment access, we will use client VM only to connect to the Openshift environment.

Note. IF you dont use shared environment access, you won't see an option to create the project in the console.

Create a new project from the console and search the image from the deploy image option.

Create the image in the openshift environment.

Create the route and map the application port, check route YAML for more information.

Access the application , make sure you use it directly and don't specify the port.

E.g if you bound 8001-->8001 of container try the http://routeurl:contextroot only 

How to import and tag an image from docker hub

If you want to craete an image stream and generate tag(latest) from the docker hub image,

you can use this YAML to create the image stream config and later run below command

STep 1 

Run YAML on openshift - -add to project 

//

kind: List  apiVersion: v1  items:    - kind: ImageStream    apiVersion: v1    metadata:      labels:        app: rhel7-weblogic      name: rhel7-weblogic    spec: {}

AFter this you can run this 

oc import-image rhel7-weblogic  --from=dockerusername/rhel7-weblogic --confirm

It will generate the latest tag and bind it to the dockerhub registry , you wont see failed builds in case yo u

have refered this imagestream in any of your other YAML in openshift 

How to Find users on a linux system ?

 cut -d: -f1 /etc/passwd

How to push an images to docker hub ?

$sudo docker images   //  will list the images 

$ docker tag imagename docker.io/<DockerHub user name>/imagename:latest

$ docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.

Username: <Dockerhub user name>

Password:

Login Succeeded

$ docker push docker.io/<DockerHub user name>/imagename:latest

will start pushin 

5cc248435d3e: Pushing [===>                                               ] 47.67 MB/635.9 MB

91f90aafa90e: Pushed

f2da66f301b3: Pushed

31239d164b59: Pushing [==================================================>] 219.1 MB

5f70bf18a086: Pushed

510c99acb643: Pushing [===========>                                       ] 81.45 MB/360 MB

82d954ed02ea: Pushing [==========================>                        ] 104.6 MB/197.2 MB

06a934ef2bcb: Pushing [=================>                                 ] 4.739 MB/13.72 MB

How to generate a public key on windows system

Go to puttygen and generate a ssh-2rsa key 2048 length and generate some randomness and use it

to access linux systems in open labs.

Used it for redhat OpenTLC.

for Private key generate one from actions menu in putty gen and save it in a folder.


Now to connect go to putty.exe -- connections--auth -- browse to private file saved above.

Now go to sessions and connect to you host, it will let you connect the environment.