Heroku Custom Trust Store for SSL Handshake

 

Working with Heroku for deploying apps (java, nodejs, etc..) is made very easy but while integrating one of the service hosted on AWS, I was getting SSLHandshakeException and could not figure it out initially how to resolve the issue.

Exception while making a request to the target system from the app hosted on Heroku:

      

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid 
certification path to requested target
      
      

Heroku allows us to customise the JDK. To Add a custom trust store

1) Download the base keystore (cacert) from following link 

download the base Heroku keystore

2) Rename downloaded file to cacerts (heroku_cacerts to cacerts)

3) Add the certificate (.pem) file of the target system by executing the following 

  keytool -import -keystore /Users/dsinghvi/Downloads/cacerts -file
  /Users/dsinghvi/Downloads/target_system_certifacte.pem
  

4) Add the cacerts file file to you project as follows (in the folder <PROJET_DIR_PATH>/.jdk-overlay/jre/lib/security/)

5) Git add, commit and push to Heroku app

git add .jdk-overlay/jre/lib/security/cacerts
git commit -m "custom trust store for certificate issue"
git push heroku master

5) Optionally verify if cacerts has been updated

Added cacerts would be overridden in the /app/.jdk/jre/lib/security (jdk1.8) and /app/.jdk/lib/security (JDK 1.9 and above)

This can be verified in the app deployed using:

heroku run bash --app <APP_NAME>
    

5) Optionally if you want to verify whether trust store has been updated or not 

Before importing target system certificate

keytool -list -v -keystore /Users/dsinghvi/Downloads/cacerts > cert1.txt
    

After importing target system certificate

keytool -list -v -keystore /Users/dsinghvi/Downloads/cacerts > cert2.txt
    

You would find the difference between cert1.txt and cert2.txt as follows:

Happy Coding

Impact of Covid for Supply Chain

 I was sharing my thoughts on the impact of Covid-19 and the new opportunity in supply chain domain.

DB Multi-Tenancy Modelling for Microservice

Multi-Tenancy with the SaaS based model does not bring any change on the application layer as multiple services deployment of application would be able to accommodate the load on application but it would matter how do we decide the modelling of the database w.r.t. Multi-Tenancy.

Following are the factors to be considered for decision making for multi-tenancy:

Cost Cost per tenants if separate DB/SCHEMA is selected per tenants.

Performance Application performance & scalability would impact based on the number of tenants, data size per tenants and the workload.

Time Development & Release migration efforts when multiple databases/schemas are in production. Complexity for managing the operations for maintaining multiple DBs.



Tenancy Modelling

Database per Tenant Single Tenant Architecture where each tenant has its own database and is isolated from other tenants.

Shared Database Multi Tenant Architecture with separate schema where all tenants share a database, but have their own database schemas and tables.

Shared Database and Shared Schema Multi Tenant Architecture with all tenants share a database and tables. Every table has a Column with the Tenant Identifier, that shows the tenant of the row.


Various multitenancy strategy supported by Hibernate are

public enum MultiTenancyStrategy {
  DISCRIMINATOR,
  SCHEMA,
  DATABASE,
  NONE;
}

DISCRIMINATOR strategy work is in progress and from the documentation of hibernate DISCRIMINATOR strategy Correlates to the partitioned (discriminator) approach. It is an error to attempt to open a session without a tenant identifier using this strategy. 

This strategy is not yet implemented and you can follow its progress via the HHH-6054 Jira issue. 

JIRA HHH-6054 is currently in OPEN Status.   

Here in the example following modules/projects shows the example to achieve the tenancy modelling using SCHEMA (shared db, separate schema) strategy per tenant based approach and SCHEMA (shared db, shared schema) strategy for different tenants.

multitenancy-op2 project shows the example with the multiple tenants share the same database but different schemas. 

multitenancy-op3 project shows the example of the hybrid approach of with the tenants are grouped against the specific database and schema. This would give an ability to map a specific tenant with single database as well when the high load is expected for a tenant in comparison to others. 

Project multitenancy-op2 UseCase is where database is same (dbA) and every TENANT is using specific Schema i.e. Tenant wise DB and Schema mapping. 

TENANTID	DB	SCHEMA
TENANT1	dbA	SCHEMA1
TENANT2	dbA	SCHEMA2

Build and run multitenancy-op2 project

cd multitenancy-op2
mvn clean package
mvn spring-boot:run

Flyway integration would be creating necessary tables in database dbA 

 Catalog creation request for TENANT1 which would insert data into table SCHEMA1.CATALOGTABLE

curl -X POST \
  http://localhost:8080/catalog \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT1' \
  -d '{"tenantId":"TENANT1","catalogName":"Catalog1","supplierId":"supplier1","source":"X-Systems"}'

Catalog creation request for TENANT2 which would insert data into table SCHEMA2.CATALOGTABLE

curl -X POST \
  http://localhost:8080/catalog \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT2' \
  -d '{"tenantId":"TENANT2","catalogName":"Catalog2","supplierId":"supplier2","source":"Y-Systems"}'

Catalog retrieval request for TENANT1

curl -X GET \
  'http://localhost:8080/catalog?catalogId=1' \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT1'

Catalog retrieval request for TENANT2

curl -X GET \
  'http://localhost:8080/catalog?catalogId=2' \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT2'

Project multitenancy-op3 UseCase is as the hybrid approach to accommodate following cases: 

1. Tenant(s) with higher load using the dedicated database and single schema (TENANT1). 
2. Tenant(s) with moderate load can use same database but different schema (approach in multenancy-op2 project). 
3. Tenant(s) with lower loads can share the DB SCHEMA (TENANT2 and TENANT3). Tenant wise DB and Schema mapping.

 

TENANTID	DB	SCHEMA
TENANT1	dbA	SCHEMA1
TENANT2	dbB	SHARED
TENANT3	dbB	SHARED

Build and run multitenancy-op3 project

cd multitenancy-op3
mvn clean package
mvn spring-boot:run

Create the following schema manually

create schema if not exists SHARED;
create table SHARED.catalogtable(id bigint, tenantid varchar(10), catalogname varchar(30), supplierid varchar(30), source varchar(10));
CREATE SEQUENCE SHARED.hibernate_sequence START WITH 1 INCREMENT BY 1 NO MINVALUE NO MAXVALUE CACHE 1;

Sample request for catalog creation
Catalog creation request for TENANT1 which would insert data into table SCHEMA1.CATALOGTABLE in database dbA

curl -X POST \
  http://localhost:8080/catalog \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT1' \
  -d '{"tenantId":"TENANT1","catalogName":"Catalog1","supplierId":"supplier1","source":"X-Systems"}'

Catalog creation request for TENANT2 which would insert data into table SHARED.CATALOGTABLE in database dbB

curl -X POST \
  http://localhost:8080/catalog \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT2' \
  -d '{"tenantId":"TENANT2","catalogName":"Catalog2","supplierId":"supplier2","source":"X-Systems"}'

Catalog creation request for TENANT3 which would insert data into table SHARED.CATALOGTABLE in database dbB

curl -X POST \
  http://localhost:8080/catalog \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT3' \
  -d '{"tenantId":"TENANT3","catalogName":"Catalog3","supplierId":"supplier3","source":"Y-Systems"}'

Inserted records can be verified at http://localhost:8080/h2-console/ by logging into 

use database dbA (jdbc:h2:mem:dbA) for TENANT1 in schema SCHEMA1  

 use database dbB (jdbc:h2:mem:dbB) for TENANT2 and TENANT3 in schema SHARED

Sample request for catalog retrieval

curl -X GET \
  'http://localhost:8080/catalog?catalogId=1' \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT1'

Catalog retrieval request for TENANT2

curl -X GET \
  'http://localhost:8080/catalog?catalogId=1' \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT2'

Catalog retrieval request for TENANT3

curl -X GET \
  'http://localhost:8080/catalog?catalogId=2' \
  -H 'content-type: application/json' \
  -H 'tenantid: TENANT3'

Download/Refer code from github repository.

Smart Light using Arduino

While working on the regular projects in office, I have joined SAP Ariba IOT Group where we try out few things which can solve some problem which could be really helpful.

I am also trying to use Raspberry Pi, Arduino and other platforms. 

Recently i have tried to build a smart light, which I want to fix it to my study table now. :-)

Circuit:
I have picked the circuit from internet which explains about the connectivity of different components.

Circuit Diagram

Demo Video:
Here is a short video about it.



Hardware Components:

Arduino Uno	1
Relay Module	1
PIR Motion Detector Sensor Module	1
Connectors/Jumper Wires	6
Power Cable	1
Power adaptor for Arduino Uno	1
Light	1

Software Components:

Arduino IDE

1

Smart Light Code

Code:



Troubleshoot:
While uploading the code there was an issue with PORT( /dev/ttyACM on ubuntu) read/write permission
Following commands helped:

deepak@deepak-B570:$ ls -l /dev/ttyACM*
crw-rw---- 1 root dialout 166, 0 May 12 19:21 /dev/ttyACM0

deepak@deepak-B570:$ ls -l /dev/ttyACM*
crw-rw---- 1 root dialout 166, 0 May 12 19:21 /dev/ttyACM0

deepak@deepak-B570:$ sudo chmod a+rw /dev/ttyACM0

[sudo] password for deepak: ********

Microservices for better scale

    

I was reading some article and which triggered the question in my mind w.r.t business services particularly "Microservices":

• What do I need from CAP theorem, should System be AP system  (consul, eureka, etc) or CP system (zookeeper, etcd, etc). How to decide about it?
• Figure out how to run, manage, and monitor these systems at scale. How to plan for it?

Some of the points were answered once I built a on the small microservice demo.

Following slide would help:

Microservices forscale from Deepak Singhvi

Download source code from github.

Eureka (Discovery Server/Service):
Eureka developed by Netflix is a REST based service that was primarily used by them in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers.

Eureka also comes with a Java-based client component,the Eureka Client, which makes interactions with the service much easier. The client also has a built-in load balancer that does basic round-robin load balancing.

Zuul (Gateway/Proxy and Load Balancer):
Zuul is a JVM based router and server side load balancer by Netflix. And Spring Cloud has a nice integration with an embedded Zuul proxy.
There are many usages of Zuul I found following would be very helpful:

• Authentication
• Dynamic Routing
• Service Migration
• Load Shedding
• Security

For proxyservice have a service and enable zuul proxy @EnableZuulProxy and define the routes.



Routing to be configured in configuration file:


Any request which catalogservice would be routed to serviceId catalogservice which in this case is catalogservice registered with Eureka.

Spring Cloud has created an embedded Zuul proxy to ease the development of a very common use case where a UI application wants to proxy calls to one or more back end services. This feature is useful for a user interface to proxy to the backend services it requires, avoiding the need to manage CORS and authentication concerns independently for all the backends.

There are few pre created filters and custom filter (see PreFilter.java) can also be created easily.

Ribbon (Load Balancer):
Ribbon is a client side load balancer which gives you a lot of control over the behaviour of HTTP and TCP clients. Feign already uses Ribbon

Feign (Web Service Client):
Feing offers a solution to dynamically generate clients from the interface. Additional benefit is that we can keep the signature of the service and client identical. Just declare an interface request mappings :




Above example would be invoking the catalogservice's (CatalogController) getItems().

The Ribbon client above would discover the physical addresses for the "catalogservice" service. If application is a Eureka client then it will resolve the service in the Eureka service registry. If Eureka is not used, than  configuring a list of servers in your external configuration would also help ribbon to find and load balance appropriately.

:-)

Visualisation using d3.js based Sunburst with Apache Zeppelin

Zeppelin provides few default visual components (pie, bar, stacked, area, line chart, etc).
If users want either they can add a new default component or create visualisation using AngularJS interpreter.

I tried to create d3 based Sunburst for preparing a report in Apache Zeppelin.

It is easy and quick.

Apache Zeppelin display system adds additional div(s) and which creates some blank area on the screen. 

You can experience this as there is a blank area between sunburst and breadcrumbs in the bottom.

********






Notebook for the above visualisation is available here which can be imported into Apache Zeppelin.
This contains the AngularJS source for sunburst visual.


Content which I downloaded from NSE historical data section and transformed it for demo purpose i.e. nsecombinedreport.csv can be downloaded from here.
This report is for the various Instrument Type, Security and the amount traded for a day.

Focusing on implementing govt policies using the big data tool zeppelin

It was good to know from the goverment that it published lots of data collected over the period of time at https://data.gov.in/

I picked and amenities data about the villages from https://data.gov.in/catalog/village-amenities-census-2011 to do some analysis.

I believe govterment is doing sufficient analysis to find where and with what force it should use its machinery to promote its schemes.

I have been doing some analysis using the Apache Spark and eco system around it. But was interested in a quick visualization, which would help to understand the data quickly. A possible use would be using R as I wanted to build the reports quickly. I explored some of the capabilities of R and Shiny App in my earlier post of Custer Analysis of banking data.

Recently I came to know about a fantastic tool, its a web based notebook, with the in-built support for Apache-Spark, with a support of multiple langues like Scala, Python, spark sql and so on and most important that this it is opensource.

"Zeppelin" 

I picked one of the csv from the the whole data, and which is for one of the district in Karnataka state is Gulbarga and started doing some analysis.

Loading the data into the dataframe/table


It is easy to accomodate spark sql also in the notebook paragraph/sections.
Following is a very simple query to show the population spread in the villages of Gulbarga district.

Goverment make policies and spend money on that, and find the effectiveness of it based on the result. We can use the collected data to understand where should be the maximum penetration of the schemes, i.e. find the villages which needs the goverment schemes most. One of the example where goverment can initiates its policies to reduce the gap of male-female ratio, we can understand from the data available, where should be the more focus.

Changed the minbenchmark to 80% and same got updated on the fly

I stated to analyse this data to check for the education facilities in the villages which is in progress, would be publishing that information in later posts.

Installation details:
a) Zeppelin was deployed on Ubuntu VirtualBox with Windows as host.
b) Set your java home (1.7) before starting Zeppelin.
c) To start execute 'zeppelin-daemon.sh start' in the ZEPPELIN_HOME\bin