Thursday, November 23, 2017

TensorFlow - Getting Started with Docker Container and Jupyter Notebook

I'm studying Machine Learning and would like to share some intro experience working with TensorFlow. To get started with TensorFlow you need to install it, easiest way (at least for me) was to run TensorFlow using Docker. Read installation instructions - Installing TensorFlow.

Once TensorFlow Docker image is installed. I suggest to create container in detach mode (--detach=true) and provide port for Jupyter UI. Make sure to provide meaningful name for Docker container:

docker run --detach=true --name RedSamuraiTensorFlowUI -it -p 8888:8888 gcr.io/tensorflow/tensorflow

Make sure to start and stop container using Docker start/stop commands, don't run and create container each time (this way you will loose your work, since new container will be created each time):

docker start RedSamuraiTensorFlowUI (docker stop RedSamuraiTensorFlowUI)

Once container is running in detached mode, you can access logs by executing docker logs command and specifying container name:

docker logs -f RedSamuraiTensorFlowUI

At this point you should see output in Docker container log, copy URL to Jupyter UI with token and paste it to the browser (for example: http://localhost:8888/?token=d0f617a4c719c40ea39a3732447d67fd40ff2028bb335823):


This will give you access to Jupyter UI. Is possible to run TensorFlow Python scripts directly through command line in Docker environment, but is more convenient to do the same through UI:


UI gives option to create new Terminal session:


Terminal allows to run Python code using command line:


Instead of using command line, more convenient is to create new notebook:


Notebook environment allows to type in Python code and execute math calculations. In the example below I multiply two arrays (1x5, 2x6, 3x7, 4x8) in Python code through TensorFlow library. Result is printed through TensorFlow session object right below and prompt for the next command is displayed - very convenient:


Jupyter UI allows to track running notebooks and terminals:


Whatever action you do in Jupyter UI, it can be tracked using log printed in Docker container log. Jupyter UI is client side JS application:


To double check Docker config, I have TensorFlow Docker image:


And Docker container, which can be started/stopped by name (see command listed above), without running new Docker container every time during restart:

Sunday, November 19, 2017

How to Setup Node.js and Oracle JET on Oracle Linux

What if you want to develop Oracle JET in Oracle Linux? Certainly this is possible - both Node.js and Oracle JET run on Oracle Linux or any other Linux distribution. If you follow Oracle JET Setup Guide, you will see Node.js is listed as prerequisite. But it may not be exactly straightforward to install Node.js on Oracle Linux. Below I will guide you through the steps.

Run command to install development tools to build native add-ons to be installed:

yum install -y gcc-c++ make


Enable Node.js yum repository:

curl -sL https://rpm.nodesource.com/setup_8.x | sudo -E bash -


Install Node.js:

sudo yum install nodejs


Verify if node and npm was installed successfully:


Next you can follow steps described in Oracle JET setup guide, all straightforward. Install Oracle JET:

sudo npm -g install @oracle/ojet-cli


Create new JET application with OJET utility:

sudo ojet create jetwinapp --template=navdrawer


Run application with OJET utility:

sudo ojet serve


JET application runs on Oracle Linux:

Friday, November 17, 2017

ADF Performance on Docker - Lighting Fast

ADF performance depends on server processing power. Sometimes ADF is blamed for poor performance, but in most of the cases real issue is related to poor server hardware, bad programming style or slow response from DB. Goal of this post is to show how fast ADF request could execute and give away couple of suggestions how to minimize ADF request time. This would apply to ADF application running on any environment, not only Docker. I'm using ADF Alta UI based list application with edit fragment.

Rule number one - enable response compression. This will allow to transfer less data and obviously response will execute faster - shorter content download time. See in the screenshot below - JS file is compressed to 87 KB from original 411 KB. Initial page load in ADF generates around 3 MB of content (if this is very first access and static content is not cached yet on client side). With compression initial load of 3 MB will be around 300 - 400 KB. Thats a big difference. In this example ADF page opens in 1.2 seconds (this is equal to client side JS applications, if static content is downloaded on first access):


You can enable content response compression in WebLogic console (will be applied for all deployed Web apps). Go to domain configuration, Web Applications section:


Select checkbox to enable GZIP compression and provide a list of content types to be compressed:


Thats it - content compression is set.

When I navigate to edit fragment - request is executed in 305 ms. Thanks to fast Docker engine (running on Digital Ocean - Oracle ADF on Docker Container) and content response compression: 3.44 KB transferred for 14.49 KB original content:


Let's try Save operation. I changed Hire Date attribute and then pressed Save button. This will trigger Commit operation in ADF, push data to ADF BC and then execute DML statement with commit in DB. All these steps are completed in 113 ms.


Don't believe anyone who says ADF is slow. As you can see, ADF request is very fast fundamentally - but of course it can become slow, if you add a lot of data fetch and processing logic on top (blame yourself). Client side JS application would not run faster, if it would call backend REST service to save data. The only advantage of JS client side application in this case would be that it executes backend REST call asynchronously, while ADF calls requests in synchronous manner. However, it all depends - sometimes asynchronous calls are not suitable for business logic either.

How come ADF BC call to DB completes so fast? For that we need to check Data Source Connection Delay Time on WLS. In Docker (Digital Ocean) environment it is ridiculously short (thats very good): 66 ms. Check same on your server (go to Data Source monitoring in WLS console), longer delay time means slower response from DB and slower ADF performance:


Navigation back to the list runs in 356 ms, with 197.96 KB of content compressed to 10.47 KB. This is very fast, 350 ms response time is something that user would not notice (almost equal to processing on client side):


To optimize ADF performance, make sure you are using ChangeEventPolicy = NONE for iterators in Page Definitions:

Friday, November 10, 2017

ADF Performance Story - This Time Developer Was Wrong

ADF is fast. If ADF application is slow, most likely this is related to development mistakes. I would like to tell you one story, based on my ADF tuning experience. Problem description: ADF application runs fast in DEV, when DB size is small. Same application runs slow in TEST/PROD, when DB size is large. Question - what is slow. Answer - slow means forms are loading slow. Ok, lets go to the story.

Developer decides to fetch data from DB and iterate over rows to process them. This is already bad practice, because it is much more effective to process large sets of rows in DB directly, without fetching to middle tier. But let's assume this is valid use case and we really need to fetch rows. Developer implements fetching using getAllRowsInRange method:


VO data is loaded on UI and displayed in the table. Method to iterate through rows is called from button, this logic is not executed on initial load. Iterator is set with Range Size = 10:


This means first 10 rows are fetched on form load and it will open fast no matter of DB size:


This is OK, but while testing row fetching functionality - developer finds a "bug". He will see that instead of fetching all rows in custom method, only 10 rows are returned. And then developer decides to implement a fix - change Range Size to -1:


Now all rows are fetched in custom method, developer is happy. But there is small side effect - for some reason table starts to display all rows. Not good, another fix is needed:


Auto Height Rows = 10 is set to prevent table displaying too many rows. But really still all rows will be fetched, because Range Size = -1:


All these fixes are wrong. Method getAllRowsInRange is not supposed to be used to iterate through all rows, it will return only currently fetched rows. Such implementation obviously will slow down form load functionality, it will fetch all rows from DB, if DB size is large - it will slow down significantly.

If all what you need is to iterate through rows, make sure you dont affect data which is displayed/fetched for UI. Keep Range Size positive:


In custom method iterate through rows by creating rowset iterator:


Download sample application - ADFRangeSizeApp.zip.

Sunday, November 5, 2017

Essential WebLogic Tuning to Run on Docker and Avoid OOM

Read my previous post about how to run ADF on Docker - Oracle ADF on Docker Container. Docker WebLogic image is based on official Oracle Docker image for FMW infrastructure - OracleFMWInfrastructure. WebLogic container created based on this image runs, but not for long - eventually JVM process eats up all memory and OOM (out of memory) exception is thrown. This is known issue related to JVM running in Docker container - Running a JVM in a Container Without Getting Killed. Good news - we can switch on WebLogic memory management functionality to prevent OOM error while running in Docker container. This WebLogic functionality is turned on with special flag -XX:+ResourceManagement. To set this flag, we need to update startWebLogic.sh script, but probably we dont want to rebuild Docker container. Read below how to achieve this.

First we need to access startWebLogic.sh script from Docker container. Make sure Docker container on your host is running and execute Docker copy command:

docker cp RedSamuraiWLS:/u01/oracle/user_projects/domains/InfraDomain/bin/startWebLogic.sh /Users/andrejusbaranovskis/infra/shared

This will copy startWebLogic.sh file from Docker container to your host system.

Search in startWebLogic.sh script content and search for resource management config. By default it is commented out. Set this string for JAVA_OPTIONS. This enables WebLogic resource management and G1GC garbage collector:

JAVA_OPTIONS="-XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC ${SAVE_JAVA_OPTIONS}"

startWebLogic.sh script contains comment, where it recommends to enable this option:


Once JAVA_OPTIONS variable is updated, copy startWebLogic.sh script back to Docker container:

docker cp /Users/andrejusbaranovskis/infra/shared/startWebLogic.sh RedSamuraiWLS:/u01/

Enter into Docker container command prompt (in my case user 501 is root user for Docker container):

docker exec -u 501 -it RedSamuraiWLS bash

Change file permissions for startWebLogic.sh:

chmod 777 startWebLogic.sh

Enter into Docker container as oracle user:

docker exec -it RedSamuraiWLS bash

Copy startWebLogic.sh script from u01 into bin folder (overwrite existing script file):

cp startWebLogic.sh /u01/oracle/user_projects/domains/InfraDomain/bin

Stop Docker container and run docker commit to create new image (which includes change in startWebLogic.sh):

docker commit RedSamuraiWLS abaranovskis/redsamurai-wls:v2

Docker image is created with delta change only, this allows to save space. Run docker images command to verify if new image is created successfully:


Run docker push to upload new image version into Docker repository. Upload will happen fast, because it will upload only delta of changes:

docker push abaranovskis/redsamurai-wls:v2

You should see new image version uploaded into Docker repository:


To run container online, we can login into Digital Ocean console and execute docker run command (I'm using container memory limit -m 4g (4 GB)) -  it will pull and run new image:


Once docker container is running, execute top command in Digital Ocean console to monitor memory consumption. Java process memory consumption should not grow, if there is no user activity in WebLogic server:

Friday, November 3, 2017

Oracle ADF on Docker Container

Want to run Oracle ADF on Docker? This is possible, I will explain how. If you are new to Docker, it may require to spend significant amount of time to get started with all different bits and pieces. I will try to explain all essential steps, so that you will get up to speed quickly.

First of all you need to have DB accessible, check my previous post explaining how to run Oracle DB on Docker - Oracle Database Docker Image in Docker Cloud (Digital Ocean). DB is required to install RCU schema for WebLogic installation with JRF files.

I have built my own Oracle Fusion Middleware Docker image using Oracle Docker images - Oracle Fusion Middleware Infrastructure on Docker.

First step is to build Oracle JDK (Server JRE) image, this is pre-requisite to build Oracle Fusion Middleware Docker image. Read through instructions documented on Oracle Fusion Middleware Infrastructure on Docker GitHub page. You should navigate to Oracle Java folder (download Oracle Docker files from GitHub link mentioned above) and copy there JDK installation file:


Run command to create JDK Docker image:

./build.sh

Command output:


Double check to verify if image was created successfully by running docker images command:


Let's move on to Oracle FMW image creation. Navigate to Oracle FMW folder and copy FMW infrastructure installation file (I'm installing 12.2.1.3):


Move one folder up and run command:

./buildDockerImage.sh -s -v 12.2.1.3

To build Oracle FMW image. I use flag -s to skip checksum verification for installation file. You should run command from this folder:


You will see long output in the log for this command:


It installs WLS into Docker image:


Run docker images command to verify if image was created successfully:


In the next step, we will create FMW domain and extend it with ADF support. But before that we need to make sure DB details are set correctly, to be able to install RCU schema. Oracle provides infraDomain file with DB and WLS properties, make sure to set correct DB details. If properties are not correct, RCU creation will fail:


Execute docker run command to startup WLS Docker container. During first start up it will create and extend WLS domain with ADF support:

docker run -d -p 7001:7001 --name RedSamuraiWLS --env-file ./infraDomain.env.list oracle/fmw-infrastructure:12.2.1.3

Flag -d means container will run in detached mode and we will be able to return to command prompt. Port with name is specified along with environment properties file. Make sure to reference FMW image which was created in the step above. Once control is returned back to the prompt, run docker command to check status of docker container (flag -a means to show all containers):

docker ps -a

Container should be in the running state. First startup takes longer, because it requires to setup and extend WLS domain:


Once domain is extended, you will see WebLogic starting:


Finally WebLogic should be in Running state:


Run again docker ps -a command to verify container state, it should be up and running:


Once WLS machine is up, you can navigate to Enterprise Manager through URL from outside of Docker container, for example from your host. Login to EM and you will see Admin server is up, but Managed Server is down. There is a way to startup Managed Server too, but if you want to run ADF apps for DEV environment, realistically speaking Admin server is more than enough for deployment too:


Simply delete (this cab done from EM) Managed Server and cluster, keep only Admin Server:


I have deployed sample ADF application:


This application is based on ADF BC, data source is defined too:


ADF application runs from WebLogic on Docker:


Now lets see how to push newly created container to Docker registry.

First we need to create new Docker image from Docker container. This can be done with docker commit command (pointing to container ID and specifying Docker repository name and tag):

docker commit da03e52b42a2 abaranovskis/redsamurai-wls:v1

Run docker images command to verify new image is created successfully. Next run docker login to authenticate with Docker repository. Run docker push to write image to Docker repository:

docker push abaranovskis/redsamurai-wls:v1

Commands execution sequence:


Pushed image should appear in docker repository:


Once image is in Docker online repository, we can startup online Docker container, so that WLS will be accessible online. This can be done through command line or using Docker Cloud UI interface. You can create new container by referencing image from Docker repository:


Our WLS docker container with ADF support runs on Digital Ocean:


Logs are accessible from Docker Cloud UI and you can see server status: