The Algorithms of Artificial Intelligent

I'm new to Python programming and - actually - it just was started in the middle of 2020 for a specific corporate purpose where I'd worked at. So, I've been learned for couple months to explore what is what about Python, and - since there's no best fit guidance about machine learning methodologies - here I summarized about 17 common algorithms I found.

A. Supervised Learning

Split into 2 methods:

  1. Regression
    • Linear
    • Logistic
    • Polynomial
  2. Classification
    • K-Nearest Neighbors (KNN)
    • Decision Tree (DT)
    • Naive Bayes (NB)
    • Support Vector Machine (SVM)

B. Unsupervised Learning 

Split into 3 methods with 2 models (ML & DL):

  • Machine Learning
    1. Clustering
      • K-Means
      • Hierarchical Clustering
      • T-SNE Clustering
      • DBScan
    2. Dimension Reduction
      • Principal Component Analysis
      • Anomaly Detection
      • Auto-Encoder
      • Hebbian Learning
  • Deep Learning
    1. Generative Models
      • Generative Adversarial Network
      • Self Organizing Maps

I wrapped this post on a video I published in Youtube:

Thank you for your reading & subs. I'll update this post as soon as I found any of new algorithms.

Labels: ,

  Post a Comment

Importing Various CSV Datasource in Python

From previous post (https://paparadit.blogspot.com/2020/11/python-common-libraries.html), we learn how to deal with libraries in Python. On this post, I'm going to show you how to import CSV data using Python. I'll update this post as soon as I discover another way how to do it:

1. CSV from local computer

import pandas as pd
from google.colab import files
uploaded = files.upload()
import io
vlog96 = pd.read_csv(io.BytesIO(uploaded['vlog96.csv']))
vlog96.head()

2. CSV from GitHub

import pandas as pd
url = 'https://raw.githubusercontent.com/kokocamp/vlog96/master/vlog96.csv'
vlog96 = pd.read_csv(url)
vlog96.head()

You can see my video below to make a practice on how to import CSV files from local computer and GitHub.


3. CSV from Google Drive (#1)

The long way:

 import pandas as pd

# Code to read csv file into Colaboratory:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

link = 'https://drive.google.com/open?id=1-2lhLcwe9QNT8okL69QL_tV-O4BN9um6'
fluff, id = link.split('=')
print (id)
downloaded = drive.CreateFile({'id':id})
downloaded.GetContentFile('vlog98.csv')
vlog98 = pd.read_csv('vlog98.csv')
vlog98.head()

4. CSV from Google Drive (#2)

The simple way: 

import pandas as pd

from google.colab import drive
drive.mount('/content/drive')

path = '/content/drive/My Drive/data/vlog96.csv'
vlog96 = pd.read_csv(path)
vlog96.head()

I wrapped both way on a video below: 


For GitHub resource, you can use CSV from my account (https://github.com/kokocamp) on or you can make it your own.

Labels: ,

  Post a Comment

Python Common Libraries

Months ago, I wrote about my first blog post about Python (https://paparadit.blogspot.com/2020/07/python-for-dummies.html) - as technically I curious about how it works as a machine learning tools. On that link, you can learn (a very) basic Python codes including: (1) How to print out on screen? (2) Learn about variable assignment and data type, (3) Sequences operation, (4) Math operation, (5) Ask for input, (6) Branches, (7) Looping, (8) Array, (9) Quit from looping Function and that's all is a good start for you to learn Python syntax structures. I wrapped it all on 2 video series below.


 

On current post, I'm going to explain you about Python common libraries I used based on my experiment to create proof-of-concept of several machine learning algorithms. I'll try to update this post as soon as I found new library algorithm used on my code.

1. Matplotlib

 A main Python library purposed for graph visualizing.

import matplotlib.pyplot as plt

2. Numpy

A famous Python library for scientific computing.

import numpy as np

I bundled about how to using those Matplotlib and Numpy libraries in a single video below:



3. Pandas

Stand for Python for Data Analysis or some said as PANel DAtaS. It mainly used for data analytical.

import pandas as pd

4. Seaborn

Commonly used for Python statistical graph and strongly integrated with Pandas data structure.

import seaborn as sns

I packed about Pandas and Seaborn example in a video below:



5. Scikit

Mostly contains core statistical algorithms and powerful package to do high-performance linear algebra and array operation with Numpy.

from sklearn.<package> [import <sub_package>]

So, I hope you exactly know what you have to import before playing around with Python. And, remember one thing, "Don't code what you can't debug" :)

Labels: ,

  Post a Comment

Python for Dummies

For anyone who want to know Python and desired to learn it, so here I write down about what I'd remembered:
  1. Python are basically interpreter, the same way with HTML processing on browser since it executed line by line.
  2. Thanks to Google because they provide Python virtual environment so that everyone can use to learn it. First thing first, open https://colab.research.google.com/ from your browser and press New Notebook. Copy and paste the following syntax and have a fun modification :)
  3. How to print out on screen?
    print("Hello","World","?")
  4. Learn about variable assignment and data type:
    #int x = 0 #float y = 4.3 #string z = "z"
    x = 0
    y = 4.3 z = "z"
    print(x,y,z)
    type(z)
  5. Python is about sequences:
    a = 1 a = a + 1 b = 2 c = a + b type(c) c = str(c) print(c)
  6. Math operation:
    print(c*2)
    49**0.5
  7. Ask for input:
    print("Name?") name = input() print("Hi",name)
  8. Branches:
    print("Name?") name = input()
    if (name=="dodol"):   print("Hi",name) else:   print("I don't know u,",name)   print("Age?")   age = input()   print("Age",name,":",age)
  9. Looping:
    for num in range(1,4):   print(4-num)
  10. Array:
    daftar_nama = ["saya","kamu","dia","mereka","kita"] for angka in range(len(daftar_nama)):   print(daftar_nama[angka])
  11. Quit from looping:
    for nama in range(len(daftar_nama)):   print(nama)   if (nama==2):     break
  12. Function:
    def add(a,b):   return a+b
    add(3,2) 
      I wrapped it on 2 videos below if you want to make a practice:


    That's it for today. Happy Python!

Labels: ,

  Post a Comment

Elevating User Account Control with Delphi

If you have medium size of Delphi based application that required some privileges access to Windows OS particularly, then you might face problem with User Account Control (UAC). User Account Control (UAC) is a technology and security infrastructure introduced since Microsoft's Windows Vista. It aims to improve the security of Windows by limiting application software to standard user privileges until an administrator authorizes an increase or elevation. However without privileges access, application that need access to system folder or database might impact to failure.

This time lesson will share you 2 alternatives how the application "playing" with UAC. The first option order user to elevating the authorization manually by executing application using Run As Administrator from right click context menu of the EXE binary. While last option inject an embedding code to elevating user automatically. Both alternatives options came with the same scenario, a login form with Login button enabled if user has full privileges access, and so on in contrary. Let the learn begin, and here the detail explanation:

1. Detecting Elevation Level with ElevationType
We know that UAC only affect with OS greater than Windows XP, so you need also to detect what kind of OS version where the application currently running on. In order so, put below function on the code:

function osver: string;
begin
result := 'Unknown (Windows ' + IntToStr(Win32MajorVersion) + '.' + IntToStr(Win32MinorVersion) + ')';
case Win32MajorVersion of
        4:
                case Win32MinorVersion of
                        0: result := 'W95';
                        10: result := 'W98';
                        90: result := 'WME';
                end;
        5:
                case Win32MinorVersion of
                        0: result := 'W2K';
                        1: result := 'WXP';
                end;
        6:
                case Win32MinorVersion of
                        0: result := 'WVista';
                        1: result := 'W7';
                end;
end;
end;

Then, you can put elevation user account detector on form onShow event just like code below :

procedure TForm1.FormShow(Sender: TObject);
const
TokenElevationType = 18;
TokenElevation     = 20;
TokenElevationTypeDefault = 1;
TokenElevationTypeFull    = 2;
TokenElevationTypeLimited = 3;

var
token: Cardinal;
ElevationType: Integer;
Elevation: DWord;
dwSize: Cardinal;
RunAsAdministrator:boolean;
level: string;
begin
SB.Panels[0].Text:='OS type: ' + osver;

// Run As Administrator Validator
RunAsAdministrator:=false;
if OpenProcessToken(GetCurrentProcess, TOKEN_QUERY, token) then
        try
        if GetTokenInformation(token, TTokenInformationClass(TokenElevationType), @ElevationType, SizeOf(ElevationType), dwSize) then
                case ElevationType of
                        TokenElevationTypeDefault:  RunAsAdministrator:=false;
                        TokenElevationTypeFull:     RunAsAdministrator:=true;
                        TokenElevationTypeLimited:  RunAsAdministrator:=false;
                else
                        RunAsAdministrator:=false;
                end
        else if (osver='W95') or (osver='W98') or (osver='WXP') then RunAsAdministrator:=true;

        if GetTokenInformation(token, TTokenInformationClass(TokenElevation), @Elevation, SizeOf(Elevation), dwSize) then
                begin
                if Elevation = 0 then RunAsAdministrator:=false
                else RunAsAdministrator:=true;
                end
        else if (osver='W95') or (osver='W98') or (osver='WXP') then RunAsAdministrator:=true;

        finally
        CloseHandle(token);
        end;

level:='(Administrator)';
btnLogin.Enabled:=true;
if not RunAsAdministrator then
        begin
        level:='(NOT Administrator)';
        btnLogin.Enabled:=false;
        end;
SB.Panels[0].Text:=SB.Panels[0].Text + ' ' + level;
end;

That's all. Try to execute the application in Windows 7, here's what I've got:



The Login button disabled since the application not executed with Run As Administrator context menu.



But, if the application running from Run As Administrator, the Login button now enabled!



Note that above code will not impact on Windows XP, since OS assumed that user running it as Administrator account even it executed from context menu or not.



2. Elevate Execution Level by Manifest file
Create a .manifest file :



Save it as .manifest. Continue to create a .rc file and named it as .rc.



Ah... don't forget to save the .manifest and .rc file in the same folder with the application source code. Then, continue to open command prompt, switch to directory where brcc32.exe (Borland Resource Compiler) exist and gives below syntax:



After the syntax succesfully compiled, it will automatically created a .res and .rec.



Back to the application source code and move to Project Manager window, right click to Project Files and point to View Source menu.



On .DPR menu, gives a line syntax after $R compiler directive just like picture below:



Finished! You may test the application and see what the different between options no #1 and #2.



Have a great fuckin' day!

Labels: ,

  Post a Comment

M8 Framework Reborn!

At the end of this year, my M8 a.k.a Method 8 tools has been finally re-evolute. It's an independent custom PHP framework built from scratch based on prior M8 core engine. Some of applications has been made from this previous M8 framework, but few of lacks found during the development phase. Such as it potentially bring  troubles for programmers when they work at the same module, too complicated programming structure, incomplete multi-users connection feature  and some other else minor issues. But at least, it's so reliable for 3.000 - 5.000 users via public internet - even in a small bandwith capacity available from clients.

Now, I've made some modification to it's core, eliminating the lacks covered from prior engine. There's major enhancement inside :
  1. Apache virtual directory and .htaccess dependency to support rewrite mod feature
  2. Multi-users connection support for database access (database class)
  3. Cacheable datasource (database class)
  4. More efficient session usage (session class)
  5. Enhancement in hi-jack and XSS filtering (secure class)
  6. 50% reducing development time by using 2 file only (presentation and controller) instead of 4 in prior engine
  7. Integrated HTML 5 usage in UI, increasing client-side validation function (ajax class)
  8. AJAX POST generator for better security
  9. PDF, CSV, XLS and print preview built-in feature from DataTables
  10. Code compressor 
  11. 100% refreshment of UI
  12. And more!
This latest framework built combined from jQuery and more persistence OOP classes. Just a little different from other framework available on the net :)

Labels: , ,

  Post a Comment

POS System Review: The Simplest, The Lightest and The Cheapest

Anyone can find cheapest Point of Sale (POS) hardware environment recently in this world (including barcode printer, barcode scanner, CPU & display monitor)? Well, I can find it for US$ 493 with all brand new devices! No kidding but here's the truth. All things has passed for future consideration, including lower electricity cost, easier to use barcoding on your store around and simple custom-made software for checkout purposed.

This is without receipts printer since this POS focused on stocking intention. So, receipt coupon for customer is infrequently or even never used, hereby changed with manual hand written.

And, here's the secret:


Zotac ZBOX ID41
WTH is this? Weird branded CPU name born in (nearly) Hong Kong with core business on mini-ITX and mini-PC since… 2006. That's why I choose it apart from it best mixed hardware combination.


This mini PC is thinner but wider than Mac Mini G4.


See link below for detail information:

http://www.zotac.com/index.php?page=shop.product_details&flypage=flypage_images-SRW.tpl&product_id=335&category_id=171&option=com_virtuemart&Itemid=100295&lang=un

Interesting?

Epson LabelWorks LW-400
This is an evolution for Epson to providing a low cost label printer. See the spoiler below:

http://www.epson.com/cgi-bin/Store/jsp/Product.do?sku=C51CB70010

The LW-400 is the minimum series to support barcode printing prior to LW-900.


The packing size slightly larger than Mac Mini G4.


A full qwerty printer at notebook size but more a little thicker. Very handy and ergonomic when hands on and feels light weight too. Just like seeing a BlackBerry with monster size :D


There're also 12mm black-on-white LC sample tape cassette including in the box, with easy way to setting-up on the printer. First press lid button to open the back cap and pull it.


Place the tape in proper way (green box) with outer ribbon attached on small slot (red circle).


Next, close the back cap slowly until it show as normal.


This series operated with dual mode; 6AA battery and power cable. You'll find a power cable included in the box. Make a power connection and press power button.


To switch to barcode printing, press ALT+barcode key as shown as picture above. Then pick one from several built-in barcode font available and specify the size. Sorry, can't tell you what else barcode font built-in included since I'm not an Epson salesman :)


Type-in some digits of numerical code to test. If you like to print copies, press copier button (red circle) several times until it displayed how much copy you want to print out. Ended with print button (green box) to start to print.


This LW-400 series also equipped with manual cutter. So, after print stopped, just press green button in right side to cut-off the paper. Simple and easy.

CipherLab 1070
Finally found a low cost scanner with fairly well-known brand name: CipherLab.


See below spoiler:

http://www.cipherlab.com/catalog.asp?CatID=8&SubcatID=10&ProdID=354

Not too worry, it can read Epson barcode print out result (by using EAN-13 small 5cm size :)


Conclusion
Building recently POS (hardware & software) system much easier than couple years ago. From above specification, there's no need to create an add-in module to print the barcode from application.


It's separate anyway. So it takes shorter time to developing the software. What you need is to mix and match everything. That's it! Thank's for reading and see you on next article...

Labels: , , , , , , , , ,

  Post a Comment

The ATM Simulator (People Hate Pearl!)

So, this is a proof-of-concept of ISO 8583 messaging in PHP. Everything you want to know about ISO 8583 (a complete) theory you can read at wikipedia page: http://en.wikipedia.org/wiki/ISO_8583. Anyway, article below will describe you a practical thing and technical work on how to understanding PHP socket programming - which is the core of ISO 8583. In short words: creating the ATM simulator with PHP!

Basically, ATM a.k.a ISO 8583 messaging using network socket to communicate between server and client. Take a look at general system activity defined below:


At first, server listening at specified address and port number, then client (for example: ATM) providing a block of ISO 8583 code to server (let say, client sending an account number). Server accepting command and parsing the code, continued to querying to backend. The result will returned back to client. Client accepting and parsing it again, and finally delivering a human-readable characters to the ATM screen.

To understand on how to developing the application, make sure that you:
  1. Familiar with PHP language.
  2. Able to use your favorite database client to create a single table.
  3.  Know the ISO 8583 philosophy.
  4. Having a PC with Apache & PHP enabled, also telnet client ready will help you much further. Windows (WAMP) and Mac OS (MAMP) is a great idea!

To start to build the application, keep in mind that there's 2 sequence things to create: SOCKET COMMUNICATION and ISO 8583 INTEGRATION.

SOCKET COMMUNICATION
This is a framework, since we need to create both different server and client applications that can interact using network socket. A server is listening and responding while client is sending and accepting. This is server code, save it with svr.php and store somewhere in your web server:

<?
error_reporting(E_ALL);
ob_implicit_flush();
$address = '10.2.2.212';
$port = 1234;
$sock = socket_create(AF_INET, SOCK_STREAM, 0);
if (socket_bind($sock, $address, $port) === false)
{
if (!socket_set_option($sock, SOL_SOCKET, SO_REUSEADDR, 1))
{
echo socket_strerror(socket_last_error($sock));
exit;
}
}
if (socket_listen($sock, 5) === false)
{
echo "socket_listen() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
}
do
{
if (($msgsock = socket_accept($sock)) === false)
{
echo "socket_accept() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
break;
}
$msg = "\nPOC-ISO Telnet Test. \n" .
"ketik 'quit' buat keluar, cuy...\n";
        socket_write($msgsock, $msg, strlen($msg));
        do
        {
        if (false === ($buf = socket_read($msgsock, 2048)))
        {
echo "socket_read() failed: reason: " . socket_strerror(socket_last_error($msgsock)) . "\n";
break 2;
}
if (!$buf = trim($buf))
{
continue;
}
if ($buf == 'quit')
{
break;
}
$talkback = 'Response Server: ' . $buf;
$talkback = $talkback . "\n";br /> socket_write($msgsock, $talkback, strlen($talkback));
//===========================================
echo "$talkback\n";
}
while (true);
socket_close($msgsock);
}
while (true);
socket_close($sock);
?>

Run svr.php code with -q parameter and try to connect to server from telnet command. What? Telnet? Yup, that's why telnet client needed in our existing article - to test the svr.php code. Look at picture below:


Send a free-string and look what the server response from telnet window or simply type 'quit' to quit. If svr.php looks fine, now you can continue to client code. Below is the client code socket core, save it with cli.php and store it to a path in your pub html web server.

<?
$host="10.2.2.212";
$port = 1234;
$message = 'HELLO 1234, apa bisa di copy ganti?';
$socket = socket_create(AF_INET, SOCK_STREAM, 0) or die("Error: SOCKET\n");
$result = socket_connect($socket, $host, $port) or die("Error: JARINGAN\n");
socket_read ($socket, 2048) or die("Error: RESP\n");

$message = $message . "\n";socket_write($socket, $message, strlen($message)) or die("Error: DATA\n");
$result = socket_read($socket, 2048) or die("Error: RESP\n");
socket_write($socket, "quit", 4) or die("Error: QUIT\n");

echo $result;?>

Also, run client code cli.php with -q parameter from second terminal while svr.php still remains active in first terminal window. If the code is right, both will response the same message sent from cli.php. And finally the first thing of socket communication is done!

ISO 8583 INTEGRATION
Before we implement ISO 8583 method in both codes above, we need to define a table test which simulate the back-end storage. In my experiment, it's only a single table. Look ak the picture below:


After creating table, continue to download a PHP class named by JAK8583 from PHPClasses.org. Download it from http://www.phpclasses.org/package/5398-PHP-Generate-and-parse-ISO-8583-transaction-messages.html. This class mainly purposed for ISO 8583 builder and parser, so you need to attach both svr.php and cli.php to this class.

Meanwhile, prepare client code (cli.php) into an ATM look-a-like interface. Contains several pages start from initiation until finish transaction (index, input, resp, output and error landing page). Let say, this below is a welcome interface (index):


The rule of this simulator is simple. User typing in 10 digit account number from input page, server will validating it from database and returning back the result to the simulator screen. So, here below is the input page.


Now, let's modify cli.php with sending and catching capabilities. Note that it also containing ISO 8583 builder and parser (and client side validation) :


<?php
if ($_POST['submit'])
{
$host="10.2.2.212";
$port = 1234;
$no_rek = $_POST['no_rek'];
//===========================================
// =========== ISO-8583 BUILDER =============
//===========================================
include_once('JAK8583.class.php');
$jak = new JAK8583();
$jak->addMTI('1800');
$jak->addData(2, $no_rek);
$jak->addData(54, '000000');
$jak->addData(43, 'XXXXXXXXX');
$jak->addData(7, '000000');
$message = $jak->getISO();
$socket = socket_create(AF_INET, SOCK_STREAM, 0) or die("Error: SOCKET\n");
$result = socket_connect($socket, $host, $port) or die("Error: JARINGAN\n");
socket_read ($socket, 2048) or die("Error: RESP\n");
$message = $message . "\n";
socket_write($socket, $message, strlen($message)) or die("Error: DATA\n");
$result = socket_read($socket, 2048) or die("Error: RESP\n");
socket_write($socket, "quit", 4) or die("Error: QUIT\n");
//===========================================
// =========== ISO-8583 PARSER ==============
//===========================================
//echo $result;
$jak = new JAK8583();
$jak->addISO($result);
if ($jak->getMTI()=='1810')
{
$data_element=$jak->getData();
$no_id=$data_element[2];
$nama=$data_element[43];
$nominal=$data_element[54];
$bln=$data_element[7];

if ($nama=='XXXXXXXXX')
{
header('Location: err.php');
exit;
}
}
else
{
header('Location: err.php');
exit;
}
}
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>POC-ISO</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>


<body bgcolor="#0066CC">
<p align="center"><font color="#FFFFFF"><strong><font size="3">KONFIRMASI PEMBAYARAN
  <br>
  ========================= </font></strong></font></p>
<p align="center">&nbsp;</p>
<form name="fInput" method="post" action="output.php">
  <div align="center">
    <table width="75%" border="0" cellspacing="0" cellpadding="0">
      <tr>
        <td width="90%" rowspan="3"><div align="center">
            <table width="100%" border="0" cellspacing="0" cellpadding="0">
              <tr>
                <td width="53%"><font color="#FFFFFF" size="3"><strong>ID PELANGGAN</strong></font></td>
                <td width="2%"><font color="#FFFFFF" size="3"><strong>:</strong></font></td>
                <td width="45%"><font color="#FFFFFF" size="3"><strong><?=$no_id?></strong></font></td>
              </tr>
              <tr>
                <td><font color="#FFFFFF" size="3"><strong>NAMA</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong>:</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong><?=$nama?></strong></font></td>
              </tr>
              <tr>
                <td><font color="#FFFFFF" size="3"><strong>BULAN TAGIHAN</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong>:</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong><?=$bln?></strong></font></td>
              </tr>
              <tr>
                <td><font color="#FFFFFF" size="3"><strong>JUMLAH TAGIHAN</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong>:</strong></font></td>
                <td><font color="#FFFFFF" size="3"><strong><?=$nominal?></strong></font></td>
              </tr>
            </table>
          </div></td>
        <td width="10%"><div align="right">
            <input type="submit" name="submit" value="--&gt; BAYAR">
          </div></td>
      </tr>
      <tr>
        <td><div align="right"> </div></td>
      </tr>
      <tr>
        <td><div align="right">
            <input type="button" name="Button3" value="--&gt; BATAL" onClick="document.location.href='input.php'">
          </div></td>
      </tr>
    </table>
  </div>
</form>


<table width="75%" border="0" align="center" cellpadding="0" cellspacing="0">
  <tr>
    <td><font color="#FFFFFF" size="3"><strong>RESP :</strong></font><font color="#FFFFFF" size="3"><strong>
      <br>
      <?=$message?>
      </strong></font></td>
  </tr>
  <tr>
    <td><font color="#FFFFFF" size="3"><strong>RECP :</strong></font><font color="#FFFFFF" size="3"><strong>
      <br><?=$result?>
      </strong></font></td>
  </tr>
</table>
</body>
</html>


Saved it as resp.php (means response). If validation returned false - for example entering only 3 digit from input box, the screen will automatically redirecting to error page, just like picture below:


Picture above displayed after we input a wrong account number. There's no such "123" account number on database. Try to typing a correct account number and re-submit again. The response page will displayed like picture below:


Take a note at both ISO 8583 code in a red box above (RESP and RECP). RESP (response) code is ISO 8583 builder code sent from client to server. While RECP (receipt) code is parser code sent from server to client. Anyway, here's below svr.php code improvement containing connection to the database (MySQL) :

<?
error_reporting(E_ALL);
ob_implicit_flush();
$address = '10.2.2.212';
$port = 1234;

//===========================================
// DB
//===========================================
include_once('JAK8583.class.php');
ini_set('display_error',1);
define("DB_HOST", "your.db.svr.host");
define("DB_USER", "db_user_name");
define("DB_PASS", "db_passwd");define("DB_NAME", "db_name");
$link=mysql_connect(DB_HOST,DB_USER,DB_PASS);
mysql_select_db(DB_NAME);
mysql_query("SET time_zone='Asia/Jakarta'");
//===========================================

$sock = socket_create(AF_INET, SOCK_STREAM, 0);
if (socket_bind($sock, $address, $port) === false)
{
if (!socket_set_option($sock, SOL_SOCKET, SO_REUSEADDR, 1))
{
echo socket_strerror(socket_last_error($sock));
exit;
}
}
if (socket_listen($sock, 5) === false)
{
echo "socket_listen() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
}


do
{
if (($msgsock = socket_accept($sock)) === false)
{
echo "socket_accept() failed: reason: " . socket_strerror(socket_last_error($sock)) . "\n";
break;
}
$msg = "\nPOC-ISO Telnet Test. \n" .
"ketik 'quit' buat keluar, cuy...\n";
        socket_write($msgsock, $msg, strlen($msg));
        do
        {
        if (false === ($buf = socket_read($msgsock, 2048)))
        {
echo "socket_read() failed: reason: " . socket_strerror(socket_last_error($msgsock)) . "\n";
break 2;
}
if (!$buf = trim($buf))
{
continue;
}
if ($buf == 'quit')
{
break;
}
$jak = new JAK8583();
$jak->addISO($buf);

if ($jak->getMTI()=='1800') // Dari Client
{
$data_element=$jak->getData();
$no_id=$data_element[2];

$perintah="select * from trx where no_id='$no_id'";

$hasil=mysql_query($perintah);
$n_data=mysql_num_rows($hasil);
$jak = new JAK8583();
$jak->addMTI('1810');
if ($n_data==1)
 {  while ($row=mysql_fetch_array($hasil))
{
$no_id=$row["no_id"];
$nama=$row["nama"];
$bln=$row["bln"];
$nominal=$row["nominal"];
}
>
$jak->addData(2, $no_id);

$jak->addData(43, $nama);
$jak->addData(54, $nominal);
$jak->addData(7, $bln);
}
else
{
$no_id = 'ERR000';$jak->addData(2, $no_id);
$jak->addData(43, 'XXXXXXXXX');
$jak->addData(54, '000000');
$jak->addData(7, '000000');
}
}
$talkback = $jak->getISO(); $talkback = $talkback . "\n"; socket_write($msgsock, $talkback, strlen($talkback)); //===========================================
echo "$n_data|$buf|$perintah|$buf|$talkback|$no_id|$nama|$bln|$nominal\n";
}
while (true);
socket_close($msgsock);
}
while (true);
socket_close($sock);
?>

Ok, so far you can learn on how to validate input from database and returning back the result to client screen. Now you can also try to extend the functionality to create a valid payable transaction if user submit from the simulator that will simulated balance subtraction. For example, after submitted, the screen will change to greeting screen or notification which say that the transaction is over and also displaying the last balance, just like picture below :


That's it for the client simulator. Meanwhile from svr.php code above, we can monitoring the throughput of the transaction in server terminal window. The red box sign tell that validation returning false result, while green color is a valid transaction.


CONCLUSION:
Since ISO 8583 messaging interoperability involving 2 or more companies (in fact the bank and merchant eventually) in a real world, each from both programmer must deal with such formats; ISO 8583 year version, header, MTI, Data Element format, server address and port. The rest, it's a character manipulation module anyway! Have a great ISO 8583 coding and share your experience below. Thank's for ever coming here.

Labels: , , , ,

  Post a Comment

One Minute Article About System Testing

So much pseudo-language articles we found on internet about software testing theories. For example, take a look at this "official" wikipedia page. There's huge of weird words that may confused some computer-college-newbies who want to learn about software testing. Also some overlapped procedures I found about several Performance Testing types on http://en.wikipedia.org/ wiki/Software_performance_testing. It's said that load testing & stress testing are types of performance testing. But it's contrary with the main page (http://en.wikipedia.org/wiki/System_testing) which said that both terms are typically system testing, so and so. Anyway, recently many peoples use these terms interchangeably. Lots talking about the words of "stress test…", "…performance test…" & "…load test", but they didn't know what the objectives is. In fact, it has quite different meanings.

Here's My Theory...

As theoretically, the best software testing model is always follow from it literatures - a standard one I meant. There are hundreds step that must be done sequentially, need additional peoples involved to work on it, also complete of infrastructure to support it, also huge of times to take and also lot of money to spend, etc, etc. So, based on my own practical experiences - in a minimal set of peoples - the software testing life-cycle can be cut to 3 sub-sequences-types (as it follow on the picture below); (1) Performance Test, (2) Load test, and (3) Stress Test.


Why? Here's the reasons:

1. Performance Test
This is baseline test. The test running in a static maximal number of concurrent user (X) as expected the system will hit. Various results at the end, will tuning some of critical points: the application level (block-code or algorithm changes), the database level (SQL code or query optimizing) and the server level (profiling OS and some other specific configurations). The network level is at the end of this boundary tuning. Tens of experiment s giving the best mix and match about the whole combinations, where the system can serve it optimum functionality. The goal of this test is : eliminating what can cause bottleneck and exactly increasing performance.

2. Load Test
Load testing often called as volume testing. The test start with maximal number of concurrent users (X) provided from test #1 and increasing to relevant number (Y) by increment of some number (Z). The goal is : to find highest number of load that the system can accept while still function constantly and properly. At the end of this test, hardware and network resources may need increased as it predicted. The scaling-up factor should improved vertically or horizontally, which give best value for money to spend efficiently and effectively.

3. Stress Test
The goal of the test is to ensure that the system recoverability functioned properly after the system down. This is important, since we also need to know how the system reacts to failure, including the interoperability of DRC infrastructure to work.

Again, these 3 software testing cycle above came from my logical thinking considering to small scale of peoples involved in a project. It is opened for debate on below comment box...

Technical Work

As basically, software testing tool purposed to measure performance, for exactly to know throughput and response time. Both number are main objectives of overall testing. Anyway, tons of software testing tool are available, but few of it are easy to operate and less, bit of it are free (See this source: http://en.wikipedia.org/wiki/Load_testing#Load_testing_tools). jMeter (http://jmeter.apache.org/) seems to be relevant to use. As is to others, it provide comprehensive way to "burn" the system anyway and providing easy-to-doing operation.

jMeter is a desktop application built from Java, such that, it's an independent platform and have ability to run on several OS platform (including my Mac :) - with Java Runtime installed of-course. To learn how to make a test, download latest version of jMeter and extract it somewhere on your PC.


Execute jMeter from bin folder by clicking ApachejMeter.jar file. After it launched - on the left pane - you'll see only 2 menu; Test Plan and WorkBench menu. The Test Plan describe about what the test physically do, while WorkBench define in-contrary (it's highly needed to create scenarios). Don't get confused, since this tools are designed for developer, you'll familiar to use in a day. Anyway, just like peoples shout, this tools lack of graphical report. A basic report viewer contains data listed on a table can be viewed from Add::Listener sub-menu:


However, it seem so hard understand to read the test result. So, before we take a test, so I hardly recommend to having a plugins called Statistical Aggregate Report. Download (StatAggVisualizer.zip ~ 1.8MB) and extract it. You'll see the same composition just like picture below:


Move the 3 files (jcommon-1.0.5.jar, jdnc-0.6-aR.jar and jfreechart-1.0.2.jar) to jMeter lib folder and move every files under ext to ext folder. After restart jMeter, you'll see the plugins in Add::Listener sub-menu:


So, we're getting ready to take a test. First, take a look picture below. Here's a basic skeleton of jMeter:


Add the appropriate 6 basic components under Test Plan and WorkBench menu just like above image. After it done, now we start to record the activities from browser and create a test scenario. But, we need to make browser's proxy heading to jMeter HTTP Proxy server (locally). So, open browser proxy configuration (I particularly using Firefox) and set to localhost with port 8080.


Close the browser configuration and back to HTTP Request Defaults menu on jMeter. Set the web server IP where we'll open the application (for this test, my application is ready on 10.2.2.212).


Until this point, we need an application to test - a single web based application, but jMeter seems able to run others too. In my experience, I use what I developed before: Yet Another Concept of Custom Manageable FTP - a PHP based FTP client. Since jMeter capable to handling FTP burning test, so the scenario is to measure uploading process simultaneously by number of users.


Now, we're doing a scenario recording by running some scene we've like to test. It's better to measure from login session until logout sequentially. If you've done it, now take view back to jMeter Recording Controller pane. See that the scenario recording process is created automatically.


At this point, we need to take a bit adjustment from the scenarios. At least (optionally) modifying the server IP on each HTTP request window. Here in my app test, I also add full path to the object file required for uploading process.


Next, let's try burning it for once. Click green start button to start the test. This test - as default - will only executing a single of thread (by means, one user for one loop). Click the Statistical Aggregate Report to show the visual graph:


Take a look at the result, the total throughput was 7.6/min. It's means that the server can accept 7.6 request per minute (from single thread). While response time showing 7,840ms (7.8 sec) as per process (by the Average column). Now, how about making a test with some number of concurrent users? For this, first you need to remove and add a new Statistical Aggregate Report. Then, modify thread number from Thread Group menu.


Here I change the number the thread (users) to 100 and leave loop time to 1. It's mean that the application will hit with 100 users simultaneously. Ready to start, click the green icon, wait until it done (from green box until it show gray) and here below the result:


From picture above, let's try to count the TPS (Transaction Per Second). This can be calculated by dividing the number of threads with total amount of times needed to finish the test (in seconds). So, the test equal with 0.91 TPS from 100 concurrent users at the same time. Note that this is local testing condition, no network-simulator used on it. For better result, please combine bandwidth limiter, sniffer or other networks tools to the test environment.

Based on previous test, some improvement needed in application level, database and OS to remove existing system bottleneck, also increasing TPS calculation in the next testing and gain higher number of concurrent users. The higher TPS calculation means more efficient system. This is what I called as Performance Test, to get the mix and match overall system configuration.

Concurrent User vs Transaction Per Second

As I said before, the thread number reflecting concurrent user accessing the system at the same time. In a real world system, calculating concurrent users can be done from additional built-in statistical software assist (eg: MRTG or other statistical software). For easy example, we can count it using Google Analytics (GA) free service. Take a look an example graph below, where I combined a web page with GA. The data provided in a specified month (April for example):


Set the appropriate time range in April to get the amount of visitor and average visit time. The concurrent users can be calculated using formula as follow:

concurrent_users = (monthly_visits x time_on_site) / (3600 x 24 x 30)

From the picture, we got 110,055 total visitor with average 2 mins and 36 sec of visit time (equal with 156 sec). So, here the concurrent users:

concurrent_users = (110,055,963 x 156 sec) / (3600 x 24 x 30)
concurrent_users = (17,168,580) / (2,592,000)
concurrent_users = 6.62 users

Conclusion

Software testing is an interesting topic - as a part of software development - and it need to be done frequently to maintain the system performance and scalability. However, some projects ignore this procedure to cut down operational cost that may bring another future possibility of  potential expense (eg: hardware replacement or anything else). The using of jMeter just for an example, it's a learning media for beginner about knowing behind-the-scene of creating the scenario work-flow. Please add your comment below and thank's for stay tune on this blog.

Labels: , , ,

  Post a Comment