UnixTech

Relational Algebra - SQL

2015-12-07T19:30:00+05:30

In the age of ORMs so many developers today doesn't know about very fundamental and basic algorithms that runs SQL. Despite being one of easiest and much useful language many people run away from using SQL directly and take shelter in using some 'wrapper' tool which is not always as good as raw SQL.

Let's start by defining very basic relation in SQL.

Database as Collection of relations ( Tables or Schemas )
Being first class predicate - State of database is final state of all relations
By joining, aggregating data from different relations one can filter out data as desired.

Relation

Relation in SQL language is defined by several terms.


Tuple	One Row in SQL Relation
Attribute	Column in Relation
Unknown	`Null` in Domain

Note:- Tuple is represented by (a, b), Attribute(Column) here will have unique domain(name - Relation name) within relation.

Relational Algebra

Relational algebra is superset of set algebra which defines formal language of relations in Database domain.
Each operation done here on relations will return new valid Relation.

This algebra has mainly two groups of operations, One it shares with set theory and other one is specific to Relational model.

	SET operations	Relation specific operations
1	UNION	SELECT
2	INTERSECTION	PROJECT
3	SET DIFFERENCE
4	CARTESIAN PRODUCT \ CROSS PRODUCT

Any operations in Relational algebra can be classified mathematically as binary and unary, this fundamental operators have all the power needed to construct complex queries as needed.
The main operators are:

SELECT ( $\sigma$ ) Can be described as below
$$ \sigma_\psi RO $$

Where:


R	Tupels sets in SQL
$\psi$	Predicate in selection retries from Tuples in R

PROJECT ( $\pi$ ) Operation which returns columnar structure in vertical dimention, If you remember this is slicing by attributes can be described as

$$ \pi _{a1,a2...an} RO $$

$_{a1,a2..an}$ are set of attributes names.

CARTESIAN\CROSS PRODUCT ( $\times$ ) This is binary operation as oppose to unary like previous two, Can be used to generate complex relations by joining each tuple operands together.

$R \times S = {r1, r2...rn,s1,s2...sn}$

UNION ($\cup$) Appends two relations together.

To be successful in this binary operations both relation needs to have same set of attributes.

$$ R \cup S = (_{r1, r2...rn}) \cup (_{s1, s2...sn}) $$

Assuming, $S \, \Sigma \, (_{s1,s2...sn}) \quad and \quad R \, \Sigma \, (_{r1,r2...rn})$

DIFFERENCE ( $\setminus or \, -$ ) A binary operation, as you may have guessed - $\cup$ only but in reverse.
Set difference can be described as

$$ R\setminus S = (_{r1,r2...rn}) \quad where \quad (_{r1,r2...rn}) \, \Sigma\, R \quad but \quad (_{r1,r2...rn}) \, \notin \, S $$

REMAME($\rho$) A unary operation that works on attributes and returns new value of attribute, This is mainly used for JOIN operations to differantiate the attributes, can be expressed as

$$ \rho_{a\setminus b}R$$

With this essential building blocks in place we can now move forward and take a look at more complex queries such as mixing many of these premitives to perform left joins, right-joins etc. In addition to these we can also add few more such as sum, multiplication to these operations on set of tuples or attributes.
These algebric math provides fundamental building block of any SQL algorithm which guarantess ACID standards are followed hence understanding them all the more important.

Interviewer's Gaffe-1

2015-06-15T16:02:00+05:30

Plight of every Interviewer

Firewall PF

2014-12-06T13:30:00+05:30

FreeBSD and OpenBSD

PF (Packet filter) is default firewall for OpenBSD and included in other OS's like FreeBSD and Apple IOS operating systems. Many other "Commercial firewall" appliances are inspired by PF.

History of PF

PF was originally designed as replacement for Darren Reed's IPFilter, from which it derives much of its rule syntax. IPFilter was removed from OpenBSD's CVS tree due to OpenBSD developers' problems with its license. Specifically, Reed distributed some versions of his software with the license clause, "Derivative or modified works are not permitted without the author's prior consent." Due to this, the OpenBSD team decided to replace the software. This decision became the subject of wrangling among the parties involved, degenerating into a discussion that failed to reach mutual understanding. On the subject, OpenBSD project leader Theo de Raadt wrote, "Software which OpenBSD uses and redistributes must be free to all... for any purpose including... modification."

PF has since evolved quickly and now has several advantages over other available firewalls. Network Address Translation (NAT) and Quality of Service (QoS) have been integrated into PF, QoS by importing the ALTQ queuing software and linking it with PF's configuration. Features such as pfsync and CARP for failover and redundancy, authpf for session authentication, and ftp-proxy to ease firewalling the difficult FTP protocol, have also extended PF.

One of the many innovative feature is PF's logging. Logging is configurable per rule within the pf.conf and logs are provided from PF by a pseudo-network interface called pflog. Logs may be monitored using standard utilities such as tcpdump, which in OpenBSD has been extended especially for the purpose, or saved to disk in a modified tcpdump/pcap binary format using the pflogd daemon.

For more info, Read - History of pf

PF setup

Usually PF is deployed in conjuction with other tools provided by OpenBSD ecosystem. These includes: HFSC Queuing system for QoS FTP-Proxy Application proxies such as Relayd ( Mainly used as HTTPs termination point ) OS detection using fingerprint - pf.os * CARP firewall failover for HA environments ( UCARP for FreeBSD users )

How to deploy PF firewall in your environment

Note: Both OpenBSD and FreeBSD OS uses different syntax for maintaining PF firewall. We will mainly focus on OpenBSD OS but there are benefits of using PF with FreeBSD OS since it provides multi-processing capable version of PF.

File - /etc/rc.conf.local

    pf=YES
    pf_rules=/etc/pf.conf
    pflogd_flags="-s 1500" # Ex. Snaplen, Log filename

File - /etc/pf.conf

    ### My master pf.conf

    ### Interfaces
    EXTIF ="em0"
    INTIF ="em1"
    DMZ = "em2"
    EXTRAIF ="em3"

    ### Hosts
    ADMIN ="10.0.11.1"
    ADMIN1 ="10.0.11.31"
    BOTHADMIN ="{" $ADMIN $ADMIN1 "}"
    EXTDNSSERVER ="4.2.2.2"
    INTDNSSERVER ="$INTIF:0"
    #DNSSERVERS ="{' $INTDNSSERVER $EXTDNSSERVER '}"
    DNSSERVER ="{$INTDNSSERVER}"
    LOGSERVER = "{ 10.0.11.22, 10.0.11.31  }"

All these variable defined are called MACROS inside pf.conf file.
These are used for convinience and ease of use
Defining nested macros are possible as well.
Take a look at INTIF macro, If you want to include that whole internal network in your rules then INTIF:network in your rule.

Now, We will have a look at some of the rules itself.

    #External Interface
    #Block all on External interface
    block log on $EXTIF

    ## Network address translation with outgoing source 
    #match out log on $EXTIF from $INTIF to any received-on $INTIF tag EGRESS nat-to ($EXTIF:0) 
    match out log on $EXTIF from $INTIF:network to any received-on $INTIF tag EGRESS nat-to ($EXTIF:0)
    #If you have difficulties with any box with static port forwarding then you should use
    #match out log on $EXTIF from $INTIF to any received-on $INTIF tag EGRESS nat-to ($EXTIF:0) static-port

    #Traffic generated from firewall it self will be tagged as EGRESS
    match out log on $EXTIF from $EXTIF to any tag EGRESS
    #More on these later on.

    #EXTIF inbound
    pass in log (to pflog1) on $EXTIF inet proto tcp from any to any port 22 
    pass in on $EXTIF inet proto tcp from any to $EXTIF port >10000

    #External interface outbound
    pass out log on $EXTIF inet  from ($EXTIF) to any $TCPSTATE $EXTIFSTO queue (bulk, ack) tagged EGRESS
    #pass out log on $EXTIF inet proto udp from ($EXTIF) to any $UDPSTATE $EXTIFSTO queue (bulk, ack) tagged EGRESS
    pass out log on $EXTIF inet proto tcp from ($EXTIF) to any port $TCPPORTS $TCPSTATE $EXTIFSTO queue (web, ack) tagged EGRESS
    pass out log on $EXTIF inet proto udp from ($EXTIF) to any port 53 $UDPSTATE $EXTIFSTO queue (dns, ack) tagged EGRESS

These are some of the rules that I have defined in my DMZ firewall to prevent other users from coming in from outside.
After deploying this ruleset only SSH is allowed from outside interface of firewall.
From inside essential end-user services such as Internet browsing, DNS are enabled.
Take a look at match out rules on EXTIF to have a look at how nat rules are working.

Turning on routing

To turn on routing functionality of the box, You need to make sure you have enabled ip forwarding in sysctl

    # To check the ip forwarding status
    sysctl net.inet.ip.forwarding
    # If it's 0 then turn it on
    sysctl net.inet.ip.forwarding=1

    #To make it permanent 
    ### /etc/sysctl.conf
    net.inet.ip.forwarding = 1

PFCTL utility

After making changes inside pf.conf file, rules are not automatically loaded. To load the rules We need to use pfctl

To load rules - Assuming rule file is /etc/pf.conf

    pfctl -vf /etc/pf.conf

To see which rules are currently loaded, It will also show related counters.

    pfctl -vsr

Conclusions

PF is one of the most popular and powerful firewall for managing your network traffic. We have barely even scratched surface of what PF can provide. It's functionality is much more then many of the commercial offerings offers.
We will also cover some extended functionality such as usage of Anchors, Preventing torrent traffic, Blacklisting and preventing brute-forcing attack etc. Being open-source it places no restrictions on usage. Users can use it any which way they would prefer.

Having used PF and OpenBSD for nearly 10 years in all of my setups I can say PF is most secure firewall there is and With combination of OpenBSD and PF you can be pretty sure you are one step ahead then rest in process of being NSA proof.

Improve PostgreSQL CLI

2013-12-07T14:30:00+05:30

When you work in Terminal/Browser workflow for almost 10 hours a day - 5 days a week, It becomes important kind of colors and configs you choose. For me Database has been important part of my workflow when configuring various business logic for applications.

I mainly use PostgreSQL for storing almost everything that has to resambles data, and mostly it will be automated through psycopg2 in python scripts but time to time I do dwell in CMD option that PostgreSQL provides through psql.
Given the configurability of psqlrc and flexibility that is allowed by PostgreSQL server, it's almost surprising that how little people take advantage of these available features. Aliases and setting up proper History files can be useful features that comes in handy.

PostgreSQL stores psqrc at various levels in system.

System wide psqlrc
- Will Affect all users
- Can be located using following

  pg_config --sysconfdir
  /usr/local/etc/postgresql

Note: This is for FreeBSD operating system, location will vary as per your own OS.

Per User psqlrc

 touch ~/.psqlrc

You can also have multiple psqlrc one per major version of PostgreSQL on your system.

if you have more then one version of PostgreSQL installed on your system, then name it accordingly. Ex. For version 9.4 - psqlrc-9.4 or psqlrc-9.4.3. This way It will enable you to have multiple configuration files for each user and per version as well.

Now, Decide on which specific configuration file you want to configure - System wide or User specific and start customizing your psqlrc.

Actual configuration file


-- This is comment.
\set PROMPT1 '%n@%/%R%x%# '
\set PROMPT2 '[more] %R > '
\pset null '[null]'
\set COMP_KEYWORD_CASE upper
\timing
\set PAGER less
\set HISTSIZE 2000
\encoding unicode
\x auto
\pset border 2
\set VERBOSITY verbose
\set version 'SELECT version();'

-- MACRO can be defined like this.
\set extensions 'select * from pg_available_extensions;'
\echo 'Welcome to Dev1 PostgreSQL \n'

Final output

Wrapping up

These are about the main settings that you would want to configure here, Apart from these settings only Aliases as per your convinience should be configured inside your PostgreSQL configuration file so repetation can be avoided. Putting it in version controlled dotfiles git repository and you will be able to sync same setting across all your DB server regardless So, Give custom configs a try!

Find - Looking for things

2010-01-03T18:02:00+05:30

The Find utility in Linux is very useful in the sense that it quickly locates and searches through list of files and directories. It can do so based on condition that you pass through arguments. Find can find files using different conditions like:

Permissions
Users
Groups
File type
Date
Size and more.

Basic Usage

Find file in current directory

 # Find by filename in Current dir
 find . -name unixtech.txt

 #Output
 ./unixtech.txt

Find file in current directory Case insensitive

 # Find by filename in Current dir
 find . -iname unixtech.txt

 #Output
 ./unixtech.txt

Recursively searching file in all in whole system

# Recurse through whole file system
find / -name $FILENAME

Find files based on permissions

Find files certain permissions

# Find only files with full 777 permissions
find / -perm 0777 -print
# Find files with SGID bit set 
find / -perm 2644
# Or
find / -perm /g+s

Find all files based on user permissions

#Find all files with READ permission
find / -perm /u=r -print

# Find all files with executable bit set
find / -perm /a=x -print

Note: Find can also execute command on found files based on given criterion. In addition to just printing list of files, You can modify, change permission and also delete files using -exec flag in find command.

So, If you want to change all the files that have permission set to 777 to something that only you can modify in your home directory, You may execute following variation of find

Find all files with 777 permission and change it to 644 inside your home directory

#Find and exec 
find ~USERNAME -perm 777 -print -exec chmod 644 {} \;


{}	Shell expander which will put current file name from list in `-exec`
\;	'\' is Shell escape and ';' is Unix chaining symbol

Note: Here thing to remember is You have to put {} symbol where you want INPUT filename to be, and chain it with \; symbol.

Same thing if you want to remove or list files

#List files that matches certain crieteria
find / -perm 777 -print -exec ls -la {} \;

#Removing files that matches certain crieteria
find / -perm 777 -print -exec rm -rf {} \;

Finding files based on user/group ownership

#Find files owned by particular user
find / -user unixtech -print

#Find files owned by group
find / -group unixgroup -print

Finding files based on modification/changed/accessed date time

Find files modified 3 days back


find / -mtime 3

find all the files those are changed last hour

#Will return all the files changed in last 60 mins
find / -cmin -60

Note: '-' sign in front of 60 includes all the files that changed within that timeframe, Ex. It will include files that are changed 3, 5, 10 mins back and so on. Notice different criterion for finding files such as -mmin, cmin, amin


Access time	If you list/delete/open this file then `atime` will be modified
Changed time	Modifying data of the file changes `ctime` parameter of file
Modification time	Same as Changed time but will also be changed upon changes in meta data of the file.

Use `find` to search files based on size

This one is quite useful in case you want to find largest files in your home directory, files that are eating away space on hard drive.

Find all the files between 10 MB - 100 MB

find /home -size +10M -size -100M

Find all the files larger then 1GB and delete em

#Find larger files and list them first
find /home -size +1G -exec ls -la {} \;

# If you see desired files then remove them
find /home -size +1G -exec rm -rf {} \;

Find all the movie files larger then 100MB and delete

# Find and list files first
find /home -size +100M -print -iname "*mp4|wmv|mov";

# After listing them just press `UP` arrow, change the CMD and  delete 
find /home -size +100M -iname "*mp4|wmv|mov" -exec rm -rf {} \;
#Be careful while executing that command.

Note: Find supports extended regular expressions too.
Regular expressions are swiss army knife for solving many kind of problem but they also come with added difficulty of maintaining and generating them. If none> of the above meets your requirement then as last resort only you should use Re>gExes in find utility.


R	Tupels sets in SQL
\(\psi\)	Predicate in selection retries from Tuples in R