Jun 23

In one of our project we needed to search for multiple words from a file and return the found words.
These files would be large and there might be large number of words to search for also.

Ruby has a good support for regexp, however it would be slow in doing these searches for patterns as compared to unix commands and hence we decided to look for certain unix commands for doing these searches.

We went through “grep“, “awk” and “boyermoore searching algorithm in ruby” for doing this search.
Each of these were used for searching patterns in a file.

grep and AWK were giving the results as the line(not he line number but the whole content of line) on which the matching pattern were present. Working on grep and awk further we found differnt ways of searching from files using these.
Continue reading »

Tagged with:
Jun 23

In one of the project we needed to store data in our data center i.e not place(store) it on cloud specifically. And this data needs to be available to multiple users/system. The operation on this data included creating/editing a file, copying a file from those stored data or copying to it.

After searching a few, we decided to use Fuse along with sshfs.

Fuse allows to use a  remote file as a local file and allows us to make changes to it that would reflect correctly on both the remote and local files.

Mounting a remote folder on ubuntu using Fuse(Filesystem in Userspace) and sshfs(Secure SHell FileSystem):

The stable release of fuse can be downloaed from : http://fuse.sourceforge.net/

Further installation can be done by :

./configure
make
sudo make install

Further installation of sshfs can be done by :

sudo apt-get install sshfs

Later we need to load the fuse using :

sudo modprobe fuse

Further the user needs to be assigned rights for performing action on fuse:

sudo adduser <username> fuse

The above command is used to add the user to the fuse module.

sudo chown root:fuse /dev/fuse
sudo chmod +x /dev/fusermount

While these command mentioned above are used to change the permission access of the file.

sudo chmod +x /dev/fusermount

might give an error namely directory not found.

In this case we can use the “whereis” command to locate usermount.

whereis fusermount

which would return a result something similar to
/usr/bin/fusermount

Using this location do :

sudo chmod +x /usr/bin/fusermount

After this do a logout and login back to the system.

Now make a directory named remoteserver to mount the remote folder.

mkdir ~/remoteserver

Now to mount the remotefolder on the remoteserver use :

sshfs <username>@<ipaddress>:/remotepath ~/remoteserver

Here the remotepath points to the remotefolder.

After this you will be prompted for the password.

Finally you will be able to use the remotefolder as if it was a part of the local system.
The remoteserver directory will contain all of the contents of the remotefolder.
Any changes made in the remoteserver will reflect in the remotefolder too.

For unmounting the remotefolder use :

fusermount -u ~/remoteserver

Using dd command to test

dd command is used to create a file :

dd if=/dev/zero of=output.dat  bs=10M  count=10240

Here the file named output.dat is created of size 100MB.

The bs option specifies the size of the file to be created.

To see the help of “dd”
Give a

dd --help

on the console.

Using dd command I created a file in the remotefolder by creating it in the remoteserver directory.
Also we tried copying the file along the network and compared the md5sum of the files to see if the files are correct.

md5sum filepath

In this case we can do it as,

md5sum output.dat

The transfer speed across the network came as 10 Mbps. This is dependent on the speed of the network.

Mounted filesystem can be used using the above explanation.

Tagged with:
Jun 14

Speex to Mp3 using ffmpeg and lame

In one of the project we receive a speex(.spx) file from client and we have to play on browser as well as in local machine.

So i am using ffmpeg and lame to do this job.It is 2 step process.

Step 1

Speex(.spx) file first need to covert into wave(.wav) file using ffmpeg.

ffmpeg -i filename.spx filename.wav

Step 2

After wave(.wav) conversion convert into mp3 using lame library

lame filename.wav filename.mp3

More info regarding lame you can find out here.

You can also manipulate ffmpeg and lame parameter as per requirement.

Tagged with:
Mar 20

Today I will demonstrate how did I manage to get Selenium (browser => “firefox”) running under headless state (click here to know more on this) .

Now we need Xvfb to accomplish this dirty task.

For Ubuntu the package is fairly straight forward just type in the command .

sudo aptitude install Xvfb

Under Fedora the same can be achieve using command .

sudo yum install xorg-x11-server-Xvfb

For more on Xvfb package  check your distribution provider .

Continue reading »

Tagged with:
Feb 02

There are various library for manipulating image,one of them is GD library.
In this post i quickly show how to resize an image or generate thumbnail of image using GD2.

Basic steps as follows

  1. First import image
  2. Resize an import image to desired size
  3. Export Image to desired file

above code is extracted from rchart.

Tagged with:
Jan 26

I have to admit this . I always fall in love with RAILS each day as I explore it to know how a complex problem can be handle with great ease and with minimum amount of code.

That what made me write this post on " Conditional Get Request."
For those who aren’t aware of it let me give them some insight in it.

Conditional GETs are a feature of the HTTP specification that provide a way for web servers to tell browsers that the response to a GET request hasn’t changed since the last request and can be safely pulled from the browser cache.

They work by using the HTTP_IF_NONE_MATCH and HTTP_IF_MODIFIED_SINCE headers to pass back and forth both a unique content identifier and thetimestamp of when the content was last changed. If the browser makes a request where the content identifier (etag) or last modified since timestamp matches the server’s version then the server only needs to send back an empty response with a not modified status.

Enough of talk now its  time for the action (Way to achieve it in Rails)

class UsersController < ApplicationController
 def show
   @user = User.find(params[:id])
    fresh_when :last_modified => @user.last_updated_at ,:etag => @user
 end
end

That it done

Now let check the logs to find, does it actually asking browser to read from its cache.

First time the page will completely render but any subsequent query for the same page will ask the browser to display from its cache(browser cache) (until modified).

Here my log :-
Conditional GET-Log
Other way to achieve the same is.

 def show
   @user = User.find(param[:id])
    if stale?(:last_modified_at => @user.updated_at,:etag => @user)
      repsond_to do |format|
        format.html
      end
    end
end
 def show
   @user = User.find(params[:id])
    response.last_modified = @user.updated_at
    response.etag = @user
    if request.fresh?(response)
      head :not_modified
    else
      respond_to do |format|
        format.html
      end
    end
 end

The Following Code should give you same result as above

That it , thats all it take to get " Conditional GET " working in Rails. Wasn’t I right when I said that I fall in love with Rails each day.

Usage :-

Now most of you  might be wondering where can they employ this technique or when should they use it

My Answer to them is think of Conditional Get a kind of Caching Mechanism ( after that what browser is doing for you)

Now suppose you have a method in controller that make large number of Database queries or ERB template that render a lot of contents on per requests .Wrapping the same with " Conditional GET " can simply do a lot of good.

That it enough on " Conditional GET " . Hope the post help you in some way or other.

Thank You .

Nov 18

In my recent work i have face some difficulties for generating different report for different organization or client using same application,because every client have their own reporting format.

Like SAAS based application if it is generate some MIS report and if client require some more or less information depending upon his requirement we can give some editable facilities to edit and generate report.

So to fulfill this kind of requirement user editable report which contains Markup + ruby code for reporting purpose.
I am using liquid ,acts_as_flying_saucer,rchart,textile/Html for generating report.

Continue reading »

Tagged with:
Oct 01

Rchart is ruby port of the slick pChart charting library.

Using Rchart we can develop various format of chart.

  • Line Chart
  • Cubic Curve Chart
  • Bar Chart
  • Pie Chart
  • Filled Line Chart
  • Stacked Bar Chart

And many more ,You can checkout examples of rchart.Some of examples are listed below.

Stacked Bar Chart

Rchart:- Stacked Bar Chart


Rchart - 3D Pie Chart

Rchart - 3D Pie Chart

Rchart -Line Chart

Rchart -Line Chart

Rchart gem require ruby-gd gem and GD library.

To install rchart gem on ubuntu following packages needed.

  • sudo apt-get install libgd-ruby
  • sudo apt-get install libpng-dev
  • sudo apt-get install libgd2-xpm-dev
  • sudo gem install ruby-gd

After installing rchart,You can checkout examples of rchart.

More examples of rchart is available on http://amardaxini.github.com/rchart/.

Online Documentation is available on http://rdoc.info/gems/rchart/1.1.1/frames

If you have found any difficulties to install or running rchart,then comment on this post or write issues on github.

Tagged with:
Aug 18

Amazon Simple Queue Service (Amazon SQS) is a distributed queue messaging service. The idea of SQS is to remove the direct associations between producer and consumer and act as mediator between them.

e.g

Consider that you have a large  application like websites monitoring which involve many stages like

  • Downloading a website
  • Processing the downloaded website
  • Generating report of the above processing

So instead of clubbing the above three one can just split them on their specific need(based on the work they do)  with each doing their respective task and on completion notify the other about it completion.

The traditional approach (called as Client-Server architecture) is to send the notification using a simple HTTPREST , SOAP etc request. But it too has some disadvantage as it require extra care and  precision  to handle all possible response e.g Time-out,Server down etc.

One can save this time and effort my using SQS.

e.g Instead of sending the direct request to Server one can use to SQS to interact with Server/Client an can eventually cut down some of drawback of traditional architecture.

Setting up Amazon SQS in Ruby: There are various library available for using Amazon SQS in Ruby (e.g right_aws, sqs)

Pre-requitise:

  • aws-access-key-id
  • aws-secret-access-key
  • ruby-library(right_aws or sqs gem)
    both of which is provide by the Amazon (or you can ask for incase) during the process of account-registration

In your irb console

require "rubygems"

require "right_aws"

aws_access_key_id  = "Your access key id"

aws_secret_access_key = "Your secret access Key"

sqs = RightAws::SqsGen2.new(aws_access_key_id,aws_secret_access_key)

This basically initialize the sqs object for you to work on.

One can create a new queue just by a single method.

 queue(queue_name, create=true, visibility=nil) 

carefully look at parameter supplied.


queue_name => name of  queue.

create => true (create a queue if it does not exist (optional)).

visibility => set visibility (which make the message reappear after a specified time if not  delete from queue).

One key point about visibility is that one can’t set visibility more than 7200 sec for a  queue or a message .But you can set so from RightScale (Cloud Computing management Platform).

e.g

my_queue = queue('queue_1')

just look at few other methods.

1. send_message or push :- Send message to the specified queue.

 
 my_queue.send_message "This is my first message"

2. receive or pop :- Receive message from the specified queue.

 
 message_received = my_queue.receive

puts message_received.body
=> "This is my first message"

2.1 receive_messages : – There is also a method called receive_messages to receive an  array of messages but it never returned me more than one message so if it work for you  it fine then syntax is like this

 
 receive_messages(number_of_message=1,visibility=nil)

 my_queue.receive_messages(2,60) => return and array of messages

3. delete : –  Delete a  message or a queue

  
 my_queue.delete => "delete the my_queue for SQS"
 
 message_received.delete => "delete the message from the intended queue"

4. size :- Specify the no of message in the queue

 
 my_queue.size => 4 ("4" message in my_queue)

5. name :- Specify name of the queue

  
 my_queue.name => "queue_1" (Name of the queue)

Note : – There is delay lag in all request and response sent/received to/from  SQS server don’t assume to work it instantaneous approximately of around 45 sec -1 minute(can be more)

e.g

If you send  a message to the queue don’t assume it to appear instantaneously in the queue

Pricing Model => $0.10 per 100,000 requests.

Message Size => 64 KB

For more information on Pricing and message size click  here.

Thank you

Tagged with:
Aug 12

In many application we want to convert one office format to another office format e.g doc to PDF , doc to html etc.We can import/export document using OpenOffice easily,but this is manual way.But standalone/Web based application we have to automate this functionality. JODConverter,the Java OpenDocument Converter, It converts documents between different office formats using OpenOffice.

JODConverter supports all conversion which is given by OpenOffice.More Info regarding format you can visit here.

Now,JODConverter is a java library so it can be used using java application,or using command line,or by making web service so any other language(RUBY,.NET,PHP,PYTHON) can used this conversion directly.
Convert Office is a ruby Wrapper of jodconverter .Which is used to convert office format to another office format.
Continue reading »

preload preload preload