Overview:

With the availability of lots of options to choose from the cost effective hardware in cloud environment, the storage has become so easy now a days. Once the storage is cheap, the immediate requirement for any software solution is to write an effective code to consume and expose the data. Agira team met a similar situation with one of our assignments last week.  The solution demanded uploading of  a very complex CSV data into an ETL Database. Improved performance was the goal and we started exploring the possibilities of developing the piece of code in Ruby on Rails and Golang.

Approach:

On the Rails side, we wrote the code on Ruby and executed the upload action using sidekiq jobs. Our good old concurrency bottlenecks of Ruby language hit the performance and the time taken to complete the job was not satisfactory. To handle the situation, we wrote the piece of code in Golang which does the similar task. On the developer’s laptop itself we could see the better performance of the Golang code.

Technology Bundle:

Ruby Code Snippet:

CSV.read('./db/samples/myfile_sample.csv', {col_sep: "," }).each_slice(200).each_with_index do |rows,i|
array_hash =[]rows.each_with_index { |row,j|
next if (i==0 && j==0);
array_hash <<         Hash[[array_header,row].transpose] }
RubyCsv.create(array_hash)
end

GoLang Code Snippet:

func (csvDataContainer *VariableInit) GenertaeString() {
// for example csvDataContainer.Lenght = 100
csvDataContainer.stringVal = ""// store the rows of data (as a string) to be inserted in table
csvDataContainer.str = ""// used for comma and semicolon separator
if len(csvDataContainer.remaining) > csvDataContainer.Lenght{
csvDataContainer.first_100 = csvDataContainer.remaining[:csvDataContainer.Lenght]
csvDataContainer.remaining = csvDataContainer.remaining[csvDataContainer.Lenght:]
}else{
csvDataContainer.first_100 = csvDataContainer.remaining
csvDataContainer.remaining = nil
}
for parentIndex,rec_values := range csvDataContainer.first_100 {
csvDataContainer.stringVal += "("
for childIndex,rec := range rec_values {
if rec = rec; rec == "" { rec = "NULL"}
checkAlphanumeric, _ := regexp.Compile("([a-zA-Z]+)")
if (checkAlphanumeric.MatchString(rec)) {rec = "'"+rec+"'"}
if csvDataContainer.str = ","; childIndex == 0 { csvDataContainer.str = "" }
csvDataContainer.stringVal += (csvDataContainer.str+rec)
}
if (len(csvDataContainer.first_100) == parentIndex+1) { csvDataContainer.stringVal += ");"
} else {csvDataContainer.stringVal += "),"}
}
psqlConn.InsertRec(csvDataContainer.stringVal)
csvDataContainer.first_100 = nil
fmt.Printf("--------------> %d\n", len(csvDataContainer.remaining))
if (len(csvDataContainer.remaining) !=0) { csvDataContainer.GenertaeString() }
}

Cloud Setup:

We wanted to do an extensive testing, so decided to have a cloud environment in AWS to test the performance in terms of huge data volumes.
The following AWS boxes configurations were considered for running the volume test:

Record Set Sample:

Observation:

The code has been executed against 100k, 500k and 1million record sets. The time taken to update the records in database are captured from the log records. Look below for the time taken in AWS for different volume of record sets for the Golang & Ruby code snippets:

Code Reference:

The complete code scripts and the sample data used are available for your knowledge at the Git repository: https://github.com/agiratech/go_vs_ruby_metrics

Conclusion:

On the considered scenario for study, to upload records into ETL database, Golang code performance is better than Ruby code.
We at Agira, always believe in suggesting the best possible solution to our clients. Here too, our team expended it’s technical knowledge, suggested required technology stack for the product/application development to the clients for improved performance.
We offer Golang development services for building world-class enterprise apps. We have expertise in building most complex software solutions using Google’s Go language. Chat with us now and hire golang developers within 72 hours.