Polymorphic Paths

One of my friends directed me to this post. Great info on how to use polymorphic paths to save yourself some confusion and work.

http://rookieonrails.blogspot.com/2008/01/sti-views-revisited-or-polymorphic.html

Pluralization Customization in Rails 2

Got stuck with a bit of an issue:

Rails thinks that weave - weaves and that weaves - weaf
Exactly what a weaf is... well I don't know either.

Fix:

ActiveSupport::Inflector.inflections do |inflect|
inflect.irregular 'weave', 'weaves'
inflect.irregular 'weaves', 'weave'
end


in your config/initializers/inflections.rb

Chainmail

I got bored over the course of last week and this weekend and decided that my usual chainmail site is starting to annoy me. It does not provide the reference I want for either my clients or for myself. I have a hard time remembering from order to order what the AR I want for my Elf Weave is. So, new project. Should be interesting and give me something to let the guy I am mentoring learn on.

Links to follow soon.

Using Spawn to circumvent Ruby OCI driver connection issues

  In our system we use Active MQ with a set of pollers to run background jobs such as importing large data sets and publishing information over the wire. For the most part this works great. You generate a message and publish it to the queue and forget about it. Normally everything rocks, except for the case where the runners (pollers) have not had any work for the past X hours, where X >= four hours. In this lovely case, Oracle and Ruby don't like each other any more. Ruby asks oracle for connection information and somewhere in the stack, there is a fifteen minute wait before both sides agree that the current connection to the database is no longer active.

  This sucks. Hard. We have tried many things in order to get Ruby to release the connection without checking with Oracle.
 

dbconfig = ActiveRecord::Base.remove_connection

ActiveRecord::Base.establish_connection(dbconfig)

and before that:

ActiveRecord::Base.establish_connection

and before that:

ActiveRecord::Base.verify_active_connections!

It was insane, everything we tried kept getting hung up in some magical part of the stack that didn't like the fact we were allowing for long running threads with no activity.

I finally found a solution that worked out for us. Amusingly it was while I was working on a section of the code for my own gratification.

Spawn. That's right, Spawn. Spawn is a simple, clean plug-in that helps to take the pain away from forking and threading in Ruby. Besides having a healthy number of fixes for threading issues in Ruby, it provides a strait forward way of creating child processes and waiting for them to complete. Of course there are a handful of options that you can specify but the base case syntax is simple:

spawn do 

   call_to_long_running_process_or_job

end

 That is it. It just works. If you want to wait for the process to complete before moving on:

fork_process = spawn do 

 call_to_long_running_process_or_job

end

wait(fork_process)

Makes life easy. It also provides a workaround for the Oracle time-out issue.

Old code:

def on_message(message)

    logger.info("#{Time.now.to_s} Received request: #{message}")
    ActiveRecord::Base.establish_connection

    #Sometimes our connection goes away when a poller has been waiting a long time for a job, this is the 15 minute hang line of code
    logger.info("#{Time.now.to_s} Finished reconnecting to the database.")

   do_something(message)

end


New code:

 def on_message(message)
    fork_process = spawn do

        do_something(message)

     end

     logger.info("Forked for processing Parent PID (#{Process.pid}) is wating for PID -- #{fork_process.handle}")
    wait(fork_process)
    logger.info("Completed message for Parent PID (#{Process.pid})")

end

No wait time for Oracle to release the connection or provide a new connection. The process gets the message out of the queue, forks itself, and runs it immediately. Makes the users happy, and provided me with enough ammo for a secondary post about threading with limits and lambdas.

Hope this helps someone out, and if not, there are a few blog links on the Spawn ReadMe that also helped me out.





Scott Persinger's blog post on how to use fork in rails for background processing. http://geekblog.vodpod.com/?p=26
Jonathon Rochkind's blog post on threading in rails.
http://bibwild.wordpress.com/2007/08/28/threading-in-rails/

Has and belongs to many through

While working on a legacy data conversion project I decided to create a Rails project in order to cheat so I didn't have to write a ton of SQL statements to dump things to flat files in order to operate on them. However, while doing this, I realized that the table names, and the entire model structure of the legacy application was jacked up hard.

This caused me to use has_and_belongs_to_many with the :through argument along with association_foreign_key and foreign_key. Let me tell you how stupid I felt when I realized I was looking at the wrong version docs. 

Brain needs to be on before turning to Google for the answer.  The code that does what I needed is below:

has_and_belongs_to_many :users, :join_table => "ib_users_accounts_link",
                                  :foreign_key => "accounts_id",
                                  :association_foreign_key => "user_id"

Custom to_xml for ActiveRecord classes

In an attempt to product client definable API returns, Dean Radcliffe, Jake Scruggs and I decided that using the local files would be a good idea. I mentioned this in a previous post. It is a great idea, and the execution is pretty sweet as well.

I am currently fighting with a way to do the sub level arrays such as

Document has many Document tags

So a layout like
<doc>
<tags>
   <tag>Tag 1</tag>
   <tag>Tag 2 </tag>
<t/ags>
</doc>

where Tags are objects themselves

I got a solution working but it is course and not using Builder XML.

It uses <<--DOCXML DOCXML tags and internally hand hacked xml arguments.

Nasty...

  def to_xmls
    xml_class_fetch = self.class.to_s.downcase
    buff = <<-EOF
    <#{xml_class_fetch}>
    #{ClientConfig.get("xml_export.#{xml_class_fetch}").map do | xml_field_name, method_call |
    case method_call
    when Symbol
      val = self.send(method_call)
    when String
      val = self.instance_eval(method_call)
    end

    if val.is_a?(Array)
      val = val.map do |element|
        if element.respond_to?(:name)
          element_call = element.name
        elsif element.respond_to?(:identifier)
          element_call = element.identifier
        end
        "<#{xml_field_name.singularize}>#{element_call}</#{xml_field_name.singularize}>"
      end
    end

    "<#{xml_field_name}>#{val}</#{xml_field_name}>"
    end.join("\n")}
    </#{xml_class_fetch}>
    EOF
  end


where the feed looks like

  xml_export:
    document:
      # TODO - may add XML element names etc ...
      - ["other-id", :other_id]
      - ['created-at', :created_at]
      - ['tags', "tags.to_a"]



As one can imagine, I am not happy with this solution at all.

If any one has any better suggestions to return valid XML with nested elements that are part of the model's has_and_belongs_to_many as well as other types of associations, please feel free to shoot me a tweet or comment.

Rails Join Tables

Wow,

Bad morning. Took me 30 min to find out that the reason I was throwing an error on a controler was not due to bad code, but due to a bad migration that I wrote.

When writing the join table migration I neglected to specify that there was no primary key. Oracle HATES this.

  def self.up
    drop_table :email_blasts_users
    create_table :email_blasts_users, :id => false do |j|
      j.references :email_blast
      j.references :user
    end
  end

Make sure to have that :id => false if you don't want your database to yell at you.

Requirement / RPM Hell

 For those of us that have been around long enough on Red Hat distros, we remember RPM Hell. A cyclic series of RPMs that depended on each other, thus RPM Hell. Trying to find the one link that would let you install the rest of the packages was often a exercise in frustration that made you think that your career path should have been in the area of Great White shark research and not software development. Our upcoming retrospective, and this XKCD made me think of RPM Hell.

There is a bit of back story require in order for the rest of this to make any kind of sense.....

Our infrastructure and staff have gotten much better over the course of the last year. When I first started on this team, it was as a solo developer attempting to transform a questionable architecture into something resembling sanely designed system.  I had some backup and support from the team's lead, Tim Galeckas <@timgaleckas>, and some direction in the most general sense from the companies CTO, but not much more than that. It was one of those projects where the general gist is "Fix it" where "Fix it" is about all the direction you can expect. As one can imagine, the project was a huge time sink with minimal return.

A few months after that project was terminated, the new effort to rebuild the system in RoR got into full swing. We still had issues of course, nothing phoenixing from a process that severely broken can be without defects. As these issues became apparent, changes came to my little world.  

The view in the team is now so divergent from the original it is hard to see how one originated from the other. We have a project manager, a user interface designer, four developers, two quality assurance members, and a true to life development manager. There is even a process in place to score, scope, and select stories <tickets, cards, issues> for development in any iteration. All that being said, there are still problems.

Our primary issues, in my opinion, is not the lack of staff, the backlog of stories, the intra-team interaction, or quality assurance backlogs. It is the quality of requirements that the development team receives. I have been converted from the waterfall style development environment that I first worked in to an agile approach, so complaining about story requirements might seem a bit odd. I do not enjoy a feature request that leaves nothing interesting for me to do. When presented with a story that tells me how to technically solve the issue, visually present the results, and what should be tested I sigh. These stories are what turns us into code monkeys and not professionals. This is a fine line to tread, I want enough information so when I deliver a piece of functionality it is complete, does what is supposed to, looks good, and is performant.  What I see as our RPM Hell in my development space is stories that look complete, but have hidden requirements or functionality that is only available if you tap the brain of the primary stakeholder.

My current example is with a story about displaying information from a third party system and how it weaves into the current system. The conversation and stories development went something like this.

End consumers of InvestorBridge what to be able to view fund level information.

Great, no problem. What does that mean? 

Well the system that we pull information from has a huge data set of fund level information. 

 Okay, we can already interface with that system so what information do you want? 


It is on the story.

That is something I love to hear. To me that means that I can load up the story, read it, and understand what they want. This is almost what happened. Almost. The story told of things like fund level returns and AUMs (Assets Under Management)  and the display of these was to follow current displays of account level information of the same type. Outstanding, easy to do, easy to validate, easy to import.

Where everything fell apart was that there happened to be a document attached to the story that contained additional requirements. That is right, the story had an attachment that modified the context of development. In our process attachments are most often images of expected display or test data. I have never before seen attachments as additional requirements. Requirements should be in plain text on the story. This is the standard and what the developers expect. Where this devolved from a process to a cyclic definition followed by more questions is when the list of additional fields broke the current display model. This brought in our UI guy as well as the project manager. Scope creep was inevitable at this point. New views, click paths, and imports were all required after this document was rediscovered.

This brought to light two things

1. We, as a team, failed to understand the story and the feature requested
2. We, as both a unit and individuals, failed to ask questions that would have mad this apparent

 This felt like RPMs all over again. If I would have known what RPM was the keystone RPM, I would have been able to easily install the software and understand its dependencies. In much the same way, if I would have known what questions to ask, I would have been able to see the full scope of the story and its underlying implications.

So, how do you know what questions to ask when you don't know that you don't what you don't know? Where to start? Where is my yum for feature requests?

SOAP, REST and XML Violence

A few of our client have requested an API to send data and verify uploaded data. This, in general, is pretty standard. As you grow you will run into clients that are more technically savvy then the others or have larger data sets with more frequent updates. If you are uploading three-hundred documents a week to a given web site by hand, well you begin to look for a better way to do things. Sometimes, and only sometimes, APIs are the way to go.


In the world, there is a huge argument over SOAP vs REST and which is the superior API. I am by no means the authority on the matter but it seems to me that there are different use cases for both of them. Now that my opinion is out in the open, I am going to clarify that position:


I HATE SOAP. Every company that I have been forced to use SOAP at drove me insane. Insane I tell you! The amount of overhead that is required to work with SOAP makes me mad. The other part of this is that during the development cycle, the WSDL kept changing without a revision number. The providers I was working with deemed that since the service was still in a development stage, it was not a requirement to version the WSDL. Every time the system was almost complete and I had conformed to all of the SOAP contracts, the WSDL changed out from under me. This might be the cause of some bias on my part....


In the same breath I am going to defend SOAP for having a contract, something that REST lacks in the formal sense. A decent REST resource can be found here.


REST is great for me as a developer in a shop that needs to have a decent velocity with a small head count. REST snaps right over the top of my already established controllers and with a few modifications to the ActiveRecord models I can customize the output of the .xml request.  Dean Radcliffe, one of the other developers has convinced me that storing client configuration for xml output in the I18n files is not a bad idea.

I found a blog who's opinions on API I tend to agree with.
Exert from REST and SOAP: When Should I Use

...Areas that REST works really well for are:

• Limited bandwidth and resources; remember the return structure is really in any format (developer defined). Plus, any browser can be used because the REST approach uses the standard GET, PUT, POST, and DELETE verbs. Again, remember that REST can also use the XMLHttpRequest object that most modern browsers support today, which adds an extra bonus of AJAX.
• Totally stateless operations; if an operation needs to be continued, then REST is not the best approach and SOAP may fit it better. However, if you need stateless CRUD (Create, Read, Update, and Delete) operations, then REST is it.
• Caching situations; if the information can be cached because of the totally stateless operation of the REST approach, this is perfect.

....If you have the following then SOAP is a great solution:

• Asynchronous processing and invocation; if your application needs a guaranteed level of reliability and security then SOAP 1.2 offers additional standards to ensure this type of operation. Things like WSRM – WS-Reliable Messaging.
• Formal contracts; if both sides (provider and consumer) have to agree on the exchange format then SOAP 1.2 gives the rigid specifications for this type of interaction.
• Stateful operations; if the application needs contextual information and conversational state management then SOAP 1.2 has the additional specification in the WS* structure to support those things (Security, Transactions, Coordination, etc). Comparatively, the REST approach would make the developers build this custom plumbing.

Mike Rozlog also makes a good point about XML. XML can be heavy, very heavy if you are transmiting a ton of data over the wire that is very verbose. Mo
on Stackoverflow makes a pointed joke at the cost of verbose XML.

"XML is like violence - If it doesn't solve your problem, you're not using enough of it."

The current application that I am working on has both APIs I am sad to report. One supports the legacy system that feeds it documents and the REST is now being provided to the clients as the API of choice for interactions on a programmatic level. I am happy to say that this is not as horrid as it sounds. The legacy SOAP code handles all kinds of stupid requests and statuses that are not needed by anyone or anything but an ill-conceived piece of stateless Java. As we trasition our clients off of that legacy system, we will be able to DRY up the API controllers, and in this case, remove the API controller entirely as it will have been relpaced by a Restful API.



I will stop ranting now and state, SOAP and REST both have their place in this world, but given the chance I would rather work with REST as a developer. Flicker and Twitter have great examples of REST working well.

--Rob

Selenium and Cucumber testing

 I got talked into a Tech Talk for our internal conference, I decided if I have to do it, I am going to do it on something that is useful to my team. Thus, cucumber and Selenium. I plan on covering the following points..

• Why cucumber
• Selenium and ruby / java
• front end verification
• good / bad practices with cucumber
• Transactional Issues
• ID Problems

More to come on this later