Formatting a 1.6gb XML file

At work today, a client gave me a 1.6gb XML file that I needed to format. Of course they didn't do anything as nice as use line breaks. None, anywhere... Being a hardcore VIM user, I looked up an easy way to sort out the file with indents and tags. Thanks to Justin Kempton I found this: Pretty XML

It works faster then the other VIM plug-ins that I have found and of course is written in easy to modify VIM code.

Custom to_xml for ActiveRecord classes

In an attempt to product client definable API returns, Dean Radcliffe, Jake Scruggs and I decided that using the local files would be a good idea. I mentioned this in a previous post. It is a great idea, and the execution is pretty sweet as well.

I am currently fighting with a way to do the sub level arrays such as

Document has many Document tags

So a layout like
<doc>
<tags>
   <tag>Tag 1</tag>
   <tag>Tag 2 </tag>
<t/ags>
</doc>

where Tags are objects themselves

I got a solution working but it is course and not using Builder XML.

It uses <<--DOCXML DOCXML tags and internally hand hacked xml arguments.

Nasty...

  def to_xmls
    xml_class_fetch = self.class.to_s.downcase
    buff = <<-EOF
    <#{xml_class_fetch}>
    #{ClientConfig.get("xml_export.#{xml_class_fetch}").map do | xml_field_name, method_call |
    case method_call
    when Symbol
      val = self.send(method_call)
    when String
      val = self.instance_eval(method_call)
    end

    if val.is_a?(Array)
      val = val.map do |element|
        if element.respond_to?(:name)
          element_call = element.name
        elsif element.respond_to?(:identifier)
          element_call = element.identifier
        end
        "<#{xml_field_name.singularize}>#{element_call}</#{xml_field_name.singularize}>"
      end
    end

    "<#{xml_field_name}>#{val}</#{xml_field_name}>"
    end.join("\n")}
    </#{xml_class_fetch}>
    EOF
  end


where the feed looks like

  xml_export:
    document:
      # TODO - may add XML element names etc ...
      - ["other-id", :other_id]
      - ['created-at', :created_at]
      - ['tags', "tags.to_a"]



As one can imagine, I am not happy with this solution at all.

If any one has any better suggestions to return valid XML with nested elements that are part of the model's has_and_belongs_to_many as well as other types of associations, please feel free to shoot me a tweet or comment.

SOAP, REST and XML Violence

A few of our client have requested an API to send data and verify uploaded data. This, in general, is pretty standard. As you grow you will run into clients that are more technically savvy then the others or have larger data sets with more frequent updates. If you are uploading three-hundred documents a week to a given web site by hand, well you begin to look for a better way to do things. Sometimes, and only sometimes, APIs are the way to go.


In the world, there is a huge argument over SOAP vs REST and which is the superior API. I am by no means the authority on the matter but it seems to me that there are different use cases for both of them. Now that my opinion is out in the open, I am going to clarify that position:


I HATE SOAP. Every company that I have been forced to use SOAP at drove me insane. Insane I tell you! The amount of overhead that is required to work with SOAP makes me mad. The other part of this is that during the development cycle, the WSDL kept changing without a revision number. The providers I was working with deemed that since the service was still in a development stage, it was not a requirement to version the WSDL. Every time the system was almost complete and I had conformed to all of the SOAP contracts, the WSDL changed out from under me. This might be the cause of some bias on my part....


In the same breath I am going to defend SOAP for having a contract, something that REST lacks in the formal sense. A decent REST resource can be found here.


REST is great for me as a developer in a shop that needs to have a decent velocity with a small head count. REST snaps right over the top of my already established controllers and with a few modifications to the ActiveRecord models I can customize the output of the .xml request.  Dean Radcliffe, one of the other developers has convinced me that storing client configuration for xml output in the I18n files is not a bad idea.

I found a blog who's opinions on API I tend to agree with.
Exert from REST and SOAP: When Should I Use

...Areas that REST works really well for are:

• Limited bandwidth and resources; remember the return structure is really in any format (developer defined). Plus, any browser can be used because the REST approach uses the standard GET, PUT, POST, and DELETE verbs. Again, remember that REST can also use the XMLHttpRequest object that most modern browsers support today, which adds an extra bonus of AJAX.
• Totally stateless operations; if an operation needs to be continued, then REST is not the best approach and SOAP may fit it better. However, if you need stateless CRUD (Create, Read, Update, and Delete) operations, then REST is it.
• Caching situations; if the information can be cached because of the totally stateless operation of the REST approach, this is perfect.

....If you have the following then SOAP is a great solution:

• Asynchronous processing and invocation; if your application needs a guaranteed level of reliability and security then SOAP 1.2 offers additional standards to ensure this type of operation. Things like WSRM – WS-Reliable Messaging.
• Formal contracts; if both sides (provider and consumer) have to agree on the exchange format then SOAP 1.2 gives the rigid specifications for this type of interaction.
• Stateful operations; if the application needs contextual information and conversational state management then SOAP 1.2 has the additional specification in the WS* structure to support those things (Security, Transactions, Coordination, etc). Comparatively, the REST approach would make the developers build this custom plumbing.

Mike Rozlog also makes a good point about XML. XML can be heavy, very heavy if you are transmiting a ton of data over the wire that is very verbose. Mo
on Stackoverflow makes a pointed joke at the cost of verbose XML.

"XML is like violence - If it doesn't solve your problem, you're not using enough of it."

The current application that I am working on has both APIs I am sad to report. One supports the legacy system that feeds it documents and the REST is now being provided to the clients as the API of choice for interactions on a programmatic level. I am happy to say that this is not as horrid as it sounds. The legacy SOAP code handles all kinds of stupid requests and statuses that are not needed by anyone or anything but an ill-conceived piece of stateless Java. As we trasition our clients off of that legacy system, we will be able to DRY up the API controllers, and in this case, remove the API controller entirely as it will have been relpaced by a Restful API.



I will stop ranting now and state, SOAP and REST both have their place in this world, but given the chance I would rather work with REST as a developer. Flicker and Twitter have great examples of REST working well.

--Rob