Vim Tricks and Toys

So after yesterdays rant about VIM and its usages I figured that I should talk about some of the toys that I use with VIM that I love.

- Spell check
- Completion
- Auto-completion
- Surround
- Ack
- Ruby Compiler
- Ctags
- Omni Completion
- Nerd Tree
- Fuzzy Finder
- NERD Commenter
- Command T
- Vicle

Yep, list I know. The configuration that goes with it is surprisingly short. God knows that the spell check plug-in is one of the most important to me, considering I spell about as well as a third grader. Auto-completion is brilliant as long as you know how to use it. There are some similar plug-ins that provide language specific shortcuts. These are pretty slick as long as you make sure you are using the one for your current programming language. Using the Perl one with Ruby has interesting results.

There are also a few things that you can do without changing your configuration file. One of the nice ones is
:! irb -r
it executes irb from inside of VIM. Makes testing changes on the fly easy. Also makes it very quick to run commands like
:! script/console generate migration
Right after that you can refresh your NERDTree and keep working.

Here is my vimrc.


"Before merge of files these existed

set cf " Enable error files & error jumping.
set clipboard+=unnamed " Yanks go on clipboard instead.
set history=256 " Number of things to remember in history.
set autowrite " Writes on make/shell commands
set timeoutlen=250 " Time to wait after ESC (default causes an annoying delay)

set nocp
set cinoptions=:0,p0,t0
set cinwords=if,else,while,do,for,switch,case
set formatoptions=tcqr
set cindent
set autoindent
set smarttab
set expandtab
set wrap

" Visual
set showmatch " Show matching brackets.
set mat=5 " Bracket blinking.
set list
" Show $ at end of line and trailing space as ~
set lcs=tab:\ \ ,eol:$,trail:~,extends:>,precedes:< "set novisualbell " No blinking . set noerrorbells " No noise. set laststatus=2 " Always show status line. " ----------------------------------------------------------------------------- " | VIM Settings | " | (see gvimrc for gui vim settings) | " | | " | Some highlights: | " | jj = Very useful for keeping your hands on the home row |
" | ,n = toggle NERDTree off and on |
" | |
" | ,f = fuzzy find all files |
" | ,b = fuzzy find in all buffers |
" | ,p = go to previous file |
" | |
" | hh = inserts '=>' |
" | aa = inserts '@' |
" | |
" | ,h = new horizontal window |
" | ,v = new vertical window |
" | |
" | ,i = toggle invisibles |
" | |
" | enter and shift-enter = adds a new line after/before the current line |
" | |
" | :call Tabstyle_tabs = set tab to real tabs |
" | :call Tabstyle_spaces = set tab to 2 spaces |
" | |
" | Put machine/user specific settings in ~/.vimrc.local |
" | CTAGS C-] - go to definition |
" | C-T - Jump back from the definition. |
" | C-W C-] - Open the definition in a horizontal split |
" | C-\ - Open the definition in a new tab |
" | A-] - Open the definition in a vertical split |
" |
" | After the tags are generated. You can use the following keys to tag into and tag out of functions:
" |
" | Ctrl-Left_MouseClick - Go to definition |
" | Ctrl-Right_MouseClick - Jump back from definition |
"

set nocompatible " We're running Vim, not Vi!

"Set Mapping to ,
"**********************************************************
let mapleader = ","
imap jj "Use jj as escape .. Eaiser?


" Tabs ************************************************************************
"set sta " a in an indent inserts 'shiftwidth' spaces
function! Tabstyle_tabs()
" Using 4 column tabs
set softtabstop=4
set shiftwidth=4
set tabstop=4
set noexpandtab
autocmd User Rails set softtabstop=4
autocmd User Rails set shiftwidth=4
autocmd User Rails set tabstop=4
autocmd User Rails set noexpandtab
endfunction

function! Tabstyle_spaces()
" Use 2 spaces
set softtabstop=2
set shiftwidth=2
set tabstop=2
set expandtab
endfunction

call Tabstyle_spaces()

set ts=2 " Tabs are 2 spaces
set bs=2 " Backspace over everything in insert mode
set shiftwidth=2 " Tabs under smart indenting

"Ctags and other shortcuts
"***********************************************
map :tab split:exec("tag ".expand(""))
map :vsp :exec("tag ".expand(""))

"Sets the tags directory to look backwards till it finds a tags dir
set tags=tags;/

au BufWritePost *.rb silent! !ctags -a --recurse -f ~/dev/tags/cuttlefish &



"Indenting *******************************************************************
set ai " Automatically set the indent of a new line (local to buffer)
set si " smartindent (local to buffer)

" Scrollbars ******************************************************************
set sidescrolloff=2
set numberwidth=4

" Windows *********************************************************************
set equalalways " Multiple windows, when created, are equal in size
set splitbelow splitright

" Vertical and horizontal split then hop to a new buffer
:noremap v :vsp^M^W^W
:noremap h :split^M^W^W

" Cursor highlights ***********************************************************
set cursorline
"set cursorcolumn

" Searching *******************************************************************
set hlsearch " highlight search
set incsearch " Incremental search, search as you type
set ignorecase " Ignore case when searching
set smartcase " Ignore case when searching lowercase

" Colors **********************************************************************
"set t_Co=256 " 256 colors
set background=dark
syntax on " syntax highlighting
colorscheme ir_black

" Status Line *****************************************************************
set showcmd
set ruler " Show ruler
"set ch=2 " Make command line two lines high
match LongLineWarning '\%120v.*' " Error format when a line is longer than 120


" Line Wrapping ***************************************************************
set nowrap
set linebreak " Wrap at word

"Line Number
"*****************************************
set nu " Line numbers on


" Misc settings ***************************************************************
set backspace=indent,eol,start
set number " Show line numbers
set matchpairs+=<:>
set vb t_vb= " Turn off bell, this could be more annoying, but I'm not sure how
set nofoldenable " Turn off folding
set noerrorbells


" File Stuff ******************************************************************
syntax enable
filetype on " Enable filetype detection
filetype indent on " Enable filetype-specific indenting
filetype plugin on " Enable filetype-specific plugins
"compiler ruby " Enable compiler support for ruby

" Ruby stuff ******************************************************************
compiler ruby " Enable compiler support for ruby
map :!ruby %


" Omni Completion *************************************************************
autocmd FileType html :set omnifunc=htmlcomplete#CompleteTags
autocmd FileType python set omnifunc=pythoncomplete#Complete
autocmd FileType javascript set omnifunc=javascriptcomplete#CompleteJS
autocmd FileType css set omnifunc=csscomplete#CompleteCSS
autocmd FileType xml set omnifunc=xmlcomplete#CompleteTags
autocmd FileType php set omnifunc=phpcomplete#CompletePHP
autocmd FileType c set omnifunc=ccomplete#Complete
" May require ruby compiled in
autocmd FileType ruby,eruby set omnifunc=rubycomplete#Complete


" Hard to type *****************************************************************
imap uu _
imap hh =>
imap aa @

" Change which file opens after executing :Rails command
" ****************************************
let g:rails_default_file='config/database.yml'

" Insert New Line *************************************************************
map O " awesome, inserts new line without going into insert mode
map o
"set fo-=r " do not insert a comment leader after an enter, (no work, fix!!)


" -----------------------------------------------------------------------------
" | Plug-ins |
" -----------------------------------------------------------------------------

" NERDTree ********************************************************************
:noremap n :NERDTreeToggle
let NERDTreeHijackNetrw=1 " User instead of Netrw when doing an edit /foobar
let NERDTreeMouseMode=1 " Single click for everything


" NERD Commenter **************************************************************
let NERDCreateDefaultMappings=0 " I turn this off to make it simple

" Toggle commenting on 1 line or all selected lines. Wether to comment or not
" is decided based on the first line; if it's not commented then all lines
" will be commented
:map c :call NERDComment(0, "toggle")

" CommandT ********************************************************
" To compile:
" cd ~/cl/etc/vim/ruby/command-t
" ruby extconf.rb
" make
let g:CommandTMatchWindowAtTop = 1
map f :CommandT


" fuzzyfinder ********************************************************
" I'm using CommandT for main searching, but it doesn't do buffers, so I'm
" using FuzzyFinder for that
map b :FufBuffer
"let g:fuzzy_ignore = '.o;.obj;.bak;.exe;.pyc;.pyo;.DS_Store;.db'
"
"




Enjoy

Vim is Life

If you are a Windows guy, don't bother to keep reading because I am about to go on a rant about development on Linux / Mac. If you are interested, feel free, just a waring.

For those of you still not using VIM as your "IDE," for Ruby development shame on you! Before the TextMate and Emacs fan boys swoop down and kill me, give me a moment to explain my position.

My requirements for a development platform:
1: Free Isn't just good, it is required
In a world where I am dropping a grand or more on a laptop to keep up with the massive amounts of abuse I deal out, a development suite that is free is a must. TextMate users take note, I enjoy TextMate, but have no desier to pay.

2: Compatibility between systems
For the past few years, I have been on a team with mixed operating systems. Macs, Linux (Ubuntu, Fedora, CentOS, Minuix) all of them are in the mix. Switching between pairs for programming requires that we have similar setups, and there for compatible development tools.

3: Bloat is bad, and slow
Current generation IDEs such as RubyMine and NetBeans (with Ruby packages) consume a good chunk of memory and cpu time. This makes them more difficult to run on lower power systems such as VMs or systems pulling double or triple duty. In my case, as with most developers, my development machine is my server, my job box, my queue serer, my poller/ background operations machine, and of course all of the standard office apps one has to run. Allowing my IDE to suck up more then a gigabyte of memory just to load is insanity. Vim and all the plugins I have attached to it, runs in under 256mb. That is right, a quarter of the footprint.

4: Portability is God
The above point brings me to this one. Small is good, small is great, small is powerful. With a development stack that is 256mb, I can now live inside of Minuix. Why? Because now it fits. Load Minuix and Vim on to a USB, now you have a walkabout development environment that fits in your pocket. Going to the girlfriends for the night and need to get some work done? USB and go. No worries about her three year old windows box. Reboot off of the USB and rock some code.


5: Simplicity is power, power is simple
If I am running RubyMine, and my pair is running Emacs, it is almost impossible for us to work on each others systems. Emacs is great, but the level of customization and macros that Emac people seem to run prevent me from taking over the keyboard without spending half the time asking which key does what. This is not something that is worth wile in my opinion. A simple setup is nice. Most of us that have been developing for a while have been exposed to vim for many things. Standard keys for standard activities. All systems that run vim have the same setup. i for insert, escape for command mode, etc. Simple setups allow for simple interaction between pairs.

6: Extensibility
RubyMine comes with everything you need. Or so it claims. In my experience, attempting to get RubyMine to do everything I want it to do becomes an exercise in masochism. You want to run sql from inside of RubyMine? Have fun, you can do it, but its setup is a pain. Just add a MySql, Sqlite3, and an Oracle database to the list of connections and wait for RubyMine to choke itself out. Not fun. Vim has thousands of plugins ready to go that are under active development. I have found everything from typeahead completion, to yaml checkers. Everything you can want you can find. Even found a plugin to do prolog completion.

7: Customization should be optional
I know a few Emacs users that swear by their tool, as they should and more power to them. However, I have issues with a N=1 setup. Emacs seems to attract the type of users that love to customize the setup to the point of obscurity. The user might be blindingly fast at developing on language, but becomes unusable for anything but the intended target. I should not have to customize my development tool in order to use it. Out of the box operation of the tool is valuable to me. I should be able to hop on a brand new install of Nix and use the editor without wondering what happened to my massive list of macros. A simple one line .vimrc file is all you need, and truthfully, it is not even required.

Yea, I know. I am proballly wrong about most of this. But it is my blog, and I needed something to rant about today that wasn't work related.

Using Spawn with a thread / fork limit

Often when you are writing software to provide parallel operations in order to improve performance or to create a non-blocking section of long running code, one tends not to think about the impact of forking many threads. The reason for this in my experience is that you tend to not create more then a handful of threads. This can however, be dangerous. In the example below, we are assuming that there are a reasonable number of Funds in the system for the current context, say ten.


fund_processes = []
Fund.all.each do |fund|
 fund_processes[fund.id] = spawn do
    ... process start(fund) ...
  end
end

wait(fund_processes)


This will not cause any real issues as long as you wish to block to wait for these to finish processing, and that there are a reasonable number of Funds in the system. The issue comes when you do something stupid without thinking about it such as request a set of forks for all users in a 150,000 user system. That is right, you just created 150,000 processes attempting to use the same memory and CPU time, database, and IO as the rest of the system. A "think-o" of this magnitude can bring down your system. Trust me, I know. It is not a pretty site when you lock yourself out of your local job box because the system (8 cores, 12gb ram) does not have enough cycles free to respond to a ssh request. Time to wait and hope, not something you ever want to do with a production system.

To address this concern, I was directed to write up some code that would allow you to pass Spawn a limit on the number of processes to create at any one time. This extension of Spawn allows you to use the same arguments as Spawn itself, with one additional parameter, group_size. Here is the new code that can be found at my git repo.

each_in_parallel_groups_of creates blocks of processing groups with a limit on them. Truly strait forward. One thing to keep in mind when using this is that each group will wait for the longest running process in its group to finish before moving on to the next X items to process.

each_in_parallel_groups_of can be called on any Enumerable. This is a nice trick to call on say:

Funds.all.each_in_parallel_groups_of(5) do |fund|
... process_start(fund) ...
end

This makes it easier to make sure that you keep in mind the amount of resources that any one code block can absorb.

Here is the main code change in the Spawn fork. A request is in to get it pulled into mainline. I will keep you up to date on that.


#Spawns a limited number of threads / forks and waits for the entire group to finish
# accepts same spawn options as spawn
# Robert Meyer / Dean Radcliffe
def each_in_parallel_groups_of(group_size=10, spawn_opts={}, &block)
spawn_opts[:argv] ||= spawn_opts[:process_label] || "default_ruby_fork_process"
spawn_opts[:method] ||= :fork

raise LocalJumpError unless block_given?

self.each_slice(group_size) do |group_job|
fork_ids = [] # reset for each loop
group_job.compact.each_with_index do |item, index|

fork_ids[index] = spawn( spawn_opts ) do
block.call(item)
end
end

logger.info "Wating for #{Process.pid}" if defined? logger
wait(fork_ids)
logger.info "Done for #{Process.pid}" if defined? logger
end
end
<\code>




Using Spawn to circumvent Ruby OCI driver connection issues

  In our system we use Active MQ with a set of pollers to run background jobs such as importing large data sets and publishing information over the wire. For the most part this works great. You generate a message and publish it to the queue and forget about it. Normally everything rocks, except for the case where the runners (pollers) have not had any work for the past X hours, where X >= four hours. In this lovely case, Oracle and Ruby don't like each other any more. Ruby asks oracle for connection information and somewhere in the stack, there is a fifteen minute wait before both sides agree that the current connection to the database is no longer active.

  This sucks. Hard. We have tried many things in order to get Ruby to release the connection without checking with Oracle.
 

dbconfig = ActiveRecord::Base.remove_connection

ActiveRecord::Base.establish_connection(dbconfig)

and before that:

ActiveRecord::Base.establish_connection

and before that:

ActiveRecord::Base.verify_active_connections!

It was insane, everything we tried kept getting hung up in some magical part of the stack that didn't like the fact we were allowing for long running threads with no activity.

I finally found a solution that worked out for us. Amusingly it was while I was working on a section of the code for my own gratification.

Spawn. That's right, Spawn. Spawn is a simple, clean plug-in that helps to take the pain away from forking and threading in Ruby. Besides having a healthy number of fixes for threading issues in Ruby, it provides a strait forward way of creating child processes and waiting for them to complete. Of course there are a handful of options that you can specify but the base case syntax is simple:

spawn do 

   call_to_long_running_process_or_job

end

 That is it. It just works. If you want to wait for the process to complete before moving on:

fork_process = spawn do 

 call_to_long_running_process_or_job

end

wait(fork_process)

Makes life easy. It also provides a workaround for the Oracle time-out issue.

Old code:

def on_message(message)

    logger.info("#{Time.now.to_s} Received request: #{message}")
    ActiveRecord::Base.establish_connection

    #Sometimes our connection goes away when a poller has been waiting a long time for a job, this is the 15 minute hang line of code
    logger.info("#{Time.now.to_s} Finished reconnecting to the database.")

   do_something(message)

end


New code:

 def on_message(message)
    fork_process = spawn do

        do_something(message)

     end

     logger.info("Forked for processing Parent PID (#{Process.pid}) is wating for PID -- #{fork_process.handle}")
    wait(fork_process)
    logger.info("Completed message for Parent PID (#{Process.pid})")

end

No wait time for Oracle to release the connection or provide a new connection. The process gets the message out of the queue, forks itself, and runs it immediately. Makes the users happy, and provided me with enough ammo for a secondary post about threading with limits and lambdas.

Hope this helps someone out, and if not, there are a few blog links on the Spawn ReadMe that also helped me out.





Scott Persinger's blog post on how to use fork in rails for background processing. http://geekblog.vodpod.com/?p=26
Jonathon Rochkind's blog post on threading in rails.
http://bibwild.wordpress.com/2007/08/28/threading-in-rails/

Has and belongs to many through

While working on a legacy data conversion project I decided to create a Rails project in order to cheat so I didn't have to write a ton of SQL statements to dump things to flat files in order to operate on them. However, while doing this, I realized that the table names, and the entire model structure of the legacy application was jacked up hard.

This caused me to use has_and_belongs_to_many with the :through argument along with association_foreign_key and foreign_key. Let me tell you how stupid I felt when I realized I was looking at the wrong version docs. 

Brain needs to be on before turning to Google for the answer.  The code that does what I needed is below:

has_and_belongs_to_many :users, :join_table => "ib_users_accounts_link",
                                  :foreign_key => "accounts_id",
                                  :association_foreign_key => "user_id"