Welcome to blog.chinoy.com Sign in | Join

tech (RSS)

Technology, in general

App A Weekend: My First Android App

posted Thursday, November 04, 2010 8:40 PM by hussain

Edit 11/08/2010 - I was inaccurate when I stated that the NRCS WAI has only a SOAP/RPC access - it has GET and POST as well. I'm currently revising my Android app to parse the WAI XML.

First things first: it’s aboot time.  I’ve spent enough time dinking around with Android that this post should’ve occurred a long time ago.  So, here it is, enjoy.

Recently, I developed an Android application to access the USDA NRCS’s public Where Am I? (WAI) service.  The WAI service provides land-related information from a geospatial location within the United States.  This took about the equivalent of a weekend (16 hours or so), but it was spread out over about a week, so I'm still calling this an "App a Weekend" entry. Deal with it. There are a few gotchas and points along the way that are worth noting as I developed this application.

The Where Am I? service provides information for a geospatial point on the map: Congressional District Code, ID, Name, Congressperson’s name, Hydrologic Unit Code, Major Land Resource Area, City, State, Zip, and Tribal Land ID/Name, among other information.  The service drills through multiple layers of geospatial data and gives back an attribute list.  Useful for some purposes when doing land-based assessment as a lot of NRCS applications may do, what with them being the Natural Resources Conservation Service.

The idea for the WAI Android application came from my work inventorying potentially enterprise-ready existing shareable services at the NRCS, which ones are being shared, and which ones were intended to be shared.  Additionally, some discussions with developers at the USDA about mobile access to NRCS data services needed some level of pragmatic understanding.  Why not write an Android app and exercise the API just enough to make a semi-useful app and exercise the end-to-end Android dev process?

Semi-Useful Design

My intended design was to have an Android application that not only displayed a map from which the visible latitude and longitude could be obtained but also use the geolocation services of the Android device (GPS, network) to obtain the latitude and longitude.  The geospatial coordinates could then be passed to WAI and the WAI attribute results would be displayed.  Pretty straighforward.

The Where Am I? Service

The WAI service is written in .NET and exposed as ASMX which provides a WSDL as an API.  A few issues with this right off the bat.  Android has no native SOAP client to dynamically create a proxy.  There is, of course the Apache Axis library, but those are huge, ~8mb, and the proxy client built by Axis is also pretty large.  Size is an issue in any mobile app and including libraries not specifcally optimized for the platform's not recommended. There’s an Android implementation of ksoap2, but there were lots of comments of its usability and feasability, so I passed on that one.  Additionally, XML parsing is reputed to be quite expensive in Android.  Another issue is that the WAI service is hosted behind a port redirector and therefore WSDL for the service indicates its endpoint at port 42500, which is incorrect.  This issue causes dynamically generated SOAP proxies to fail.  The work around is to manually generate the proxy (using wsdl.exe in .NET for example) and then manually correcting the port.  SOAP access to WAI seems less and less likely.

For a mobile device, it really seems like a REST/document style interface with json would be better than an RPC style interface, given the potential size of a proxy client and supporting libraries. Why not write one?

The REST Service for WAI

I wrote a very quick REST service using ASP.NET MVC2 and pushed this app to Azure, http://wai.cloudapp.net

It has a very basic human user interface and some documentation on the single existing REST method.  Currently, there's one method on the controller that takes lat and lon value, calls the USDA NRCS SOAP webservice via a manual proxy class, and returns a full set of data attributes as a JsonResult object.

Creating an Azure app from an existing MVC2 app was cake - add a Windows Azure Cloud Service project to the existing solution. Publishing to Azure was surprizingly easy, with one hitch: our worktops are XPSP3 and the latest Azure SDK for VisualStudio 2010 will only work on Vista/7.  I published to the cloud using my personal laptop (Windows 7).

It might’ve been slicker to use Google App Engine to host the REST proxy, but the nerd humor of a GOOG app calling MSFT service can't be beat.

Finally, the Android App

With the REST service in place, utilizing in an Android app was a snap.  Obtaining a GPS/Network location from the device is well explained in the online SDK.  Writing a REST client in Java within a thread is as you’d expect (although this is not the recommended pattern, see this very informative talk from I/O on Developing REST Client Applications; a talk I opted not to attend and am grateful they posted the talk and pdf slides online).

I made a main Activity class that contains the Maps MapView, one for the Aboot and WAI Results dialog boxes, and another for the Preferences (an upcoming enhancement). Two straight Java classes, a REST client to house static methods around issuing an HttpClient GET call, and an interface class for my RestRequestCallback that my main Activity implemented in order to get the info back from the threat that starts the REST call.

I obtained a Google Maps API key in order to utilize the Google Maps API, which is separate from the default Android API.  It’s used in order to enable the tile service of the Maps API to function.  Also, I had a development (“debug”) key per machine I was developing on (I used both my desktop and laptop to develop the app).  This meant i had to keep changing the hash string that enabled the Maps API in the view layout, which wasn’t hard, just slightly inconvenient.

Testing on the emulator's really easy and I'm always impressed with the SDK's qemu running Android. Once started, it's snappy and responsive. The emulator has no GPS, so you've got to push a lat, lon to it via the DDMS emulator interface ui or via telnetting to the emulator and issuing a "geo fix lon, lat" command to simulate a location fix. Incidentally, there's a Galaxy Tab add-on for the SDK so you can skin the emulator to look like the upcoming 7" device, if you're so inclined.

Signing an application is actually really easy from within Eclipse -  it’s a menu item added to the right-click context menu by the Android SDK plug-in.  It’s not really that hard via the command line, either.

Publishing to the Android Market was also an interesting experience.  You upload the apk (compiled Android app) and a few screenshots (optional).  Once you do, there’s an entry in the developer listing.  And it can’t be removed.  Even if you unpublish your app, the app remains in your list (not on the Market, but in your developer dashboard for the Market).  I initially compiled my app in a very generic namespace, net.bespokesystems, and then wanted to change it to net.bespokesystems.demo.RestLocation, a much better namespace.  Seems like the namespace is the key for the developer dashboard, so no changing that - got to publish a new app.  And wouldn’t you know, can’t remove the app from my dashboard.  So, although unpublished, there’s an app entry that’s there.  At the time, there were no users of the app (I’d installed the apk manually on my Nexus One), but after a few days, oddly, there were!  It seems like there must be people who automate installation of apps as they’re published, maybe some feed that I’m unaware of plus a delay in reporting of installations.  I’d literally had the poorly-namespaced app published in the Market for less than 20 min.  I’ll have to follow up on either on the dev mailing lists or irc channel to determine some of the Market's logistics. One thing I didn't have to do was wait (I'm looking at you Apple App Store).

Using the Google Chart API for QR codes, I created the clever little QR image above that, when scanned with the Barcode Scanner app on your Android phone, directs you to the download for the app on the Market.

There are some interesting bits in the Android app lifecycle - rotating the screen pauses and restarts the app - and as a developer one has to accomodate for saving state.  I got a bit into the Preferences API, alternate layouts and screen depths, and making a simple icon.

I hope to place both the Android app and the Azure MVC app in a public repository soon for anyone who wants to see some awesome slapdash code.

A few ideas on future updates:

... for the Android app

  • Use a better pattern for accessing REST services
  • Allow the user to change the map type, from satellite to road
  • Reduce REST query timeout to accomodate for NRCS WAI service response timeouts
  • General polish (release notes, proper liste provider for WAI results)
  • Better control over GPS/Network/Wifi location behavior

... for the WAI REST service include

  • REST methods to allow the request to specify which WAI attributes to return
  • Google map for the UI piece
  • Reduce timeout to NRCS WAI service and add a single retry to accomodate for the NRCS WAI service’s timeout issues
0 Comments

Wolfram|Alpha & AZ Crime Rates

posted Wednesday, August 04, 2010 2:15 PM by hussain

This is a test of a very simple Wolfram|Alpha beta widget about Arizona Crime Rates:

0 Comments

Illumina to sequence genome for $19.5k

posted Wednesday, June 16, 2010 6:27 AM by hussain

Your Genome Is Coming, June 3, 2010, Forbes
Illumina, maker of sequencing machines, announced a MD-requested full genome sequence price of $19,500.

A great chart from Gregory Lucifer of Life Technologies showing the cost of sequencing a human genome vs. Moore's Law, below.

0 Comments

H+ Summit Links

posted Monday, June 14, 2010 8:24 AM by hussain

The H+ Summit was this weekend at Harvard and was live broadcast on the web. In a while, H+'ll have sessions on line. Until then, a few links:

Of random interest:

0 Comments

Ubuntu 10.4 Netbook Edition on the Lenovo S10-2

posted Sunday, May 02, 2010 7:47 AM by hussain

The Lenovo S10-2, with it's oversmall keyboard and smallish screen, has taken a permanent back seat to the iPad for livingroom and wandering around computing. The S10-2 has Windows 7 on it, a process which took a while, so I figured that I'd experiment and put the latest Ubuntu 10.4 Netbook Edition on it to see if it was snappy and responsive. UNE is supposed to have a new layout / launcher that's geared towards netbooks.

The download site has a link that describes how to create a bootable USB, which is great. Great because 1) the Lenovo S10-2 has no CD drive, 2) there weren't any good clear instructions on how to do this for the Windows 7 upgrade I'd performed and 3) the instructions were provided right there, on the download page, not some "go to this shady site" reference.

Creating a USB stick went fairly smoothly and I was able to get a stick that'd boot, granted I had to hit F12 in the startup process to choose the USB stick, and perusing the UNE interface in livecd mode gave me confidence. The wacky "application as tabs" sort of thing and the left tab-section for applications as "apps" seemed to work for the small screen.

There were multiple issues, though, when I wanted to pull the trigger and install the OS on to this machine.

First, Avira antivirus had to be disabled to use usb-creator. Pretty minor, really. The usb-creator.exe is provided on the iso of ubuntu, so mounting the iso, grabbing the exe, and then running it was no problem. Avira chirped at an unknown autorun.inf, which is a good thing for antiviruses to do.

Second, my USB stick was NTFS formatted and usb-creator likes FAT32, so I reformatted it.

Third, there's an option in usb-creator to store documents and settings on the USB when starting up from the USB. Don't choose it, choose instead the "discarded on shutdown, unless you save them elsewhere" option. Ubuntu won't boot in livecd mode or install mode with that option chosen. It'll drop to a command prompt with the error "can not mount /dev/loop1 on /cow". Really frustrating. I googled around until I found a few helpful hints and was able to get past that.

Fourth, and most importantly, the Lenovo partition scheme seems to be the blocker when installing. Ubuntu can't figure it out and "parted_server" crashes after the "choose your keyboard setup" screen in the wizard. I found a few references to that on the internets, but nothing was terribly helpful, so I ended up booting in livecd mode and using Ubuntu's disk management Disk Utility to wipe all the partitions. I wasn't going to be using Win7 and the Lenovo restore partition was only good for XP, so bye-bye Windows 7, helo Ubuntu.

Until ChromeOS comes around.

0 Comments

Genetics Roundup

posted Friday, April 30, 2010 11:15 PM by hussain

Stanford bioengineer explores own genome, 04/30/2010, Silicon Valley Mercury News

Lucky bastard.

A decade ago, sequencing of the first-ever whole genome by the federal government took many years and cost $400 million to $500 million. Quake's machine, the size of a freezer, sequenced his human genome in only four weeks, for $50,000. The procedure is expected to cost $10,000 by the end of this year.

U. scientist links one gene to intelligence, 04/22/2010, Salt Lake Tribune

Interesting. STX1A has a variant on SNPedia, Rs3793243.

The gene in question plays a central role in neuro-transmission, particularly in the areas of the brain associated with learning, memory and fear.
"We're talking about a basic utility when we look at STX1A," [Julie] Korenberg said. "This study shows in part how nature's hand shapes intelligence at the synapse."
... She and colleagues at California institutions performed genetic testing on 65 patients with Williams Syndrome, an uncommon congenital developmental disorder that appears in only one in 20,000 births. People with the syndrome are genetically similar to other individuals except they are missing 27 genes.

SNPwatch: Genetic Variant May Impact Rate of Cognitive Decline in the Elderly, 04/26/2010, The Spittoon, 23andme's blog

Variants of Rs4680 have different effects over time.

New research, published recently in the journal Neurology, has found the surprising result that a genetic variant previously associated with better cognitive function in young people appears to have the opposite effect as people get older.
I'll be interested to see what my snps reveal, and also what sort of mitigation environmental effects might have.

0 Comments

DNA Day Chat Transcription Expresses Humor

posted Sunday, April 25, 2010 5:01 PM by hussain

DNA Day chat room transcribes humor:

  • Q: James Woods Elementary in MA (5th grade student): My mom says she can never fit her genes. Can you fix her DNA so she can fit them?
  • A: Sarah Harding, M.P.H: Unfortunately, no...your genes pretty much stay the same from the day you are born. It's your environment that can change...Tell her to buy bigger pants.

2010 National DNA Day Online Chatroom Transcript, genome.gov
0 Comments

Today's favorite iPad App: Epicurious

posted Monday, April 05, 2010 10:25 PM by hussain

... and it's free!

From iPad Apps

0 Comments

iPad Thoughts

posted Saturday, April 03, 2010 10:20 PM by hussain
Watching a video on the iPad is really nice - great clear, colorful HD screen - with one exception: the glare.  I watched a handbraked episode of Stargate Universe, as well as Jamie Oliver's Food Revolution and an episode of Modern Family via the ABC Player iPad application.

Overall, the apps for the iPad - ones designed to take advantage of the larger screen - are wonderfully useful.

Specific iPad applications

The Yahoo! Entertainment app is much more useful than the web version.  It's too bad neither TiVo or MSN have iPad formatted apps like this.  Customizing TV listings is very intuitive, poking at on/off choices with a finger is even more intuitive than pointing and clicking with a mouse.

The NPR app is fun, too, with streaming access to affiliates as well as a triple stacked stacked news story layout.

The Yahoo! Entertainment app crashed a few times, as did the NPR app, returning to the home screen.

The IMDB application was really fun.  The wife and I sat around looking up movies and reading trivia and being silly running lines from The Karate Kid and Office Space.  All on the couch, without a keyboard.  That part truly shows off how the same data, different interface and in a movie-watching setting (couch) really can make the power of the internets click.  I understand why Steve says it's a magical device.

ABC Player is fantastic, access to watch shows on demand is reminiscent of Hulu.  I have to say that I was right on the edge of buying an iPad and the Modern Family episode featuring it had a little bit to contribute to pushing me over.  ABC being owned by Disney with Steve Jobs as a 7% shareholder probably had a something to do with that.

Cool Hunting and Gilt's apps are also really great; easy browsing of the articles and items to buy are formatted to take advantage of the screen and input style.  Here, also, Gilt's application has crashed once or twice on me.

With the larger storage space (I got the 32gb model), I think what might be an issue soon is more organization for certain applications.  It sure is nice to have space for a lot of music, but music doesn't need screen size as the successful iPod line can attest - videos and pictures do.  And both videos and pictures need some level of organization and categorization.  On iTunes, on the desktop, I can only select a single level of folders for photos, whereas on the iPad, it has a more sophisticated level of organization: all thumbnails, or view by date or view by location.  This is clever and pleasing.  With videos, it's just by all thumbnails and once people gear up and buy tv shows and movies via iTunes, some better level of organization will be needed.

Some Technical Thoughts

Using the various applications on the iPad makes me think of the early days of the World Wide Web when various different browsers were coming out and HTML implementations were fragmenting and CSS was implemented spottily.  The big angst and handwringing was about having to design a website multiple times just to cover all the browsers that were out there.  It's clear that there's a divide between web browsers made for desktop and laptop devices (15"+ lcds, but more likely 19"+) and handheld/phone devices (3.7" screens or so) - things that fit in your pocket.  The iPad is a new class of device - can't fit it in your pocket, but you don't want it to be a traditional computer - at about 9.5".  This is the size of the "netbook" which hasn't really caused any inspiration at all, except maybe in the NGO sector as cheap laptops for the developing world (see XO, etc.)

The varying interfaces are also of concern.  Right now, there're very few (free) iPad formatted apps.  That, of course, will change.  When Flash rose to prominence (and even now) the 

Another thought comes to mind with Windows 7 Phone Series - Microsoft's touting that you can write Silverlight (browser-based) or XNA (XBox 360) applications and have them run on WinMo7 devices.  This hearkens back to the horrible fail of Sun's tagline for Java "write once, deploy anywhere."  No one will want to run the same silverlight browser-based application (19"+ screen) or XBox game in the same format as a mobile device (3.7" screen).  They're two different types of media.

USB charging is an issue - it just doesn't work on (some) computers.  On all the ones I have, the iPad won't charge - I have to use a wall outlet.  There are apple support articles on this: http://support.apple.com/kb/HT4060 and http://support.apple.com/kb/HT4049.

I took a peek at some of the crash logs and most of them appear to be Y! Entertainment out of memory ones and one Mobile Safari one.

Accessories

Seems like there're more accessories to shake a stick at yet, never the right ones.  In particular, there needs to be an unobtrusive and heavy stand that can hold the 1.5 lb iPad in landscape and, most importantly, the angled, top heavy portrait mode.  An ideal stand height would be about the height of the iPad's bezel, and black in color.  I'm temporarily using the WD TV stand I have and it's too light (can't hold the iPad in portrait mode) and too high (about twice the height of the iPad bezel), but it's functional.

The Macally ViewStand looks quite clever, mimicing an iMac type appearance for a landscape view: http://www.macally.com/EN/Product/ArticleShow.asp?ArticleID=325

The M-Edge Trip Jacket appeals to the Molskine lover in me: http://www.medgestore.com/products/ipad-trip.psp

I'd also like to see more clamp-type accessories so I can have the iPad at various angles in various places.
0 Comments

Notes for Today

posted Sunday, March 07, 2010 7:14 PM by hussain

The Collatz Conjecture is a mathematical problem which states that given any number, and following a simple formula, converges to 1.

Take any number and do this:

  • If odd, multiply by 3 and add 1
  • If even, divide by 2
Repeat. You'll get to 1. Every time.

A comment about the latest xkcd comic's reference to the Collatz Conjecture, which I had never heard of before, lead me to create Hailstone sequence; my first Google AppEngine web app.

1 Comments

Notes for Today, March 6th

posted Saturday, March 06, 2010 8:17 PM by hussain
Notes for Today: March 6, 2010

Back when I started my blog, I would have entries that were simply lists of links that I'd come across and thought worth sharing or brief events during the day.  Never mind "sharing with whom."  The impetus to share, presuming some sort of tenuous permanence seems like a decent rationale for blogging.

Today, one of the tabs open was a Wolfram|Alpha preview search for Academy Award nominations for the recent Star Trek movie (makeup, sound editing, sound mixing, and visual effects).  That lead me to the Star Trek wikipedia page where I read about some of the backstories on casting, etc.  I spent a few hours rewatching that awesome movie.

Saturday's my day to work myself into a little frenzy about savings etc, so I listened to Marketplace Money and called TiVo to follow up on cancelling my subscription from a long while back and that an acceptable refund was issued.  I still sort of want one of the new super cool HD TiVo Premiers because Comcast's DVR is just awful.

Later, I watch the latest episode of Caprica and lamented (privately) that the Facebook fan page for Caprica showed the closing climactic scene of Friday's episode as a preview last week, pretty much making episode 6 literally anticlimactic. I also looked up the word apotheosis that Sister Clarisse likes to say.

I read a recent first hand report of someone who attended Singularity U's executive conference and got to thinking about small-cap biotech ETFs as the next investment bubble.  A bit of Googling came to a decent seekingalpha article that mentioned XBI, BBH, and FBT.  Apparently, the Chinese government's bought $96m worth of Illumina genetic sequencing machines (@ $750k a pop) - the same machines used by personal genomics companies 23andme, decodeme, and counsyl. Will the new phrase be "cheap chinese genomes"?

Back to Singularity U, I watched Dr. Daniel Reda's talk on Biotechnology Fundamentals and wondered if I could memorize the RNA codes for all the amino acids. May be.  It's got to be like learning hex or anything else computational.

Optimization efficacy of evolutionary techniques

Natural selection
  • Slow!
  • Optimized for selecting the best replicators
  • Builds on previous adaptations (doesn't optimize best adaptations)
  • Optimization principle: Just good enough - ie selected for whatever's just good enough to pass on genes, not for any longer (healthy life, etc.)
Human Intelligence
  • Recombinant DNA technology - cut & paste via enzyme restriction endonuclease + ligase
  • DNA printer - writes DNA
  • http://www.bio-era.net/
Recursive AI


 If you haven't seen Harvard's BioVisions animation of the cell, you should.




Protein folding
    Game: http://fold.it/portal/

http://online.wsj.com/article/SB124207326903607931.html

"Killer Apps"
  • Drug metabolism
  • High-risk drugs

Edit, 04/25/2010 (DNA Day)

The new machine, the HiSeq2000, will begin shipping next month with a cost of $690,000 vs. $500,000 for Illumina's current model. It is being unveiled today at J.P. Morgan's investment conference in San Francisco. The Beijing Genomics Institute will be the first customer, purchasing 128 of the new machines.
Illumina's Cheap New Gene Machine Matthew Herper, 01.12.10, 03:00 PM EST
0 Comments

Multiple Issues with WCF Makes the Choice Easy

posted Thursday, January 07, 2010 9:39 PM by hussain
Here's the situation: I've a stateless web app that uses jQuery to hit another application that's just a set of services.  I chose this model because I wanted to separate out the user interaction code from the information interface - the services simply give out data upon request and the UI allows the user to interact with it enough to make calls to the service interface.  Seems rational, simple and clean.

I chose WCF for the service layer and began playing around.  I'm a rather late adopter in learning WCF, primarily because it seemed so heavyweight.  Wrapping all sorts of communications methodologies into one sounded fairly ambitious.  To do a simple thing like return XML or JSON, there're a lot of pieces that need to be put in place: interfaces, somewhat arcane attributes - good practices, mind you with encapsulation of the attributes and the use of interfaces - but this could all be done and simpler with good old asmx's.  Once ASP.NET MVC came along, most of the web-facing usage for WCF is pretty much trumped and made obsolete.  Regardless, I persisted and made a nice front end UI with jQuery that posted json info to the WCF services and received json back.  All's well and good.

The problem comes in putting WCF on IIS7 in what I call development and production modes or, as other people have envisoned using it, in internal and external modes.  Another example scenario is URI aggregation and forwarding for a SOA scenario.  I have external responses on 80, but I want internal service traffic to occur on a different port.

My IIS was configured to respond to dev:80 and dev:8080.  Seems to respond ok when I hit localhost:80 or localhost:8080, but I receive a "This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.Parameter name: item" error when trying to go to dev:8080.

Looking around, the post "Can't host WCF service in a website with multiple identities" at Microsoft Connect sums it up nicely: Microsoft's aware of this issue posted by a user and closes the issue without a fix, calling the behavior "by design." To be fair, they changed the bug's status to "Closed as Fixed" without providing a fix.  (Like "hotfix" - neither hot nor a fix.)  Eventually in the comments on this bug they state that this issue is fixed in .NET 4.0 beta 2.  Great, that's what, WCF 3.0?  (or 3.0, if .NET 3.5 was 2.0).

Internally, I can run my application quite well, but users outside my network can't access it.  Either I put up two copies of the service (ugh) or I consolidate my service on one port.

I've come away thinking that I want that part of my brain back that is stuffed with knowing anything about WCF.

Here's my cynical opinion about it (aka hating). WCF comes out of Microsoft's "enlightenment, wave 2" attitude, after they'd finally adopted an OO methodology with .NET, where they appeared to be in such a tizzy that they slapped on some skis, got in a pool and proceeded to jump the shark, converting all their grand unification ideas into monolithic pieces (see their much delayed ORM, Entity Framework, and the poorly executed Workflow Foundation) and half-reworks (see SharePoint 2007, CRM 4.0, BizTalk 2006 - only the pieces that were easy to veneer with .NET were done, the rest was done incompletely, leaving them looking like Matt Dillon in There's Something About Mary trying to impress).  Honestly, it works great, you just gotta know where to step.  This is sort of a typical experience I've had with Microsoft products - early adopters evangelize and late ("late 1.0") adopters get bitten - except usually it's their packaged products that suffer this malaise, not libraries so close to the core framework.  "Knowing where to step" is not expert-level services from a software framework, it's rote memorization without logic behind it

No wonder Microsoft has no native SOA offering, preferring to partner with other SOA vendors (HP/Systinet, webMethods/SAP, or SOA Software - all of whom use Java).  The multiple offerings of BizTalk / WF / WCF (in some combination / competition) are these huge elephants in the room of that space.

Phew.  Hating's hard work.

I've more or less resigned myself to consolidating services, but I intend to, immediately afterwards, rewrite the app into ASP.NET MVC which will allow me to consolidate the stateless UI project and the services project into one.
0 Comments

codeBeamer Timeline Builder

posted Tuesday, December 22, 2009 10:28 PM by hussain
Background

Project dashboarding - viewing the progress of a software development project from different viewpoints - has been an interest of mine ever since I started developing software. After one particularly difficult project involving issues with requirements, development and operations deployment and environmental difficulties, with no shortage of fingerpointing going on, I thought it would be good to step back and see the project from a linear timeline perspective and gain some objectivity. Additionally and especially for long running projects, viewing time linearly helps us as humans see the scope of events.

Simile Timeline + Google Calendar

I'd already had a calendar that was a great repository of significant events, which meetings I'd scheduled, etc. in Google Calendar. The data originally came from the corporate Outlook, synced up to Google Calendar with the Google Calendar Sync application. Using the Simile Timeline Javascript widget, I wrote a quick export from Google Calendar using the Google Calendar Data API to format the calendar events into the Simile Timeline json format. Since the events were hand coded Client Requirements (grey), Impediment (red) and Success (green), viewing the events linearly helped clear up the discussion around where the issues were in deploying the application. This quick web application was very well received and project managers in both development and operations, as well as other non-project related developers and managers, were able to see the timeline of events that occurred for this particular project. I hadn't fully automated the import from Google Calendar to the Project Timeline page, and that's what led to the next step.

Dynamic Timeline Generation

The timeline view of the project was very useful and I thought it'd be a great perspective for other projects. Most of the projects at the organization use codeBeamer by Intland as an issue tracker and document repository. CodeBeamer has a much richer ability to code tasks / tracker items with statuses as well as start dates, end dates, and changed dates. Being able to dynamically pull project info via tracker lists and view them in a linear timeline looked to be a great start for a project dashboard.

The Timeline Builder was constructed with two picklists, one that displayed the lists of projects available and the second picklist that was contextual to the project's actual tracker lists. The codeBeamer repository is organized such that every project has multiple "tracker lists" such as Business Requirements, Change Requests, Production Releases, and Defects. Project administrators can also add tracker lists as needed. When a project is selected from the first picklist, an AJAX call is made to the codeBeamer services, returning the project-specific tracker lists. When a user selects a tracker list, the application issues an AJAX query and retrieves the list of tracker events and then displays them as a timeline. The timeline has three horizontally scrollable bands: a weekly view, a monthly view and a yearly view. Each of them can be dragged left or right and the display of events will be synchronized. The display is "coded" by status: tems with a status of "closed" are represented as a solid blue ribbon, individual events have a circle icon, "in progress" events are a slightly transparent blue ribbon, "open" items are represented by a slightly transparent red ribbon with a solid red circle icon. Selecting a timeline event yields a link to the original codeBeamer tracker item as well as a short description along with the open and closed/last updated information. Below the timeline is a tabular representation of all the event data.

Technology Decisions

The organization uses .NET predominantly so I decided on using WCF and the codeBeamer .NET SDK to serve up the Simile Timeline JSON and ASP.NET (without WebForms) and jQuery to make AJAX requests to the WCF codeBeamer services. Additionally, the organization is standardized on Windows 2003 and IIS6, so I passed on using ASP.NET MVC on IIS6. Each Codebeamer project can have multiple task trackers ("tracker lists"), so there were three total JSON services: GetAllProjects, GetTrackerListsForProject, and the last, GetTimelineForTrackerList, which retrieved all tracker items for a particular tracker list as Simile JSON events. Additionally, I used two jQuery plugins - jTemplates to populate portions of the page, and flexigrid to show the same events in a table below the timeline.

Findings and Stumblings

Looking at the variety of projects that we have in a linear format brought some interesting insights, the first of which is that almost no two projects use Tracker lists the same way. Not every project uses codeBeamer the same way, even though we have default tracker lists for Business Requirements, Change Requests, Production Releases, and Defects. Not every Task Lead uses the default statuses the same way - some close all tracker items only when a project has deployed, and create a separate status - "development complete" - for developers to use. Tracker items stay open throughout the iteration. For long-running, multi-year projects, cyclicality was shown quite well in a linear timeline - periods of project activity were clearly mapped to variety of business cycles.

With the differences in usage of codeBeamer trackers, the high level of ability to customize tracker item templates, and the variability in conforming to the SDLC in the organization, comparing project-to-project is difficult in general, not just with a linear timeline.

The decision to use .NET WCF and a jQuery-driven front-end separated the codeBeamer Tracker List JSON generation service from the UI application, creating two projects which may or may not have been a good idea - although service-oriented, it's two distinct codebases to maintain. Another interesting challenge was the codeBeamer API documentation for .NET - there isn't really any, for either Hessian C# or the codeBeamer Remote API. Using Reflector and referring to the codeBeamer Java SDK Javadocs did help, but a Sandcastle generated documentation set would've been useful. Thankfully, the Java and .NET API's are extremely similar, so it wasn't a problem interpreting what should've happened.

Project Next Steps / Directions

For the current iteration of this timeline builder I have a few minor technical issues I'd like to address. I'm planning on having better integration between the flexigrid table of events and the simile timeline so that when you select an item from the table, it scrolls the timeline to the relevant event. Another enhancement would be to allow stacking timelines of multiple projects for juxtaposition.

The organization also uses VersionOne's on-line agile project tracking application which has similar data to codeBeamer. A future rev to this application may include pulling from VersionOne project data dynamically in a similar manner (choose a project, see a timeline). Similar "coding" issues occur with VersionOne as with codeBeamer use, but since VersionOne is more focused on an agile project management lifecycle, I expect representing the variety of task types to be somewhat easier. A first version is pictured below (using jstree to visualize the project hierarchy, at the left). Coding (designating the display of open, closed, in progress, etc.) is a bigger issue, and relates more to the choice of software project management structure - agile, etc. - but is something that's greatly needed to get a consistent level of display. Other project tracking software, which I'm familiar with from a user standpoint, that may be usable include Redmine and Assembla/Trac.

From the technical framework, I may experiment with ASP.NET MVC next (which would remove the need for a separate WCF project) and then GWT (with the codeBeamer Java SDK) to see which one is more code-efficient.

0 Comments

Google Analytics - Account Deleted!

posted Monday, December 07, 2009 11:13 PM by hussain

I had all my Google Analytics accounts deleted (not all, just my personal ones, not the ones for a client or two). Very devastating. There's no direct e-mail for analytics support (just if you've forgotten your login).

Google directs you to their forums, where there're a few reports of people suddenly losing their analytics accounts without reason. From what I can tell, either they've "drunk deleted" (they deleted, but can't remember they did), or there's an actual glitch. I don't drink, so who knows.

Their recommendation is to restart and rebuild. I've sent an e-mail to their "I can't remember my login" support (via a form) and have been promised a response within "usually one business day."

I had about 22 sites and subsites being tracked so that's a lot of data (for me, at least). The bigger thing is the difficulty about support around this. It may be an edge condition (possibly the only way to have something deleted is by an actual deletion), but a few posts on the support forum doesn't leave me satisfied as a recourse.

I'm reconstructing my tracking items by looking through my sites and determining which sites were tracked. This time, I'm segmenting out my tracked sites - personal, business, clients - into separate Analytics accounts (UA-*'s) to (hopefully) minimize the impact of whatever caused this issue. (A stab in the dark, I know.)

It does make me wonder, tangentially, about mission-critical cloud data, though. Is there a backup for marketing/ad companies using Analytics? Google, oddly, doesn't back up the accounts and once an account is deleted it can't be recovered - all data is lost. There is a Google Analytics Data Export API, so I guess that's an option. My experience with Google APIs tells me that this one'd be pretty easy to use, but I can't fathom wanting to back up the tracking data for the "unknown unknown" use case of accounts just disappearing.

To be fair, Microsoft SQL Azure doesn't offer backups, either. Their strategy is "resiliency" - with 3 copies of each database underneath their fabric. They expose almost everything in SQL Server 2008 with the addition of the Sync Framework. Backups, they claim, aren't needed but, if you desperately want backups, use the apis and make your own. (Why not also in the cloud, they suggest?) Maybe on Amazon?

On a different note, there's a new async tracking script so the code can be placed at different spots on the page rather than just the recommended bottom.

Other posts of mine referring to Analytics and tracking

0 Comments

A Timeline of Enterprise Architectures

posted Thursday, November 26, 2009 10:58 PM by hussain

In honor of my upcoming TOGAF 9 certification, I've put together a Timeline of Enterprise Architectures - Zachman FEA, TOGAF, Gartner EA, from 1987 to today.

0 Comments

Google's Chrome OS - The Quickening

posted Thursday, October 15, 2009 11:15 AM by hussain

So, yesterday, someone released a beta of what was supposedly a Google Chrome OS build. The twittersphere was all twitterpated. I downloaded this .deb file and stood up an Ubuntu 9.04 vm (VirtualBox) to see what it really was.

The Google Chrome OS, as announced in July, is a lightweight linux-based OS with the Chrome browser as its centerpiece.

Turns out, it's a Chromium build (4.0.222.6) with a compact navigation bar and a Google-icon menu on the top, left side. Clicking said icon brings up Google's Short Links.

Here're some pictures:


Starting it up


It's chromium++: note compact top bar with Google menu icon on left, clock bar on right


From ChromeOS


Compact nav bar


Clicking the google icon maximizes the browser and directs you to Short Links, but underlying OS's top bar is still visible.


Short Links info, "?" icon button

There's also this anomalous download of a SUSE vm with Chrome+OpenOffice, etc. Which is not the Chrome OS.

Google's planning on a Chrome OS announcement tonight, so hopefully there'll be an official release and more info other than just these previews.

I, for one, welcome the singularity.

Edit (10/16/2009):
I built the latest Chromium version (4.0.223.1 29191) and the leaked version is pretty different, as the latest browser does not have the compact navigation or the Google icon.

"This is actually just a small recruiting event and we won't be talking about Chrome OS at all," the spokesperson told Betanews moments ago, "just one engineer talking about UI design for Google Chrome (the browser)." The implication that Chrome OS was the subject was chocked up as a "false alarm." - betanews
0 Comments

Mark Twain, still relevant

posted Monday, August 31, 2009 4:44 PM by hussain

All you need in this life is ignorance and confidence, and then success is sure.
- Mark Twain

Letter to Mrs Foote, Dec. 2, 1887

Feel free to apply to coworkers, consultants, management, work, politicians, etc.

0 Comments

Android Apps

posted Saturday, August 29, 2009 6:44 PM by hussain

I haven't announced my issues with the G1 hardware I've had, primarily because they've been due to my stupidity, but, of late, I've had to hard reset my Android phone a few times in the last year or so. It always takes me a while to figure out what I had on there and then go through the painful addition of those apps. So here's a list:
Amazon Mobile
Google Finance
CompareEverywhere
Shop savvy
Twidgit Lite
Google Sky Map
Astrid
Last.fm
Shazam
Layar
ConnectBot - ssh
Movies by Flixter
PicSay
Google Voice
Weather forecast widget v2, Francois DESLANDES
Digital Clock Widget, Maize
FBReaderJ - an epub reader


Amazon

Astrid

Compare Anywhere

Google Finance

Google Sky Map

Layar

0 Comments

Upgrading the IdeaPad S10-2 to Windows 7

posted Monday, August 24, 2009 6:24 PM by hussain

Jack had such a hard time putting XP Pro on his Lenovo IdeaPad S10 via bootable USB that I was a bit concerned about trying to put W7 on my S10-2.

I followed the instructions on ditii.com using a 4 Gb Patriot USB removable stick that I'd been using on my Vista desktop as a readyboost helper. A few parts of the instructions were a bit odd - the referenced zip's .ini file had multiple lines for the %Microdrive_devdesc% part when only one was needed. I had to run through the diskpart bit to assign a drive letter to the newly recognized usb drive a few times.

I used Slyfox's VirtualClone drive to mount the W7 RTM iso and copied the files over to the G: drive (the usb drive's letter) with xcopy. When installing W7, the machine had the main hard drive partitioned into three parts, a primary for the os, a 2nd for drivers (xp home, the original os) and a recovery partition. I left the 2nd and 3rd partitions alone. The installation went smoothly and completed within 45 minutes.

Lenovo has a forum dedicated to W7 RTM and on it I came across this thread which states there're only a few drivers that the W7 dvd or Windows Update don't include.

I haven't added any other drivers just yet, but the machine is working fine. Of course, there're the Lenovo specific buttons that don't work - QuickStart for the IdeaPad and the Restore button, but the camera and the touchpad work fine.

Jack ended up borrowing a coworker's USB DVD player to install XP Pro.

0 Comments

Testing Wolfram|Alpha

posted Monday, August 24, 2009 5:15 PM by hussain

Wolfram|Alpha has a preview program where you can sign up and view their latest version at a beta url before they push their build to their live site, www.wolframalpha.com. I have to admit, I'm a sucker for testing new things, early adopter syndrome and all, so I don't mind getting a sneak peek. Over the weekend I started looking into Mathematica, too, since Wolfram|Alpha's written in Mathematica.

During my Six Sigma course, the examples have been given in Matlab and I've really wanted to start using R. After watching an intro presentation on the newness in Mathematica 7, it looks like there's quite a bit more to it than just math. There're stat functions in there, too. And it looks like it's free "player" would be good to show off created workbooks to non-Mathematica havers.

Unfortunately, I don't know enough Mathematica to determine whether Wolfram|Alpha takes query inputs in the Mathematic syntax, but it bears investigation - especially since their yet-released SDK's signup page asks that very question: Describe your experience with Mathematica?

They have some preliminary dev documentation (pdf) available, but oddly their API doesn't drop json back. Curious, since I'd think that's what they'd be using in their ajax-y pods and incremental loading. Maybe prior to full reveal.

More as I play around.

0 Comments