Changing Software Support Model

The changing role of the CIO and the central IT organization is certainly a hot topic at the moment.  One item I have spoken about is the changing software model.  I do believe we are moving away from the traditional IT requirements-based approach that worked (and probably still works) well for software development, to a model based on consumerization.  Our constituents fall in love and buy software.

A few years ago I participated with Educause in writing about reviewing software and managing alternative sourcing at Oakland University.  We marveled in that we looked at over 25 pieces of software in a year, and most were not coming through traditional project methods.  We had to review our purchasing process.  This past year the university community looked at over 325 pieces of software, that we know about.  We've done pretty well at making sure we are part of the purchasing process so that we avoid redundant purchases and software installations and support aren't unexpectedly dropped in our laps.  Some software applications are installed on campus, and some solutions are hosted.  Some are desktop installs, some are server installs, and now we have the App family.

Nevertheless, there is an emerging issue about software support.  Obviously, my staff of 33 IT professionals cannot be experts on supporting over 300 pieces of software.  The university cannot hire a system administrator or application support specialist for each piece of software.  Our traditional support expectations do not scale with that volume of purchases. We can't even be the sole-vendor contact. Yet I had an uncomfortable situation last week where a senior staff member told me that calling a vendor for support was "not a lean process."

In discussions with my staff, we've identified about a dozen key software systems that we've installed where we only offer hardware and operating system support.  There may be a disconnect between the services we offer and the expectations of our community.

I'm trying to put together tiers of support for software, something like:

• Outsourced.  Contacts go directly from client to vendor.  Central IT gets involved for contract review and assists with resolution in technical disputes.
• Campus install in datacenter with full support.
• Campus install in datacenter with some support.
• Campus install in datacenter with only operating system and hardware support.
• Departmental server install.
• Local device install.

As I'm looking at this, there is a developing support matrix.  In addition to location and local support resources, we also have to look at server, storage, operating system, web server applications, identity access management, database management applications, code management architectures, security applications, backup services, and the actual software application.  Server and storage architectures are a lot more complicated to manage with the virtualized environments we now install.

In some cases our central IT operation provides and supports the entire list.  The closest is our ERP implementation, even though we do push using the vendor support options (including mailing lists and online community) before contacting us.  In other cases, we split the work with another unit; for example, for Moodle, we provide the server, storage, operating system, identity access management connection, security applications, and backup services.  Our E-Learning operations does the rest.  We jointly work on tuning.

In other cases, particularly for research, we are hosting hardware and really not doing much else.

The issue we see emerging is when a software package starts locally, in a department, and gains adoption.  We then need some sort of escalation and approval process to move the software to a support model.  A few years ago we looked at this from a server support model, with levels described on our web site.  We really need to rethink this for tiers of software support, and the governance model that moves a software solution to a higher tier of support.  This needs to be logical and rational.  We need to include the staffing needed to provide support in the evaluation.  That piece is really missing, as we've been resourced constrained so long.  

We can't keep adding solutions to the VM architecture without analyzing the impact on staff resources, and we haven't fully identified the impact on staff.  We can respond to a new server request with agility in the VM environment.  When we do so, we need to be clear that it doesn't mean we can answer questions about the software internals or how the application really works.  As we get a more and more complex virtual environment with expanded storage and complex disaster recovery scenarios, we need systems engineers and storage architects, but these additional staff will still not be able to answer the questions about how the application really works.  Then we get a community disconnect;  we got additional technical staff, but the client support expectations may still not be met.  We aren't there yet, but we seem to see new challenges for communications, particularly communications needed to justify additional resources.

Our path is not yet clear, but we are thinking about it.

Who needs the web?

http://www.nytimes.com/2012/04/11/technology/instagram-deal-is-billion-dollar-move-toward-cellphone-from-pc.html?_r=1&nl=todaysheadlines&emc=edit_th_20120411 

The timing of this article connected to my thinking about future projects on our enterprise systems team and allocating resources from that team to the wide variety of identified projects.  I had lunch yesterday with the Director for Enterprise Systems, Lori Tirpak, and two team members who worked on mobility projects for the last year.

The team has finished a strong set of projects this past year. Projects included improved messaging to students about advising and progress to degree.  Also, the team upgraded the portal architecture to the uPortal platform that is mobile-ready.  Finally, the group completed and launched the first OU App in the Apple store.  We spent time talking about projects for the coming year, and we talked about when we might no longer support the portal and instead totally focus our resources (a team of 8 people) on mobile initiatives.  Obviously, something like a transcript isn't going to translate well to a smartphone.  But many services will transfer to mobile devices.

The phrases that caught my eye in this article:
“For decades, the center of computing has been the desktop, and software was modeled after the experience of using a typewriter,” said Georg Petschnigg, a former Microsoft employee who is one of the creators of Paper, a new sketchbook app for the iPad. “But technology is now more intimate and pervasive than that. We have it with us all the time, and we have to reimagine innovative new interfaces and experiences around that.”


"Who needs the Web?"


It reminds me of the era when we kept our old mainframe databases running, due to the cost of transitioning those databases to new platforms.  Instead, we dropped the mainframe screen interfaces and replaces the user interfaces with new designs and platforms.


So we can keep those huge ERP databases or learning management systems behind the scenes.
Then focus on small service bites that are important in the moment and that can be provided by mobile devices.  That has to be the priority.  Using the points in the article, put your resources into mobile from the start, rather than starting with the web and redesigning for mobile.
Any left over resources - and that isn't much on our team - updates the web experience.  That is quite a shift in thinking.  


This really has us thinking about project priority for the next year.

Academic Records and the Changing Education Model

I think, increasingly, what we will be buying from individual college and universities, is the credential, not the full learning experience that we associate with traditional college.  How we store records supporting the credential, and put those records together to indicate evidence of learning, may change in the not too distant future.

Imagine a space where you, as an individual, take some courses from MIT or Stanford, online, open and free.  You build your knowledge for knowledge sake.  There are two reasons we pursue knowledge:

• personal growth
• obtaining meaningful work that pays a salary that meets our lifestyle expectations

The open course-ware model works perfectly for personal growth.  But how does an employer know that you've assembled a body of knowledge that leads to you being a talented individual that is knowledgeable, capable, and hire-able?  Right now, degrees provide that university tattoo of knowledge.

Today, each university owns the academic records that make up that tattoo.  The student has to meet the requirements of that university to obtain the tattoo.

What if the world was different:
Each student owned their academic records, rather than the educational institution owning the records.
Students might hire a company that is an "academic records repository" or "academic records bank".
Some records in the repository are graded; some are not.
Some records may be combined to present a "Badge" of learning.  A Google search on the two words "learning" and "badge" show this growing concept.
Some records, graded and accredited, may be assembled into a traditional degree.
The student assembles a personal tattoo that is not defined by a university, but by the student.

What is the role of the university in this environment?  The university may offer some courses for credit and some as open (like Stanford is doing now).  The student may obtain the knowledge in either path.  One path, the credit path, achieves a higher standard of knowledge certification (i.e., grades).  So what the university is really providing is knowledge certification, not just a diploma that represents completion. 

What if I, as a student, can build knowledge from several sources, some certified, others open, and record that with the third-party academic records bank that is independent of any particular university?  The records bank doesn't certify, but provides the transcript of all the work, instead of the university transcript.  The university credit transcript would feed into the academic records bank, as well as open course work, technical certifications and whatever.  The employer would request knowledge verification from the academic records bank, not any one university.

The "Big Blue Button" idea of medical records could apply to student records.  But if institutions could send directly to the academic records bank, what would be the point of the "Big Blue Button"?  Perhaps to make a record withdraw to send to a potential employer.

We are now sending all academic records to the State of Michigan for storage in a state-wide database.  Perhaps certifications and other learning mechanisms could feed into that database.  So perhaps, down the road, the state will provide the academic records bank.

What does this do to the academic records we store on campus?  Is there any reason to keep years and years of campus history if we can store the records in a single centralized bank?

There is potential for a shift in where and how we store student academic history.

Storage

Steve Glowacki, Director Systems Engineering at OU, and I have had several interesting conversations since the holiday break.   Steve ended up doing trouble-shooting and problem resolution (with his team) over the break.    I thought I would interview Steve for this blog post.  This follows on to my earlier impacts post, as we talk about data and content, and the need to store ever-larger files.  Nothing is ever deleted, it seems.  What is back-up and disaster planning in this growing data environment?  We had a discussion about problems and tactics.

Steve, can you describe some of the challenges experienced over the break?
We noticed that the back-up amounts had basically quadrupled started just before the holiday.  We went from approximately 500GB differential per night to usually slightly more than 2TB per night.

Did you find out the source and stop it?
Finding the source wasn't terribly difficult but we couldn't stop it.  It was valid stuff to back-up.  There were a lot of changes going on.   Over the break, we allocated an additional 3 TB of storage to back-ups, which was everything that remained within the architecture, and added 10 additional LT04 tapes, which the system promptly consumed.  We placed an emergency order for 20 additional tapes while still on break.

What next steps did you take when you returned?
We started reviewing how to consolidate tape utilization for optimized tape utilization.  As tapes are written to, the tape is consumed or full.   Then as data ages off, portions of the tape are freed, but the physical tape is still consumed.  Out of 130+ tapes, a good portion have low percentage physical utilization.

What we are looking for now is a process or procedure that will consolidate the data in use, distributed across several tapes, to a single highly utilized tape.

The other thing we are looking at is migrating to a new complete new architecture for backups, restoration, disaster recovery and data de-duplication.

What are you looking at?
The architecture is based on VMWare / NetApp, so the backup environment needs to work very closely with that architecture.


So, a minute ago, you said you were trying to review something and said:

"In some obscure way you log into this thing and control backups, restorations, bare metal restores, or tape archiving....  The tech team is having discussions about how this all works."

Tell me more.

What I found funny was that it took two university engineers and a vendor sales engineer three days to find the compatibility list for tape libraries.  It is complicated.

What makes it complicated?

Tape is becoming very limited use.  

What should we be doing?

Remote site de-duplicated replication is one option.  The remote site may be here or the cloud.  Remote site is likely phase two.  Phase one is implementing solutions to work with the VMWare / NetApp architecture.  Phase two is the remote site capabilities.  The reason we are breaking it into two pieces is timing and procurement process.  De-duplication and how that is done technically is extremely important to consider.  

So several purchases have been made over the past couple days.

We purchased another shelf of high speed disks.  This will allow us to optimize server performance and through-put.  Looking at the total input/output per second across the two types of shelves we have, and made a decision to improve performance by going with smaller, faster disks for specific services.  For example, we have a virtual server which may have enhanced performance by being located on faster disks while the connected storage may be on slower disks.

A lot of analysis about what to put where...

It's an ongoing thing.  

And your second purchase?

Several software options driven by recent requests.  One request was video streaming, so one of the options is for turning on native shares for CIFS.   Another software request was for NFS, to allow for UNIX-based mounts.  This has the potential to additionally augment throughput.  Another is SnapVault, but it is just for swarming snapshots, and that takes us back to where we started this conversation.

Is your head spinning?

A bit.  It's my job, though.  A lot of conversations will be needed with the Network team too.  

Favorite News Stories - 2011

My favorite tech-related news stories of the year:

The year of nature web cams
This was the year that watching nature via web cam caught my attention.  My favorite story was of Violet and Bobby, the red-tail hawks at NYU.  Watching Violet continue to sit after all experts said that the egg wasn't viable was heart-touching.  And when Pip hatched on Mother's Day, it was a "wow" moment.  And there were others:  the wonderful Decorah, Iowa eagles next, a hummingbird in Hawaii, and the Roosevelt Arch camera from the Yellowstone Association.  I am so amazed that I can go to the computer and watch wildlife, live, up-close and in natural settings.

Young people and meaningful messages
Meaningful personal stories shared by young people using YouTube as the medium really touched me this year.  Zach Wahls and Ben Breedlove had a lot to say about life and living it well, with open hearts and open minds.

People and gadgets
This was the year that iPads really took off, and iPhones and Droid smart phones became pervasive around campus.  The willingness to carry a gadget everywhere you go surprises me.  The guru of understanding the connection of people and technology, Steve Jobs, passed this year.  The story of his life was like a walk down my own tech memory lane.
 

Holiday Break - Slowing Down

We started a tradition a few years ago that has become a part of our culture.  The last work day before the break is designated as "Cleaning Day."  We stop work on projects and we don't go to meetings.  We bring in coffee and snacks, and we provide pizza for lunch.  Staff are encouraged to dress casually.

We provide a variety of cleaning products and suggest that everyone pitch in and clean up.  This is physical as well as virtual.   The result is a winding down of activity to the break.  There's laughter as we find really old software diskettes or manuals.  There's a lot of "Why did we keep this?" talk.  And we leave the space really fresh for our return in the new year.

The list of suggested activities includes:

Update all assigned help and project tickets with current status and clearly described next steps.

Update any status reports.

Update the wiki or other documentation.

Clean up old email and reorganize your labels. 

Review and delete unneeded files from your desktop or laptop computer.

Delete and organize files on shares.

Reorganize your desk and drawers.

Backup your computer.

Clean out and drop in the shred bin paper files and documents from your desk.

Wipe the coffee and pop spills off your desk - we provide cleaning spray and paper towels.

Dust off the tops of cubes, window sills, window blinds.

Wipe off door knobs, your phone and your cell phone.

Join in the clean-up of common areas such book shelves, refrigerator, microwave, kitchen and such.

Note that the refrigerator and freezer will be emptied and anything left will be thrown in the trash.

Recycle old periodicals.

Clean up areas we'll identify, reorganize, put everything in its place.

Take time to review university emergency response materials here http://www.police.oakland.edu/emergencypreparedness/resources

Go through the short online training for Preventing Workplace Violence on campus:  http://www2.oakland.edu/training/cal.cfm?courseid=525

If you have any departmental rituals for your organization, please share.

Happy holidays!

Forming Phase: IT meets HR

Our senior IT directors and I have started regularly scheduled round-table meetings with key university Human Resource leaders.  Our first two meetings have been something like “clearing of the air” meetings, broad discussions about the challenges we face.  We’ve learned some things already.

One of the first things we learned is that our IT leaders and HR leaders have  different perspectives on absence management.  IT leaders view high absenteeism as a disconnect that makes it difficult for an individual to keep up with changing technology and assigned projects.   We look at the outcomes.  Someone who is missing a lot of time misses out on how technology is changing and can’t keep their skills current.  We don’t typically reassign work unless we have a confirmed leave, so projects can get far behind schedule.  We were surprised to learn that our HR organization does not monitor for high absenteeism across the university, and we’ve learned this is something we have to do as directors.  That also means we have to define the level of absenteeism that is disruptive and above average for our organization. 

Our HR department considers scheduled and unscheduled absenteeism a measure to track; when we asked about monitoring, they want us to monitor how often someone calls off at the last minute.  This surprised us, because with project work that spans months, it doesn’t matter if someone calls off at the last minute or if they schedule the time off.   It is the absence that is disruptive, not the type of absence reporting.  It seems that our HR looks at the last-minute disruption as important, because they thought work would need rescheduling.  We had a discussion about how we can’t reschedule the majority of our work given a last minute call-off, as the work is project-based.    And with online calendars, tracking last minute call-offs is problematic.  The result of this discussion is that we are re-thinking how we look at absenteeism in our department and how we want to approach conversations with the employee. 

Another issue where our perspectives differ is employee retention.  I like this short description at BrightHub of an overall view on employee retention: 
http://www.brighthub.com/office/human-resources/articles/102224.aspx 

We work hard to hire the right people.  We know that potential employees who value balance between work and personal life often like working in the university central IT organization.  Our benefits for managed time-off (vacation, personal, sick) and flexible scheduling are strong recruiting tools.  Employees who value salary above managed time-off will not likely stay working with us over the long term, as our salaries are not competitive with area business.   

We are often surprised when there’s occasional university talk about reducing benefits to be comparable with some  businesses in the area.  In terms of overall compensation, we are competitive, but the balance between salary and benefits is different in our university compared to area business.  If we reduce the benefits and don’t correspondingly increase the salary, we will not be able to attract the IT talent we need for the projects defined by university leadership.

Our university HR discussed retaining solid employees, and particularly engaging in activities to support retention of employees with five years or more seniority.  However, from an IT perspective, five years is an eternity; that’s at least two technology change cycles.  We had churn in about half of our professional positions about a year ago.  We have 40 positions in central IT; of that 40, 20 have been with the university 5 years or less.  Of the remaining 20, 13 have changed responsibilities within the last 5 years.   Much of this churn happened after June 2010.  We agree we want to retain employees, but we'd like to see retention efforts  occur earlier and not five years into employment.

In an earlier blog, I posted about the technology sea change we are observing and the need for technology skill development and retooling to keep current.  We are at a point of investing thousands of dollars into skill and professional development.  If we are going to see return on that investment, we need to keep technically talented staff longer that 3 years.  This can’t be an ad-hoc “lazy-loser” system, where the default retention keeps those too lazy to look for another job or those  too technically out-of-date to find another job.  Our reward mechanisms need to favor those who are enthusiastic about university technology initiatives, adopting and excelling in the technical skill areas that we really need to move university projects forward.  The retention strategy needs to align with supporting technical change.