Wednesday, October 26, 2011

OIG Review of Visa Lottery Snafu Details Failure of the Consular Bureau's CST Office

The State Department's OIG has released one of its shortest limited scope reports on the visa lottery failure this past summer.  Not only was the visa lottery a public relations debacle, there was also a case filed in federal court on it; later dismissed in mid-July. Excerpts from the report:

The OIG team found three problems that led to this failure, all of which stem from the lack of adherence to sound project management and systems development principles. First, CA's Office of Consular Systems and Technology (CST) implemented a system programming change without performing adequate testing. Second, CST changed contract task orders without notifying the Office of Acquisition Management (AQM). Third, CST management failed to adequately discuss the changes with all stakeholders and thus did not fully understand how overseas consular officers administer the DV program.
The primary reason for the DV 2012 program failure was that CST did not adequately test the new computer program for the random selection of potential DV program participants. Section 203(c) of the Immigration and Nationality Act (INA), as amended, 3limits the number of DVs that may be available by both region and country. Therefore, 22 CFR § 42.33(c) requires that selection be based on random rank-ordering of participants by region through a computer program designed for this purpose. When participants submit their records, the computer program assigns a sequential number to each record based on the participant's region. Subsequently, the selection process uses the sequential numbers to randomly rank-order the participants' records. CST management decided in November 2010 not to use the commercial off-the-shelf statistics analysis program that it had used successfully for random rank-ordering in numerous previous years. Instead, CST management asked one of its contractors to develop a program. This new computer program had a coding error that produced a nonrandom rankordering and thus failed to meet INA requirements. The program not only selected 98 percent of the applicants from the first two dates of the allowed submission dates, it also selected multiple individuals from the same families.

According to CST management and the contractor staff who developed the new DV computer program, testing scenarios were limited to validating that all geographic regions were assigned the correct numerical limitation and that the total number of selectees to be drawn was accurate. In addition, the development, testing, and production implementation of the program were done exclusively by one contracting company that, due to poor planning and failure to consult with all DV stakeholders, did not have adequate information to create a complete test plan for the computer program. Key stakeholders such as CST's independent validation and verification team, the Visa Office, and the contractor that operated and managed the legacy computer program were not involved in planning and implementing the new computer program.

CST management acknowledged that it did not follow the standard systems development life cycle (SOLC) process, which establishes standards for planning, developing, testing, and deploying systems. Sound systems and computer program development calls for the use of the SOLC process, and in its absence, the project was implemented without the opportunity to identify and assign the appropriate resources to support and adequately test this critical computer program. As a result, while the new contractor was developing the new program, the incumbent contractor tasked with operating and maintaining the legacy selection program was unaware of the new development effort and was preparing to run the old program. CST did not notify the incumbent contractor until it was time to run the selection process. Thus there was no opportunity for the two contracting companies to share information and plan the implementation.

Furthermore, the decision to move to an internally developed DV randomization computer program changed the contractors' scopes of work, and the CST director did not notify AQM of these tasking changes. This lapse provides further support for the OIG May 2011 inspection report recommendation that CA and AQM conduct independent audits of CST contracts.

CST failed to follow the Department's 5 FAM 610 requirements for project development, integration, modification, and maintenance of information technology systems, products, and services. These requirements apply to all Department personnel and to contractors involved in Department systems and program planning. Additionally, the U.S. Government Accountability Office in its internal controls standards identifies Application Software Development and Change Control as one of the internal management controls to mitigate risk for fraud, waste, abuse, and mismanagement. One of the requirements is that all new or revised software be thoroughly tested and approved. By not adhering to the SDLC process and Department guidance, CST failed to meet established quality standards and business objectives and added unnecessary and costly steps to the program development and testing stages.
The report provides three brief recommendations but as if unable to help itself, the report points out to one final issue about CST management:
A final issue is that CST managers demonstrated a lack of understanding of visa issuance procedures and failed to consult appropriately with all stakeholders. For example, CST managers believed it was necessary to centralize all DV petition data because they erroneously thought the legacy stand-alone system allowed unqualified applicants to be issued visas without proper security checks. In actual practice, no visas are issued using the selection database. Consular officers adjudicate all DV applicants in the same system and with the same clearance processes as all other immigrant visas issued overseas.

Principals in the Visa Office were not aware that changes had been made to the computer program until after it failed and the results had to be voided. CST management further stated that it is not clear to them which office is responsible for administering the DV program.
The OIG review of the DV 2012 program emphasizes the need for CST to implement the recommendations made in the May 2011 report. At the time of the current review, CST management had neither provided OIG with an after-action report nor conducted a lessons learned analysis to share with CST staff or other stakeholders. CST was also unable to provide the OIG team with a plan for future implementation of the DV program. These documents are vital to the effective oversight and management of the DV program and will help avoid this and similar incidents, as well as their potentially costly consequences, in the future.
If CA's Office of Consular Systems and Technology (CST) seems familiar, that's because we have posted about it here, here and most recently, here.  Hard to imagine that there won't be any fall out on this over there in terms of top management shuffle. 

Here's the rather shocking part.  CST apparently is not sure who is responsible for administering the Diversity Visa program. Dammit, all my gray hairs jumped as if pulled by electric sheeps! All these years, CST wasn't sure and it never bothered to find out? Holy molly guacamole!

Which of the following belong together?  a) Diversity Visa, b) Visa Office, c) Overseas Citizens Services. 

Bingo! I can do this with my eyes close - see? 

Related item:
Review of the FY 2012 Diversity Visa Program Selection Process, ISP-I-12-01

Related posts:

Tuesday, June 14, 2011 | Inadequate Oversight of State Dept's Consolidated Consular Database-- Is This WikiLeaks Waiting to Happen, Again?

Monday, July 18, 2011 | US Court Dismisses Lawsuit Against State Dept for Voided Green Card Lottery

Wednesday, October 12, 2011 | State Dept CA CIO Kirit Amin Rants on FSOs, Cronyism, Nepotism, "Doormats"... Two Days Later, Apologizes

1 comment:

Samuel Beckett said...

Thank you for sharing useful information.