Government Big Data Award Special Mention: US Department of State, Bureau of Consular Affairs

The Government Big Data Solutions Award was established to highlight innovative solutions and facilitate the exchange of best practices, lessons learned and creative ideas for addressing Big Data challenges. The top five nominees and overall winner was announced at Hadoop World in New York City on November 8 2011.

The Government Big Data Solutions Award Program is coordinated by CTOLabs.com.  The 2011 judging panel included: Doug Cutting, creator of Apache Hadoop and architect at Cloudera, Alan Wade, former CIA and IC CIO, Ryan LaSalle, Director of Accenture Cyber R & D, Ed Granstedt, Senior VP Director of the QinetiQ Strategic Solutions Center and Chris Dorobek, founder, editor and publisher of DorobekInsider.com.

The top five honorees of the Government Big Data Solutions Award are:

  • USA Search: Hosted search services over more than 500 government sites.  Provides search and suggestion services plus analytical tool dashboards.
  • GCE Federal: Cloud-based financial management solutions.
  • PNNL Bioinformatics: Advancing understanding of health, biology, genetics and computing.
  • SherpaSurfing: A cybersecurity solution that analyzes trends, finds malware, and writes alerts.
  • US Department of State, Bureau of Consular Affairs: Large data set with critically important applications for citizen service and national security.

Because of its critical mission impact and creative ability to leverage legacy architecture, the US Department of State, Bureau of Consular Affairs earned a special mention. Department of State (DoS), Bureau of Consular Affairs (CA) uses a suite of software applications  to collect information on applicants for immigrant visas, non-immigrant visas, and United States passports at consular posts abroad, domestic processing centers, and government agencies. It must use that massive amount of data to  issue travel documents to U.S. and foreign citizens. So that it may use this data effectively, CA stores it in the  Consular Consolidated Database (CCD), a robust, economical, and analytically-powerful data platform.

The data stored on the CCD is extensive and complex. Currently, there are 115 terabytes of information in the CCD, growing at a rate of 6-8 terabytes a month. That data is gathered by over 170 software applications processing travel documents and interfacing with partner agencies. Data from these applications comes in different formats and is organized by the case rather than by the individual. It also contains unstructured data such as comments, case notes, and images such as photos and scanned documents. The CCD synthesizes that information and finds all possible identity matches to give users the ability to make informed decisions, detect and prevent fraud, and identify potential national security threats.

The CCD is also crucial for information sharing between national security agencies. It is the  is the single most important and frequently-used source of data for the DHS and gets hundreds of thousands of hits a month from the DoD and FBI. The CCD has made sharing information like fingerprints automated, simplified, routine, and almost real-time.

To achieve this performance, CCD is built around flexible and scalable architecture.  It provides a web-enabled, directly-accessible single database platform that is easily integrated wit external systems. The CCD  centralizes information from 270 consular posts using Oracle Multimaster Replication, making it  the “largest connected/replicating database structure in the government” according to Oracle. The CCD still manages to save energy and $1.4 million annually by reducing redundancy in servers, data centers, and storage networks.

Using  powerful analytical tools and a set of custom-built services, the CCD performs rapid background and identity checks. By linking a variety of information from biometrics such as facial recognition to points of contact on applications and strings of unstructured data in notes, it can counteract fraud, see through aliases, and vet individuals against other government databases. It brings all relevant information on an individual together and pre-screens applicants, making it very easy to use. In an environment  where fragmentation and inefficiency had been the norm, the Consular Consolidated Database breaks the  paradigm of data isolated in independent databases to facilitate information sharing, fight fraud, and help CA complete its mission effectively and efficiently.

CTOvision Pro Special Technology Assessments

We produce special technology reviews continuously updated for CTOvision Pro members. Categories we cover include:

  • Analytical Tools - With a special focus on technologies that can make dramatic positive improvements for enterprise analysts.
  • Big Data - We cover the technologies that help organizations deal with massive quantities of data.
  • Cloud Computing - We curate information on the technologies enabling enterprise use of the cloud.
  • Communications - Advances in communications are revolutionizing how data gets moved.
  • GreenIT - A great and virtuous reason to modernize!
  • Infrastructure  - Modernizing Infrastructure can have dramatic benefits on functionality while reducing operating costs.
  • Mobile - This revolution is empowering the workforce in ways few of us ever dreamed of.
  • Security  -  There are real needs for enhancements to security systems.
  • Visualization  - Connecting computers with humans.
  • Hot Technologies - Firms we believe warrant special attention.

 

solid

Comments

  1. I wanted to add that in many ways this database is like the first line of defense against terrorism for the US. It helps a team of great people in a very important mission. 

  2. I wanted to add that in many ways this database is like the first line of defense against terrorism for the US. It helps a team of great people in a very important mission. 

Trackbacks

  1. [...] Hadoop in a cybersecurity solution for DoD networks, and the Bureau of Consular Affairs for the Consular Consolidated Database, which searches and analyzes travel documents from around the world for fraud and security [...]

  2. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  3. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  4. [...] system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) Seven.  DNA profiles on more than 10 million people are available in the FBI coordinated [...]

  5. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  6. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  7. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  8. [...]   Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  9. [...] towers that can aid agencies and law enforcement in locating Americans anytime, anywhere. 5. The US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]

  10. [...] Six.  More than 110 million people have their visas and more than 90 million have their photographs entered into the US Department of State Consular Consolidated Database (CCD).  This system grows by adding about 35,000 people a day.  This system serves as a gateway to the Department of State Facial Recognition system, IDENT and IAFSIS. (Source) [...]

  11. [...] US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]

  12. [...] US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]

  13. [...] US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]

  14. [...] US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]

  15. [...] US Department of State Consular Consolidated Database (CCD) has more than 90 million people’s photographs data based with the continuous use of the [...]