Searching for relevant knowledge across heterogeneous imagery intelligence databases requires an extensive knowledge of the semantic meaning of images, a keen eye for visual patterns, efficient strategies for collecting and analyzing data, and a deep understanding of related tradecrafts (signal intelligence, measurement intelligence, etc.). For example, "What are possible annotations for this building complex given the construction methods and materials used and the trace measurement of these chemical signatures?" Answering this question requires an extensive knowledge of the perceptual categories used for image annotation and the ability to draw information from one tradecraft modality to another.

The size of current intelligence datasets far exceeds what manual intelligence analysis can handle to correlate findings among multi-modal databases. This will only get worse moving forward as future imagery collection systems (FIA, WorldView, SBR, etc.) and other intelligence methods come online to deal with the post 9-11 requirements of more ubiquitous and persistent surveillance.

In meeting the needs of a modern intelligence community in this area, we face two central technological challenges. The first is to develop new approaches for interoperability that can function and flourish in today's landscape of diverse and constantly changing databases, relational schema, and terminology. By interoperability we mean the ability of disparate computational resources to reliably exchange knowledge, data, and computations with minimized intervention of the analyst. The second is to develop new methods that answer complex queries analysts have and facilitate knowledge discovery. By complex queries, we mean a combination of scientific expertise and algorithms to detect and retrieve information related by image features, annotations, geospatial information, vectors, human intelligence, etc. even when those data are not uniformly present in the partner databases.

We are applying our multi-disciplinary research experience in high-resolution earth image feature extraction, content-based image retrieval, semantic modeling and knowledge exchange, and visualization to develop prototype systems for content-based information retrieval (CBIR) and exploitation of multi-modal databases.

In this work we are adapting a multi-modal database architecture for CBIR that was previously developed and operationally used in medical information retrieval and diagnostic applications. The CBIR architecture shown in Fig. 1 accesses both text (DNA, patient history, pathology, etc.) and image (HRCT) databases and allows hybrid (image and semantic text) based queries.

The CBIR engine in Fig. 1 is supported by four components:

(1) The Domain Knowledge (DK) module provides a web-based graphical user interface (GUI) for delineating regions of interest (ROI) and annotating text-based categories for the ROI’s. The annotation process is validated by the CBIR engine if the ROI matches the majority of database ROI’s under the same or synonymous annotations. This tool has been tested in medical domain and is expected to be valuable for multi-modal earth images.

(2) The Image Content Formulation (ICF) module collects images initially deposited and annotated by the domain experts. A well-organized computer vision and image-processing library customizes image features that distinguish image patterns among different categories in the DK module. Statistical methods that use human perceptual categories are used test the efficiency of higher-level descriptions of image content from primitive image features in the image database retrieval system.

(3) The Machine Learning and Data Mining (MLDM) module utilizes expert labeling of partial image data to build statistical k-d (SKD) trees to index and cluster unlabeled images for fast and accurate retrievals. Associations among different perceptual categories used by domain experts and distributions of image features are studied to provide multi-source evidence for assisting diagnoses.

(4) The Spatial and Temporal Indexing (STI) module applies recently developed histogram of force algorithms to model spatial relationships among lesions and anatomical landmarks for medical images and an SKD tree with temporal features for tracking lesions across different time stamps.

The CBIR architecture was used to implement WebHIQS (Web-based Hybrid Inquiry Query System) for lung disease diagnosis. WebHIQS is currently operational and is being used for diagnosis for 13 types of lung disease containing 32 perceptual categories. The average retrieval accuracy, as measured by physicians using the system, is presently 85%.

The CBIR architecture in Fig. 1 is very flexible and can be adapted to a wide variety of other multi-modal domains. ProteinDBS is a good example that demonstrates the feasibility of applying and adapting the system to a new domain. This project was initiated in early October 2003 and launched in mid February 2004. The motivation for this research was to speed up searches of similar protein structures for understanding functions of unknown proteins. Using current search engines it normally takes hours or even days to receive results from these engines. Thus there was an urgent need to intelligently extract features from 3D protein structures and search the results in seconds. Indexing protein structures has been shown to provide a scalable solution for structure-to-structure comparisons in large protein structure retrieval systems. To conduct similarity searches against 46,075 polypeptide chains in a database with real-time responses, two critical issues were addressed, information extraction and suitable indexing.

First, we applied computer vision techniques to extract the predominant information encoded in each 2D distance matrix, generated from 3D coordinates of protein chains. Distance matrices are capable of representing specific protein structural topologies, and similar proteins will generate similar matrices. Once meaningful features were extracted from distance images, an advanced indexing structure, Entropy Balanced Statistical (EBS) k-d tree, was then utilized to index the multidimensional data. With a limited amount of training data from domain experts, we applied various techniques in the pattern recognition field to determine clusters of proteins in the multi-dimensional feature space.

The ProteinDBS system shown in Fig. 3 is now operational and is able to recall search results in a ranked order from the protein database in only seconds while, while exhibiting a high degree of precision (relevant retrieval). Our performance in efficiency is at least 120 times faster than the latest reported protein structure search method and approximately 30% higher in precision with 100% recalls.

We are presently conducting research to adapt the highly successful CBIR architecture in Fig. 1 to the domain of geospatial information retrieval. Work is being done to:

(1) Develop, test, and integrate efficient and relevant image feature extraction algorithms appropriate for multi-modal (optical, radar, etc.) medium and high-resolution earth images.

(2) Build a hybrid query system for image-based and text-based information retrieval appropriate for the geospatial intelligence domain.

(3) Build a semantic and knowledge sharing information mining hub for geospatial analysts and analysts from other tradecrafts.

(4) Develop a distributed visualization system for displaying and hierarchically browsing multi-modal database retrieval results and incorporate relevance feedback for query focus.