Results, so far...
vendredi, 07 décembre 2007

TRENDS' goal is the achievement of an interactive software for the elaboration of design trend boards dedicated to product designers in business to consumer markets such as for the automotive and original equipment manufacturers. This software aims at improving the early design process in acting on the design watch process thanks to the digitalisation of the Conjoint Trends Analysis method. The main research challenges will be met through the integration of three innovative technologies:

  1. web search agent
  2. visual content indexing and retrieval
  3. semantic multimedia search engine

All of these will be integrated into a cutting edge interface. The main innovation is related to knowledge/semantics and to the integration into a cutting edge interface of different search algorithms.

Summary of activities

TRENDS is a 36-month targeted research project which investigated the inspirational cognitive process of the designers and the related Conjoint Trends Analysis method developed by SERAM before setting out the initial components of the TRENDS system. The second prototype was developed and evaluated by the end-users. This first interactive software aims at demonstrating the feasibility of the technical architecture and assessing the first version of the interface and its functionalities: random search, search by image sample, search by relevance feedback, and search by text. It is currently improved with the integration of mixed text/image search. It is constituted of the user interface and two servers: the text search engine and the image search engine. The basis of the communications protocols and formats for the exchanges between the modules are defined to match the requirements of the functionalities implemented. Semantic developments such as ontology tags, pertimmizers, co-occurences are present in the text search engine but not visible from the user interface. In this prototype, the database index is built by steps integrating text indexation, images validation and ontology tags but there is no automated process to link these actions. The prototype has been implemented with two distant servers, the image search engine located at INRIA and the text search engine located at PERTIMM. A copy of the database stands on both servers. Only the user interface, made by ROBOTIKER, stands on the client machine by ROBOTIKER. The maintenance of the prototype is then dispatched between the three partners.

Important work areas

The main work areas involved in TRENDS project are two:

  1. the integration of the cognitive design process of the designers with related design knowledge (Kansei information: semantics + affective matter) and the digitalisation of the Conjoint Trends Analysis method;
  2. the integration of specific technologies in semantics and linguistics to search, categorize and analyse pictures and texts corresponding to potential inspirational information for design. This all was put forward through the development of a cutting edge interface.

Cognitive Design Process

The cognitive activity of the designers from the CENTRO RICERCHE FIAT (CRF) and STILE BERTONE (SB) was studied in the earliest phases of the TRENDS project.

 
Figure 1: Stile Bertone and Centro Ricerche FIAT Interviews snapshots (MARCH 2006 – TORINO)

When selecting inspirational materials, the core activity of a designer is to link semantic adjectives with design parameters and vice-versa. Designer’s expertise consists of giving emotional effects through new design solutions characterized by their semantic expression. Kansei oriented methods cover all the various approaches for translating semantic words (Kansei words) into design parameters, by measuring semantics and showing the correlation with some design properties. Kansei words mainly include semantic adjectives, objects names and low-level descriptors. The TRENDS system will be based on design rules according to the extraction of Kansei words by the end-users in order to build Design-oriented ontology. Designers already proposed specific links between semantic words and product images during pre-defined observations, here they were asked to associate words and pictures.The results of this data are illustrated in Microsoft Excel spreadsheets showing words which illustrate the relationship between pictorial information and high level attributes like semantic adjectives, and, middle and low level attributes. Merging textual and pictorial approaches to develop semantic queries will allow the user to browse the database using concepts, ideas, emotional or semantic terms.


      trends_images.jpg

Figure 2: Kansei words extraction: manual annotation of images from

magazines by the --users (JULY 2006 – TORINO)

Integration of Design Knowledge and Process

A study of the cognitive activity of the designers from the CENTRO RICERCHE FIAT (CRF) and STILE BERTONE (SB) enabled to investigate designers’ information process and extract the following essential data:

  • Designers sectors of influence,
  • Design cognitive structures and processes (emotional / affective processes),
  • Domain specific knowledge (semantic adjectives, Kansei information),

The sectors of influence were used for the database elaboration and appear also on the GUI. The way the designers search for information was formalised and transferred into procedures. The latter gave birth to specific trends functionalities like the TRENDS search capabilities going from open (with serendipity) to focused. The design cognitive structures and the domain specific knowledge were integrated into pertimmizers. Moreover, TRENDS system integrates flexible content-based image retrieval facilities that utilise ontological referencing, and software able to realise main procedures relating to Conjoint Trends Analysis. This method which was developed by SERAM enables the identification of design trends through the investigation of sectors of influence. The main functionalities are the trends identification and the pallets generation.

table5.png

Figure 3: From Conjoint Trends Analysis to TRENDS software

Image description technology

Work related to image content indexing, i.e. the choice of image descriptors, has been finalized. The relevant software modules (image retrieval server and the indexing module) have been implemented and delivered for Prototype 2. The query module also implements a state of the art database exploration paradigm based on relevance feedback (see figure 3). The system has been tested with real users and their feedback integrated into the system. In parallel, the image part continued with work on the construction of pallets and clustering of images. These results will be used for image categorization to generate summaries of classes of images and for the definition of harmonies of colour and texture. Also, a review is under way concerning existing techniques based on psychological aspects that should reliably represent the way of thinking of most designers.

image5.jpg
image6.jpg
Figure 4: Relevance feedback function

TRENDS Prototype 2 software integrates relevance feedback: the user selected some positive (green frame on the left) and some negative (red frame on the left) examples; the results are presented on the right.

Semantic text and image search technology

Work on semantic search technology by combined use of text and visual descriptors has started with the investigation of existing state of the art algorithms, such as data and feature fusion techniques and bindings of results on each modality. For the necessities of TRENDS system, the most adapted method to perform hybrid text and image queries is based in late fusion of results obtained separately for each modality (see figure 2). This avoids the very high dimensional vector spaces associated to textual features and the curse of dimensionality. Moreover, this method is very well adapted when combining results from different technologies which are not open source (as is the case in the TRENDS project).

image7.jpg

Figure 5: Fusion of results obtained for each modality

The TRENDS system offers text and visual rankings based on different query formulations: keywords search and query by example. Each modality provides a complete ordering of the dataset and returns a list of most fitted elements to the query subject. The work continues with experimenting several late merging and re-ranking algorithms.

Database and Meta Search Engine

The second version of the database gathered approximately 500.000 good quality of images that illustrate products in many sectors: automotive, architecture, aeronautics, fashion, sailing, sport, etc.

image8.jpg

Figure 6: TRENDS database V2, sector of Architecture (NOVEMBER 2006, 14 Websites, 2,2 Go, 65 000 files)

During the second year of the project, this database has been widely improved by grabbing fifteen sectors. The evolution of the statistics for the database is presented in the following table. The third list refers to the database used for prototype 2. The grabbing for the last version of the database has begun and will be completed by the end of the year. We can expect a really rich database of about a million images (on the basis of more than 6200 websites in 24 sectors).

 

image9.jpg
Table 1: Database results

 

The table shows that the filtering process removed 87% of the text and only 26% of the images resulting in a more efficient index. In the last months major improvements have been realized to automate the grabbing and filtering processes. New tools have been implemented thus improving the robustness, the error recovery, and providing statistics. Moreover, the last version of the grabber is able to process flash websites which was missing in the previous versions. The core of the meta-search engine enables to test queries in a Pertimmizer mode. There is not yet a user interface that could allow wide user testing. The meta-search engine is able to interpret Pertimmizer queries and emulate web search engines to retrieve results (Linguistic segmentation of a complex request). The collected results are then processed thanks to the Pertimmizers (Semantic filtering of the results) and the resulting scores are used for presenting the better results. This meta-search engine fully rely on the semantic dimension of Pertimm's products. This semantic approach is one of the major innovations of TRENDS project. Moreover, the indexation process has been changed so that the anchors, legends, images file names are indexed with a higher weight, thus improving the results to user queries. The system will enable the users to annotate the images. These user comments will be included in the index so that they will be searched. The user comments will be shared and editable by all users.

Cutting edge interface

During the second year of the project, the interface of prototype 2 has been developed. This interface was co-built with the designers, since the design and development process was as tight as possible to their expressed requirements gathered through the different creativity sessions and preliminary tests that took place during the first year. This interface constitutes the interactive software that allows the user to request and receive information from the text and image search engines. In comparison with the interfaces usually used, it can be appreciated that it is graphically advanced, and it manages a big amount of data every time, however, it is capable of maintaining a very good response time, totally satisfactory for the user, as it has been confirmed in the tests.

The user interface of prototype 2 has been developed in C#. C# is an object-oriented programming language developed and standardised by Microsoft as part of its platform .NET. It syntaxes results from C/C++ and uses the model of objects of platform .NET. The integration of the different servers under the user interface being executed in the client’s side has been carried out and validated in prototype 2. This prototype 2 proves the feasibility of the integration of the different modules which constitutes one of the main achievements of this prototype, and key issue in the final TRENDS software.

Moreover, the user interface of prototype 3, that gathers all the prototype 2 limitations detected during the end users’ tests and includes the proper improvements, is even more cutting edge. Its wide functionality, careful aesthetic and fast interaction with the users turn it into a useful and nice tool that manages to automate successfully manual actions in first design stages.

 

image10.jpg

Figure 7: TRENDS User Interface

Architecture and components integration

One of the main innovations in TRENDS lies on the fact that several servers and modules have to communicate. An intensive work of design has been performed in order to define the exchange protocols among all these modules. Prototype 2 proved the feasibility of this architecture. The performances of the system are good, even with disseminated search servers. A first and important step has been achieved with the online search on Prototype 2. A permanent work is conducted to improve the protocols, reduce the weight of the exchanges, and make them more efficient and less bandwidth consuming. All the communications between TRENDS servers and client-side IHM will go through the Request Management Server via HTTP request. The following schema illustrates the integration of all the modules.

 

image11.jpg
Figure 8: TRENDS Architecture

User Involvement, Promotion and Awareness

Continuous test sessions had been carried out with the end-users CENTRO RICERCHE FIAT and STILE BERTONE. These included: the initial needs analysis, the validation of TRENDS functions, the early evaluation of the Graphical User Interface (GUI) concept and of the non interactive prototype, and finally the evaluation of the second prototype which is interactive.

The currently main sector targeted by TRENDS is industrial design. The project website gives an overview of the project outside the consortium. The TRENDS partners can find more detailed information at by logging into the secured area of the site. A project flyer is available on the website in the project website section; this also gives an overview of the project. In addition, a video demonstration is available on the public part of the website, showing the main functionalities of prototype 2.

The following papers were published and presented in the framework of international conferences.

Future Work or Exploitation Prospects, as appropriate

Concerning TRENDS prototype 2, the user centred approach focused on assessing the interface design with items like efficiency, satisfaction, aesthetic, affective and cognitive responses to GUI design, search performance and database quality.

The next year will focus on the development of the final prototype and of the final software. This will include the current functionalities on the basis of a new improved and cutting edge GUI also integrating the new functionality of supported visual and semantic categorisation and semi-automatic pallets generation. The performance of the full prototype will be evaluated by the end-users in June-July 2008 and its acceptance will also be validated.

The end-users here will be manufacturers we are currently working with (CRF and SB), and others such as French and Italian car suppliers. Designers from other industrial design sectors (computers, PDA’s, mobile phones, fashion), and more widely stakeholders working on markets linked to the need for cross-lingual and mixed semantic text and image queries will also be approached.