The AIDocumentLibraryChat mission has been prolonged to incorporate a picture database that may be questioned for photographs. It makes use of the LLava mannequin of Ollama, which might analyze photographs. The picture search makes use of embeddings with the PGVector extension of PostgreSQL.
Structure
The AIDocumentLibraryChat mission has this structure:
The Angular front-end exhibits the add and query options to the person. The Spring AI Backend adjusts the mannequin’s picture measurement, makes use of the database to retailer the information/vectors, and creates the picture descriptions with the LLava mannequin of Ollama.
The circulation of picture add/evaluation/storage appears to be like like this:
The picture is uploaded with the front-end. The back-end resizes it to a format the LLava mannequin can course of. The LLava mannequin then generates an outline of the picture based mostly on the supplied immediate. The resized picture and the metadata are saved in a relational Desk of PostgreSQL. The picture description is then used to create Embeddings. The Embeddings are saved with the outline within the PGVector database with metadata to search out the corresponding row within the PostgreSQL Desk. Then the picture description and the resized picture are proven within the frontend.
The circulation of picture questions appears to be like like this:
The person can enter the query within the front-end. The backend converts the query to Embeddings and searches the PGVector database for the closest entry. The entry has the row ID of the picture desk with the picture and the metadata. The picture desk information is queried mixed with the outline and proven to the person.
Backend
To run the PGVector database and the Ollama framework the recordsdata runPostgresql.sh and runOllama.sh comprise Docker instructions.
The backend wants these entries in application-ollama.properties:
# picture processing
spring.ai.ollama.chat.mannequin=llava:34b-v1.6-q6_K
spring.ai.ollama.chat.choices.num-thread=8
spring.ai.ollama.chat.choices.keep_alive=1sThe applying must be constructed with Ollama help (property: ‘useOllama’) and began with the ‘ollama’ profile and these properties have to be activated to allow the LLava mannequin and set a helpful keep_alive. The num_thread is just wanted if Ollama doesn’t choose the correct amount robotically.
The Controller
The ImageController incorporates the endpoints:
@RestController
@RequestMapping("rest/image")
public class ImageController {
...
@PostMapping("/query")
public Record postImageQuery(@RequestParam("query") String
question,@RequestParam("type") String sort) {
var outcome = this.imageService.queryImage(question);
return outcome;
}
@PostMapping("/import")
public ImageDto postImportImage(@RequestParam("query") String question,
@RequestParam("type") String sort,
@RequestParam("file") MultipartFile imageQuery) {
var outcome =
this.imageService.importImage(this.imageMapper.map(imageQuery, question),
this.imageMapper.map(imageQuery));
return outcome;
}
}The question endpoint incorporates the ‘postImageQuery(…)’ methodology that receives a type with the question and the picture sort and calls the ImageService to deal with the request.
The import endpoint incorporates the ‘postImportImage(…)’ methodology that receives a type with the question(immediate), the picture sort, and the file. The ImageMapper converts the shape to the ImageQueryDto and the Picture entity and calls the ImageService to deal with the request.
The Service
The ImageService appears to be like like this:
@Service
@Transactional
public class ImageService {
...
public ImageDto importImage(ImageQueryDto imageDto, Picture picture) {
var resultData = this.createAIResult(imageDto);
picture.setImageContent(resultData.imageQueryDto().getImageContent());
var myImage = this.imageRepository.save(picture);
var aiDocument = new Doc(resultData.reply());
aiDocument.getMetadata().put(MetaData.ID, myImage.getId().toString());
aiDocument.getMetadata().put(MetaData.DATATYPE,
MetaData.DataType.IMAGE.toString());
this.documentVsRepository.add(Record.of(aiDocument));
return new ImageDto(resultData.reply(),
Base64.getEncoder().encodeToString(resultData.imageQueryDto()
.getImageContent()), resultData.imageQueryDto().getImageType());
}
public Record queryImage(String imageQuery) {
var aiDocuments = this.documentVsRepository.retrieve(imageQuery,
MetaData.DataType.IMAGE, this.resultSize.intValue())
.stream().filter(myDoc -> myDoc.getMetadata()
.get(MetaData.DATATYPE).equals(DataType.IMAGE.toString()))
.sorted((myDocA, myDocB) ->
((Float) myDocA.getMetadata().get(MetaData.DISTANCE))
.compareTo(((Float) myDocB.getMetadata().get(MetaData.DISTANCE))))
.toList();
var imageMap = this.imageRepository.findAllById(
aiDocuments.stream().map(myDoc ->
(String) myDoc.getMetadata().get(MetaData.ID)).map(myUuid ->
UUID.fromString(myUuid)).toList())
.stream().accumulate(Collectors.toMap(myDoc -> myDoc.getId(),
myDoc -> myDoc));
return imageMap.entrySet().stream().map(myEntry ->
createImageContainer(aiDocuments, myEntry))
.sorted((containerA, containerB) ->
containerA.distance().compareTo(containerB.distance()))
.map(myContainer -> new ImageDto(myContainer.doc().getContent(),
Base64.getEncoder().encodeToString(
myContainer.picture().getImageContent()),
myContainer.picture().getImageType())).restrict(this.resultSize)
.toList();
}
personal ImageContainer createImageContainer(Record aiDocuments,
Entry myEntry) {
return new ImageContainer(
createIdFilteredStream(aiDocuments, myEntry)
.findFirst().orElseThrow(),
myEntry.getValue(),
createIdFilteredStream(aiDocuments, myEntry).map(myDoc ->
(Float) myDoc.getMetadata().get(MetaData.DISTANCE))
.findFirst().orElseThrow());
}
personal Stream createIdFilteredStream(Record aiDocuments,
Entry myEntry) {
return aiDocuments.stream().filter(myDoc -> myEntry.getKey().toString()
.equals((String) myDoc.getMetadata().get(MetaData.ID)));
}
personal ResultData createAIResult(ImageQueryDto imageDto) {
if (ImageType.JPEG.equals(imageDto.getImageType()) ||
ImageType.PNG.equals(imageDto.getImageType())) {
imageDto = this.resizeImage(imageDto);
}
var immediate = new Immediate(new UserMessage(imageDto.getQuery(),
Record.of(new Media(MimeType.valueOf(imageDto.getImageType()
.getMediaType()), imageDto.getImageContent()))));
var response = this.chatClient.name(immediate);
var resultData = new
ResultData(response.getResult().getOutput().getContent(), imageDto);
return resultData;
}
personal ImageQueryDto resizeImage(ImageQueryDto imageDto) {
...
}
}Within the ‘importImage(…)’ methodology the tactic ‘createAIResult(…)’ known as. It checks the picture sort and calls the ‘resizeImage(…)’ methodology to scale the picture to a measurement that the LLava mannequin helps. Then the Spring AI Immediate is created with the immediate textual content and the media with the picture, media sort, and the picture byte array. Then the ‘chatClient’ calls the immediate and the response is returned within the ‘ResultData’ document with the outline and the resized picture. Then the resized picture is added to the picture entity and the entity is continued. Now the AI doc is created with the embeddings, description, and the picture entity ID within the metadata. Then the ImageDto is created with the outline, the resized picture, and the picture sort and returned.
Within the ‘queryImage(…)’ methodology the Spring AI Paperwork with the bottom distances are retrieved and filtered for AI paperwork of picture sort within the metadata. The Paperwork are then sorted for the bottom distance. Then the picture entities with the metadata IDs of the Spring AI Paperwork are loaded. That allows the creation of the ImageDtos with the matching paperwork and picture entities. The picture is supplied as a Base64 encoded string. That allows the MediaType the simple show of the picture in an IMG tag.
To show a Base64 Png picture you need to use: ‘’
Outcome
The UI outcome appears to be like like this:
The applying discovered the big airplane within the vector database utilizing the embeddings. The second picture was chosen due to an analogous sky. The search took solely a fraction of a second.
Conclusion
The help of Spring AI and Ollama allows the usage of the free LLava mannequin. That makes the implementation of this picture database straightforward. The LLava mannequin generates good descriptions of the pictures that may be transformed into embeddings for quick looking. Spring AI is lacking help for the generate API endpoint, due to that the parameter ‘spring.ai.ollama.chat.options.keep_alive=1s’ is required to keep away from having previous information within the context window. The LLava mannequin wants GPU acceleration for productive use. The LLava is just used on import, which implies the creation of the descriptions might be finished asynchronously. The LLava mannequin on a medium-powered Laptop computer runs on a CPU, for 5-10 minutes per picture. Such an answer for picture looking is a leap ahead in comparison with earlier implementations. With extra GPUs or CPU help for AI such Picture search options will change into rather more common.