Forum OpenACS Development: Semantic Search in OpenACS

In short, an embedding vector is a list of numbers that captures some of the semantics of the input by placing semantically similar inputs close together. This list of numbers depends on the language model and how it was trained. Embedding vectors help us find phrases that are relevant to a query, even if they have different words.

Here's an example of what the results might look like to a given query (note that the first result contains only one of the keywords of the user, yet it is the most similar according to the language model used):

Search query: "Should I get health insurance?"

Search results:
1. Should I sign up for Medicare Part B if I have Veterans' Benefits?
(similarity score: 0.5152)
2. Can I sign up for Medicare Part B if I am working and have health insurance through an employer?
(similarity score: 0.4782)
3. How can I get help with my Medicare Part A and Part B premiums?
(similarity score: 0.4490)

If you have any questions, please do not hesitate and let me know.

git clone --recurse-submodules https://github.com/jerily/tbert.git
cd tbert
docker build . -t tbert:latest
docker run --rm -it --entrypoint bash tbert:latest

Then, inside the container:

cd tbert/build
tclsh8.6 ../example.tcl ../bert.cpp/models/all-MiniLM-L12-v2/ggml-model-q4_0.bin

You can even copy the model as follows:

id=$(docker create tbert:latest)
docker cp $id:/tbert/bert.cpp/models/all-MiniLM-L12-v2/ggml-model-q4_0.bin .
docker rm -v $id
ls -la ggml-model-q4_0.bin

This is likely the approach to go with models i.e. download a couple of models inside the docker container and then copy them over to the host machine to try them out on your openacs instance.

docker build . -t pgvector-driver:latest
docker run --network host pgvector-driver:latest

After you execute above commands, all you have to do is point your browser to the following url and create some content in xowiki - you can then try to search for it from search: http://localhost:8000/

Notes:

1. Please make sure you are running nothing on port 5432 of your host machine. In other words, stop postgresql service on your host machine in order to try this.

2. It only indexes titles for now.

3. I will try to populate xowiki with some CSV data that I have and update the postgres dump that I'm using for this.

That's all for now. If there are any questions, please do not hesitate and contact me.

Anyway, the two vector similarity packages I provided together with tbert are put out there as proof of concept that TCL/OpenACS has these capabilities now.

Here's my plan for the next couple of days:

1. Resolve the issue with the current version of tsearch2-driver (see above - ranking is lost - search results are NOT in order). Most likely I'll provide a drop in replacement for it i.e. pgfts-driver.

2. Polish tbert, pgvector-driver, and pgembedding-driver. For example, cmake does not seem to be the preferred choice for NaviServer modules. I'll sort it out. Just wanted to have all dependencies installed from the same git repo. Furthermore, I need to add the index for the pgvector-driver package and add permission checking.

3. Ideally, the vector similarity packages should be used to provide additional search results (similar to your search kind of thing). This would require some changes to the search package. They are easy to make but they are out of scope for now.

4. I won't do solr and faiss unless someone really needs them. Both solr and faiss have the downside that they won't have the acs objects. So, searching will slow things.

PS. If there is no interest from the community about vector similarity search, I might as well turn my attention to a two-factor authentication solution for OpenACS. It would require a C-based module (maybe two naviserver modules) as well but I have that under control.

For example, cmake does not seem to be the preferred choice for NaviServer modules. I'll sort it out. Just wanted to have all dependencies installed from the same git repo.

This is done (added NaviServer module Makefile - thanks Gustaf) plus I've made the cmake installation more robust for TCL installation. If you have trouble installing tbert on your system please do not hesitate and contact me via email or here. I'm using Ubuntu Linux and so are the docker images.

You will be able to play with the docker image demo once am done.

You can try it all with docker, including the food hunter reviews that I populated in xowiki and you can search them already.

I forgot to mention that content is indexed both in pgvector-driver and pgembedding-driver and you can switch between the two in the demo by changing the new VectorDriver parameter of the search package (patch available in git repo). So, you can compare the results for yourselves.

That said, there's no point polishing these packages further until there is some sort of response from the community on where they want to take this. All the code is available in the aforementioned git repositories. Once I have some feedback, I'll improve them.

My time is limited as well starting next week as I'll be back to work but the OpenACS packages (pgvector-driver & pgembedding-driver) and tbert (the C-based module for vector embeddings) are in good shape and so I can make any kind of small changes or improvements to them based on the feedback I receive.

I'll switch to multi-factor authentication now in the hope that I can provide a proof of concept in the next couple of days.

The other thing to note is that the results from pgvector-driver were poor in the demo after I added an index on the table (does approximate search in that case). So, I decided to switch the VectorDriver parameter in the search package to pgembedding-driver that produces better results now. In other words, the demo now uses pgembedding-driver for vector similarity search.

Finally, you cannot switch language model after data has been indexed without migrating to the new language model. You have to make a choice from the beginning and go with it.

Instructions on how to convert the better model would be great. Any kind of documentation about what is happening and where would be much appreciated.

I will try to write up something over the weekend. In the meantime, you might want to check out this video on the way they are trained as part of a language model: https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture

In short embeddings encode semantic relations (bring relevant words in meaning together). So the vector of a word will be very close in distance (e.g. euclidean, cosine) to the vector of a similar word. For example, cat and dog are similar in at least one dimension i.e. they are both animals.

This is done when the language model is trained via a neural network. tbert is based on bert.cpp that does inference of BERT neural net architecture with pooling and normalization from SentenceTransformers (https://www.sbert.net/ - this is what I used in Python). tbert computes the embeddings vector based on a language model. There are lots of them in huggingface.

When some title is indexed, pgvector-driver and pgembedding-driver ask tbert to compute the vector based on the language model that is used and the result is stored in pgvector or pgembedding columns in the database. Upon search, tbert again computes the vector of the query of the user and then asks pgvector or pgembedding to rank them by similarity (basically euclidean distance between the vectors in both of these drivers).

We were able to build the image after the change you made, but when trying to run the container we weren't able to access it outside of the container itself. We added this line: RUN sed -i 's/127.0.0.1/0.0.0.0/g' /usr/local/ns/config-oacs-5-10-0.tcl

Will check it out and make the change. Thanks.

Oh, also, at one point we accidentally were trying to build using the dockerfile in the pgembedding-driver directory and the build was failing.

I forgot I had it there as well. I was updating the one in openacs-packages and copying to pgvector-driver. Fixed in pgembedding-driver as well.

Looking forward to improve it based on your feedback.

As far as the NaviServer module is concerned, I think we are good as it uses the NaviServer Makefile.

The problem is the TCL extension that is currently being built with cmake. I have no way to check it on a Mac - it builds and installs fine on Ubuntu. Apparently, macOS distinguishes between shared libraries and loadable modules. So, I have added MODULE in CMakeLists.txt (line 14). If someone can get the latest and try building the TCL extension on a Mac and let us know of their findings, it would be great.

In other words, when you do "make install" for TCL build, it shows what libraries it installed. The question is if it tried to install a .dylib or a .so or both on a Mac.