Using Condenser models with pyserini

[vicpara]

Hi, I'm trying to use the pyserini framework with the pre-trained condenser models available here https://github.com/luyug/Condenser .

I can encode the corpus yet I get errors when I am trying to create a FaissSearcher because the model cannot be found and downloaded from HF. This happens in spite of successfully running the encoding.

Encoding runs ok. It downloads the model, and encodes the output in spite of warning for not having all the layers initialized.

python -m pyserini.encode \
  input   --corpus data/raw-collection/ \
          --fields title text \
          --delimiter "\n" \
  output  --embeddings embeddings/condenser/ \
          --to-faiss \
  encoder --encoder Luyu/condenser \
          --fields title text \
          --batch 32 \
          --max-length 512 \
          --device cpu

When creating the Searcher the 'Luyu/condenser ' cannot be found in spite of being used during the encoding phase.

from pyserini.search import FaissSearcher

searcher = FaissSearcher(
        'index/condenser',
        'Luyu/condenser'
)

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:239, in hf_raise_for_status(response, endpoint_name)
    238 try:
--> 239     response.raise_for_status()
    240 except HTTPError as e:

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/luyu/condenser/resolve/main/config.json

The above exception was the direct cause of the following exception:

RepositoryNotFoundError                   Traceback (most recent call last)
File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/utils/hub.py:409, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, use_auth_token, revision, local_files_only, subfolder, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash)
    407 try:
    408     # Load from URL or cache if already cached
--> 409     resolved_file = hf_hub_download(
    410         path_or_repo_id,
    411         filename,
    412         subfolder=None if len(subfolder) == 0 else subfolder,
    413         revision=revision,
    414         cache_dir=cache_dir,
    415         user_agent=user_agent,
    416         force_download=force_download,
    417         proxies=proxies,
    418         resume_download=resume_download,
    419         use_auth_token=use_auth_token,
    420         local_files_only=local_files_only,
    421     )
    423 except RepositoryNotFoundError:

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:124, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    120     kwargs = smoothly_deprecate_use_auth_token(
    121         fn_name=fn.__name__, has_token=has_token, kwargs=kwargs
    122     )
--> 124 return fn(*args, **kwargs)

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/file_download.py:1065, in hf_hub_download(repo_id, filename, subfolder, repo_type, revision, library_name, library_version, cache_dir, user_agent, force_download, force_filename, proxies, etag_timeout, resume_download, token, local_files_only, legacy_cache_layout)
   1064 try:
-> 1065     metadata = get_hf_file_metadata(
   1066         url=url,
   1067         token=token,
   1068         proxies=proxies,
   1069         timeout=etag_timeout,
   1070     )
   1071 except EntryNotFoundError as http_error:
   1072     # Cache the non-existence of the file and raise

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:124, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    120     kwargs = smoothly_deprecate_use_auth_token(
    121         fn_name=fn.__name__, has_token=has_token, kwargs=kwargs
    122     )
--> 124 return fn(*args, **kwargs)

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/file_download.py:1374, in get_hf_file_metadata(url, token, proxies, timeout)
   1365 r = _request_wrapper(
   1366     method="HEAD",
   1367     url=url,
   (...)
   1372     timeout=timeout,
   1373 )
-> 1374 hf_raise_for_status(r)
   1376 # Return

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:268, in hf_raise_for_status(response, endpoint_name)
    260     message = (
    261         f"{response.status_code} Client Error."
    262         + "\n\n"
   (...)
    266         + "\nIf the repo is private, make sure you are authenticated."
    267     )
--> 268     raise RepositoryNotFoundError(message, response) from e
    270 elif response.status_code == 400:

RepositoryNotFoundError: 401 Client Error. (Request ID: e9fE-BJwy_9Sxf9QCJ4DM)

Repository Not Found for url: https://huggingface.co/luyu/condenser/resolve/main/config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If the repo is private, make sure you are authenticated.
Invalid username or password.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
Cell In [107], line 3
      1 from pyserini.search import FaissSearcher
----> 3 searcher = FaissSearcher(
      4     'index/condenser',
      5         query_encoder='Luyu/condenser'
      6 )
      7 hits = searcher.search('Who created a lot of standards for internet and networking')
      9 for i in range(0, 10):

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pyserini/search/faiss/_searcher.py:360, in FaissSearcher.__init__(self, index_dir, query_encoder, prebuilt_index_name)
    358     self.query_encoder = query_encoder
    359 else:
--> 360     self.query_encoder = self._init_encoder_from_str(query_encoder)
    361 self.index, self.docids = self.load_index(index_dir)
    362 self.dimension = self.index.d

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pyserini/search/faiss/_searcher.py:522, in FaissSearcher._init_encoder_from_str(encoder)
    520     return AutoQueryEncoder(encoder_dir=encoder, pooling='mean', l2_norm=True)
    521 else:
--> 522     return AutoQueryEncoder(encoder_dir=encoder)

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pyserini/search/faiss/_searcher.py:288, in AutoQueryEncoder.__init__(self, encoder_dir, tokenizer_name, encoded_query_dir, device, pooling, l2_norm, **kwargs)
    286 if encoder_dir:
    287     self.device = device
--> 288     self.model = AutoModel.from_pretrained(encoder_dir)
    289     self.model.to(self.device)
    290     try:

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py:434, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    432 hub_kwargs = {name: kwargs.pop(name) for name in hub_kwargs_names if name in kwargs}
    433 if not isinstance(config, PretrainedConfig):
--> 434     config, kwargs = AutoConfig.from_pretrained(
    435         pretrained_model_name_or_path,
    436         return_unused_kwargs=True,
    437         trust_remote_code=trust_remote_code,
    438         **hub_kwargs,
    439         **kwargs,
    440     )
    441 if hasattr(config, "auto_map") and cls.__name__ in config.auto_map:
    442     if not trust_remote_code:

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py:776, in AutoConfig.from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    774 kwargs["name_or_path"] = pretrained_model_name_or_path
    775 trust_remote_code = kwargs.pop("trust_remote_code", False)
--> 776 config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    777 if "auto_map" in config_dict and "AutoConfig" in config_dict["auto_map"]:
    778     if not trust_remote_code:

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/configuration_utils.py:559, in PretrainedConfig.get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    557 original_kwargs = copy.deepcopy(kwargs)
    558 # Get config dict associated with the base config file
--> 559 config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
    560 if "_commit_hash" in config_dict:
    561     original_kwargs["_commit_hash"] = config_dict["_commit_hash"]

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/configuration_utils.py:614, in PretrainedConfig._get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    610 configuration_file = kwargs.pop("_configuration_file", CONFIG_NAME)
    612 try:
    613     # Load from local folder or from cache or download from model Hub and cache
--> 614     resolved_config_file = cached_file(
    615         pretrained_model_name_or_path,
    616         configuration_file,
    617         cache_dir=cache_dir,
    618         force_download=force_download,
    619         proxies=proxies,
    620         resume_download=resume_download,
    621         local_files_only=local_files_only,
    622         use_auth_token=use_auth_token,
    623         user_agent=user_agent,
    624         revision=revision,
    625         subfolder=subfolder,
    626         _commit_hash=commit_hash,
    627     )
    628     commit_hash = extract_commit_hash(resolved_config_file, commit_hash)
    629 except EnvironmentError:
    630     # Raise any environment error raise by `cached_file`. It will have a helpful error message adapted to
    631     # the original exception.

File ~/.pyenv/versions/3.8.15/lib/python3.8/site-packages/transformers/utils/hub.py:424, in cached_file(path_or_repo_id, filename, cache_dir, force_download, resume_download, proxies, use_auth_token, revision, local_files_only, subfolder, user_agent, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash)
    409     resolved_file = hf_hub_download(
    410         path_or_repo_id,
    411         filename,
   (...)
    420         local_files_only=local_files_only,
    421     )
    423 except RepositoryNotFoundError:
--> 424     raise EnvironmentError(
    425         f"{path_or_repo_id} is not a local folder and is not a valid model identifier "
    426         "listed on 'https://huggingface.co/models'\nIf this is a private repository, make sure to "
    427         "pass a token having permission to this repo with `use_auth_token` or log in with "
    428         "`huggingface-cli login` and pass `use_auth_token=True`."
    429     )
    430 except RevisionNotFoundError:
    431     raise EnvironmentError(
    432         f"{revision} is not a valid git identifier (branch name, tag name or commit id) that exists "
    433         "for this model name. Check the model page at "
    434         f"'https://huggingface.co/{path_or_repo_id}' for available revisions."
    435     )

OSError: luyu/condenser is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

[crystina-z]

Hi @vicpara , thanks for reporting! the issue should be fixed now with #1355.

[vicpara]

Yaay