Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

google-genai [feature]: Context Caching #7169

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

google-genai [feature]: Context Caching #7169

wants to merge 5 commits into from

Conversation

KevinZJN
Copy link
Contributor

@KevinZJN KevinZJN commented Nov 7, 2024

Fixes #5841

Implemented conetxt caching feature for google-genai, we now allow user to caching a file and create gen-ai model base on the cached content

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Nov 7, 2024
@vercel Vercel
Copy link

vercel bot commented Nov 7, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchainjs-docs ✅ Ready (Inspect) Visit Preview Nov 19, 2024 6:14pm
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
langchainjs-api-refs ⬜️ Ignored (Inspect) Nov 19, 2024 6:14pm

@dosubot dosubot bot added the auto:improvement Medium size change to existing code to handle new use-cases label Nov 7, 2024
@KevinZJN KevinZJN marked this pull request as draft November 7, 2024 11:45
@KevinZJN
Copy link
Contributor Author

Hi @jacoblee93, we've implemented the foundational structure for context caching with files. Could you take a look and let us know if the code aligns with our intended design? Thank you!

@KevinZJN KevinZJN marked this pull request as ready for review November 13, 2024 17:30
@jacoblee93
Copy link
Collaborator

Looks reasonable to me - @afirstenberg can you have a look? It looks similar to some work you've done on Vertex.

We will also want to write up some docs!

jacoblee93
jacoblee93 reviewed Nov 17, 2024
View reviewed changes
libs/langchain-google-genai/src/tests/context_caching.int.test.ts
model.enableCachedContent(cachedContent);
}, 10 * 60 * 1000); // Set timeout to 10 minutes to upload file

test("Test Google AI", async () => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be part of the same test? Otherwise it will fail in isolation

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking the time to review this. Regarding your comment, could you please clarify how to reproduce the test failure? My understanding is that the beforeAll hook, which is handling the setup and file uploading, runs before any tests, so I'm not entirely sure what you mean by suggesting that they should be part of the same test.

@jacoblee93 jacoblee93 added the close PRs that need one or two touch-ups to be ready label Nov 17, 2024
@chaunguyenm
Copy link

Looks reasonable to me - @afirstenberg can you have a look? It looks similar to some work you've done on Vertex.

We will also want to write up some docs!

We will be working on the docs now, but please keep us updated if there's any suggestion on the design or the tests. Thank you both!

@chaunguyenm
Copy link

@jacoblee93 @afirstenberg We currently add a wrapper around GoogleAIFileManager and GoogleAICacheManager to support context caching, but the wrapper doesn't provide additional functionalities. This is because we are not sure if we can rely on users installing and using google/generative-ai/server package directly. Do you have a suggestion on how we can structure this? Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chaunguyenm chaunguyenm chaunguyenm left review comments

@jacoblee93 jacoblee93 jacoblee93 left review comments

Assignees
No one assigned
Labels
auto:improvement Medium size change to existing code to handle new use-cases close PRs that need one or two touch-ups to be ready size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

google-common google-genai [feature]: Context Caching
3 participants
@KevinZJN @jacoblee93 @chaunguyenm