proxy support for fetch methods

[sahuguet]

When HTTP_PROXY or HTTPS_PROXY env variables are present, the fetch method is enriched with a proxy from https-proxy-agent.

Tested locally using tinyproxy and the following script test.genai.mjs

script({
    title: "Test for proxy",
    description: "Simple script to test fetch with proxy",
    model: "github:o1-mini",
    temperature: 0,
})


const url = "http://neverssl.com";    // HTTTP
const urls = "https://raw.githubusercontent.com/sahuguet/genaiscript/refs/heads/main/.gitignore"; // HTTPS
const { text, file } = await fetchText(url)

console.log(text);

using HTTP_PROXY=http://127.0.0.1:3128 node packages/cli/built/genaiscript.cjs run test.genai.mjs

Note: when I created the commit, some changes were added, not sure why.

-    const cmd = `curl "${url}" \\
---no-buffer \\
+    const cmd = `curl ${url} \\
 ${Object.entries(headers)
     .map(([k, v]) => `-H "${k}: ${v}"`)
-    .join(" \\\n")} \\
+    .join("\\\n")} \\

[sahuguet]

Not sure why the changes 166-172 were introduced. I did not mean to make them happen.

[pelikhan]

are these common names? should it be GENAISCRIPT_HTTPS_PROXY / GENAISCRIPT_HTTP_PROXY?

[sahuguet]

There are the ones used by Python and NodeJS as default env variables.
But we can add more in the expression.

[pelikhan]

i think we want to keep the space here?

[pelikhan]

probably a merge issue

[pelikhan]

move to devDependencies

[sahuguet]

This is a feature I need in production. Any GenAIScript that wants to fetch content will need to go through the proxy and will need this.
This is not related to the issue I had with installing packages. And I don't think this will fix it.

[pelikhan]

LGTM, a few comments.

• we need to add docs for this. maybe a dedicated "proxy" page in the docs?

[pelikhan]

@sahuguet take a peek at 82333ed . can you test and add some docs?

[sahuguet]

For the documentation, I am not sure we need a dedicated page.
I would add a short either under installation or configuration.

One question I don't have the answer for: when using GenAIScript from VS Code, how does GenAIScript get the VS Code proxy config info?

The proxy setting to use. If not set, will be inherited from the http_proxy and https_proxy environment variables.

[pelikhan]

For the documentation, I am not sure we need a dedicated page. I would add a short either under installation or configuration.

One question I don't have the answer for: when using GenAIScript from VS Code, how does GenAIScript get the VS Code proxy config info?

The proxy setting to use. If not set, will be inherited from the http_proxy and https_proxy environment variables.

I need to research this. This is where the server process is created. https://github.com/microsoft/genaiscript/blob/main/packages/vscode/src/servermanager.ts#L83

[pelikhan]

so we can probably read the "http.proxy" setting and send it to the new process

[sahuguet]

I tested the master branch on my work intranet.
fetchText() works fine. But fetch does not.
Is there an easy way of enriching the default NodeJS fetch with one that supports the proxy?
I would like const fetch = await createFetch() to be called when GenAIScript starts to make sure all future calls to fetch use the proxy version?
@pelikhan

[pelikhan]

It's dangerous to replace the global fetch as any 3rd party library would be also using it which may lead to some unexpected results.

[sahuguet]

How do I reference createFetch from a GenAI script?

const response = await fetchText(url); works; const fetch = await createFetch(); gives me a "is not defined".

[pelikhan]

See https://microsoft.github.io/genaiscript/reference/scripts/fetch/#hostfetch

I put them on the host object