flexo Posted October 21, 2023 Share Posted October 21, 2023 openai 0.27.4 -> 0.28.1 (to current) - all requirements satisfied this should be non-breaking for anyone, the old version is obsolete --- then -> tiktoken 0.5.1 (pip install tiktoken) - will need regex>=2022.1.18 tiktoken installation will download 3MB caches upon first use, run: >>> import tiktoken >>> encoding = tiktoken.get_encoding('p50k_base') >>> encoding = tiktoken.get_encoding('r50k_base') >>> encoding = tiktoken.get_encoding('cl100k_base') >>> encoding.encode("hello") [15339] hopefully the system cache can be made readable from user space, using os in the library and os.environ. self-documentation, from tiktoken's load.py: def read_file_cached(blobpath: str) -> bytes: if "TIKTOKEN_CACHE_DIR" in os.environ: cache_dir = os.environ["TIKTOKEN_CACHE_DIR"] elif "DATA_GYM_CACHE_DIR" in os.environ: cache_dir = os.environ["DATA_GYM_CACHE_DIR"] else: cache_dir = os.path.join(tempfile.gettempdir(), "data-gym-cache") if cache_dir == "": # disable caching return read_file(blobpath) cache_key = hashlib.sha1(blobpath.encode()).hexdigest() cache_path = os.path.join(cache_dir, cache_key) if os.path.exists(cache_path): with open(cache_path, "rb") as f: return f.read() contents = read_file(blobpath) os.makedirs(cache_dir, exist_ok=True) tmp_filename = cache_path + "." + str(uuid.uuid4()) + ".tmp" with open(tmp_filename, "wb") as f: f.write(contents) os.rename(tmp_filename, cache_path) return contents Otherwise the system installed code would download for each object instantiation or bork if no `os` is available. unmodified user code would also seem to need os. Link to comment Share on other sites More sharing options...
Unknown025 Posted October 21, 2023 Share Posted October 21, 2023 I'll escalate your request for the administrators to take a look at those Python packages for you. Link to comment Share on other sites More sharing options...
flexo Posted October 21, 2023 Author Share Posted October 21, 2023 Thanks! Link to comment Share on other sites More sharing options...
Krydos Posted October 21, 2023 Share Posted October 21, 2023 You'll need a VPS for all of that. Link to comment Share on other sites More sharing options...
flexo Posted October 22, 2023 Author Share Posted October 22, 2023 Thanks. `openai` should update ok (with the same level of function already present) I'll see if I can do "all that" as a shared host user on my own VM and move it over if I need to try out some chatbot chatting from another IP.🤐 Link to comment Share on other sites More sharing options...
Krydos Posted October 22, 2023 Share Posted October 22, 2023 The Python 3.10 module openai on Johnny has been updated https://krydos2.heliohost.org/pyinfo/info3.10.py 1 Link to comment Share on other sites More sharing options...
flexo Posted October 25, 2023 Author Share Posted October 25, 2023 Thanks, you're the best! Link to comment Share on other sites More sharing options...
Recommended Posts