Feature/utl file#1168
Feature/utl file#1168yasir-hussain-shah wants to merge 19 commits intoIvorySQL:masterIvorySQL/IvorySQL:masterfrom yasir-hussain-shah:feature/utl_fileyasir-hussain-shah/IvorySQL:feature/utl_fileCopy head branch name to clipboard
Conversation
📝 WalkthroughWalkthroughThis PR adds a complete Oracle-compatible UTL_FILE package to IvorySQL, enabling file I/O operations. It includes C implementation with file handle management and security checks, SQL package definitions with multiple function overloads, test cases, build configuration updates, and expected test output changes. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as SQL Client
participant SQLLayer as SQL API Layer
participant CImpl as C Implementation
participant FileOps as File System
participant SysTable as sys.utl_file_directory
Client->>SQLLayer: FOPEN(location, filename, mode)
SQLLayer->>CImpl: ora_utl_file_fopen()
CImpl->>SysTable: Query safe directory paths
SysTable-->>CImpl: Validate location
CImpl->>FileOps: Open file with security checks
FileOps-->>CImpl: File handle + slot allocation
CImpl-->>SQLLayer: Return ORA_UTL_FILE_FILE_TYPE
SQLLayer-->>Client: File handle
Client->>SQLLayer: PUT_LINE(file_handle, text)
SQLLayer->>CImpl: ora_utl_file_put_line()
CImpl->>FileOps: Write to buffered handle
FileOps-->>CImpl: Write status
CImpl-->>SQLLayer: Success/Error
SQLLayer-->>Client: Result
Client->>SQLLayer: FGETATTR(location, filename)
SQLLayer->>CImpl: ora_utl_file_fgetattr()
CImpl->>FileOps: Query file metadata
FileOps-->>CImpl: Exists, length, block_size
CImpl-->>SQLLayer: ora_utl_file_fgetattr_result
SQLLayer-->>Client: Attributes
Client->>SQLLayer: FCLOSE(file_handle)
SQLLayer->>CImpl: ora_utl_file_fclose()
CImpl->>FileOps: Close and flush
FileOps-->>CImpl: Confirmation
CImpl->>CImpl: Free slot
CImpl-->>SQLLayer: Return status
SQLLayer-->>Client: Closed
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@bigplaice Once you okay, I will start adding new functions. |
|
@yasir-hussain-shah |
checking this. Will update shortly. |
f5304d6 to
60868e3
Compare
08e2b48 to
f738ad9
Compare
spell correction
regression fixes
8acd25e to
f945064
Compare
| int encoding = 0; | ||
|
|
||
| CHECK_FILE_HANDLE(); | ||
| fd = get_file_handle_from_slot(PG_GETARG_UINT32(0), &max_linesize, &encoding); |
There was a problem hiding this comment.
If we are putting raw content, then the encoding should be unnecessary.
There was a problem hiding this comment.
@bigplaice Right. With default file handles (opened with utl_file.FOPEN) that inherit system encoding, the encoding parameter has no practical effect here.
…e.umask fix: error: no previous declaration for ‘utl_file_umask_str’
9102573 to
01f85b1
Compare
There was a problem hiding this comment.
Actionable comments posted: 16
🤖 Fix all issues with AI agents
In `@contrib/ivorysql_ora/expected/utl_file.out`:
- Around line 103-106: The test's expected output hardcodes "block_size = 4096"
which is filesystem-dependent; update the expected result handling for the
utl_file test so it no longer asserts a specific block_size: either mask or
wildcard the block_size token in contrib/ivorysql_ora/expected/utl_file.out
(e.g., replace the numeric value with a placeholder or regex) or remove the
block_size field from the NOTICE lines, and/or add a comment documenting the
filesystem assumption; locate occurrences of "block_size =" and "fexists" in the
expected output to make the change.
In `@contrib/ivorysql_ora/sql/utl_file.sql`:
- Around line 1-165: The test file omits coverage for new APIs fopen_nchar,
put_line_nchar, get_line_nchar, and put_raw and doesn't remove generated files;
add small positive-path cases that open a file via
utl_file.fopen_nchar('data_directory', 'regress_nchar.txt', 'w'), write with
utl_file.put_line_nchar and utl_file.put_raw, close, then reopen with
utl_file.fopen_nchar(...,'r') and read back using utl_file.get_line_nchar to
assert content, and finally remove created files with
utl_file.fremove('data_directory', 'regress.txt'),
utl_file.fremove('data_directory', 'regress2.txt'),
utl_file.fremove('data_directory', 'regressflush.txt'), and the new
regress_nchar.txt to ensure cleanup.
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file--1.0.sql`:
- Around line 168-171: The SQL-language wrapper for sys.ora_utl_file_put_line
incorrectly references the parameter by name ("autoflush") instead of position;
update the function body to use positional parameters so the third argument is
$3 (e.g., call sys.ora_utl_file_put_line($1, $2::text, $3)); adjust only the SQL
function AS $$...$$ to replace the bare identifier "autoflush" with $3 so the
parameters match the signature.
- Around line 60-66: The composite type sys.ora_utl_file_fgetattr_result
declares file_length as integer (32-bit) but the C function
sys.ora_utl_file_fgetattr returns a 64-bit value (Int64GetDatum); update the
composite type so file_length is bigint (or int8) to match the C return and
avoid overflow for files >2GB, leaving fexists and block_size unchanged; ensure
the SQL CREATE TYPE line for sys.ora_utl_file_fgetattr_result is modified
accordingly.
- Around line 173-176: The C implementation of ora_utl_file_put_raw is ignoring
the third parameter (autoflush) declared in the SQL wrapper; update the
implementation in utl_file.c (function ora_utl_file_put_raw) to fetch the third
argument (autoflush) and, when true, perform a flush/sync on the file after
writing (mirror existing flush logic used by other put/close functions), or
alternatively update the SQL declaration to remove/document the unused
parameter; prefer changing ora_utl_file_put_raw to read the boolean arg and call
the appropriate flush function after the write so callers using autoflush =>
true actually get a flush.
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c`:
- Around line 687-698: ora_utl_file_ftell currently calls ftell(fd) (which
returns long) and returns it via PG_RETURN_INT32 causing silent truncation for
offsets > INT32_MAX; update ora_utl_file_ftell to capture the ftell() result
into a long (or off_t) variable, check for errors (ftell == -1L) and for
overflow beyond INT32_MAX/INT32_MIN, and if overflow or error occur raise an
error (or alternatively return a 64-bit PG type using PG_RETURN_INT64 if
compatible with the API), ensuring you still call CHECK_FILE_HANDLE(),
get_file_handle_from_slot(), and use INVALID_FILEHANDLE_EXCEPTION() as before.
- Around line 1406-1409: The string passed to GUC_check_errdetail in the digits
> 3 branch is missing a closing parenthesis; update the error message string in
that conditional (the GUC_check_errdetail(...) call) to read "number is too big
(only four digits are allowed)" so the parentheses are balanced and the message
is correct.
- Around line 425-444: The ora_utl_file_fclose function is using PG_GETARG_INT32
to retrieve a slot ID although slot IDs are uint32; replace the arg retrieval
with PG_GETARG_UINT32(0) so you call free_slot with the correct unsigned slot ID
(i.e., change free_slot(PG_GETARG_INT32(0)) to free_slot(PG_GETARG_UINT32(0))).
Keep the rest of the error handling (EBADF/STRERROR_EXCEPTION) unchanged; this
aligns ora_utl_file_fclose with fopen/fflush/fseek which use
PG_RETURN_UINT32/PG_GETARG_UINT32.
- Around line 1227-1249: The function get_file_handle_from_slot can return NULL
on a stale slot (it currently calls INVALID_FILEHANDLE_WARNING() and returns
NULL), leading to callers like ora_utl_file_fflush, ora_utl_file_get_line,
ora_utl_file_new_line, ora_utl_file_putf, do_put (and transitive
ora_utl_file_put, ora_utl_file_put_line), and ora_utl_file_put_raw invoking
stdio functions on NULL; change the behavior so a missing slot triggers an
error/exception instead of just a warning—replace the
INVALID_FILEHANDLE_WARNING() path with the same error path used for
INVALID_SLOTID (e.g., call INVALID_FILEHANDLE_EXCEPTION() or otherwise raise an
error before returning) so get_file_handle_from_slot never returns NULL and
callers need not be changed.
- Around line 1050-1090: In close_all_files(), the loop can overwrite errno
across iterations so STRERROR_EXCEPTION(WRITE_ERROR) may report a stale or
incorrect error; modify close_all_files to record the first failing errno (e.g.,
add an int first_errno = 0), on each fclose failure if first_errno == 0 set
first_errno = errno and set report_error = true, and after the loop pass that
saved errno into the error-reporting path (use strerror(first_errno) or
otherwise ensure STRERROR_EXCEPTION uses that saved value) while still cleaning
up slots and closing other fds; reference symbols: close_all_files, slots[], fd,
fclose, report_error, STRERROR_EXCEPTION(WRITE_ERROR), errno.
- Around line 196-231: reserve_slot currently calls NEXT_SLOTID(slotid) but
never updates the global slotid on wrap, so after UINT32_MAX all new slots get
id 1; fix by updating slotid and using the updated value when assigning
slots[i].id — e.g., compute the new id into slotid (slotid =
NEXT_SLOTID(slotid)) and then set slots[i].id = slotid (or change NEXT_SLOTID to
assign back to its argument), referencing NEXT_SLOTID, slotid, reserve_slot and
slots[i].id to locate the change.
- Around line 663-685: The implementation of ora_utl_file_fseek incorrectly sums
absolute and relative offsets and always seeks from SEEK_SET; change it to
follow Oracle semantics: if the absolute_offset argument is not null (check
PG_ARGISNULL(1) / use the same presence check used elsewhere) call fseek(fd,
absolute_offset, SEEK_SET); else if the relative_offset argument is not null
call fseek(fd, relative_offset, SEEK_CUR); if both are null simply return
without seeking; keep the existing error path (raise INVALID_OFFSET via
CUSTOM_EXCEPTION when fseek returns non-zero) and preserve use of
get_file_handle_from_slot/CHECK_FILE_HANDLE/INVALID_FILEHANDLE_EXCEPTION for
locating fd.
- Around line 500-524: In fcopy, when copy_text_file(srcfile, dstfile,
start_line, end_line) returns non-zero you must close both raw FILE* handles
before raising the IO_EXCEPTION to avoid leaking srcfile and dstfile; update the
error path in the fcopy function to call fclose(srcfile) and fclose(dstfile) (or
use a PG_TRY/PG_CATCH around the operation and ensure fclose is called in the
cleanup/exception path) prior to invoking IO_EXCEPTION so both handles are
always closed on error.
- Around line 1095-1135: In copy_text_file, the do/while loops assume each
fgets() read ends with '\n' and treat fgets()==NULL as an error (returning
errno), which drops or prematurely stops copying a final line without a trailing
newline; update both the skip and copy loops inside copy_text_file to check if
fgets() returns NULL: for the skip loop, break out of the inner loop (and outer
skip loop) when fgets()==NULL (EOF) instead of returning errno; for the copy
loop, if fgets()==NULL write any partial buffer content already read (ensure
len>0 before accessing buffer[len-1]) and then break to finish normally
returning 0 rather than returning errno; keep references to buffer, fgets,
fwrite, start_line and end_line when locating the change.
- Around line 1001-1048: get_safe_path currently skips locality validation when
safe_named_location() returns a mapped directory (variable location); after
canonicalize_path(fullname) you must ensure the canonicalized fullname actually
stays under that resolved location: canonicalize and store the resolved location
string (location) as an absolute path and then verify fullname starts with that
resolved location plus a path separator (or equals it); if not, throw the
appropriate error/raise exception (same semantics as check_secure_locality)
instead of returning the path; retain the existing behavior of calling
check_secure_locality(fullname) when location is NULL.
- Around line 861-873: ora_utl_file_put_raw currently calls do_write which
performs text encoding conversion (via encode_text/pg_do_encoding_conversion)
and will corrupt binary bytea data; change ora_utl_file_put_raw to bypass
do_write and instead extract the raw byte buffer and length from the bytea
argument using VARDATA_ANY and VARSIZE_ANY_EXHDR, then call fwrite to write that
exact byte buffer to the FILE* returned by get_file_handle_from_slot (keep
existing CHECK_FILE_HANDLE and NOT_NULL_ARG checks), check fwrite result for
short/failed writes and report errors similarly to other file ops; remove any
encoding conversion references (encode_text/pg_do_encoding_conversion) from this
path so raw bytes are written unchanged.
🧹 Nitpick comments (3)
contrib/ivorysql_ora/src/ivorysql_ora.c (1)
52-54: Consider moving extern declarations to a shared header.These
externdeclarations forutl_file_umask_str,utl_file_umask_assign_hook, andutl_file_umask_check_hookare declared directly in the.cfile. For better maintainability and to avoid declaration drift, consider placing them in a header (e.g., autl_file.h) that bothivorysql_ora.candutl_file.ccan include.contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c (1)
294-397:callback_registeredpersists across transactions and cannot be re-registered if callbacks are invalidated.The
static bool callback_registeredis local tofopenand is set once. PostgreSQL'sRegisterResourceReleaseCallbackandbefore_shmem_exitregister callbacks for the lifetime of the backend, so registering once is correct for a single backend session. This looks intentional and correct.However, there's a subtle issue: if the extension is loaded, unloaded (e.g., during
ALTER EXTENSION ... UPDATE), and reloaded, the static variable resets tofalsein the new DSO image while stale callbacks pointing to the old code may still be registered. This is an edge case but could cause a crash.Consider moving the flag and registration to the module's
_PG_initfunction instead of insidefopen, which is the conventional pattern for PostgreSQL extensions.contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file--1.0.sql (1)
329-533: Package body:FCLOSEdoes not nullify the file handle (IN OUT parameter).In Oracle,
FCLOSEsets the file handle toNULLafter closing. Here,file.idretains the stale slot ID afterPERFORM sys.ora_utl_file_fclose(file.id). WhileIS_OPENwill correctly returnfalse(the slot is freed), the stale handle could confuse user code that checks the handle value directly. Consider settingfile := NULLorfile.id := NULLafter the PERFORM.Proposed fix
PROCEDURE FCLOSE( file IN OUT ORA_UTL_FILE_FILE_TYPE ) IS BEGIN PERFORM sys.ora_utl_file_fclose(file.id); + file := NULL; END;
| NOTICE: fexists = t, file_length = 75, block_size = 4096 | ||
| NOTICE: fexists = t, file_length = 75, block_size = 4096 | ||
| NOTICE: fexists = f, file_length = <NULL>, block_size = <NULL> | ||
| NOTICE: fexists = t, file_length = 75, block_size = 4096 |
There was a problem hiding this comment.
block_size in expected output is filesystem-dependent — test may be fragile.
The expected block_size = 4096 (line 103, 104, 106) depends on the underlying filesystem's block size. Systems using ZFS, Btrfs, or certain cloud storage may report different block sizes, causing spurious test failures. Consider either masking this value in the test output or documenting the assumption.
🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/expected/utl_file.out` around lines 103 - 106, The
test's expected output hardcodes "block_size = 4096" which is
filesystem-dependent; update the expected result handling for the utl_file test
so it no longer asserts a specific block_size: either mask or wildcard the
block_size token in contrib/ivorysql_ora/expected/utl_file.out (e.g., replace
the numeric value with a placeholder or regex) or remove the block_size field
from the NOTICE lines, and/or add a comment documenting the filesystem
assumption; locate occurrences of "block_size =" and "fexists" in the expected
output to make the change.
| -- | ||
| -- tests for utl_file package | ||
| -- | ||
|
|
||
| -- insert a few directory objects for testing | ||
| insert into sys.utl_file_directory(dirname, dir) | ||
| select 'data_directory', current_setting('data_directory'); | ||
|
|
||
| -- test case for fopen with invalid directory object name | ||
| declare | ||
| f sys.ora_utl_file_file_type; | ||
| begin | ||
| -- an invalid directory object name | ||
| f := utl_file.fopen('wrong_directory','regress.txt','a',1024); | ||
| end; | ||
| / | ||
|
|
||
| -- test fopen, put, putf, put_line, new_line, get_line, is_open, fclose, fclose_all | ||
| declare | ||
| f sys.ora_utl_file_file_type; | ||
| line text; | ||
| i integer; | ||
| begin | ||
| f := utl_file.fopen('data_directory', 'regress.txt', 'w'); | ||
|
|
||
| -- data writing | ||
| utl_file.put_line(f, 'abc'); | ||
| utl_file.put_line(f, '123'::numeric); | ||
| utl_file.put_line(f, '-----'); | ||
| utl_file.new_line(f); | ||
| utl_file.put_line(f, '-----'); | ||
| utl_file.new_line(f, 0); | ||
| utl_file.put_line(f, '-----'); | ||
| utl_file.new_line(f, 2); | ||
| utl_file.put_line(f, '-----'); | ||
| utl_file.put(f, 'a'); | ||
| utl_file.put(f, 'b'); | ||
| utl_file.new_line(f); | ||
| utl_file.putf(f, '[1=%s, 2=%s, 3=%s, 4=%s, 5=%s]', '1', '2', '3', '4', '5'); | ||
| utl_file.new_line(f); | ||
| utl_file.put_line(f, '1234567890'); | ||
| utl_file.fclose(f); | ||
|
|
||
| f := utl_file.fopen('data_directory', 'regress.txt', 'r'); | ||
|
|
||
| -- data reading | ||
| i := 1; | ||
| utl_file.get_line(f, line); | ||
| while line is not null and i < 11 loop | ||
| raise notice '[%] >>%<<', i, line; | ||
| utl_file.get_line(f, line); | ||
| i:=i+1; | ||
|
|
||
| end loop; | ||
|
|
||
| while line is not null loop | ||
| raise notice '[%] >>%<<', i, line; | ||
| utl_file.get_line(f, line, 3); | ||
| i:=i+1; | ||
| end loop; | ||
|
|
||
| raise notice '>>%<<', line; | ||
| utl_file.fclose(f); | ||
|
|
||
| exception | ||
| when others then | ||
| raise notice 'exception: % ', sqlerrm; | ||
| raise notice 'is_open(f) = %', utl_file.is_open(f); | ||
| utl_file.fclose_all(); | ||
| raise notice 'is_open(f) = %', utl_file.is_open(f); | ||
| end; | ||
| / | ||
|
|
||
| -- test fgetattr, fcopy, fremove, frename | ||
| declare | ||
| fexists boolean; | ||
| file_length integer; | ||
| block_size integer; | ||
| begin | ||
| utl_file.fgetattr('data_directory', 'regress.txt', fexists, file_length, block_size); | ||
| raise notice 'fexists = %, file_length = %, block_size = %', fexists, file_length, block_size; | ||
| utl_file.fcopy('data_directory', 'regress.txt', 'data_directory', 'regress2.txt'); | ||
| utl_file.fgetattr('data_directory', 'regress2.txt', fexists, file_length, block_size); | ||
| raise notice 'fexists = %, file_length = %, block_size = %', fexists, file_length, block_size; | ||
| utl_file.fremove('data_directory', 'regress2.txt'); | ||
| utl_file.fgetattr('data_directory', 'regress2.txt', fexists, file_length, block_size); | ||
| raise notice 'fexists = %, file_length = %, block_size = %', fexists, file_length, block_size; | ||
| utl_file.frename('data_directory', 'regress.txt', 'data_directory', 'regress2.txt', true); | ||
| utl_file.fgetattr('data_directory', 'regress2.txt', fexists, file_length, block_size); | ||
| raise notice 'fexists = %, file_length = %, block_size = %', fexists, file_length, block_size; | ||
| end; | ||
| / | ||
|
|
||
| -- test fflush, fseek | ||
| declare | ||
| f sys.ora_utl_file_file_type; | ||
| f1 sys.ora_utl_file_file_type; | ||
| line text; | ||
| i integer; | ||
| line2_pos integer; | ||
| begin | ||
| f := utl_file.fopen('data_directory', 'regressflush.txt', 'w'); | ||
| f1 := utl_file.fopen('data_directory', 'regressflush.txt', 'r'); | ||
|
|
||
| -- data writing | ||
| for i in 1..5 loop | ||
| utl_file.put_line(f, 'line ' || i::text); | ||
| end loop; | ||
|
|
||
| -- data reading before fflush | ||
| raise notice 'data read before fflush'; | ||
| utl_file.get_line(f1, line); | ||
| while line is not null loop | ||
| raise notice '>>%<<', line; | ||
| utl_file.get_line(f1, line); | ||
| end loop; | ||
| raise notice '>>%<<', line; | ||
|
|
||
| utl_file.fflush(f); | ||
|
|
||
| -- data reading after fflush | ||
| utl_file.fseek(f1); -- reset off_set to 0 | ||
| raise notice 'data read after fflush'; | ||
| utl_file.get_line(f1, line); | ||
| line2_pos := utl_file.fgetpos(f1); -- after reading line 1, get position | ||
| while line is not null loop | ||
| raise notice '>>%<<', line; | ||
| utl_file.get_line(f1, line); | ||
| end loop; | ||
| raise notice '>>%<<', line; | ||
| raise notice 'lets print line#2 again'; | ||
| utl_file.fseek(f1, 0, line2_pos); -- seek to line 2 position | ||
| utl_file.get_line(f1, line); | ||
| raise notice '>>%<<', line; | ||
|
|
||
| utl_file.fclose_all(); | ||
|
|
||
| exception | ||
| when others then | ||
| raise notice 'exception: % ', sqlerrm; | ||
| raise notice 'is_open(f) = %', utl_file.is_open(f); | ||
| raise notice 'is_open(f1) = %', utl_file.is_open(f1); | ||
| utl_file.fclose_all(); | ||
| raise notice 'is_open(f) = %', utl_file.is_open(f); | ||
| raise notice 'is_open(f1) = %', utl_file.is_open(f1); | ||
| end; | ||
| / | ||
|
|
||
| -- test cases for automatic file closing on session terminated/rollback etc | ||
| declare | ||
| f sys.ora_utl_file_file_type; | ||
| begin | ||
| f := utl_file.fopen('data_directory','regress.txt','a',1024); | ||
|
|
||
| raise notice 'is_open = %', utl_file.is_open(f); | ||
| commit; | ||
| raise notice 'is_open = %', utl_file.is_open(f); | ||
| rollback; | ||
| raise notice 'is_open = %', utl_file.is_open(f); | ||
| end; | ||
| / | ||
|
|
||
| -- clean up | ||
| truncate sys.utl_file_directory; | ||
| / |
There was a problem hiding this comment.
Missing test coverage for several UTL_FILE functions.
The commit messages mention fopen_nchar, put_line_nchar, get_line_nchar, and put_raw as added API functions, but none of these are exercised in this test file. Consider adding at least basic positive-path tests for these functions.
Additionally, the tests don't clean up the generated files (regress.txt, regress2.txt, regressflush.txt) from the data directory. Consider adding fremove calls at the end to avoid leaving artifacts. As per coding guidelines, ensure the test aligns with all new APIs.
🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/sql/utl_file.sql` around lines 1 - 165, The test file
omits coverage for new APIs fopen_nchar, put_line_nchar, get_line_nchar, and
put_raw and doesn't remove generated files; add small positive-path cases that
open a file via utl_file.fopen_nchar('data_directory', 'regress_nchar.txt',
'w'), write with utl_file.put_line_nchar and utl_file.put_raw, close, then
reopen with utl_file.fopen_nchar(...,'r') and read back using
utl_file.get_line_nchar to assert content, and finally remove created files with
utl_file.fremove('data_directory', 'regress.txt'),
utl_file.fremove('data_directory', 'regress2.txt'),
utl_file.fremove('data_directory', 'regressflush.txt'), and the new
regress_nchar.txt to ensure cleanup.
| -- Create composite types for ora_utl_file_fgetattr return values | ||
| CREATE TYPE sys.ora_utl_file_fgetattr_result AS (fexists bool, file_length integer, block_size integer); | ||
|
|
||
| CREATE FUNCTION sys.ora_utl_file_fgetattr(location text, filename text) | ||
| RETURNS sys.ora_utl_file_fgetattr_result | ||
| AS 'MODULE_PATHNAME','ora_utl_file_fgetattr' | ||
| LANGUAGE C VOLATILE; |
There was a problem hiding this comment.
file_length type mismatch: SQL type is integer (32-bit) but C returns Int64GetDatum.
In the C function ora_utl_file_fgetattr (line 580 of utl_file.c), the file length is returned as Int64GetDatum(st.st_size), which is a 64-bit integer. But the composite type here declares file_length integer — a 32-bit type. For files > 2 GB this will either overflow or error.
Proposed fix
-CREATE TYPE sys.ora_utl_file_fgetattr_result AS (fexists bool, file_length integer, block_size integer);
+CREATE TYPE sys.ora_utl_file_fgetattr_result AS (fexists bool, file_length bigint, block_size integer);🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file--1.0.sql` around
lines 60 - 66, The composite type sys.ora_utl_file_fgetattr_result declares
file_length as integer (32-bit) but the C function sys.ora_utl_file_fgetattr
returns a 64-bit value (Int64GetDatum); update the composite type so file_length
is bigint (or int8) to match the C return and avoid overflow for files >2GB,
leaving fexists and block_size unchanged; ensure the SQL CREATE TYPE line for
sys.ora_utl_file_fgetattr_result is modified accordingly.
| CREATE FUNCTION sys.ora_utl_file_put_line(file INTEGER, buffer anyelement, autoflush bool) | ||
| RETURNS bool | ||
| AS $$SELECT sys.ora_utl_file_put_line($1, $2::text, autoflush); $$ | ||
| LANGUAGE SQL VOLATILE; |
There was a problem hiding this comment.
Bug: autoflush is referenced by name in a LANGUAGE SQL function — should be $3.
In SQL-language functions, parameters are referenced positionally ($1, $2, $3), not by name. The bare identifier autoflush will be interpreted as a column reference, causing a runtime error like "column autoflush does not exist".
Proposed fix
CREATE FUNCTION sys.ora_utl_file_put_line(file INTEGER, buffer anyelement, autoflush bool)
RETURNS bool
-AS $$SELECT sys.ora_utl_file_put_line($1, $2::text, autoflush); $$
+AS $$SELECT sys.ora_utl_file_put_line($1, $2::text, $3); $$
LANGUAGE SQL VOLATILE;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| CREATE FUNCTION sys.ora_utl_file_put_line(file INTEGER, buffer anyelement, autoflush bool) | |
| RETURNS bool | |
| AS $$SELECT sys.ora_utl_file_put_line($1, $2::text, autoflush); $$ | |
| LANGUAGE SQL VOLATILE; | |
| CREATE FUNCTION sys.ora_utl_file_put_line(file INTEGER, buffer anyelement, autoflush bool) | |
| RETURNS bool | |
| AS $$SELECT sys.ora_utl_file_put_line($1, $2::text, $3); $$ | |
| LANGUAGE SQL VOLATILE; |
🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file--1.0.sql` around
lines 168 - 171, The SQL-language wrapper for sys.ora_utl_file_put_line
incorrectly references the parameter by name ("autoflush") instead of position;
update the function body to use positional parameters so the third argument is
$3 (e.g., call sys.ora_utl_file_put_line($1, $2::text, $3)); adjust only the SQL
function AS $$...$$ to replace the bare identifier "autoflush" with $3 so the
parameters match the signature.
| CREATE FUNCTION sys.ora_utl_file_put_raw(file integer, buffer bytea, autoflush bool) | ||
| RETURNS void | ||
| AS 'MODULE_PATHNAME','ora_utl_file_put_raw' | ||
| LANGUAGE C VOLATILE; |
There was a problem hiding this comment.
put_raw C function ignores the autoflush parameter.
The SQL declaration passes 3 arguments (file, buffer, autoflush), but the C implementation (ora_utl_file_put_raw in utl_file.c lines 861–873) only reads arguments 0 and 1. The autoflush boolean (argument 2) is silently ignored, so callers who pass autoflush => true get no flush. Either the C code should honor it, or the SQL signature should document this limitation.
Proposed C fix (in utl_file.c)
Datum ora_utl_file_put_raw(PG_FUNCTION_ARGS)
{
FILE *fd;
+ bool autoflush;
CHECK_FILE_HANDLE();
fd = get_file_handle_from_slot(PG_GETARG_UINT32(0), NULL, NULL);
NOT_NULL_ARG(1);
- do_write(fcinfo, 1, fd, max_linesize, encoding);
+ /* write raw bytes (see separate put_raw fix) */
+ ...
+
+ autoflush = PG_GETARG_IF_EXISTS(2, BOOL, false);
+ if (autoflush)
+ do_flush(fd);
+
PG_RETURN_VOID();
}🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file--1.0.sql` around
lines 173 - 176, The C implementation of ora_utl_file_put_raw is ignoring the
third parameter (autoflush) declared in the SQL wrapper; update the
implementation in utl_file.c (function ora_utl_file_put_raw) to fetch the third
argument (autoflush) and, when true, perform a flush/sync on the file after
writing (mirror existing flush logic used by other put/close functions), or
alternatively update the SQL declaration to remove/document the unused
parameter; prefer changing ora_utl_file_put_raw to read the boolean arg and call
the appropriate flush function after the write so callers using autoflush =>
true actually get a flush.
| static char * | ||
| get_safe_path(text *location_or_dirname, text *filename) | ||
| { | ||
| char *fullname; | ||
| char *location; | ||
| bool check_locality; | ||
|
|
||
| NON_EMPTY_TEXT(location_or_dirname); | ||
| NON_EMPTY_TEXT(filename); | ||
|
|
||
| location = safe_named_location(location_or_dirname); | ||
| if (location) | ||
| { | ||
| int aux_pos = size2int(strlen(location)); | ||
| int aux_len = VARSIZE_ANY_EXHDR(filename); | ||
|
|
||
| fullname = palloc(aux_pos + 1 + aux_len + 1); | ||
| strcpy(fullname, location); | ||
| fullname[aux_pos] = '/'; | ||
| memcpy(fullname + aux_pos + 1, VARDATA(filename), aux_len); | ||
| fullname[aux_pos + aux_len + 1] = '\0'; | ||
|
|
||
| /* location is safe (ensured by dirname) */ | ||
| check_locality = false; | ||
| pfree(location); | ||
| } | ||
| else | ||
| { | ||
| int aux_pos = VARSIZE_ANY_EXHDR(location_or_dirname); | ||
| int aux_len = VARSIZE_ANY_EXHDR(filename); | ||
|
|
||
| fullname = palloc(aux_pos + 1 + aux_len + 1); | ||
| memcpy(fullname, VARDATA(location_or_dirname), aux_pos); | ||
| fullname[aux_pos] = '/'; | ||
| memcpy(fullname + aux_pos + 1, VARDATA(filename), aux_len); | ||
| fullname[aux_pos + aux_len + 1] = '\0'; | ||
|
|
||
| check_locality = true; | ||
| } | ||
|
|
||
| /* check locality in canonizalized form of path */ | ||
| canonicalize_path(fullname); | ||
|
|
||
| if (check_locality) | ||
| check_secure_locality(fullname); | ||
|
|
||
| return fullname; | ||
| } |
There was a problem hiding this comment.
Path traversal vulnerability: named-location paths skip check_secure_locality, allowing directory escape via filename.
When safe_named_location finds a match (line 1012), check_locality is set to false (line 1024) and the concatenated path is only canonicalized but never validated to remain under the named directory. A user can pass filename = '../../etc/passwd' which, after canonicalization, resolves outside the allowed directory, completely bypassing the security check.
Example: if dirname MY_DIR maps to /safe/uploads, calling FOPEN('MY_DIR', '../../etc/passwd', 'R') produces /safe/uploads/../../etc/passwd → canonicalized to /etc/passwd — and no locality check is performed.
Proposed fix: verify the canonicalized path starts with the resolved directory
if (location)
{
int aux_pos = size2int(strlen(location));
int aux_len = VARSIZE_ANY_EXHDR(filename);
fullname = palloc(aux_pos + 1 + aux_len + 1);
strcpy(fullname, location);
fullname[aux_pos] = '/';
memcpy(fullname + aux_pos + 1, VARDATA(filename), aux_len);
fullname[aux_pos + aux_len + 1] = '\0';
- /* location is safe (ensured by dirname) */
- check_locality = false;
+ /* Still need to verify the final path is under the named directory */
+ check_locality = true;
pfree(location);
}Alternatively, after canonicalization, explicitly verify that fullname starts with location:
canonicalize_path(fullname);
- if (check_locality)
- check_secure_locality(fullname);
+ check_secure_locality(fullname);🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c` around lines
1001 - 1048, get_safe_path currently skips locality validation when
safe_named_location() returns a mapped directory (variable location); after
canonicalize_path(fullname) you must ensure the canonicalized fullname actually
stays under that resolved location: canonicalize and store the resolved location
string (location) as an absolute path and then verify fullname starts with that
resolved location plus a path separator (or equals it); if not, throw the
appropriate error/raise exception (same semantics as check_secure_locality)
instead of returning the path; retain the existing behavior of calling
check_secure_locality(fullname) when location is NULL.
| static void | ||
| close_all_files(void) | ||
| { | ||
| FILE *fd = NULL; | ||
| bool report_error = false; | ||
|
|
||
| for (int i = 0; i < MAX_SLOTS; i++) | ||
| { | ||
| if (slots[i].id != INVALID_SLOTID) | ||
| { | ||
| fd = slots[i].fd; | ||
|
|
||
| slots[i].id = INVALID_SLOTID; | ||
| slots[i].fd = NULL; | ||
| slots[i].max_linesize = 0; | ||
| slots[i].encoding = 0; | ||
|
|
||
| /* | ||
| * Oracle documentation says: FCLOSE_ALL does not alter the state of the | ||
| * open file handles held by the user. This means that an IS_OPEN test on | ||
| * a file handle after an FCLOSE_ALL call still returns TRUE, even though | ||
| * the file has been closed. No further read or write operations can be | ||
| * performed on a file that was open before an FCLOSE_ALL. | ||
| * | ||
| * However, | ||
| * | ||
| * we choose to close the file handles and free the slots here | ||
| * to avoid dangling file handles. | ||
| */ | ||
|
|
||
| if(fd != NULL && fclose(fd) == 0) | ||
| continue; | ||
| else | ||
| report_error = true; | ||
| } | ||
| } | ||
|
|
||
| /* some file(s) were not closed, throw exception */ | ||
| if(report_error) | ||
| STRERROR_EXCEPTION(WRITE_ERROR); | ||
| } |
There was a problem hiding this comment.
close_all_files: stale errno reported after the loop.
When multiple files fail to close, or a failed fclose is followed by a successful one, errno is overwritten. The STRERROR_EXCEPTION on line 1089 may report a misleading error message from a later (possibly successful) call rather than the actual failure.
Consider saving the first failing errno and reporting that, or reporting errors per-file.
Proposed fix
static void
close_all_files(void)
{
FILE *fd = NULL;
- bool report_error = false;
+ int saved_errno = 0;
for (int i = 0; i < MAX_SLOTS; i++)
{
if (slots[i].id != INVALID_SLOTID)
{
fd = slots[i].fd;
slots[i].id = INVALID_SLOTID;
slots[i].fd = NULL;
slots[i].max_linesize = 0;
slots[i].encoding = 0;
- if(fd != NULL && fclose(fd) == 0)
- continue;
- else
- report_error = true;
+ if (fd != NULL && fclose(fd) != 0)
+ {
+ if (saved_errno == 0)
+ saved_errno = errno;
+ }
}
}
- /* some file(s) were not closed, throw exception */
- if(report_error)
+ if (saved_errno != 0)
+ {
+ errno = saved_errno;
STRERROR_EXCEPTION(WRITE_ERROR);
+ }
}🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c` around lines
1050 - 1090, In close_all_files(), the loop can overwrite errno across
iterations so STRERROR_EXCEPTION(WRITE_ERROR) may report a stale or incorrect
error; modify close_all_files to record the first failing errno (e.g., add an
int first_errno = 0), on each fclose failure if first_errno == 0 set first_errno
= errno and set report_error = true, and after the loop pass that saved errno
into the error-reporting path (use strerror(first_errno) or otherwise ensure
STRERROR_EXCEPTION uses that saved value) while still cleaning up slots and
closing other fds; reference symbols: close_all_files, slots[], fd, fclose,
report_error, STRERROR_EXCEPTION(WRITE_ERROR), errno.
| static int | ||
| copy_text_file(FILE *srcfile, FILE *dstfile, int start_line, int end_line) | ||
| { | ||
| char *buffer; | ||
| size_t len; | ||
| int i; | ||
|
|
||
| buffer = palloc(MAX_LINESIZE); | ||
|
|
||
| errno = 0; | ||
|
|
||
| /* skip first start_line. */ | ||
| for (i = 1; i < start_line; i++) | ||
| { | ||
| CHECK_FOR_INTERRUPTS(); | ||
| do | ||
| { | ||
| if (fgets(buffer, MAX_LINESIZE, srcfile) == NULL) | ||
| return errno; | ||
| len = strlen(buffer); | ||
| } while(buffer[len - 1] != '\n'); | ||
| } | ||
|
|
||
| /* copy until end_line. */ | ||
| for (; i <= end_line; i++) | ||
| { | ||
| CHECK_FOR_INTERRUPTS(); | ||
| do | ||
| { | ||
| if (fgets(buffer, MAX_LINESIZE, srcfile) == NULL) | ||
| return errno; | ||
| len = strlen(buffer); | ||
| if (fwrite(buffer, 1, len, dstfile) != len) | ||
| return errno; | ||
| } while(buffer[len - 1] != '\n'); | ||
| } | ||
|
|
||
| pfree(buffer); | ||
|
|
||
| return 0; | ||
| } |
There was a problem hiding this comment.
copy_text_file: last line without trailing newline is silently dropped during skip and copy.
If a line doesn't end with \n (common for the last line of a file), the inner do-while loop re-enters, fgets returns NULL (EOF), and the function returns errno (0 for normal EOF), exiting the outer for loop. During the skip phase, this is benign. During the copy phase, the partial last line IS written (line 1127) before the NULL return exits, so it's actually copied — but the outer loop terminates early, which may skip subsequent lines within the range if the file is somehow extended concurrently.
More importantly, the do-while pattern assumes every line ends with \n. Consider adding a break condition when fgets returns NULL to handle the last-line-no-newline case cleanly:
Proposed fix for both loops
for (i = 1; i < start_line; i++)
{
CHECK_FOR_INTERRUPTS();
do
{
if (fgets(buffer, MAX_LINESIZE, srcfile) == NULL)
- return errno;
+ return errno; /* EOF or error */
len = strlen(buffer);
- } while(buffer[len - 1] != '\n');
+ } while(len > 0 && buffer[len - 1] != '\n');
}
for (; i <= end_line; i++)
{
CHECK_FOR_INTERRUPTS();
do
{
if (fgets(buffer, MAX_LINESIZE, srcfile) == NULL)
- return errno;
+ return errno; /* EOF or error */
len = strlen(buffer);
if (fwrite(buffer, 1, len, dstfile) != len)
return errno;
- } while(buffer[len - 1] != '\n');
+ } while(len > 0 && buffer[len - 1] != '\n');
}🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c` around lines
1095 - 1135, In copy_text_file, the do/while loops assume each fgets() read ends
with '\n' and treat fgets()==NULL as an error (returning errno), which drops or
prematurely stops copying a final line without a trailing newline; update both
the skip and copy loops inside copy_text_file to check if fgets() returns NULL:
for the skip loop, break out of the inner loop (and outer skip loop) when
fgets()==NULL (EOF) instead of returning errno; for the copy loop, if
fgets()==NULL write any partial buffer content already read (ensure len>0 before
accessing buffer[len-1]) and then break to finish normally returning 0 rather
than returning errno; keep references to buffer, fgets, fwrite, start_line and
end_line when locating the change.
| static FILE * | ||
| get_file_handle_from_slot(uint32 sid, size_t *max_linesize, int *encoding) | ||
| { | ||
| int i; | ||
|
|
||
| if (sid == INVALID_SLOTID) | ||
| INVALID_FILEHANDLE_EXCEPTION(); | ||
|
|
||
| for (i = 0; i < MAX_SLOTS; i++) | ||
| { | ||
| if (slots[i].id == sid) | ||
| { | ||
| if (max_linesize) | ||
| *max_linesize = slots[i].max_linesize; | ||
| if (encoding) | ||
| *encoding = slots[i].encoding; | ||
| return slots[i].fd; | ||
| } | ||
| } | ||
|
|
||
| INVALID_FILEHANDLE_WARNING(); | ||
| return NULL; | ||
| } |
There was a problem hiding this comment.
get_file_handle_from_slot can return NULL, but most callers don't check.
When a stale/invalid slot ID is passed (valid but not found in slots[]), this function emits only a WARNING (line 1247) and returns NULL. Callers ora_utl_file_fflush, ora_utl_file_get_line, ora_utl_file_new_line, ora_utl_file_putf, do_put (and transitively ora_utl_file_put, ora_utl_file_put_line), and ora_utl_file_put_raw all pass the result directly to fflush/fputc/fwrite/fgetc without a NULL check, causing undefined behavior (likely crash).
The simplest fix: promote the WARNING to an ERROR so the function never returns NULL.
Proposed fix
}
- INVALID_FILEHANDLE_WARNING();
- return NULL;
+ INVALID_FILEHANDLE_EXCEPTION();
+ pg_unreachable(); /* ereport(ERROR) does not return */
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| static FILE * | |
| get_file_handle_from_slot(uint32 sid, size_t *max_linesize, int *encoding) | |
| { | |
| int i; | |
| if (sid == INVALID_SLOTID) | |
| INVALID_FILEHANDLE_EXCEPTION(); | |
| for (i = 0; i < MAX_SLOTS; i++) | |
| { | |
| if (slots[i].id == sid) | |
| { | |
| if (max_linesize) | |
| *max_linesize = slots[i].max_linesize; | |
| if (encoding) | |
| *encoding = slots[i].encoding; | |
| return slots[i].fd; | |
| } | |
| } | |
| INVALID_FILEHANDLE_WARNING(); | |
| return NULL; | |
| } | |
| static FILE * | |
| get_file_handle_from_slot(uint32 sid, size_t *max_linesize, int *encoding) | |
| { | |
| int i; | |
| if (sid == INVALID_SLOTID) | |
| INVALID_FILEHANDLE_EXCEPTION(); | |
| for (i = 0; i < MAX_SLOTS; i++) | |
| { | |
| if (slots[i].id == sid) | |
| { | |
| if (max_linesize) | |
| *max_linesize = slots[i].max_linesize; | |
| if (encoding) | |
| *encoding = slots[i].encoding; | |
| return slots[i].fd; | |
| } | |
| } | |
| INVALID_FILEHANDLE_EXCEPTION(); | |
| pg_unreachable(); /* ereport(ERROR) does not return */ | |
| } |
🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c` around lines
1227 - 1249, The function get_file_handle_from_slot can return NULL on a stale
slot (it currently calls INVALID_FILEHANDLE_WARNING() and returns NULL), leading
to callers like ora_utl_file_fflush, ora_utl_file_get_line,
ora_utl_file_new_line, ora_utl_file_putf, do_put (and transitive
ora_utl_file_put, ora_utl_file_put_line), and ora_utl_file_put_raw invoking
stdio functions on NULL; change the behavior so a missing slot triggers an
error/exception instead of just a warning—replace the
INVALID_FILEHANDLE_WARNING() path with the same error path used for
INVALID_SLOTID (e.g., call INVALID_FILEHANDLE_EXCEPTION() or otherwise raise an
error before returning) so get_file_handle_from_slot never returns NULL and
callers need not be changed.
| if (digits > 3) | ||
| { | ||
| GUC_check_errdetail("number is too big (only four digits are allowed"); | ||
| return false; |
There was a problem hiding this comment.
Missing closing parenthesis in error message.
Line 1408: "number is too big (only four digits are allowed" is missing the closing ).
Fix
- GUC_check_errdetail("number is too big (only four digits are allowed");
+ GUC_check_errdetail("number is too big (only four digits are allowed)");📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (digits > 3) | |
| { | |
| GUC_check_errdetail("number is too big (only four digits are allowed"); | |
| return false; | |
| if (digits > 3) | |
| { | |
| GUC_check_errdetail("number is too big (only four digits are allowed)"); | |
| return false; |
🤖 Prompt for AI Agents
In `@contrib/ivorysql_ora/src/builtin_packages/utl_file/utl_file.c` around lines
1406 - 1409, The string passed to GUC_check_errdetail in the digits > 3 branch
is missing a closing parenthesis; update the error message string in that
conditional (the GUC_check_errdetail(...) call) to read "number is too big (only
four digits are allowed)" so the parentheses are balanced and the message is
correct.
Final draft for review
Summary by CodeRabbit
Release Notes
New Features
utl_file.umaskconfiguration option to control file permissions for file operations.Tests