|
| 1 | +# Vector Concat and Slice Functions for libSQL |
| 2 | + |
| 3 | +This project adds two new vector functions to libSQL to maintain compatibility with Turso: |
| 4 | + |
| 5 | +1. `vector_concat(X, Y)` - Concatenates two vectors of the same type. |
| 6 | +2. `vector_slice(X, start_idx, end_idx)` - Extracts a subvector from start_idx (inclusive) to end_idx (exclusive). |
| 7 | + |
| 8 | +## Implementation |
| 9 | + |
| 10 | +The implementation is based on the existing vector functions in libSQL. The new functions are added to the `vector.c` file in the `libsql-sqlite3/src` directory. |
| 11 | + |
| 12 | +### `vector_concat(X, Y)` |
| 13 | + |
| 14 | +This function concatenates two vectors of the same type. It performs the following steps: |
| 15 | + |
| 16 | +1. Parse the two input vectors |
| 17 | +2. Check that both vectors are of the same type |
| 18 | +3. Allocate a new vector with dimensions equal to the sum of the dimensions of the input vectors |
| 19 | +4. Copy the data from both vectors into the new vector |
| 20 | +5. Return the new vector |
| 21 | + |
| 22 | +### `vector_slice(X, start_idx, end_idx)` |
| 23 | + |
| 24 | +This function extracts a slice of a vector from start_idx (inclusive) to end_idx (exclusive). It performs the following steps: |
| 25 | + |
| 26 | +1. Parse the input vector |
| 27 | +2. Validate the start and end indices: |
| 28 | + - Both must be non-negative |
| 29 | + - start_idx must not be greater than end_idx |
| 30 | + - Both must be within the bounds of the vector |
| 31 | +3. Allocate a new vector with dimensions equal to end_idx - start_idx |
| 32 | +4. Copy the appropriate slice of data from the input vector to the new vector |
| 33 | +5. Return the new vector |
| 34 | + |
| 35 | +Note: FLOAT1BIT vectors are not yet supported for the slice operation due to the complexity of bit-by-bit extraction. |
| 36 | + |
| 37 | +## Usage |
| 38 | + |
| 39 | +```sql |
| 40 | +-- Create a test table with a vector column |
| 41 | +CREATE TABLE test_vectors ( |
| 42 | + id INTEGER PRIMARY KEY, |
| 43 | + vec VECTOR |
| 44 | +); |
| 45 | + |
| 46 | +-- Insert some test vectors |
| 47 | +INSERT INTO test_vectors VALUES (1, vector32(1, 2, 3, 4, 5)); |
| 48 | +INSERT INTO test_vectors VALUES (2, vector32(6, 7, 8, 9, 10)); |
| 49 | + |
| 50 | +-- Concatenate vectors |
| 51 | +SELECT vector_extract(vector_concat(vec, vector32(11, 12, 13))) FROM test_vectors WHERE id = 1; |
| 52 | +-- Returns: [1.0, 2.0, 3.0, 4.0, 5.0, 11.0, 12.0, 13.0] |
| 53 | + |
| 54 | +-- Slice a vector |
| 55 | +SELECT vector_extract(vector_slice(vec, 1, 4)) FROM test_vectors WHERE id = 1; |
| 56 | +-- Returns: [2.0, 3.0, 4.0] |
| 57 | +``` |
| 58 | + |
| 59 | +## Testing |
| 60 | + |
| 61 | +A test file `test_vector_functions.sql` is provided to verify the implementation. |
| 62 | + |
| 63 | +## Building |
| 64 | + |
| 65 | +The implementation is integrated directly into the libSQL SQLite fork. To build it, follow the standard libSQL build process. |
| 66 | + |
| 67 | +## References |
| 68 | + |
| 69 | +- Turso Implementation: https://github.com/tursodatabase/turso/pull/2336 |
| 70 | +- libSQL Issue: https://github.com/tursodatabase/libsql/issues/2136 |
0 commit comments