?h!#|dZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZddlm Z m!Z!dd l"m#Z$dd l"m%Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,m-Z-m.Z.ddl/m0Z0ddlm1Z1m2Z2m3Z3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBddlmCZCee2e3fZDe GddZEe GddZFGdde ZGGddZHedeeIeJef ZKd!eKd"eLd#eeKd$dfd%ZMd!eNd"eLd$dfd&ZOd'eId(eId)ePd$dfd*ZQd+eRd$eRfd,ZSd-eLd$eLfd.ZTy)/z Batch class definitions. N)deque)ThreadPoolExecutor as_completedFuture) dataclassfield)Real) AnyCallableDequeDictListOptionalSequenceSetTupleTypeTypeVarUnioncast) ReadTimeoutResponse)ConnectionError) HTTPError) Connection)ConsistencyLevel)_find_value_typeVALUE_ARRAY_TYPESWHERE_OPERATORS)UUID) BatchRequestObjectsBatchRequestReferenceBatchRequest BatchResponse)Cluster)&BATCH_REF_DEPRECATION_NEW_V14_CLS_NS_W&BATCH_REF_DEPRECATION_OLD_V14_CLS_NS_WBATCH_EXECUTOR_SHUTDOWN_W)UnexpectedStatusCodeException)_capitalize_first_lettercheck_batch_result_check_positive_num_decode_json_response_dict_decode_json_response_list) _WarningscFeZdZUeed<edZeeed<defdZ y)Shard class_nameN)defaulttenantreturncDt|j|jfSN)hashr4r6selfs W/home/chris/cleankitchens-env/lib/python3.12/site-packages/weaviate/batch/crud_batch.py__hash__zShard.__hash__?sT__dkk233) __name__ __module__ __qualname__str__annotations__rr6rintr>r?r=r3r3:s&O!$/FHSM/4#4r?r3c\eZdZUdZdZeed<dZee e ed<dZ ee e ed<ddZ y) WeaviateErrorRetryConfaConfigures how often objects should be retried when Weavaite returns an error and which errors should be included or excluded. By default, all errors are retried. Parameters ---------- number_retries: int How often a batch that includes objects with errors should be retried. Must be >=1. errors_to_exclude: Optional[List[str]] Which errors should NOT be retried. All other errors will be retried. An object will be skipped, when the given string is part of the weaviate error message. Example: errors_to_exclude =["string1", "string2"] will match the error with message "Long error message that contains string1". errors_to_include: Optional[List[str]] Which errors should be retried. All other errors will NOT be retried. An object will be included, when the given string is part of the weaviate error message. Example: errors_to_include =["string1", "string2"] will match the error with message "Long error message that contains string1". number_retriesNerrors_to_excludeerrors_to_includecv|j$|jt|jdzt |j dt dtttddfd}||j||j|j$t|jdk(r tdyy)Nz% can either include or exclude errorsrJ error_listr7cD|ytd|Dr tdy)Nc3>K|]}t|t ywr9) isinstancerC).0entrys r= zLWeaviateErrorRetryConf.__post_init__..check_lists..hsF%z%--FszList entries must be strings.)any ValueError)rNs r= check_listsz9WeaviateErrorRetryConf.__post_init__..check_listses+!F:FF !@AAGr?rz=errors_to_include has 0 entries and no error will be retried.) rKrLrVrAr.rJrErrrClen)r<rWs r= __post_init__z$WeaviateErrorRetryConf.__post_init___s  ! ! -$2H2H2TT__/VVW WD//1A3G BHT#Y$7 BD B D**+D**+  ! ! -#d6L6L2MQR2R\] ]3S -r?r7N) r@rArB__doc__rJrErDrKrrrCrLrYrFr?r=rHrHCsA,NC-1xS *1-1xS *1^r?rHceZdZdZdefdZy) BatchExecutorz Weaviate Batch Executor to run batch requests in separate thread. This class implements an additional method `is_shutdown` that us used my the context manager. r7c|jS)z Check if executor is shutdown. Returns ------- bool Whether the BatchExecutor is shutdown. ) _shutdownr;s r= is_shutdownzBatchExecutor.is_shutdownxs~~r?N)r@rArBr[boolr`rFr?r=r]r]rs T r?r]ceZdZdZdefdZdeddfdZddd d ded d df d e e d e e de de de e de e eegdfdede de eddfdZd]dZ d^dedede ede ede edef dZ d_dededed ed!e ede eddfd"Zd#ed$edefd%Zd&eddfd'Zd#ed$edefd(Zd$edefd)Z d$e!de!fd*Z"de#fd+Z$de#fd,Z%d#ed$ede&e ee ffd-Z'd.eddfd/Z(d]d0Z)d]d1Z* d`ded2ed3ed4ede edef d5Z+de fd6Z,de fd7Z-dad8e defd9Z.dad8e defd:Z/d]d;Z0d]d<Z1defd=Z2defd>Z3e4de&e e ffd?Z5e4de e fd@Z6e6jndAe e ddfdBZ6e4defdCZ8e8jndAeddfdDZ8e4de9edffdEZ:e:jndFe e9eefddfdGZ:e4de e fdHZ;e4de e fdIZdbdLZ?dMedNedOeddfdPZ@ dcdQe eeAdRe ddfdSZBdTeAdeefdUZCe4de fdVZDeDjndAe ddfdWZDe4de fdXZEeEjndAe ddfdYZEe4de fdZZFeFjndAe ddfd[ZFd&ed#ede&eGeffd\ZHy)dBatcha Batch class used to add multiple objects or object references at once into weaviate. To add data to the Batch use these methods of this class: `add_data_object` and `add_reference`. This object also stores 2 recommended batch size variables, one for objects and one for references. The recommended batch size is updated with every batch creation, and is the number of data objects/references that can be sent/processed by the Weaviate server in `creation_time` interval (see `configure` or `__call__` method on how to set this value, by default it is set to 10). The initial value is None/batch_size and is updated with every batch create methods. The values can be accessed with the getters: `recommended_num_objects` and `recommended_num_references`. NOTE: If the UUID of one of the objects already exists then the existing object will be replaced by the new object. This class can be used in 3 ways: Case I: Everything should be done by the user, i.e. the user should add the objects/object-references and create them whenever the user wants. To create one of the data type use these methods of this class: `create_objects`, `create_references` and `flush`. This case has the Batch instance's batch_size set to None (see docs for the `configure` or `__call__` method). Can be used in a context manager, see below. Case II: Batch auto-creates when full. This can be achieved by setting the Batch instance's batch_size set to a positive integer (see docs for the `configure` or `__call__` method). The batch_size in this case corresponds to the sum of added objects and references. This case does not require the user to create the batch/s, but it can be done. Also to create non-full batches (last batch/es) that do not meet the requirement to be auto-created use the `flush` method. Can be used in a context manager, see below. Case III: Similar to Case II but uses dynamic batching, i.e. auto-creates either objects or references when one of them reached the `recommended_num_objects` or `recommended_num_references` respectively. See docs for the `configure` or `__call__` method for how to enable it. Context-manager support: Can be use with the `with` statement. When it exists the context- manager it calls the `flush` method for you. Can be combined with `configure`/`__call__` method, in order to set it to the desired Case. Examples -------- Here are examples for each CASE described above. Here `client` is an instance of the `weaviate.Client`. >>> object_1 = '154cbccd-89f4-4b29-9c1b-001a3339d89d' >>> object_2 = '154cbccd-89f4-4b29-9c1b-001a3339d89c' >>> object_3 = '254cbccd-89f4-4b29-9c1b-001a3339d89a' >>> object_4 = '254cbccd-89f4-4b29-9c1b-001a3339d89b' For Case I: >>> client.batch.shape (0, 0) >>> client.batch.add_data_object({}, 'MyClass') >>> client.batch.add_data_object({}, 'MyClass') >>> client.batch.add_reference(object_1, 'MyClass', 'myProp', object_2) >>> client.batch.shape (2, 1) >>> client.batch.create_objects() >>> client.batch.shape (0, 1) >>> client.batch.create_references() >>> client.batch.shape (0, 0) >>> client.batch.add_data_object({}, 'MyClass') >>> client.batch.add_reference(object_3, 'MyClass', 'myProp', object_4) >>> client.batch.shape (1, 1) >>> client.batch.flush() >>> client.batch.shape (0, 0) Or with a context manager: >>> with client.batch as batch: ... batch.add_data_object({}, 'MyClass') ... batch.add_reference(object_3, 'MyClass', 'myProp', object_4) >>> # flush was called >>> client.batch.shape (0, 0) For Case II: >>> client.batch(batch_size=3) >>> client.batch.shape (0, 0) >>> client.batch.add_data_object({}, 'MyClass') >>> client.batch.add_reference(object_1, 'MyClass', 'myProp', object_2) >>> client.batch.shape (1, 1) >>> client.batch.add_data_object({}, 'MyClass') # sum of data_objects and references reached >>> client.batch.shape (0, 0) Or with a context manager and `__call__` method: >>> with client.batch(batch_size=3) as batch: ... batch.add_data_object({}, 'MyClass') ... batch.add_reference(object_3, 'MyClass', 'myProp', object_4) ... batch.add_data_object({}, 'MyClass') ... batch.add_reference(object_1, 'MyClass', 'myProp', object_4) >>> # flush was called >>> client.batch.shape (0, 0) Or with a context manager and setter: >>> client.batch.batch_size = 3 >>> with client.batch as batch: ... batch.add_data_object({}, 'MyClass') ... batch.add_reference(object_3, 'MyClass', 'myProp', object_4) ... batch.add_data_object({}, 'MyClass') ... batch.add_reference(object_1, 'MyClass', 'myProp', object_4) >>> # flush was called >>> client.batch.shape (0, 0) For Case III: Same as Case II but you need to configure or enable 'dynamic' batching. >>> client.batch.configure(batch_size=3, dynamic=True) # 'batch_size' must be an valid int Or: >>> client.batch.batch_size = 3 >>> client.batch.dynamic = True See the documentation of the `configure`( or `__call__`) and the setters for more information on how/why and what you need to configure/set in order to use a particular Case. connectioncVd|_d|_||_t|_t |_td|_td|_ g|_ g|_ tj|_t|_d|_d|_t't(t+|jj,ddz d|_d |_d |_d |_|j$|_|j$|_t;|_d|_d|_ d|_!y) ag Initialize a Batch class instance. This defaults to manual creation configuration. See docs for the `configure` or `__call__` method for different types of configurations. Parameters ---------- connection : weaviate.connect.Connection Connection object to an active and running weaviate instance. NT)maxlen2r! r&rIdynamic)"_shutdown_background_event_new_dynamic_batching _connectionr#_objects_batchr$_reference_batchr_objects_throughput_frame_references_throughput_frame _future_pool_reference_batch_queue threadingLock_callback_lockr- _callback_weaviate_error_retry _batch_sizerr mintimeout_config_creation_time_timeout_retries_connection_error_retries_batching_type_recommended_num_objects_recommended_num_referencesset_Batch__imported_shards _num_workers_consistency_level _executor)r<rds r=__init__zBatch.__init__ s FJ'%)"%13 5 77B26r?kwargsr7c &|jdi|S)av WARNING: This method will be deprecated in the next major release. Use `configure` instead. Parameters ---------- batch_size : Optional[int], optional The batch size to be use. This value sets the Batch functionality, if `batch_size` is None then no auto-creation is done (`callback` and `dynamic` are ignored). If it is a positive number auto-creation is enabled and the value represents: 1) in case `dynamic` is False -> the number of data in the Batch (sum of objects and references) when to auto-create; 2) in case `dynamic` is True -> the initial value for both `recommended_num_objects` and `recommended_num_references`, by default None creation_time : Real, optional How long it should take to create a Batch. Used ONLY for computing dynamic batch sizes. By default None timeout_retries : int, optional Number of retries to create a Batch that failed with ReadTimeout, by default 3 weaviate_error_retries: Optional[WeaviateErrorRetryConf], by default None How often batch-elements with an error originating from weaviate (for example transformer timeouts) should be retried and which errors should be ignored and/or included. See documentation for WeaviateErrorRetryConf for details. connection_error_retries : int, optional Number of retries to create a Batch that failed with ConnectionError, by default 3 callback : Optional[Callable[[dict], None]], optional A callback function on the results of each (objects and references) batch types. By default `weaviate.util.check_batch_result`. dynamic : bool, optional Whether to use dynamic batching or not, by default False num_workers : int, optional The maximal number of concurrent threads to run batch import. Only used for non-MANUAL batching. i.e. is used only with AUTO or DYNAMIC batching. By default, the multi-threading is disabled. Use with care to not overload your weaviate instance. Returns ------- Batch Updated self. Raises ------ TypeError If one of the arguments is of a wrong type. ValueError If the value of one of the arguments is wrong. rF) configure)r<rs r=__call__zBatch.__call__4sZt~~'''r?rhNrITr! batch_size creation_timetimeout_retriesconnection_error_retriesweaviate_error_retriescallbackrj num_workersconsistency_levelc | |_|t|dt||_n9t tt |j jddz d|_t|dtt|dt||_ ||_ ||_ ||_ ||sd|_d|_|St|dtt|d tt!|d |j"|k7r7|j%|j'||_|j)||_|d urd |_n9d |_|d n||_|d n||_|j.|j1|j3|S)a} Warnings -------- - It has default values and if you want to change only one use a setter instead or provide all the configurations, both the old and new ones. - This method will return `None` in the next major release. If you are using the returned `Batch` object then you should start using the `client.batch` object instead. Parameters ---------- batch_size : Optional[int], optional The batch size to be use. This value sets the Batch functionality, if `batch_size` is None then no auto-creation is done (`callback` and `dynamic` are ignored). If it is a positive number auto-creation is enabled and the value represents: 1) in case `dynamic` is False -> the number of data in the Batch (sum of objects and references) when to auto-create; 2) in case `dynamic` is True -> the initial value for both `recommended_num_objects` and `recommended_num_references`, by default 50 creation_time : Real, optional How long it should take to create a Batch. Used ONLY for computing dynamic batch sizes. By default None timeout_retries : int, optional Number of retries to create a Batch that failed with ReadTimeout, by default 3 connection_error_retries : int, optional Number of retries to create a Batch that failed with ConnectionError, by default 3 weaviate_error_retries: WeaviateErrorRetryConf, Optional How often batch-elements with an error originating from weaviate (for example transformer timeouts) should be retried and which errors should be ignored and/or included. See documentation for WeaviateErrorRetryConf for details. callback : Optional[Callable[[dict], None]], optional A callback function on the results of each (objects and references) batch types. By default `weaviate.util.check_batch_result` dynamic : bool, optional Whether to use dynamic batching or not, by default True num_workers : int, optional The maximal number of concurrent threads to run batch import. Only used for non-MANUAL batching. i.e. is used only with AUTO or DYNAMIC batching. By default, the multi-threading is disabled. Use with care to not overload your weaviate instance. Returns ------- Batch Updated self. Raises ------ TypeError If one of the arguments is of a wrong type. ValueError If the value of one of the arguments is wrong. Nrr!rir&rrrrrjFfixedrh)rr.r r|rrzrmr{_check_non_negativerErwr}r~rxryr _check_boolrflushshutdownstartrrrk_update_recommended_batch_size _auto_create) r<rrrrrrrjrrs r=rzBatch.configurecshz"3  $   E"/D "&tS1A1A1P1PQR1SVX1XZ[-\"]D O->D46PRUV! /)A&%;"  g#D "&D KJ c:K<GY'    + JJL MMO +D  JJL% e ")D "+D 2<2DB*D )5?5GrZD ,..6335  r?ctj_dfd }tj|dd}|j y)zUCreate a background thread that periodically checks how congested the batch queue is.Nctj}j9jjs |j }d|dvs d|ddvrd_y|ddd}|j z }|ddd}|dk(r,jtjdzdz_n_||z }d |cxkDrd kDr nn|_nD|d kr%tjd z|dz|z _n|d kr|dz|z _nd_d}tj|jjjsd _d_y#ttf$rd }Y_wxYw)Nstatsr ratePerSecondF batchStats queueLengthr&g@gffffff?g?rig?) r'rmrkis_setget_nodes_statusrlrrrzRequestsHTTPErrorrtimesleep)clusterstatusraterate_per_worker batch_lengthratio refresh_timer<s r=periodic_checkz.periodic_checksd../G//;77>>@'$557FfQi/?&QR)T[J\3\5:2!!9\2?CD&*T->->&>O#)!9\#:=#IL#q(8<8U8UX[ 99A=rY95!-t 3%-#->@F-/D ).2D + *;7'#&L's(E,)B7E,,F?FTbatchSizeRefresh)targetdaemonnamerZ)rtEventrkThreadr)r<rdemons` r=rz$Batch._update_recommended_batch_sizes>*3//*;'( 3T  !#   r? data_objectr4uuidvectorr6c|jjt|||||}|jjt |||j r|j |S)a Add one object to this batch. NOTE: If the UUID of one of the objects already exists then the existing object will be replaced by the new object. Parameters ---------- data_object : dict Object to be added as a dict datatype. class_name : str The name of the class this object belongs to. uuid : Optional[UUID], optional The UUID of the object as an uuid.UUID object or str. It can be a Weaviate beacon or Weaviate href. If it is None an UUIDv4 will generated, by default None vector: Sequence or None, optional The embedding of the object that should be validated. Can be used when: - a class does not have a vectorization module. - The given vector was generated using the _identical_ vectorization module that is configured for the class. In this case this vector takes precedence. Supported types are `list`, 'numpy.ndarray`, `torch.Tensor` and `tf.Tensor`, by default None. Returns ------- str The UUID of the added object. If one was not provided a UUIDv4 will be generated. Raises ------ TypeError If an argument passed is not of an appropriate type. ValueError If 'uuid' is not of a proper form. r4rrrr6)rnaddr,rr3rr)r<rr4rrr6s r=add_data_objectzBatch.add_data_objectskX""&&/ ;# '  ""5V#<=        r?from_object_uuidfrom_object_class_namefrom_property_nameto_object_uuidto_object_class_namec|jjdk\}|"|r tjtt d|X|s"tjt t dd}|r2t|tstdt|t|}|jjt|||||||jr|jyy)ay Add one reference to this batch. Parameters ---------- from_object_uuid : UUID The UUID of the object, as an uuid.UUID object or str, that should reference another object. It can be a Weaviate beacon or Weaviate href. from_object_class_name : str The name of the class that should reference another object. from_property_name : str The name of the property that contains the reference. to_object_uuid : UUID The UUID of the object, as an uuid.UUID object or str, that is actually referenced. It can be a Weaviate beacon or Weaviate href. to_object_class_name : Optional[str], optional The referenced object class name to which to add the reference (with UUID `to_object_uuid`), it is included in Weaviate 1.14.0, where all objects are namespaced by class name. STRONGLY recommended to set it with Weaviate >= 1.14.0. It will be required in future versions of Weaviate Server and Clients. Use None value ONLY for Weaviate < v1.14.0, by default None tenant: str, optional Name of the tenant. Raises ------ TypeError If arguments are not of type str. ValueError If 'uuid' is not valid or cannot be extracted. z1.14Nr!messagecategory stacklevelz@'to_object_class_name' must be of type str or None. Given type: )rrrrrr6)rmserver_versionwarningswarnr(DeprecationWarningr)rQrC TypeErrortyper,rorrr)r<rrrrrr6is_server_version_14s r= add_referencezBatch.add_reference=sT $//>>&H  ',@ MM>+  +' B/  (,$#!"6<#''+,@'A&BD(@@T'U$ !!#;-0J0J0Y0YYCGCWCW)9D@(@~.2 ../GH-2-,:M$&&}5   3 &O+gi[ ,JHUUo#/+$($9$9# "Q&M$($C$CI}$]M=)Q.#+:/2,+3+=+= ,,;;A>B,( //*/.$($B$B# %)$ *1d' \)*LMS[ [ 1 {#99=9I9I9X9XYZ9[8\]::=m:L9MNPP  g&D 0 1sUG/D BG G  BF G G %GGG  G H+ G&&AH+rc|jy|j5|j|dddy#1swYyxYwr9)rwrv)r<rs r=rzBatch._run_callbacks9 >> !    % NN8 $ % % %s5>c|dk(r#t|tsJ|j|St|tsJ|j |S)a Readds items (objects or references) that were not added due to a timeout. Parameters ---------- data_type : str The Batch Request type, can be either 'objects' or 'references'. batch_request : BatchRequest The Batch Request that TimeOuted. Returns ------- BatchRequest New Batch Request with objects that were not added or not updated. objects)rQr#_readd_objects_after_timeoutr$_readd_references_after_timeout)r<rrs r=rz Batch._batch_retry_after_timeoutsP&  !m-@A AA44]C Cm-BC CC77 F Fr?c t}|jdD]'}|d}|jdd}|d}|d|ind}|jj d|zdz|z|}|j d k(r2|j t||d ||jd d |jjd|zdz|z|} t| d } | J| d |d k7s&|jd d| jd dk7s|j t||d ||jd d|*|S)a Read all objects that were not created or updated because of a TimeOut error. Parameters ---------- batch_request : ObjectsBatchRequest The ObjectsBatchRequest from which to check if items where created or updated. Returns ------- ObjectsBatchRequest New ObjectsBatchRequest with only the objects that were not created or updated. rclassr6Nidz /objects//)rri propertiesr)r4rrrzRe-add objectsr) r#rgetrmheadrrr,r/) r<r new_batchobjr4r6rr response_headrobj_weavs r=rz"Batch._readd_objects_after_timeouts"()  113I>% CWJWWXt,Ft9D+1+=h'4F ,,11 :-3d:2M ((C/ 7 C #L 1778T2  ''++ :-3d:,H 2(>> client.batch.add_data_object({}, 'NonExistingClass') >>> client.batch.add_data_object({}, 'ExistingClass') Note that 'NonExistingClass' is not present in the client's schema and 'ExistingObject' is present and has no proprieties. 'client.batch.add_data_object' does not raise an exception because the objects added meet the required criteria (See the documentation of the 'weaviate.Batch.add_data_object' method for more information). >>> result = client.batch.create_objects(batch) Successful batch creation even if one data object is inconsistent with the client's schema. We can find out more about what objects were successfully created by analyzing the 'result' variable. >>> import json >>> print(json.dumps(result, indent=4)) [ { "class": "NonExistingClass", "creationTimeUnix": 1614852753747, "id": "154cbccd-89f4-4b29-9c1b-001a3339d89a", "properties": {}, "deprecations": null, "result": { "errors": { "error": [ { "message": "class 'NonExistingClass' not present in schema, class NonExistingClass not present" } ] } } }, { "class": "ExistingClass", "creationTimeUnix": 1614852753746, "id": "b7b1cfbe-20da-496c-b932-008d35805f26", "properties": {}, "vector": [ -0.05244319, ... 0.076136276 ], "deprecations": null, "result": {} } ] As it can be noticed the first object from the batch was not added/created, but the batch was successfully created. The batch creation can be successful even if all the objects were NOT created. Check the status of the batch objects to find which object and why creation failed. Alternatively use 'client.data_object.create' for Object creation that throw an error if data item is inconsistent or creation/addition failed. To check the results of batch creation when using the auto-creation Batch, use a 'callback' (see the docs `configure` or `__call__` method for more information). Returns ------- list A list with the status of every object that was created. Raises ------ requests.ConnectionError If the network connection to weaviate fails. weaviate.UnexpectedStatusCodeException If weaviate reports a none OK status. rrrrr!zbatch add objects)rXrnr1manual_batchingrr#rpappendr total_secondssummaxroundfloatr|rr0)r<robj_per_secondress r=create_objectszBatch.create_objectsjsl t"" #q (  % % '((#"11)H#6"7D   * * 1 1D''(8+;+;+I+I+KK !!?!?@3..DN-0nuT-@-@'AABA-D )-X7JKC? "?J r?ct|jdk7rtj|j d|j}t |_|j jt|j|jjz t|j t|j z }t|t|jz|_t|d}|J|SgS)a Creates multiple References at once in Weaviate. Adding References in batch is faster but it ignores validations like class name and property name, resulting in a SUCCESSFUL reference creation of a nonexistent object types and/or a nonexistent properties. If the consistency of the References is wanted use 'client.data_object.reference.add' to have additional validation against the weaviate schema. See Examples below. Examples -------- Here `client` is an instance of the `weaviate.Client`. Object that does not exist in weaviate. >>> object_1 = '154cbccd-89f4-4b29-9c1b-001a3339d89d' Objects that exist in weaviate. >>> object_2 = '154cbccd-89f4-4b29-9c1b-001a3339d89c' >>> object_3 = '254cbccd-89f4-4b29-9c1b-001a3339d89a' >>> object_4 = '254cbccd-89f4-4b29-9c1b-001a3339d89b' >>> client.batch.add_reference(object_1, 'NonExistingClass', 'existsWith', object_2) >>> client.batch.add_reference(object_3, 'ExistingClass', 'existsWith', object_4) Both references were added to the batch request without error because they meet the required criteria (See the documentation of the 'weaviate.Batch.add_reference' method for more information). >>> result = client.batch.create_references() As it can be noticed the reference batch creation is successful (no error thrown). Now we can inspect the 'result'. >>> import json >>> print(json.dumps(result, indent=4)) [ { "from": "weaviate://localhost/NonExistingClass/ 154cbccd-89f4-4b29-9c1b-001a3339d89a/existsWith", "to": "weaviate://localhost/154cbccd-89f4-4b29-9c1b-001a3339d89b", "result": { "status": "SUCCESS" } }, { "from": "weaviate://localhost/ExistingClass/ 254cbccd-89f4-4b29-9c1b-001a3339d89a/existsWith", "to": "weaviate://localhost/254cbccd-89f4-4b29-9c1b-001a3339d89b", "result": { "status": "SUCCESS" } } ] Both references were added successfully but one of them is corrupted (links two objects of nonexisting class and one of the objects is not yet created). To make use of the validation, crete each references individually (see the client.data_object.reference.add method). Returns ------- list A list with the status of every reference added. Raises ------ requests.ConnectionError If the network connection to weaviate fails. weaviate.UnexpectedStatusCodeException If weaviate reports a none OK status. r referencesrzCreate references)rXror1rrr$rqrrrr r r r|rr0)r<r ref_per_secrs r=create_referenceszBatch.create_referencessT t$$ % *  % % '((&"33)H%:$;D !  - - 4 4D))*X-=-=-K-K-MM d??@311DK05[5I\I\C]5]/^D ,,X7JKC? "?J r?c`t|dk7r |j||}|t|fSy)a Flush BatchRequest in current thread/process. Parameters ---------- data_type : str The data type of the BatchRequest, used to save time for not checking the type of the BatchRequest. batch_request : weaviate.batch.BatchRequest Contains all the data objects that should be added in one batch. Note: Should be a sub-class of BatchRequest since BatchRequest is just an abstract class, e.g. ObjectsBatchRequest, ReferenceBatchRequest Returns ------- Tuple[requests.Response, int] The request response and number of items sent with the BatchRequest as tuple. rr)Nr)rXr)r<rrrs r=_flush_in_threadzBatch._flush_in_thread:sA0 }  "((#+)HS// /r? force_waitc |j|jnJ|jjr0tjt t d|j|jJ|jj|jd|j}|jj|t|jdkDr%|jj|jt|_ t!|_ |s2|j"dkDr#t|j|j"kryd}t%|jD]P}|j'\}}|7|j(j||j*j-z Od}R|r+|j.t1|j.d zd|_nt|j(dk7r|j.|j2sut5|j(t|j(z d z}t1t7t9|t;|j<z|j.d zd|_g}|jD];} |jj|jd | }|j|=d}t%|D]P}|j'\} } | 7|j>j| | j*j-z Od}R|r+|j@t1|j@d zd|_ nt|j>dk7rt|j@ht5|j>t|j>z } t7t9| t;|j<z|j@d z|_ g|_ g|_y) a Send BatchRequest in a separate thread/process. This methods submits a task to create only the ObjectsBatchRequests to the BatchExecutor and adds the ReferencesBatchRequests to a queue, then it carries on in the main thread until `num_workers` tasks have been submitted. When we have reached number of tasks to be equal to `num_workers` it waits for all the tasks to finish and handles the responses. After all ObjectsBatchRequests have been handled it created separate tasks for each ReferencesBatchRequests, then it handles their responses as well. This mechanism of creating References after Objects is constructed in this manner to eliminate potential error when creating references from a object that does not yet exists (object that is part of another task). Parameters ---------- force_wait : bool Whether to wait on all created tasks even if we do not have `num_workers` tasks created Nr!rrrrFTr&g?r)!rrr`rrr*RuntimeWarningsubmitrrnrrrrXrorsr#r$rrresultrprrrr rlr rzr r r|rqr) r<rfuturetimeout_occurred done_futureresponse_objects nr_objectsr reference_future_poolreference_batchresponse_references nr_referencesrs r=_send_batch_requestszBatch._send_batch_requestsZs" >> ! JJL ^^ ' ' ) MM1'  JJL~~)))&&  ! !--'    ( t$$ % )  ' ' . .t/D/D E13 5 7d//!3D (K1<1C1C1E . #.1188!$7$?$?$M$M$OO$(  (  @ @ L/243S3SWX3XZ[/\D , 11 2a 700<d??@311DK03kE$*=*=$>>?00140D , &(#r?c|jdk(rC|jJt|j|jk\r|j dy|jdk(r|j |j k\s|j|jk\rF|j dk(r%tjd|j dk(r%|j dytd|jd ) ad Auto create both objects and references in the batch. This protected method works with a fixed batch size and with dynamic batching. For a 'fixed' batching type it auto-creates when the sum of both objects and references equals batch_size. For dynamic batching it creates both batch requests when only one is full. rNFrrjrr!zUnsupported batching type "") rryr shaper% num_objectsrnum_referencesrrrrVr;s r=rzBatch._auto_creates   ' )##/ //4::$"2"22))U);   I -  "d&C&CC&&(D,L,LL33q8JJqM33q8))U); 6t7J7J6K1MNNr?c(|jdy)z Flush both objects and references to the Weaviate server and call the callback function if one is provided. (See the docs for `configure` or `__call__` for how to set one.) Tr'N)r%r;s r=rz Batch.flushs !!T!2r?whereoutputdry_runcpt|tstdt|dt|tstdt|dt|tstdt|dt|t stdt|di}|j |j j|d<|||d<t|t|d ||d } |jjd || }t|d} | J| S#t$r} td | d} ~ wwxYw)a Delete objects that match the 'match' in batch. Parameters ---------- class_name : str The class name for which to delete objects. where : dict The content of the `where` filter used to match objects that should be deleted. output : str, optional The control of the verbosity of the output, possible values: - "minimal" : The result only includes counts. Information about objects is omitted if the deletes were successful. Only if an error occurred will the object be described. - "verbose" : The result lists all affected objects with their ID and deletion status, including both successful and unsuccessful deletes. By default "minimal" dry_run : bool, optional If True, objects will not be deleted yet, but merely listed, by default False Examples -------- If we want to delete all the data objects that contain the word 'weather' we can do it like this: >>> result = client.batch.delete_objects( ... class_name='Dataset', ... output='verbose', ... dry_run=False, ... where={ ... 'operator': 'Equal', ... 'path': ['description'], ... 'valueText': 'weather' ... } ... ) >>> print(json.dumps(result, indent=4)) { "dryRun": false, "match": { "class": "Dataset", "where": { "operands": null, "operator": "Equal", "path": [ "description" ], "valueText": "weather" } }, "output": "verbose", "results": { "failed": 0, "limit": 10000, "matches": 2, "objects": [ { "id": "1eb28f69-c66e-5411-bad4-4e14412b65cd", "status": "SUCCESS" }, { "id": "da217bdd-4c7c-5568-9576-ebefe17688ba", "status": "SUCCESS" } ], "successful": 2 } } Returns ------- dict The result/status of the batch delete. z.'class_name' must be of type str. Given type: .z*'where' must be of type dict. Given type: z*'output' must be of type str. Given type: z,'dry_run' must be of type bool. Given type: Nrr6)rr-)matchr.dryRunz/batch/objectsrz Batch delete was not successful.zDelete in batch)rQrCrrdictrarrr,_clean_delete_objects_wherermdeleterr/) r<r4r-r.r/r6rpayloadrrrs r=delete_objectszBatch.delete_objectssbd*c*LTR\M]L^^_`a a%&He UVWX X&#&HfVWXY Y'4(J4PW=/YZ[\ \!#  " " .*.*A*A*G*GF& '  %F8 2*=4U;   \''..% '/H)3DE ' \)*LMS[ [ \s+D D5$ D00D5c,t|jS)z Get current number of objects in the batch. Returns ------- int The number of objects in the batch. )rXrnr;s r=r*zBatch.num_objectsds4&&''r?c,t|jS)z Get current number of references in the batch. Returns ------- int The number of references in the batch. )rXror;s r=r+zBatch.num_referencesps4(())r?indexc8|jj|S)a Remove and return the object at index (default last). Parameters ---------- index : int, optional The index of the object to pop, by default -1 (last item). Returns ------- dict The popped object. Raises ------- IndexError If batch is empty or index is out of range. )rnpopr<r;s r= pop_objectzBatch.pop_object|s(""&&u--r?c8|jj|S)a Remove and return the reference at index (default last). Parameters ---------- index : int, optional The index of the reference to pop, by default -1 (last item). Returns ------- dict The popped reference. Raises ------- IndexError If batch is empty or index is out of range. )ror=r>s r= pop_referencezBatch.pop_references($$((//r?c8|jjy)z8 Remove all the objects from the batch. N)rnemptyr;s r= empty_objectszBatch.empty_objectss !!#r?c8|jjy)z; Remove all the references from the batch. N)rorCr;s r=empty_referenceszBatch.empty_referencess ##%r?c6|jjS)z Check if batch contains any objects. Returns ------- bool Whether the Batch object list is empty. )rnis_emptyr;s r=is_empty_objectszBatch.is_empty_objectss""++--r?c6|jjS)z Check if batch contains any references. Returns ------- bool Whether the Batch reference list is empty. )rorHr;s r=is_empty_referenceszBatch.is_empty_referencess$$--//r?cVt|jt|jfS)a" Get current number of objects and references in the batch. Returns ------- Tuple[int, int] The number of objects and references, respectively, in the batch as a tuple, i.e. returns (number of objects, number of references). )rXrnror;s r=r)z Batch.shapes%D''(#d.C.C*DEEr?c|jS)aF Setter and Getter for `batch_size`. Parameters ---------- value : Optional[int] Setter ONLY: The new value for the batch_size. If NOT None it will try to auto-create the existing data if it meets the requirements. If previous value was None then it will be set to new value and will change the batching type to auto-create with dynamic set to False. See the documentation for `configure` or `__call__` for more info. If recommended_num_objects is None then it is initialized with the new value of the batch_size (same for references). Returns ------- Optional[int] Getter ONLY: The current value of the batch_size. It is NOT the current number of data in the Batch. See the documentation for `configure` or `__call__` for more info. Raises ------ TypeError Setter ONLY: If the new value is not of type int. ValueError Setter ONLY: If the new value has a non positive value. )ryr;s r=rzBatch.batch_sizes:r?rc|d|_d|_yt|dt||_|jd|_|j||_|j ||_|j y)Nrr)ryrr.rErrrr<rs r=rzBatch.batch_sizesv =#D "&D  E<5     &")D   ( ( 0,1D )  + + 3/4D , r?c |jdk(S)a) Setter and Getter for `dynamic`. Parameters ---------- value : bool Setter ONLY: En/dis-able the dynamic batching. If batch_size is None the value is not set, otherwise it will set the dynamic to new value and auto-create if it meets the requirements. Returns ------- bool Getter ONLY: Wether the dynamic batching is enabled. Raises ------ TypeError Setter ONLY: If the new value is not of type bool. rj)rr;s r=rjz Batch.dynamic s.""i//r?c||jyt|d|durd|_nd|_|jy)NrjTr)rrrrOs r=rjz Batch.dynamic%s>    & E9% D="+D ")D  r?cJ|j|jjSdSr9)rrr;s r=rzBatch.consistency_level2s%040G0G0St&&,,]Y]]r?xc8|t||_yd|_yr9)rr)r<rSs r=rzBatch.consistency_level6s9:"21"5Dr?c|jS)z The recommended number of objects per batch. If None then it could not be computed. Returns ------- Optional[int] The recommended number of objects per batch. If None then it could not be computed. )rr;s r=recommended_num_objectszBatch.recommended_num_objects:s,,,r?c|jS)a The recommended number of references per batch. If None then it could not be computed. Returns ------- Optional[int] The recommended number of references per batch. If None then it could not be computed. )rr;s r=recommended_num_referencesz Batch.recommended_num_referencesGs///r?c|j|jjrt|j|_|jdk(r6|j |j j r|j|S)z Start the BatchExecutor if it was closed. Returns ------- Batch Updated self. ) max_workersrj)rr`r]rrrkrrr;s r=rz Batch.startTsk >> !T^^%?%?%A*t7H7HIDN   ) +  + + 3t7V7V7]7]7_  / / 1 r?c|j4|jjs|jj|j|jj yy)z- Shutdown the BatchExecutor. N)rr`rrkrr;s r=rzBatch.shutdownhsP&$..*D*D*F NN # # %  * * 6  + + / / 1 7r?c"|jSr9)rr;s r= __enter__zBatch.__enter__rszz|r?exc_typeexc_valexc_tbcD|j|jyr9)rr)r<r^r_r`s r=__exit__zBatch.__exit__us  r?shardshow_many_failurescH(ttstdtd+tdtstdtddt dt ffd ds*tdtjd ds)yy) ayWait for the all the vectors of the batch imported objects to be indexed. Upon network error, it will retry to get the shards' status for `how_many_failures` times with exponential backoff (2**n seconds with n=0,1,2,...,how_many_failures). Parameters ---------- shards {Optional[List[Shard]]} -- The shards to check the status of. If None it will check the status of all the shards of the imported objects in the batch. how_many_failures {int} -- How many times to try to get the shards' status before raising an exception. Default 5. Nz2'shards' must be of type List[Shard]. Given type: r1rhow_manyr7c  tfdxs jDS#t$rK}td|dd|zd||k(r|t j d|z|dzcYd}~Sd}~wwxYw)Nc3RK|]}tj| ywr9)all_get_shards_readiness)rRshardr<s r=rTzCBatch.wait_for_vector_indexing..is_ready..s)2259:s$'z+Error while getting class shards statuses: z, trying again with 2**n=r&zs exponential backoff with n=r!)rirrprintrr)rferdis_readyr<rcs r=rnz0Batch.wait_for_vector_indexing..is_readys .!'!A4+A+A+ .A!D]^_ai^i]jkHIQHRS%0G 1h;'1 -- .s!% A9AA4.A94A9z'Waiting for async indexing to finish...g?) rQlistrrr3rErarlrr)r<rcrdrns```@r=wait_for_vector_indexingzBatch.wait_for_vector_indexingys  j&>PQUV\Q]P^^_`a a  jE&BPQUV\Q]P^^_`a a .s .t . .1+ ; < JJt 1+r?rkc *t|jts"tdt |jddt |jd|j dnd|j } |jj|}t|d }|J|Dcgc]G}tt|jd d k(tt|jd d k(zIc}S#t$r}td|d}~wwxYwcc}w)Nz9'class_name' argument must be of type `str`! Given type: r1z/schema/z/shardsz?tenant=)rzDClass shards' status could not be retrieved due to connection error.zGet shards' statusrREADYvectorQueueSizer) rQr4rCrrr,r6rmrrr0rrE)r<rkrrrrs r=rjzBatch._get_shards_readinesssE%**C0#E$4$456a9  253C3CDEWSXS_S_SgRowx}yEyExFnGMHI ''+++6H )3GH #uyy* +w 6C#456!; =  ' )V   s4C3$A D3 D < DD c|jS)a Setter and Getter for `creation_time`. Parameters ---------- value : Real Setter ONLY: Set new value to creation_time. The recommended_num_objects/references values are updated to this new value. If the batch_size is not None it will auto-create the batch if the requirements are met. Returns ------- Real Getter ONLY: The `creation_time` value. Raises ------ TypeError Setter ONLY: If the new value is not of type Real. ValueError Setter ONLY: If the new value has a non positive value. )r|r;s r=rzBatch.creation_times2"""r?ct|dt|j=0).N)rQrarrV)rrrs r=rr1sS* eY ':eT+B!H:%7 {!DEE qy1XJ&`abbr?cBt|tstd|dy)a" Check if bool. Parameters ---------- value : bool The value to check. arg_name : str The name of the variable from the original function call. Used for error message. Raises ------ TypeError If the `value` is not of type bool. rz' must be of type bool.N)rQrar)rrs r=rrLs(" eT "!H:%<=>> #r?rrrc ||k\r|td|jjd|dzdzd|dzd|d tjd t j |dzdzy ) a Handle errors that occur in Batch creation. This function is going to re-raise the error if number of re-tries was reached. Parameters ---------- retry : int Current number of attempted request calls. max_retries : int Maximum number of attempted request calls. error : Exception The exception that occurred (to be re-raised if needed). Raises ------ Exception The caught exception. z[ERROR] Batch z! Exception occurred! Retrying in r!r&zs. [r]T)filerN)rl __class__r@sysstderrrrrs r=rrass$   1122S AI? 4 {!K= ; ZZ   JJ Qr?r-cd|vrt|}t|}d|vrtd||dtvrtd|ddt|d}d|vrd|vrtd|d |d t|j |||<|Sd |vr!|d Dcgc] }t |c}|d <|Std |cc}w) a1Converts the Python-defined where filter type into the Weaviate-defined where filter type used in the Batch REST request endpoint. Parameters ---------- where : dict The Python-defined where filter. Returns ------- dict The Weaviate-defined where filter. roperatorz:Where filter is missing required field `operator`. Given: z Operator z( is not allowed. Allowed operators are: ContainsArrayz Operator 'z#' is not supported for value type 'z'. Supported value types are: operandsz>Where is missing required fields `path` or `operands`. Given: )r_convert_value_typerVrrr=r5)r- py_value_typeweaviate_value_typeroperands r=r5r5~s8(/ 1-@ U "OPUwW   O 3E*-./**9):< $  !g5H&HXJ&IJ]I^^|~O}PQ &+YY}%=!" L u QVWaQbcg8Acj L OPUw W  dsB>_typecN|dk(ry|dk(ry|dk(ry|dk(ry|d k(ry |d k(ry |S) a9Converts the Python-defined where filter type into the Weaviate-defined where filter type used in the Batch REST request endpoint. Parameters ---------- _type : str The Python-defined where filter type. Returns ------- str The Weaviate-defined where filter type. valueTextListvalueTextArrayvalueStringListvalueStringArray valueIntList valueIntArrayvalueNumberListvalueNumberArrayvalueBooleanList valueDateListvalueDateArrayrF)rs r=rrsN  # #! .  # #! $ $! / ! r?)Ur[rrrtrr collectionsrconcurrent.futuresrrr dataclassesrrnumbersr typingr r r r rrrrrrrrrrequestsrrrequests.exceptionsrrrrweaviate.connectrweaviate.data.replicationrweaviate.gql.filterrrrweaviate.typesr r"r#r$r%rr' error_msgsr(r)r* exceptionsr+utilr,r-r.r/r0r1rr3rHr]rcrEr rrCrrar Exceptionrr4r5rrFr?r=rs  GG( +J>'6TT]] 7!,.CCD 44 4 +^+^ +^\&&f/f/R5 CuS%-./cqcCcDGcc6?t?s?t?* s  Y SW :&t&&Rssr?