ZTh? ddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z ddl Z ddlZ ddlZddlmZdd lmZmZdd lmZdd lmZdd lmZdd l m!Z!GddZ"de e#efde#dedefdZ$de e#efdedejJfdZ&dejJdejJdejJfdZ'dejPde#de)de edejJf dZ*dedejVdejVfd Z,ded!ejJdefd"Z-dede efd#Z.y#e$rZededZ[wwxYw#e$rZededZ[wwxYw)$N) defaultdict) ExitStack)Path)DictListTupleUnioncastzThe h5py package is required to use the HDF5 upload service. Please include this dependency in your project by specifying `sift-stack-py[hdf5]`.zThe polars package is required to use the HDF5 upload service. Please include this dependency in your project by specifying `sift-stack-py[hdf5]`.) Hdf5DataCfg) CsvConfig Hdf5Config)CsvUploadService)DataImportService)NamedTemporaryFile)SiftRestConfigc deZdZUdZeed<eed<defdZ d de ee fde de d e fd Zd efd Zy )Hdf5UploadServicez' Service to upload HDF5 files. _csv_upload_service _prev_run_id rest_confc2t||_d|_y)N)rrr)selfrs ]/home/www/backend.miabetepe.com/venv/lib/python3.12/site-packages/sift_py/data_import/hdf5.py__init__zHdf5UploadService.__init__)s#3I#> path hdf5_config show_progressreturnct|tr t|n|}|jst d|dt |}t 5}g}|D]R}|jtdd} t|| j|} |j| j| fT|s t d|jjdk7rc|jj|jj} |D]'\} } d| j _ | | j _)| |_n<|jj"dk7r|jj"|_nd|_d} |D]9\}} |jj'|| | }| |} )| j)|; ddd | St d #1swYxYw) aU Uploads the HDF5 file pointed to by `path` using a custom HDF5 config. Args: path: The path to the HDF5 file. hdf5_config: The HDF5 config. show_progress: Whether to show the status bar or not. Returns: DataImportService used to get the status of the import zProvided path, 'z$', does not point to a regular file.wz.csv)modesuffixz2No data found for upload during processing of filerN)rzNo data uploaded by service) isinstancestrris_file Exception_split_hdf5_configsr enter_contextr_convert_to_csv_filenameappend _hdf5_configrun_namer _create_run _csv_configrun_idruploadextend)rrrr posix_path split_configsstack csv_itemsconfig temp_file csv_configr2_import_servicefilenamenew_import_services rr3zHdf5UploadService.upload-s$$.dC#8T$Zd !!#.tf4XYZ Z ,K8 [( >E57I' ?!//0BTZ0[\ 1NN   )..*!=> ? TUU ''00B611==k>V>V>_>_`%.;MAz68J**34:J**1;%+!))00B6$/$<$<$C$C!$&!"N(1 >$*%)%=%=%D%Dj &E&"")%7N"))*<= >C( >T  %! !9: :[( >( >s EG  Gc|jS)z-Return the run_id used in the previous upload)r)rs rget_previous_upload_run_idz,Hdf5UploadService.get_previous_upload_run_id{s   rN)T)__name__ __module__ __qualname____doc__r__annotations__r&rrr rr boolrr3rArrrr!sl*).# L;CIL; L; L;  L;\!C!rrsrc_pathdst_filerr cXt||}t||}|j||S)aConverts the HDF5 file to a temporary CSV on disk that we will upload. Args: src_path: The source path to the HDF5 file. dst_file: The output CSV file path. hdf5_config: The HDF5 config. Returns: The CSV config for the import. )_convert_hdf5_to_dataframes_create_csv_config write_csv)rIrJr merged_dfcsv_cfgs rr+r+s/ ,HkBI i8G ! Nrctt}|jjD].}|j|j f}||j |0g}tj|ddd5}|jD]'\\}}} t|||| } |j | ) dddt|dkDr~g} tdt|dD]R} | dzt|kr+|| } || dz}t| |}| j |?| j || T| }t|dkDr~|djd }|S#1swYxYw) zConvert the HDF5 file to a polars DataFrame. Args: src_path: The source path to the HDF5 file. hdf5_config: The HDF5 config. Returns: A polars DataFrame containing the data. rlatestT)libverswmrNr timestamp)rlistr.data time_dataset time_columnr-h5pyFileitems_extract_hdf5_data_to_dataframelenrange_merge_timeseries_dataframessort)rIrdata_cfg_ts_mapdata_cfg map_tuple data_framesh5f time_pathtime_col data_cfgsdf next_roundidf1df2mergedrOs rrLrLs{ALD@QO,,114**H,@,@A  "))(34K 8S =#0?0E0E0G # , !Y90i9UB   r " ## k Q  q#k*A. 2A1us;''!!n!!a%(5c3?!!&)!!+a.1 2!  k Q A##K0I +##s 2;EE!rprqc |jDcgc] }|dk7s | }}|jDcgc] }|dk7s | }}t|t|z}|rtj}|j }|D]}|j ||d|i}|j |ddd}|D]s}|d|} |jtjtj|tj| gj|j| }u|S|j |ddd}|Scc}wcc}w)zDMerge two timeseries dataframes together. Handles duplicate channelsrXr<fullT)onhowcoalesce) columnssetuuiduuid4clonerenamejoin with_columnsplrwcolaliasdrop) rprqr df1_channels df2_channels dup_channelsuid df2_renamedrO temp_col_names rrcrcsN$';;EC#2DCELE#&;;EC#2DCELE|$s<'88Ljjliik  DC%,,ccU!C5>-BCK DHH[[ftHT   "C"e1SENM!.. RVVC["&&*?@AGGLd=!  " HHS[ftHL 1FEs EE EE hdf5_filerjrkhdf5_data_configsc ||vrtd|ttj||}t j |dd}|dz }|j d|krt|d|d||j|}|jtjk(r|jtj}t j d|i}|D] } | j|vrtd| j|| jk(sJd|d | j|| jk(sJd |d | jttj|| j} t j | dd} | jdz } | j d| kr1t| jd | jd | j| | j| } t!|t!| k7r0t| jdt!|dt!| d| jtjk(r| jtj} | j"dk(r9t%| j&Dcgc]}|j(c}r t+| | } |j-| j/| j}|Scc}w)ahExtract data from an hdf5_file to a polars DataFrame. Args: hdf5_file: HDF5 File time_path: HDF5 time array path time_col: HDF5 time array col (1-indexed) hdf5_data_config: List of HDF5 Data Configs being extracted Returns: A multi-column polars DataFrame containing the timestamps and associated channels z#HDF5 file does not contain dataset NrVz: time_column=z out of rangerX)rZzWorking time dataset z) does not match data cfg defined dataset zWorking time col z% does not match data cfg defined col z: value_column=z out of range for z1: time and value columns have different lengths (z vs )CHANNEL_DATA_TYPE_ENUM)r(r r]Datasetr DataFrameshaperxdtypeBinaryString value_datasetr[r\ value_columnr,ra data_typeany enum_types is_signed_convert_signed_enumsrr)rrjrkrr[df_timetime_idx time_series data_framehdf5_data_configrdf_valueval_idx value_series enum_types rr`r`s $  !=i[IJJ i &:;Lll<?+G!|H}}Q8#9+^H:]KLL'//(34KBII%!&&ryy1 K#=>J-)X--:ABRB`B`Aabc c,999 #I;.WXhXuXuWv w 9+777 z)NO_OkOkNl m 7T\\95E5S5S+TU << a 01"//!3 >>!  '#(()9I9V9V8WWijzkIkIjJK   0 0 9: { s<0 0#(())Z[^_j[kZllpqtvBrCqDDEF     *',,RYY7L  % %)A Ac2B2M2M NYY NG 11A<PL,,\-?-?@P@U@U-VW S)XV  OsK+ rfrZc*t|jDcgc]}|jc}}|jD]}|jr|jdk\r|jdkr%t |j d|jd|jdz}||vr't |j d|jd|||_t j|jjtjScc}w)a' Convert signed enums to unsigned ints for ingestion Ignores keys >= 0, such as those which may have been converted previously by the user Will raise an exception if casting will cause a collision with an existing key Or otherwise cannot cast signed negative int to a uint32 riz: Cannot convert key z/ to uint32 due to being below valid int32 rangelz: Converting key z, to unsigned int collides with existing key ) ryrkeyrr(r,rSeriesto_numpyastypenpuint32)rfrZr cur_enum_keys unsigned_keys rrr2s8K8KL9LMM(( % ""immq&8  ==> )==/!6y}}oEtu !}}0 = (==/!29==/Amnzm{| %  % 99T]]_++BII6 77#MsDrOc|jj|jj|jjd|jjj d|jjj dd}|jjDcic]}|j|}}|jddk(sJd|jdi}t|jddD]W\}}||}|dz} |j|j|j|j|j|jd || <Y||d <t!|Scc}w) zConstruct a CsvConfig from a Hdf5Config Args: hdf5_config: The HDF5 config merged_df: The merged dataFrame of data Returns: The CSV config. rWrV)format column_numberrelative_start_time) asset_namer/r2first_data_rowr\rrXzOUnexpected merged DataFrame layout. Expected first column to be timestamp, not N)r,runits descriptionrbit_field_elements data_columns)r.rr/r2timerrrZr,rx enumeraterrrrrr ) rrOcsv_config_dictd_cfg config_mapridx channel_namerfcol_nums rrMrMMsj"..99,,55**11!..33::#.#;#;#@#@#T#T O2=1I1I1N1NO%**e#OJO   Q ; . YZcZkZklmZnYop .L&y'8'8'<=  \l+'MM!++^^#//"--"*"="= ! W  '3ON# _ %%+PsE c |jj|jj|jj|jj|jj Dcgc]}|j dk7r|c}d}g}|dr|jt||jj D]}|j dk7rt|jj|jj|jj|jj|gd}|j||Scc}w)a+ Split up hdf5_config into separate configs used to generate each CSV file Needed as string channels cannot be merged without creating empty string data points in the app Args: hdf5_config: The HDF5 config. Returns: List of HDF5Configs for later CSV conversion CHANNEL_DATA_TYPE_STRING)rr/r2rrZrZ) r.rr/r2rrZrr-r )rrfnon_string_config_dictfiltered_hdf5_configs string_configs rr)r)}s;"..99,,55**11((--(4499 !!%??  f%$$Z0F%GH,,11 4   !; ; ")66AA'44==%2299#0055!    $$]3 4 ! 7 s-E )/rz collectionsr contextlibrpathlibrtypingrrrr r numpyrr] ImportErrore RuntimeErrorpolarsrsift_py.data_import._configr sift_py.data_import.configr r sift_py.data_import.csvrsift_py.data_import.statusrsift_py.data_import.tempfiler sift_py.restrrr&r+rrLrcr^intr`rrrMr)rHrrrs # 11  4<48;'\!\!~CI 0)CI)-7)\\)Xbll",,