Workflow of the class ===================== The workflow of the class involves several key steps, each contributing to the overall process of handling measurement data. Here is a general description of the workflow: 1. **Initialization**: Set up the class with the necessary information about the radionuclide and the measurement period. 2. **Parsing readings**: Read and organize raw measurement data from CSV files into a structured format. 3. **Processing measurements**: Categorize and process the parsed data into background, sample, and net measurements. 4. **Summarizing data**: Generate a concise overview of the processed data, including key statistics and information. 5. **Exporting results**: Save the processed data and visualizations to CSV and PNG files for further analysis and reporting. 6. **Comprehensive analysis**: Combine all steps into a single workflow for a complete analysis of the measurement data. Initialization -------------- The workflow begins with the initialization of the ``Hidex300`` class. During this step, the user provides information about the radionuclide being measured, such as the name of the radionuclide, and the year and month of the measurements. This setup ensures that the class is configured with the correct context for the data it will process. During the initialization, the constructor (``__init__``) is called with parameters for the ``radionuclide``, ``year``, and ``month`` of the measurements. These parameters are stored as the configuration attributes of the class instance (``radionuclide``, ``year``, and ``month``). Data storage attributes (``readings``, ``background``, ``sample``, ``net``) and measurement attributes (``cycles``, ``cycle_repetitions``, ``repetition_time``, ``total_measurements``, ``measurement_time``) are initialized to ``None``. Parsing readings ---------------- Once the class is initialized, the next step is to parse the measurement data. The user specifies the folder containing the CSV files provided by the Hidex 300 SL automatic counter from the measurement of the radionuclide. Then, the ``parse_readings`` method orchestrates the parsing of measurement data from CSV files located in this folder. It reads the CSV files in this folder, extracts the relevant information, and organizes it into a structured format. Parsing CSV files ^^^^^^^^^^^^^^^^^ The ``parse_readings`` method calls the ``_parse_readings`` private method to handle the detailed parsing logic, which includes reading the files, extracting relevant data, and organizing it into a structured DataFrame. Getting CSV files """"""""""""""""" The ``_parse_readings`` method retrieves a list of relevant CSV files that need to be processed calling to the ``_get_csv_files`` helper function. This function iterates over all the files in the given folder, checks if each file has a ``.csv`` extension, and appends the absolute path of each CSV file to a list. The method then prints the number of CSV files found and returns the list of full paths to these files. Identifying information blocks """""""""""""""""""""""""""""" Then, the ``_parse_readings`` method iterate over the CSV files. For each file, it reads the file line by line, and extracts relevant data based on predefined parameters. These parameters are defined as class private constants. The method assign to each file an identification number. The method skips 4 of initial metadata lines (specified by the ``_ID_LINES`` class constant). It identifies the start of new data blocks by the key ``Sample start`` (defined in the ``_BLOCK_STARTER`` class constant). Extracting block information """""""""""""""""""""""""""" Then, for each new data block, the ``_parse_readings`` method extracts the values in the rows that starts with ``Samp.``, ``Repe.``, ``CPM``, ``Counts``, ``DTime``, ``Time`` or ``EndTime``. These keys are defined in the ``_ROWS_TO_EXTRACT`` class constant. These values represent respectively the type of sample (background or radionuclide sample), the number of the repetition, the count rate, the total counts, the dead time, the measurement time and the end time of the measurement. The delimiter to use when parsing the values is ``;``, defined in the ``_DELIMITER`` class constant. The method compiles the extracted information for each block into a dictionary, adding also the file identification number. Then, it compiles the extracted information for all blocks into a list. Structuring the data """""""""""""""""""" Then, ``_parse_readings`` method structures all the extracted data in a DataFrame, being the columns the key words defined in the ``_ROWS_TO_EXTRACT`` class constant, and each row corresponding to a single measurement of background or radionuclide sample. Then, it converts the strings of all the columns to numeric format, except for the ``EndTime`` column. It converts date and time strings of the ``EndTime`` column to ``datetime`` objects using the format ``%d/%m/%Y %H:%M:%S`` (specified in the ``_DATE_TIME_FORMAT`` class constant). Then, it sorts the DataFrame by end time of the measurements, and reassign the file identification number to match the chronological order of the measurements. It ensures consistency in repetitions per cycle. If not it will raise a ``ValueError``. Finally, the ``_parse_readings`` method renames columns for clarity and returns the structured DataFrame. The ``parse_readings`` method assign this DataFrame to the ``readings`` class attribute. Getting readings statistics ^^^^^^^^^^^^^^^^^^^^^^^^^^^ After parsing the readings to a structured DataFrame, the ``parse_readings`` method generates a summary of the measurement cycles and their key attributes using the ``_get_readings_statistics`` private method. Summary of measurement cycles """"""""""""""""""""""""""""" The ``_get_readings_statistics`` starts generating a summary of the measurement cycles and their key attributes using the ``_get_readings_summary`` private method. The ``_get_readings_summary`` iterates over each unique cycle in the readings DataFrame, stored in the ``readings`` class attribute. For each cycle, it filters the DataFrame, determines the number of repetitions per cycle, retrieves the real time of measurement for the repetitions, and identifies the earliest end time of the measurements. It compiles these details into a list of results, which is then converted into a summary DataFrame containing the cycle, repetitions per cycle, real time of the measurement, and the date and time of the measurement cycle. If the real time values are not consistent for all measurements, it will raise a ``ValueError``. If the ``readings`` class attribute is ``None``, it will raise a ``ValueError``, indicating that the readings must be parsed first from the CSV files. Summary of all measurements """"""""""""""""""""""""""" After getting the measurement cycles summary DataFrame, the ``_get_readings_statistics`` method checks for consistency in repetitions per cycle. If they are not consistent, it will raise a ``ValueError``. Then, from this Dataframe, it calculates the number of cycles, the total number of measurements, the total measurement time, the time per repetition, and the number of repetitions per cycle. These statistics are compiled into a dictionary and returned. Finally, the ``parse_readings`` method assigns these statistics to the corresponding measurement attributes of the class. Summarizing measurements ------------------------ With the raw data of the readings parsed and the statistics of the readings generated, the class can generate a summary of the parsed readings in a user friendly format. This summary includes the key aspects of the measurement cycles as well as of the complete set of measurements including all cycles. The ``summarize_readings`` method handles this task, providing a concise overview of the measurements structure as well as of the class's current data and processing state. This method can show the summary directly in the console or write it to a text file. This behaviour is determined by the parameter ``save``. - By default, the ``save`` parameter is set to False, then the method print the summary through the console. - However, if the ``save`` parameter is set to ``True``, it writes the summary to a text file called ``summary.txt`` in the folder specified in the parameter ``folder_path``. In any case, the method calls the ``__str__`` dunder method to create the summary as a string. The summary string generated depends on the class's current data and processing state. - If the readings of the measurements **have not been parsed** from the Hidex 300 SL CSV files (meaning the ``readings`` attribute is ``None``), the method compile the information of the configuration attributes (radionuclide, year, month) and returns them in a readable way. - If the readings of the measurements **have been parsed** from the Hidex 300 SL CSV files (meaning the ``readings`` attribute is not ``None``), the method compile the information of the configuration attributes, the measurement attributes (number of cycles, repetitions per cicle, measurement time per repetition, total number of measurements and total measurement time) and the summary of the measurement cycles returned by the ``_get_readings_summary`` private method and returns them in a readable way. Processing measurements ----------------------- With the raw data of the readings parsed and the statistics of the readings generated, the class can proceeds to process the measurements. **Processing the readings** involves computing certain quantities of interest for the characterization of the radionuclide being measured from the quantities extracted from the CSV files provided by the Hidex 300 SL. This involves categorizing the data into **different types**: background measurements, sample measurements, and net measurements. The ``process_readings`` method handles this task. It processes different types of measurements (background, sample, net, or all) and updates the corresponding attributes with the processed data for further use. It calls specific private methods to handle each type of measurement and raises an error if an invalid type is provided. The **type of measurement** to proces is defined by the ``kind`` parameter. Options are 'background', 'sample', 'net', or 'all'. In any case, one of the quantities to calculate is the **elapsed time** between measurements of the same type. The user can choose the unit of this quantity setting it in the ``time_unit`` parameter. Default is 's' (seconds). Other options include 'min' (minutes), 'h' (hours), 'd' (days), 'wk' (weeks), 'mo' (months), and 'yr' (years). To **process background measurements**, the ``kind`` must be set to ``background`` Then the method calls the ``_get_background_sample`` with the same ``kind`` and ``time_unit`` parameters. This method returns the processed background measurements as a DataFrame, which is stored in the ``background`` class attribute. The processed background measurements are stored in the ``background`` class attribute. To **process sample measurements**, the ``kind`` must be set to ``sample`` Then the method calls the ``_get_background_sample`` with the same ``kind`` and ``time_unit`` parameters. This method returns the processed sample measurements as a DataFrame, which is stored in the ``sample`` class attribute. To **process net measurements**, the ``kind`` must be set to ``net`` Then the method calls the ``_get_background_sample`` with the same ``kind`` and ``time_unit`` parameters. This method returns the processed background measurements as a DataFrame, which is stored in the ``net`` class attribute. If the ``kind`` parameter is set to ``all``, the method processes all types of measurements (background, sample and net) and stores in the corresponding class attributes. Processing background/sample measurements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``_get_background_sample`` method processes either background or sample measurements from the parsed measurement readings and returns them as a DataFrame. It takes a parameter ``kind`` to specify whether it is processing background or sample data. Options are ``background`` or ``sample``. It also take a ``time_unit`` parameter to specify the unit to calculate the elapsed time between measurements. In the class workflow, both parameters are set by the ``process_readings`` method. The method first **check for data availability**. If ``readings`` class attribute is ``None``, it raises a ``ValueError``, indicating that no readings data is available. Then, the method **filters the** ``readings`` **DataFrame** based on the ``kind`` parameter to isolate the background or sample measurements in a new Dataframe. In order to do this, it defines a dictionary that maps the measurement types (background and sample) to their respective identifiers (set in the ``_BACKGROUND_ID`` and ``_SAMPLE_ID`` class constants). It then calculates the **quantities of interest** for each repetition within each cycle from those stored in the filtered Dataframe. - It calculates the **live time** (in seconds) by dividing the real time (in seconds) by the dead time (:math:`\ge 1`) - It calculates the **elapsed time** between measurements in ``datetime`` format and in the specified unit by calling the ``_get_elapsed_time`` helper function and passing the filtered DataFrame and the specified ``time_unit``. - It calculates the **counts** by multiplying the count rate (in counts per minute) by the live time (in seconds) and dividing by 60. - It calculates the **counts uncertainty** as the square root of the counts. - It calculates the **counts relative uncertainty** (in %) by dividing the counts uncertainty by the counts and multiplying by 100. These quantities are added as new columns to the filtered DataFrame, which is then returned. Processing net measurements ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``_get_net_measurements`` method processes net measurements from the background and sample measurements and returns them as a DataFrame. It takes a parameter ``time_unit`` to specify the unit to calculate the elapsed time between measurements. In the class workflow, both parameters are set by the ``process_readings`` method. The method first **check for data availability**. If ``background`` and ``sample`` class attribute are ``None``, it raises a ``ValueError`` indicating that the necessary data is not available. Then, some of the quantities **quantities of interest** for each repetition within each cycle are taken directly from the ``sample`` DataFrame: cycle, repetition, and elapsed time in ``datetime`` format and in the specified unit. Some other **quantities of interest** for each repetition within each cycle from those stored in the ``background`` and ``sample`` Dataframes. - It calculates the **net count rate** (in cpm) by subtracting the background count rate from the sample count rate. - It calculates the **net counts** by subtracting the background counts from the sample counts. - It calculates the **net counts uncertainty** as the square root of the sum of sample and background counts. - It calculates the **relative net counts uncertainty** (in %) calculated as the counts uncertainty divided by the net counts and multiplied by 100. The method compiles the results in a dictionary which is finally returned as a Dataframe. Computing elapsed time ^^^^^^^^^^^^^^^^^^^^^^ The ``_get_elapsed_time`` method calculates the elapsed between measurements in ``datetime`` format from a specified DataFrame (set in the ``df`` parameter) and converts it to the specified time unit (set in the ``time_unit``). In the class workflow, both parameters are set by the ``_get_background_sample`` method. First, the method get the **initial time** of the measurement by finding the earliest measurement end time in the ``background`` or ``sample`` DataFrame. Then it calculates the elapsed time between measurements by subtracting the initial_time from the end time values. This results in a pandas Series of time deltas. Then, the method defines a **time conversion dictionary** that maps each time unit to its corresponding conversion factor from seconds. The next conversion factors are used: - 1 minute = 60 seconds - 1 hour = 60 minutes - 1 day = 24 hours - 1 week = 7 days - 1 month = 30.44 days - 1 year = 365.25 days Then, the method **validate the time unit**, checking if the provided ``time_unit`` is present in the time conversion dictionary. Options are 's' (seconds), 'min' (minutes), 'h' (hours), 'd' (days), 'wk' (weeks), 'mo' (months), and 'yr' (years). If it is not, it raises a ``ValueError`` exception. Then, the method **converts the elapsed time** from ``datetime`` to seconds using the ``datetime`` builtin function ``total_seconds()``. Then it converts the elapsed time from seconds to the specified unit by multiplying by the corresponding time conversion factor. Finally, the method returns a tuple containing the original elapsed time (as time deltas) and the elapsed time converted to the specified unit. Plotting measurements --------------------- After parsing the readings from the CSV files provided by the Hidex 300 SL and processing the different types of measurements, the class is ready to plot relevant information of the measurements in terms of time. The ``plot_measurements`` method handles this task. It uses ``matplotlib`` to generate the plots. It determines the type of measurements to plot based on the ``kind`` parameter (``background``, ``sample``, or ``net``). It then calls the appropriate helper function for the specified measurement type (``_plot_background_sample_measurements`` for background or sample measurements or ``_plot_net_measurements`` for net measurements) and providing the corresponding attribute DataFrame (``background``, ``sample``, or ``net``). It raises a ``ValueError`` if an invalid ``kind`` is provided. Plotting background or sample measurements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``_plot_background_sample_measurements`` helper function creates a series of plots for background or sample measurements. It determines the type of measurements to plot based on the ``kind`` parameter (``background`` or ``sample``). It gathers the measurements data from the corresponding attribute DataFrame (``background``, ``sample``, or ``net``). In the class workflow, both parameters are set by the ``plot_measurements`` method. It extracts the ``End time`` column from the appropriate measurement DataFrame for the x-axis. Then it creates a 3x2 grid of subplots to plot the next quantities in terms of the end time: count rate, dead time, real time, live time, counts, and counts uncertainty, extracting the corresponding column from the appropriate measurement DataFrame. It set the x-axis labels ``End time`` and the y-axis labels to the corresponding quantity and unit. It sets the title of the figure to ``Background measurements`` or ``Sample measurements`` depending on the kind of measurement to plot. Plotting net measurements ^^^^^^^^^^^^^^^^^^^^^^^^^ The ``_plot_net_measurements`` helper function creates a series of plots for net measurements. It gathers the measurements data from the ``net`` attribute DataFrame In the class workflow, this parameter is set by the ``plot_measurements`` method. It extracts the ``Elapsed time`` (with its unit) column from the attribute DataFrame for the x-axis. Then it creates a 2x1 grid of subplots to plot the next quantities in terms of the elapsed time: counts and counts uncertainty, extracting the corresponding column from the attribute DataFrame. It set the x-axis labels ``Elapsed time`` (with its unit) and the y-axis labels to the corresponding quantity and unit. It sets the title of the figure to ``Net quantities measurements``. Exporting measurements ---------------------- After parsing the readings from the CSV files provided by the Hidex 300 SL and processing the different types of measurements, the class is ready to save the processed data and visualizations to CSV and PNG files for further analysis and reporting. Exporting tables ^^^^^^^^^^^^^^^^ The ``export_table`` method saves specified types of measurements as a CSV file to the specified folder path. It determines the type of measurements to plot based on the ``kind`` parameter (``readings``, ``background``, ``sample``, ``net``, or ``all``). First, the method maps the ``kind`` parameter to the corresponding DataFrame attributes (``readings``, ``background``, ``sample``, ``net``, or a the compiled DataFrame returned by the ``_compile_measurements`` method for ``all``). Then, it validates the ``kind`` parameter by ensuring it exists in the dictionary keys. If the ``kind`` is invalid, it raises a ``ValueError`` exception. Then, it retrieves the corresponding DataFrame from the dictionary and exports it to a CSV file in the specified ``folder_path`` using the ``to_csv`` method of the DataFrame. The CSV file is named after the ``kind`` parameter. Finally, it prints a confirmation message indicating the successful export. Exporting plots ^^^^^^^^^^^^^^^ The ``export_plot`` method class saves the plot of a specified type of measurements as a PNG file to the specified folder path. It determines the type of measurements to plot based on the ``kind`` parameter (``background``, ``sample``, or ``net``). First, the method maps the ``kind`` parameter to the corresponding DataFrame attributes (``readings``, ``background`` or ``sample``). Then, it validates the ``kind`` parameter by ensuring it exists in the dictionary keys. If the ``kind`` is invalid, it raises a ``ValueError`` exception. Then, it calls the ``plot_measurements`` method to generate the plot and exports it to a PNG file in the specified ``folder_path`` using the ``savefig`` method of ``matplotlib``. The PNG file is named after the ``kind`` parameter. Finally, it prints a confirmation message indicating the successful export. Comprehensive analysis ---------------------- The ``Hidex300`` class allows to run a full comprehensive analysis of the readings, combining all the previous steps into a single workflow, including including parsing, processing and summarizing the readings and exporting the results to CSV and PNG files. The ``analyze_readings`` method handles this task. First, the method **parses the readings** of the CSV files provided by the Hidex 300 SL from the specified ``input_folder``, by calling the calling the ``parse_readings`` method. Then, it **processes all types of measurements** (background, sample and net) by calling the ``process_readings`` method with ``kind='all'`` and the specified ``time_unit``. Finally, it **prints a summary** of the measurements by calling the ``__str__`` method. Optionally, the method can **save the results** (CSV files and plots) to the specified ``output_folder`` if the ``save`` parameter is ``True``. The method checks if the ``output_folder`` exists and creates it if it doesn't. It also creates a subfolder for the specific radionuclide, year, and month to store the results. Then it saves the background, sample, net, and compiled measurements as CSV files to this folder by calling the ``export_table`` method. It also saves the summary of the readings to a text file by calling the ``summarize_readings``. Finally, it saves plots of background, sample, and net measurements to PNG files by calling the ``export_plot`` method.