vikit.common.file_tools ======================= .. py:module:: vikit.common.file_tools Overview -------- .. list-table:: Function :header-rows: 0 :widths: auto :class: summarytable * - :py:obj:`copy_file_from_gcs `\ (bucket, blob_path, destination_file_name) - Copy a file from Google Cloud storage using the API * - :py:obj:`download_or_copy_file `\ (url, local_path, force_download, enable_gcs_url_rewrite) - Download a file from a URL to a local file asynchronously * - :py:obj:`file_url_exists `\ (url) - Check if a file URL exists locally * - :py:obj:`gcs_file_exists `\ (bucket, file_path) - \- * - :py:obj:`get_canonical_name `\ (file_path) - Get the canonical name of a file, without the extension * - :py:obj:`get_max_path_length `\ (path) - get the max file name for the current OS * - :py:obj:`get_max_remote_path_length `\ () - Get the maximum length of a remote path * - :py:obj:`get_path_type `\ (path) - Validate the path and return its type * - :py:obj:`get_public_uri_from_gcs_path `\ (gcs_path) - \- * - :py:obj:`get_safe_filename `\ (filename) - \- * - :py:obj:`is_valid_filename `\ (filename) - Check if the provided string is a valid filename for the local file system. * - :py:obj:`is_valid_path `\ (path) - Check if the path is valid: could be a local path or a remote one * - :py:obj:`upload_to_bucket `\ (source_file_name, destination_blob, destination_file_name, bucket_name) - \- * - :py:obj:`url_exists `\ (url) - Check if a URL exists somewhere on the internet or locally. To be superseded by a more * - :py:obj:`web_url_exists `\ (url) - \- .. list-table:: Attributes :header-rows: 0 :widths: auto :class: summarytable * - :py:obj:`TIMEOUT ` - \- Functions --------- .. py:function:: copy_file_from_gcs(bucket: str, blob_path: str, destination_file_name: str = 'downloaded_file') Copy a file from Google Cloud storage using the API params: gcs_full_url: the full URL of the file to copy blob_path: the name of the object to copy destination_file_name: the local path of the copied file return: the local path of the copied file .. py:function:: download_or_copy_file(url: str, local_path: str, force_download: bool = False, enable_gcs_url_rewrite: bool = True) -> str :async: Download a file from a URL to a local file asynchronously :param url: The URL to download the file from (supported: http, https, gs, local) :type url: str :param local_path: The filename to save the file to. :type local_path: str :param force_download: If True, the file will be downloaded even if it already exists :type force_download: bool :param enable_gcs_url_rewrite: If True, the GCS Cloud Storage libraries will be used for HTTP(S) URLs pointing to cloud storage resource. If False, the file will be downloaded using the HTTP(S) protocol. This parameter exists mainly for testing purposes. We recommend using the GCS library whenever possible. :type enable_gcs_url_rewrite: bool :returns: The filename of the downloaded file. :rtype: str .. py:function:: file_url_exists(url) Check if a file URL exists locally .. py:function:: gcs_file_exists(bucket: str, file_path: str) -> bool .. py:function:: get_canonical_name(file_path: str) Get the canonical name of a file, without the extension .. py:function:: get_max_path_length(path='.') get the max file name for the current OS params: path: the file path return: file name max length. .. py:function:: get_max_remote_path_length() Get the maximum length of a remote path .. py:function:: get_path_type(path: Optional[Union[str, os.PathLike]]) -> dict Validate the path and return its type :param path: The path to validate :type path: str, os.PathLike, :returns: The path type and the path itself Path type can be local, http, https, s3, gs, None, undefined, error, error : message if the path is invalid, None if no error :rtype: dict .. py:function:: get_public_uri_from_gcs_path(gcs_path: str) -> str .. py:function:: get_safe_filename(filename) .. py:function:: is_valid_filename(filename: str) -> bool Check if the provided string is a valid filename for the local file system. :param filename: The filename to check. :type filename: str :returns: True if valid, False otherwise. :rtype: bool .. py:function:: is_valid_path(path: Optional[Union[str, os.PathLike]]) -> bool Check if the path is valid: could be a local path or a remote one (http, etc). We don't test the actual access and credentials at this stage, just the path format. :param path: The path to validate :type path: str, os.PathLike :returns: True if the path is valid, False otherwise :rtype: bool .. py:function:: upload_to_bucket(source_file_name, destination_blob, destination_file_name, bucket_name) :async: .. py:function:: url_exists(url: str) Check if a URL exists somewhere on the internet or locally. To be superseded by a more versatile and unified library in the future. :param url: The URL to check :type url: str :returns: True if the URL exists, False otherwise :rtype: bool .. py:function:: web_url_exists(url) Attributes ---------- .. py:data:: TIMEOUT :value: 10