Uploaded image for project: 'StreamSets Data Collector'
  1. StreamSets Data Collector
  2. SDC-7361 Improve SFTP Origin
  3. SDC-9626

SFTP Origin to support File Post Processing

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: P3 (Limited Impact)
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.8.0
    • Component/s: None
    • Labels:
    • Sprint:
      Mystifying Boba
    • Doc Impact Details:
      Hide
      Should be similar to Directory origin doc

      The paths can get a bit confusing because of the "Path Relative to User Home Directory" config in the "SFTP/FTP" tab, and also in the "Post Processing" tab; so it might be good to have some extra explanation. The path that the data ends up in will be <archive-dir>/<data-dir>. <archive-dir> can either be an absolute path (if "Path Relative to User Home Directory" under "Post Processing" is false) or a path relative to the user's home dir. The <data-dir> comes from the URI under "SFTP/FTP" plus any subdirs that got processed. Regardless of the value of "Path Relative to User Home Directory" under "SFTP/FTP", the <data-dir> will always be relative to whatever was in the URI (the config only determines if the path in the URI is relative to the user's home dir or not).

      For example, if you have data in /home/username/data/file.txt on a remote host, and the URI is set to (s)ftp://host:port/data with "Path Relative to User Home Directory" under "SFTP/FTP" set to true, then the data will be archived to <archive-dir>/file.txt. Equivalently, if the "Path Relative to User Home Directory" under "SFTP/FTP" was set to false and the URI was set to (s)ftp://host:port/home/username/data, then it would still be archived to <archive-dir>/file.txt. If you are processing subdirectories and the URIs were instead (s)ftp://host:port/ and (s)ftp://host:port/home/username/, respectively, then it would be archived to <archive-dir>/data/file.txt.

      Let me know if that's not clear.
      Show
      Should be similar to Directory origin doc The paths can get a bit confusing because of the "Path Relative to User Home Directory" config in the "SFTP/FTP" tab, and also in the "Post Processing" tab; so it might be good to have some extra explanation. The path that the data ends up in will be <archive-dir>/<data-dir>. <archive-dir> can either be an absolute path (if "Path Relative to User Home Directory" under "Post Processing" is false) or a path relative to the user's home dir. The <data-dir> comes from the URI under "SFTP/FTP" plus any subdirs that got processed. Regardless of the value of "Path Relative to User Home Directory" under "SFTP/FTP", the <data-dir> will always be relative to whatever was in the URI (the config only determines if the path in the URI is relative to the user's home dir or not). For example, if you have data in /home/username/data/file.txt on a remote host, and the URI is set to (s) ftp://host:port/data with "Path Relative to User Home Directory" under "SFTP/FTP" set to true, then the data will be archived to <archive-dir>/file.txt. Equivalently, if the "Path Relative to User Home Directory" under "SFTP/FTP" was set to false and the URI was set to (s) ftp://host:port/home/username/data, then it would still be archived to <archive-dir>/file.txt. If you are processing subdirectories and the URIs were instead (s) ftp://host:port/ and (s) ftp://host:port/home/username/, respectively, then it would be archived to <archive-dir>/data/file.txt. Let me know if that's not clear.
    • Testing Status:
      Required
    • Testing Required Description:
      Presume same as Directory origin on deletes and archive

      Description

      Would like "File Post Processing" support in SFTP origin similar to current capabilities in Directory origin.  Screenshot snippet attached.

        Attachments

        1. post-processing.png
          post-processing.png
          21 kB
        2. screenshot2.png
          screenshot2.png
          38 kB
        3. scrreenshot1.png
          scrreenshot1.png
          25 kB

          Issue Links

            Activity

              People

              Assignee:
              rkanter@streamsets.com Robert Kanter
              Reporter:
              todd Todd McGrath
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: