Uploaded image for project: 'StreamSets Data Collector'
  1. StreamSets Data Collector
  2. SDC-15869

Google OAuth2 JWT auth token renew errors in HTTP client stage

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2 (Critical with Workaround)
    • Resolution: Fixed
    • Affects Version/s: 3.18.1
    • Fix Version/s: 3.20.0
    • Component/s: http
    • Labels:
    • Environment:

      SDC on Linux Docker. Tested on 3.18.1 but this might be impacted in latest versions too

    • Testing Status:
      Not Required
    • Team:
      Data Plane
    • Stage:
      HTTP Client destination, HTTP Client origin, HTTP Client processor

      Description

      Once when JWT token expires and SDC renews, seeing the below error in sdc.log As per quick check by Santhosh Kumar and discussing with him, seeing that there are 2 issues lurking:

      1. When auth token expires, its renew is not handled before 400/403 error. Before the JWT exp time lapses we need to refresh the token
      2. When renew is run, the JWT exp and iat values are reused from the first time JWT auth token was requested, causing the Invalid JWT token issue

      Error:

      2020-10-01 01:57:30,238 [user:*admin@engproductivity] [pipeline:STFTS (STF Test Scheduler) (SCH Test Run) copy/STFTSSTFfd95a99e-8d9b-49e3-9d44-563dea84a672] [runner:0] [thread:STFTSSTFfd95a99e-8d9b-49e3-9d44-563dea84a672_Worker-1] [stage:HTTPClient_01] ERROR HttpStageUtil - OAuth2 Authentication failed
      com.streamsets.pipeline.lib.http.AuthenticationFailureException: Authentication failed with error Code: 400 and  error message: {
        "error": "invalid_grant",
        "error_description": "Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values and use a clock with skew to account for clock differences between systems."
      }
      	at com.streamsets.pipeline.lib.http.oauth2.OAuth2ConfigBean.processResponse(OAuth2ConfigBean.java:425)
      	at com.streamsets.pipeline.lib.http.oauth2.OAuth2ConfigBean.obtainAccessToken(OAuth2ConfigBean.java:398)
      	at com.streamsets.pipeline.lib.http.oauth2.OAuth2ConfigBean.reInit(OAuth2ConfigBean.java:527)
      	at com.streamsets.pipeline.stage.util.http.HttpStageUtil.getNewOAuth2Token(HttpStageUtil.java:88)
      	at com.streamsets.pipeline.stage.processor.http.HttpProcessor.processResponse(HttpProcessor.java:769)
      	at com.streamsets.pipeline.stage.processor.http.HttpProcessor.process(HttpProcessor.java:397)
      	at com.streamsets.pipeline.api.base.SingleLaneProcessor.process(SingleLaneProcessor.java:95)
      	at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
      	at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:299)
      	at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
      	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
      	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
      	at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:864)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:908)
      	at com.streamsets.datacollector.runner.PipeRunner.acceptConsumer(PipeRunner.java:207)
      	at com.streamsets.datacollector.runner.PipeRunner.forEachInternal(PipeRunner.java:152)
      	at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:132)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:907)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:885)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:515)
      	at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:383)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at com.streamsets.datacollector.runner.StageRuntime.processBatch(StageRuntime.java:379)
      	at com.streamsets.datacollector.runner.StageContext.processBatch(StageContext.java:293)
      	at com.streamsets.pipeline.stage.origin.scheduler.SchedulerJob.execute(SchedulerJob.java:50)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      2020-10-01 01:57:30,239 [user:*admin@engproductivity] [pipeline:STFTS (STF Test Scheduler) (SCH Test Run) copy/STFTSSTFfd95a99e-8d9b-49e3-9d44-563dea84a672] [runner:0] [thread:STFTSSTFfd95a99e-8d9b-49e3-9d44-563dea84a672_Worker-1] [stage:HTTPClient_01] ERROR PipeRunner - Failed executing stage 'HTTPClient_01': com.streamsets.pipeline.api.StageException: HTTP_21 - OAuth2 authentication failed. Please make sure the credentials are valid.
      com.streamsets.pipeline.api.StageException: HTTP_21 - OAuth2 authentication failed. Please make sure the credentials are valid.
      	at com.streamsets.pipeline.stage.util.http.HttpStageUtil.getNewOAuth2Token(HttpStageUtil.java:92)
      	at com.streamsets.pipeline.stage.processor.http.HttpProcessor.processResponse(HttpProcessor.java:769)
      	at com.streamsets.pipeline.stage.processor.http.HttpProcessor.process(HttpProcessor.java:397)
      	at com.streamsets.pipeline.api.base.SingleLaneProcessor.process(SingleLaneProcessor.java:95)
      	at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
      	at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:299)
      	at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
      	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
      	at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
      	at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:864)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:908)
      	at com.streamsets.datacollector.runner.PipeRunner.acceptConsumer(PipeRunner.java:207)
      	at com.streamsets.datacollector.runner.PipeRunner.forEachInternal(PipeRunner.java:152)
      	at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:132)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:907)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:885)
      	at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:515)
      	at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:383)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at com.streamsets.datacollector.runner.StageRuntime.processBatch(StageRuntime.java:379)
      	at com.streamsets.datacollector.runner.StageContext.processBatch(StageContext.java:293)
      	at com.streamsets.pipeline.stage.origin.scheduler.SchedulerJob.execute(SchedulerJob.java:50)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      

        Attachments

          Activity

            People

            Assignee:
            sebas Sebastian Sanchez
            Reporter:
            srid Sridhar Banoor
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: